syncstate.Config held one carrier field typed as the NATS client, so a
pgbus.Bus could not be handed to a SyncedMap at all: it satisfies
messaging.Broadcaster and not MessagingClient. The durable re-hydration
path built for the responses map therefore had a NATS-only consumer and
nothing in the build said so.
The field becomes Bus messaging.Broadcaster, SubscribeJSON moves to its
own file and relaxes its parameter to Broadcaster, and the four adopters
fan out over PostgreSQL LISTEN/NOTIFY: fine-tune jobs, quantization jobs,
agent tasks with their per-tenant children, and Open Responses metadata.
A new spec proves it on a real database, over two Bus instances on two
pinned listener connections: a Set and a Delete carry, a payload past the
8000-byte notification cap comes back byte identical through the spill
row, two families sharing one LISTEN channel stay separate, and a
terminated listener re-hydrates a row written while it was gone.
The five sites that each chose a carrier for an adopter are collapsed
into one DistributedServices.Broadcast() accessor. Five field reads were
five chances to leave one family on NATS with nothing failing, because
messaging.Client satisfies Broadcaster too. The accessor also refuses to
hand out a nil pgbus.Bus wrapped in a non-nil interface, which every
adopter would read as "broadcast" and dereference on the first Set.
SetTaskSyncNATS and SetJobSyncNATS are renamed to SetTaskSyncBus and
SetJobSyncBus so a missed wiring site fails to compile.
The response metadata table gains a retention of its own, defaulting to
24 hours. It inherited the Open Responses store TTL, which defaults to 0
meaning no expiration. Zero is defensible for a map that dies with the
process and is not for a table: the table grew for the life of the
deployment and a restarting replica re-hydrated every response the
cluster had ever created. A row that names its own expiry is still judged
on that column alone, and "this row is dead" now has one SQL spelling
that PurgeExpired deletes by and ListUnexpired is the negation of, so a
hydrate cannot resurrect what a sweep has already retired.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Unsubscribe decided a channel had lost its last subscriber under one
lock and issued the UNLISTEN after releasing it. A Subscribe on the same
root could decide to LISTEN in that window, and the two reached the
connection in the wrong order: the root ended up not listened with a
live subscription on it. It does not heal, because the next Subscribe
sees the registration already there and never re-LISTENs, so the whole
root stays deaf on that replica until the connection drops.
The decision and the statement it implies now happen under one lock,
held across both, at both call sites. A second lock and not the
registration lock: issuing waits on the listener goroutine, delivery
takes the registration lock, and holding that across the wait deadlocks
the carrier.
The race is spec'd through a barrier seam rather than by racing
goroutines. The natural window is microseconds wide, and a spec that
waits for it to open passes by luck; the seam scripts the interleaving,
so the spec decides in both directions.
Resolving a spilled message moved off the listener. PostgreSQL keeps
undelivered notifications in a shared, fixed-size queue, so a listener
that stops draining it can block COMMIT for every publisher on the
server, not only this one. The listener now only drains; one resolver
goroutine reads the row back and dispatches, which also keeps a spilled
message and an inline one on the same subject in the order they were
published.
Three wiring lines that could be deleted with the suite staying green:
the sweeper's start is now pinned by a Config interval, and the two
lines that carry the bus into the deployment now refuse to boot when
either is missing. A subscription can also report what it dropped, so
the party that missed a message is the party that can see it.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Distributed mode needs an operator to run a NATS cluster. This adds the
carrier that replaces its fan-out half, so a deployment eventually needs
PostgreSQL and its own HTTP listener and nothing else.
pgbus holds one PostgreSQL session per replica, pinned for the life of
the process because LISTEN registrations belong to one backend session
and a pooled handle would lose them on the next checkout. Publishes go
out on the pool with pg_notify.
Subjects map onto a channel by their first token, from a closed set of
roots. A subject outside the set is refused at publish AND at subscribe
rather than mapped to a channel of its own: a channel name is capped at
63 bytes, and one LISTEN per job id would be unbounded. Refused rather
than dropped, because a subject that goes nowhere and reports nothing is
the class of defect this work exists to remove.
PostgreSQL refuses a notify payload of 8000 bytes or more, and several
subjects on this bus exceed that in normal operation: a job result
carries a whole LLM output, a gallery progress event carries one entry
per node. Those are written to a row and the notification carries the
id. What is measured against the cap is the ENCODED notification, not
the caller's payload, because the subject and the envelope travel too.
The filter grammar is not respelled here. Subscribe asks
messaging.ValidFilter and delivery asks messaging.SubjectMatches, which
makes this the first production caller of a matcher that had only test
doubles. New refuses a DSN that names a different database from the
pool: that pairing publishes successfully, delivers nothing, on every
replica, and reports no error anywhere.
Nothing publishes on it and nothing subscribes yet. The construction is
wired anyway, because the DSN has exactly one legitimate source and a
setting that decides whether any broadcast is delivered should not be
invented by whichever call site is migrated first.
Delivery is at-most-once, like NATS core. Nothing downstream may read a
message it did not receive as evidence about a node: a carrier that
cannot deliver is not a worker that is gone.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Ettore Di Giacinto