Unsubscribe decided a channel had lost its last subscriber under one
lock and issued the UNLISTEN after releasing it. A Subscribe on the same
root could decide to LISTEN in that window, and the two reached the
connection in the wrong order: the root ended up not listened with a
live subscription on it. It does not heal, because the next Subscribe
sees the registration already there and never re-LISTENs, so the whole
root stays deaf on that replica until the connection drops.
The decision and the statement it implies now happen under one lock,
held across both, at both call sites. A second lock and not the
registration lock: issuing waits on the listener goroutine, delivery
takes the registration lock, and holding that across the wait deadlocks
the carrier.
The race is spec'd through a barrier seam rather than by racing
goroutines. The natural window is microseconds wide, and a spec that
waits for it to open passes by luck; the seam scripts the interleaving,
so the spec decides in both directions.
Resolving a spilled message moved off the listener. PostgreSQL keeps
undelivered notifications in a shared, fixed-size queue, so a listener
that stops draining it can block COMMIT for every publisher on the
server, not only this one. The listener now only drains; one resolver
goroutine reads the row back and dispatches, which also keeps a spilled
message and an inline one on the same subject in the order they were
published.
Three wiring lines that could be deleted with the suite staying green:
the sweeper's start is now pinned by a Config interval, and the two
lines that carry the bus into the deployment now refuse to boot when
either is missing. A subscription can also report what it dropped, so
the party that missed a message is the party that can see it.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Distributed mode needs an operator to run a NATS cluster. This adds the
carrier that replaces its fan-out half, so a deployment eventually needs
PostgreSQL and its own HTTP listener and nothing else.
pgbus holds one PostgreSQL session per replica, pinned for the life of
the process because LISTEN registrations belong to one backend session
and a pooled handle would lose them on the next checkout. Publishes go
out on the pool with pg_notify.
Subjects map onto a channel by their first token, from a closed set of
roots. A subject outside the set is refused at publish AND at subscribe
rather than mapped to a channel of its own: a channel name is capped at
63 bytes, and one LISTEN per job id would be unbounded. Refused rather
than dropped, because a subject that goes nowhere and reports nothing is
the class of defect this work exists to remove.
PostgreSQL refuses a notify payload of 8000 bytes or more, and several
subjects on this bus exceed that in normal operation: a job result
carries a whole LLM output, a gallery progress event carries one entry
per node. Those are written to a row and the notification carries the
id. What is measured against the cap is the ENCODED notification, not
the caller's payload, because the subject and the envelope travel too.
The filter grammar is not respelled here. Subscribe asks
messaging.ValidFilter and delivery asks messaging.SubjectMatches, which
makes this the first production caller of a matcher that had only test
doubles. New refuses a DSN that names a different database from the
pool: that pairing publishes successfully, delivers nothing, on every
replica, and reports no error anywhere.
Nothing publishes on it and nothing subscribes yet. The construction is
wired anyway, because the DSN has exactly one legitimate source and a
setting that decides whether any broadcast is delivered should not be
invented by whichever call site is migrated first.
Delivery is at-most-once, like NATS core. Nothing downstream may read a
message it did not receive as evidence about a node: a carrier that
cannot deliver is not a worker that is gone.
Assisted-by: Claude Opus 5 [claude-code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Ettore Di Giacinto