4 Commits

Include renames

Author	SHA1	Message	Date
Ettore Di Giacinto	eb7ed9b910	refactor(distributed): delete MessagingClient and shrink the NATS client to fan-out ... Nothing in the tree publishes, subscribes, queue-subscribes or requests through the MessagingClient interface any more, so it is deleted rather than shrunk to Broadcaster: two exported names for one method set in one package is an invitation for the next author to pick whichever the surrounding file already imported. $ grep -rn 'messaging\.MessagingClient' --include='*.go' . core/services/syncstate/syncstate.go:54: // It is messaging.Broadcaster rather than messaging.MessagingClient because (one hit, a comment; no live referent. The naive grep in the plan also matches prose and the local test type names fakeMessagingClient and countingMessagingClient, so it can never be empty.) *messaging.Client is shrunk to exactly Broadcaster plus its own lifecycle. QueueSubscribe, QueueSubscribeReply, SubscribeReply, Request, Conn and the package helpers QueueSubscribeJSON and RequestJSON go with it; none had a production caller. Deleting the methods rather than only the call sites is what makes putting a family back on this carrier a build error instead of a line that compiles, publishes successfully, and is delivered onto a carrier the deployment is being taken off. Conn is in that list because while it existed every other name was one c.Conn().X() away; the flush-and-verdict that its real consumers needed is now ConfirmRoundTrip, which keeps the NATS JWT permission specs armed. The client, its options and its TLS plumbing are NOT deleted, and both processes stay on the bus. agent.<name>.cancel is the one fan-out family that could not move: its only subscriber is the agent worker, which has no database and cannot join the PostgreSQL carrier at all, so a cancel published there would reach no worker and be reported as sent. The frontend passes the client to newFanoutBridges as its cancelCarrier and the worker subscribes on it, so --nats-url stays required on agent-worker. Both go with the tunnel cancel verb. The struct field is renamed Nats -> CancelCarrier to say what it is for, and agentpool loses the messaging.Publisher it held only to be non-nil: it never published on it, and it was gating whether a frontend runs agents distributed or in an in-process pool. Retiring the bus would have flipped every replica back to the in-process pool silently. The gate now reads the agent store, which is the dependency the mode actually requires. Also deletes four subject builders with no production publisher (SubjectFineTuneProgress, SubjectFineTuneCancel, SubjectCacheInvalidateSkills, SubjectCacheInvalidateCollection), the queue and request/reply halves of the shared test double, and the e2e specs that were their only callers. Every surviving subject is now pinned to its exact literal, because a subject is a cross-version wire format and a rename that looks internal stops half a fleet hearing the other half. Docs: distributed-mode.md and cli-reference.md no longer claim NATS carries the agent-worker job subjects, the frontend's cross-replica events, or an agent worker's real work. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-09-27 03:05:13 +00:00
Ettore Di Giacinto	affad9c06d	feat(distributed): carry the state.*.delta families on PostgreSQL ... syncstate.Config held one carrier field typed as the NATS client, so a pgbus.Bus could not be handed to a SyncedMap at all: it satisfies messaging.Broadcaster and not MessagingClient. The durable re-hydration path built for the responses map therefore had a NATS-only consumer and nothing in the build said so. The field becomes Bus messaging.Broadcaster, SubscribeJSON moves to its own file and relaxes its parameter to Broadcaster, and the four adopters fan out over PostgreSQL LISTEN/NOTIFY: fine-tune jobs, quantization jobs, agent tasks with their per-tenant children, and Open Responses metadata. A new spec proves it on a real database, over two Bus instances on two pinned listener connections: a Set and a Delete carry, a payload past the 8000-byte notification cap comes back byte identical through the spill row, two families sharing one LISTEN channel stay separate, and a terminated listener re-hydrates a row written while it was gone. The five sites that each chose a carrier for an adopter are collapsed into one DistributedServices.Broadcast() accessor. Five field reads were five chances to leave one family on NATS with nothing failing, because messaging.Client satisfies Broadcaster too. The accessor also refuses to hand out a nil pgbus.Bus wrapped in a non-nil interface, which every adopter would read as "broadcast" and dereference on the first Set. SetTaskSyncNATS and SetJobSyncNATS are renamed to SetTaskSyncBus and SetJobSyncBus so a missed wiring site fails to compile. The response metadata table gains a retention of its own, defaulting to 24 hours. It inherited the Open Responses store TTL, which defaults to 0 meaning no expiration. Zero is defensible for a map that dies with the process and is not for a table: the table grew for the life of the deployment and a restarting replica re-hydrated every response the cluster had ever created. A row that names its own expiry is still judged on that column alone, and "this row is dead" now has one SQL spelling that PurgeExpired deletes by and ListUnexpired is the negation of, so a hydrate cannot resurrect what a sweep has already retired. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-09-27 03:05:12 +00:00
Ettore Di Giacinto	d9c5d031a8	fix(distributed): order LISTEN and UNLISTEN on one lock ... Unsubscribe decided a channel had lost its last subscriber under one lock and issued the UNLISTEN after releasing it. A Subscribe on the same root could decide to LISTEN in that window, and the two reached the connection in the wrong order: the root ended up not listened with a live subscription on it. It does not heal, because the next Subscribe sees the registration already there and never re-LISTENs, so the whole root stays deaf on that replica until the connection drops. The decision and the statement it implies now happen under one lock, held across both, at both call sites. A second lock and not the registration lock: issuing waits on the listener goroutine, delivery takes the registration lock, and holding that across the wait deadlocks the carrier. The race is spec'd through a barrier seam rather than by racing goroutines. The natural window is microseconds wide, and a spec that waits for it to open passes by luck; the seam scripts the interleaving, so the spec decides in both directions. Resolving a spilled message moved off the listener. PostgreSQL keeps undelivered notifications in a shared, fixed-size queue, so a listener that stops draining it can block COMMIT for every publisher on the server, not only this one. The listener now only drains; one resolver goroutine reads the row back and dispatches, which also keeps a spilled message and an inline one on the same subject in the order they were published. Three wiring lines that could be deleted with the suite staying green: the sweeper's start is now pinned by a Config interval, and the two lines that carry the bus into the deployment now refuse to boot when either is missing. A subscription can also report what it dropped, so the party that missed a message is the party that can see it. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-09-27 03:05:12 +00:00
Ettore Di Giacinto	17a1fa312e	feat(distributed): carry fan-out on PostgreSQL LISTEN/NOTIFY ... Distributed mode needs an operator to run a NATS cluster. This adds the carrier that replaces its fan-out half, so a deployment eventually needs PostgreSQL and its own HTTP listener and nothing else. pgbus holds one PostgreSQL session per replica, pinned for the life of the process because LISTEN registrations belong to one backend session and a pooled handle would lose them on the next checkout. Publishes go out on the pool with pg_notify. Subjects map onto a channel by their first token, from a closed set of roots. A subject outside the set is refused at publish AND at subscribe rather than mapped to a channel of its own: a channel name is capped at 63 bytes, and one LISTEN per job id would be unbounded. Refused rather than dropped, because a subject that goes nowhere and reports nothing is the class of defect this work exists to remove. PostgreSQL refuses a notify payload of 8000 bytes or more, and several subjects on this bus exceed that in normal operation: a job result carries a whole LLM output, a gallery progress event carries one entry per node. Those are written to a row and the notification carries the id. What is measured against the cap is the ENCODED notification, not the caller's payload, because the subject and the envelope travel too. The filter grammar is not respelled here. Subscribe asks messaging.ValidFilter and delivery asks messaging.SubjectMatches, which makes this the first production caller of a matcher that had only test doubles. New refuses a DSN that names a different database from the pool: that pairing publishes successfully, delivers nothing, on every replica, and reports no error anywhere. Nothing publishes on it and nothing subscribes yet. The construction is wired anyway, because the DSN has exactly one legitimate source and a setting that decides whether any broadcast is delivered should not be invented by whichever call site is migrated first. Delivery is at-most-once, like NATS core. Nothing downstream may read a message it did not receive as evidence about a node: a carrier that cannot deliver is not a worker that is gone. Assisted-by: Claude Opus 5 [claude-code] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>	2026-09-27 03:05:12 +00:00