* feat(config): add a limit for concurrent backend operations
Add --max-concurrent-backend-operations
(LOCALAI_MAX_CONCURRENT_BACKEND_OPERATIONS) and the runtime setting
max_concurrent_backend_operations. It sets how many backend install,
upgrade and delete operations run at the same time. The default is 1,
which keeps the current behaviour. Values below 1 become 1.
The runtime value is also kept in an atomic, so the scheduler that reads
it never races with a settings change. The setting shows in the Settings
page under Backends. This commit only adds the setting; the next one
uses it.
Assisted-by: Claude:claude-sonnet-5-5 go-vet
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
* feat(galleryop): run backend operations in a bounded pool
One goroutine read both gallery channels and ran each operation inline,
so one slow backend operation held every operation behind it, model
operations included. In distributed mode the manager also visited the
nodes one at a time and waited for each up to the install timeout.
Backend operations now go through a scheduler that runs up to
max_concurrent_backend_operations of them at once. Operations start in
the order they arrive. Install, upgrade and delete of one backend share
a key and never overlap; an operation that waits for its backend does
not hold a slot, so independent operations pass it. Cancelling a queued
operation drops it at once. The limit is read on every scheduling step,
so a runtime change applies without a restart. With the default of 1 the
behaviour is the old single worker. Model operations get their own loop
and stay serial.
In distributed mode the manager sends one operation to its nodes in
parallel, up to the same limit, and keeps the result in node order. A
per (node, backend) lock keeps a live operation and the reconciler from
sending the same operation to a node together; the reconciler skips a
row that is locked and checks that the row still exists before it sends.
Timeouts are unchanged.
The download progress timers were plain package variables. They are
atomics now, because operations can run in parallel.
Assisted-by: Claude:claude-sonnet-5-5 go-vet go-test
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
---------
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
