Ettore Di Giacinto 05c0a08e24 fix(galleryop): persist cancellation + periodically reap orphaned ops ...

Two distributed gaps surfaced when a replica was killed mid-upgrade on a live
cluster, leaving the backend stuck 'processing' in the UI forever:

1. CancelOperation flipped the in-memory status to cancelled and broadcast a
   NATS event but never persisted the terminal status. On the next replica
   restart the still-active row re-hydrated straight back into
   processingBackends and the UI spun again. It now calls store.Cancel(id) so
   the cancel survives a restart.

2. CleanStale (which marks abandoned active ops failed) only ran once on
   startup, so an op orphaned AFTER startup - its owning replica's foreground
   handler goroutine gone - was never reaped until the next restart. Add
   GalleryService.ReapStaleOperations and run it on a 15m ticker (CleanStale
   now returns the reaped count for observability).

Neither is covered by the OpCache self-evict fix: an orphaned op never reaches
Processed, so it would never self-evict.

Assisted-by: Claude:claude-opus-4-8 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

2026-06-07 23:10:53 +00:00

..

finetune.go

feat: add distributed mode (#9124)

2026-03-30 00:47:27 +02:00

gallery.go

fix(galleryop): persist cancellation + periodically reap orphaned ops

2026-06-07 23:10:53 +00:00

init.go

feat: add distributed mode (#9124)

2026-03-30 00:47:27 +02:00

skills.go

feat: add distributed mode (#9124)

2026-03-30 00:47:27 +02:00