fix(distributed): resync stored config revisions at startup

The controller pins a model's replicas to a stored revision and rejects
any request carrying a different one. Nothing ever re-derived that value
from the configuration on disk: it moved only on an edit, a gallery
install, or a peer's change broadcast. An inference request may only
establish a revision, never replace one.

So any other way for the two to diverge left the model permanently
unroutable. A configuration edited while a frontend was down lands
there, and so does a change in what the revision is computed over: an
upgrade that alters the hashed form leaves every stored revision
describing a configuration that no longer exists. The only recovery was
deleting the row by hand, which is not something a cluster should need.

Each frontend now reconciles the stored revisions against the loaded
configurations at startup and republishes the ones that disagree. Only
those: republishing quarantines every replica loaded under the old
revision, so doing it for a model that did not drift would unload a
healthy replica for nothing. A model with no stored revision has never
been served and is left for its first request to establish.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [golangci-lint]

core/application/startup.go

@@ -373,6 +373,16 @@ func New(opts ...config.AppOption) (*Application, error) {
 			cfgLoaderOpts := options.ToConfigLoaderOptions()
 			modelRevisionLifecycle := modeladmin.NewDistributedModelRevisionLifecycle(distSvc.Registry, distSvc.ModelCleanup)
 			gs.SetModelRevisionLifecycle(modelRevisionLifecycle)
+			// Bring the controller's stored revisions back in line with the
+			// configuration on disk. An inference request may only establish a
+			// revision, never replace one, so a model whose stored value had
+			// drifted stayed unroutable until someone deleted the row.
+			if err := modeladmin.ResyncModelConfigRevisions(options.Context,
+				application.ModelConfigLoader(),
+				modeladmin.NewRevisionStore(distSvc.Registry, modelRevisionLifecycle),
+			); err != nil {
+				xlog.Warn("Failed to resync model config revisions", "error", err)
+			}
 			gs.OnModelsChanged = func(evt messaging.CacheInvalidateEvent) {
 				// ApplyRemoteChange honors the op: a "delete" prunes the element
 				// (a reload-from-path is additive and cannot drop it), anything

core/services/modeladmin/revision_resync.go

@@ -0,0 +1,118 @@
+package modeladmin
+
+import (
+	"context"
+	"errors"
+	"fmt"
+
+	"github.com/mudler/xlog"
+	"gorm.io/gorm"
+
+	"github.com/mudler/LocalAI/core/config"
+)
+
+// ErrNoStoredRevision reports that the controller holds no revision for a
+// model, which is the normal state for one that has never been served.
+var ErrNoStoredRevision = gorm.ErrRecordNotFound
+
+// RevisionStore is the controller state this resync reads and corrects.
+type RevisionStore interface {
+	GetModelConfigRevision(ctx context.Context, modelName string) (string, error)
+	ApplyConfigRevisions(ctx context.Context, transitions []ModelRevisionTransition) (int, error)
+}
+
+// RevisionReader is the read half, satisfied by the node registry.
+type RevisionReader interface {
+	GetModelConfigRevision(ctx context.Context, modelName string) (string, error)
+}
+
+type revisionStore struct {
+	RevisionReader
+	lifecycle ModelRevisionLifecycle
+}
+
+func (s revisionStore) ApplyConfigRevisions(ctx context.Context, t []ModelRevisionTransition) (int, error) {
+	return s.lifecycle.ApplyConfigRevisions(ctx, t)
+}
+
+// NewRevisionStore pairs the registry that holds the stored revisions with the
+// lifecycle that publishes new ones. Returns nil when either half is missing,
+// which ResyncModelConfigRevisions treats as "nothing to reconcile".
+func NewRevisionStore(reader RevisionReader, lifecycle ModelRevisionLifecycle) RevisionStore {
+	if reader == nil || lifecycle == nil {
+		return nil
+	}
+	return revisionStore{RevisionReader: reader, lifecycle: lifecycle}
+}
+
+// ResyncModelConfigRevisions makes the controller's stored revision for each
+// model agree with what this build computes from the configuration on disk.
+//
+// The stored revision is what every inference request is checked against, but
+// nothing ever re-derived it from the persisted configuration: it moved only on
+// an edit, a gallery install, or a peer's change broadcast. Any other way for
+// the two to diverge left the model permanently unroutable, because an
+// inference request may only establish a revision, never replace one. A
+// configuration edited while this frontend was down, or a change in what the
+// revision is computed over, both landed there, and the only recovery was
+// deleting the row by hand.
+//
+// Running this at startup makes that self-correcting. Only a model whose stored
+// revision disagrees is republished, so replicas of models that did not drift
+// keep serving: republishing is not free, it quarantines every replica loaded
+// under the old revision.
+//
+// A model with no stored revision is left alone. It has never been served, and
+// inventing controller state for it here would quarantine nothing and describe
+// a model that may never be requested.
+func ResyncModelConfigRevisions(ctx context.Context, loader *config.ModelConfigLoader, store RevisionStore) error {
+	if loader == nil || store == nil {
+		return nil
+	}
+
+	var transitions []ModelRevisionTransition
+	for _, cfg := range loader.GetAllModelsConfigs() {
+		want, err := config.ModelConfigRevision(&cfg)
+		if err != nil {
+			return fmt.Errorf("compute config revision for %q: %w", cfg.Name, err)
+		}
+
+		stored, err := store.GetModelConfigRevision(ctx, cfg.Name)
+		if errors.Is(err, ErrNoStoredRevision) {
+			continue
+		}
+		if err != nil {
+			return fmt.Errorf("read stored config revision for %q: %w", cfg.Name, err)
+		}
+		if stored == want {
+			continue
+		}
+
+		xlog.Warn("Stored model config revision disagrees with the configuration on disk, republishing",
+			"model", cfg.Name, "stored", shortRevision(stored), "computed", shortRevision(want))
+		transitions = append(transitions, ModelRevisionTransition{
+			ModelName: cfg.Name, ConfigRevision: want, Disabled: cfg.IsDisabled(),
+		})
+	}
+
+	if len(transitions) == 0 {
+		return nil
+	}
+	if _, err := store.ApplyConfigRevisions(ctx, transitions); err != nil {
+		return fmt.Errorf("republish model config revisions: %w", err)
+	}
+	xlog.Info("Republished model config revisions to match the configuration on disk", "models", len(transitions))
+	return nil
+}
+
+// shortRevision trims a revision for log output; the leading bytes identify it
+// well enough to tell two apart.
+func shortRevision(revision string) string {
+	if revision == "" {
+		return "(none)"
+	}
+	if len(revision) > 12 {
+		return revision[:12]
+	}
+	return revision
+}

core/services/modeladmin/revision_resync_test.go

@@ -0,0 +1,142 @@
+package modeladmin
+
+import (
+	"context"
+	"errors"
+	"os"
+	"path/filepath"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+
+	"github.com/mudler/LocalAI/core/config"
+	"github.com/mudler/LocalAI/pkg/system"
+)
+
+// stubRevisionStore stands in for the controller's stored revisions.
+type stubRevisionStore struct {
+	stored  map[string]string
+	getErr  error
+	applied []ModelRevisionTransition
+	applyEr error
+}
+
+func (s *stubRevisionStore) GetModelConfigRevision(_ context.Context, name string) (string, error) {
+	if s.getErr != nil {
+		return "", s.getErr
+	}
+	rev, ok := s.stored[name]
+	if !ok {
+		return "", ErrNoStoredRevision
+	}
+	return rev, nil
+}
+
+func (s *stubRevisionStore) ApplyConfigRevisions(_ context.Context, t []ModelRevisionTransition) (int, error) {
+	s.applied = append(s.applied, t...)
+	return 0, s.applyEr
+}
+
+// The controller pins a model's replicas to a stored revision and rejects any
+// request carrying a different one. Nothing ever re-derived that stored value
+// from the configuration on disk: it only moved on an edit, a gallery install
+// or a peer's change event. So whenever the stored value stopped matching what
+// this build computes for an unchanged file, every request for that model was
+// rejected until an operator deleted the row by hand.
+var _ = Describe("ResyncModelConfigRevisions", func() {
+	var (
+		dir       string
+		loader    *config.ModelConfigLoader
+		store     *stubRevisionStore
+		appConfig *config.ApplicationConfig
+	)
+
+	write := func(name, body string) {
+		Expect(os.WriteFile(filepath.Join(dir, name+".yaml"), []byte(body), 0o600)).To(Succeed())
+	}
+
+	revisionOf := func(name string) string {
+		cfg, ok := loader.GetModelConfig(name)
+		Expect(ok).To(BeTrue())
+		rev, err := config.ModelConfigRevision(&cfg)
+		Expect(err).ToNot(HaveOccurred())
+		return rev
+	}
+
+	BeforeEach(func() {
+		dir = GinkgoT().TempDir()
+		appConfig = config.NewApplicationConfig()
+		appConfig.SystemState = &system.SystemState{Model: system.Model{ModelsPath: dir}}
+		loader = config.NewModelConfigLoader(dir)
+		store = &stubRevisionStore{stored: map[string]string{}}
+	})
+
+	load := func() {
+		Expect(loader.LoadModelConfigsFromPath(dir, appConfig.ToConfigLoaderOptions()...)).To(Succeed())
+	}
+
+	It("republishes the revision when the stored one no longer matches the config on disk", func() {
+		write("drifted", "name: drifted\nbackend: llama-cpp\ncontext_size: 4096\n")
+		load()
+		store.stored["drifted"] = "a-revision-from-an-earlier-build"
+
+		Expect(ResyncModelConfigRevisions(context.Background(), loader, store)).To(Succeed())
+
+		Expect(store.applied).To(HaveLen(1))
+		Expect(store.applied[0].ModelName).To(Equal("drifted"))
+		Expect(store.applied[0].ConfigRevision).To(Equal(revisionOf("drifted")))
+	})
+
+	It("leaves a model alone when the stored revision already matches", func() {
+		write("agreed", "name: agreed\nbackend: llama-cpp\n")
+		load()
+		store.stored["agreed"] = revisionOf("agreed")
+
+		Expect(ResyncModelConfigRevisions(context.Background(), loader, store)).To(Succeed())
+
+		Expect(store.applied).To(BeEmpty(), "republishing an unchanged revision would quarantine live replicas for nothing")
+	})
+
+	// A model nobody has served has no stored revision. Creating one here would
+	// invent controller state for a model that may never be requested; the first
+	// request establishes it.
+	It("does not create state for a model that has never been served", func() {
+		write("never-served", "name: never-served\nbackend: llama-cpp\n")
+		load()
+
+		Expect(ResyncModelConfigRevisions(context.Background(), loader, store)).To(Succeed())
+
+		Expect(store.applied).To(BeEmpty())
+	})
+
+	It("republishes only the models that actually drifted", func() {
+		write("drifted", "name: drifted\nbackend: llama-cpp\n")
+		write("agreed", "name: agreed\nbackend: llama-cpp\ncontext_size: 2048\n")
+		load()
+		store.stored["drifted"] = "stale"
+		store.stored["agreed"] = revisionOf("agreed")
+
+		Expect(ResyncModelConfigRevisions(context.Background(), loader, store)).To(Succeed())
+
+		Expect(store.applied).To(HaveLen(1))
+		Expect(store.applied[0].ModelName).To(Equal("drifted"))
+	})
+
+	It("reports a store failure instead of continuing silently", func() {
+		write("drifted", "name: drifted\nbackend: llama-cpp\n")
+		load()
+		store.stored["drifted"] = "stale"
+		store.applyEr = errors.New("database is down")
+
+		Expect(ResyncModelConfigRevisions(context.Background(), loader, store)).ToNot(Succeed())
+	})
+
+	It("skips a model whose stored revision cannot be read rather than guessing", func() {
+		write("unreadable", "name: unreadable\nbackend: llama-cpp\n")
+		load()
+		store.getErr = errors.New("connection reset")
+
+		Expect(ResyncModelConfigRevisions(context.Background(), loader, store)).ToNot(Succeed())
+		Expect(store.applied).To(BeEmpty())
+	})
+})

docs/content/features/distributed-mode.md

@@ -1041,8 +1041,9 @@ Notes:
 **Requests fail with `stale model config revision` although nobody edited the model:**
 - A model's stored revision must describe its persisted configuration. Releases before this fix also hashed the per-request prediction parameters, so the first request after a restart pinned the revision to its own `temperature`, `top_p`, `stop` and similar values. Every later request that sent different values was then rejected.
 - Upgrade the frontend replicas first. After the upgrade the revision is stamped when the configuration is loaded, so it no longer depends on the request body.
-- The stored revision does not heal on its own, because the recorded value belongs to no persisted configuration. Clear it once per affected model so the next request establishes the correct revision: `DELETE FROM model_config_states WHERE model_name = '<model>';`
-- Saving any edit for the model through the API or the WebUI has the same effect, because an edit publishes the current revision.
+- Each frontend now reconciles the stored revisions against the configuration on disk at startup, and republishes any that disagree, so a drifted revision heals on the next restart. Only models that actually drifted are republished, because republishing quarantines the replicas loaded under the old revision.
+- A model that has never been served has no stored revision and is left alone; its first request establishes one.
+- On a release without that reconciliation, clear the row once per affected model so the next request establishes the correct revision: `DELETE FROM model_config_states WHERE model_name = '<model>';` Saving any edit through the API or the WebUI has the same effect.
 
 **Port conflicts on workers:**
 - Each model gets its own gRPC process on an incrementing port (50051, 50052, ...)