fix(gallery): verification follow-ups for oci:// galleries (#12243)

* fix(gallery): verification follow-ups for oci:// galleries

Follow-ups from the post-merge review of #12238 and #12239.

Only a policy decision is a refusal now. cosignverify wraps
ErrPolicyRejected around a failed signature check, an identity or
source-repository mismatch, a not_before cutoff and a missing or
unparseable bundle. A TUF, registry or network failure during
verification, or a timeout, is an outage: the gallery falls back to the
copy verified under the current policy, as it does when the registry is
down.

An oci:// gallery with a verification block, or any oci:// gallery under
strict integrity, is no longer answered by an https://, github: or
file:// mirror. Such a mirror is ignored with a warning, because nothing
can check its signature. The index of an HTTP gallery, whose policy only
covers its backend images, is cached under the URL-only name again, so no
unchecked body is stored under a policy-keyed name.

The in-memory index cache key now includes the policy. After a runtime
policy change the index is fetched again, and entries with a relative url
install again.

The registry digest lookups after install and upgrade, and in the
upgrade check, run only for real registry references (new
URI.LooksLikeRegistryOCI), not for ollama:// or ocifile://.

The refusal message names strict integrity when that is the cause, and
the gallery name is no longer repeated.

Specs pin the URL-only cache name for galleries without a policy, a fixed
key for a fixed policy, and that every GalleryVerification field changes
the key. The docs describe refusal, outage, mirrors and strict integrity.

Assisted-by: Claude:claude-opus-5-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* fix(gallery): reset listings on gallery changes, classify referrer outages

Review follow-ups for this PR.

The React UI lists from AvailableGalleryModelsCached, which is keyed by
nothing. A gallery change through the settings API or a
runtime_settings.json edit now drops that listing when the model or
backend gallery configuration differs. Before, the UI kept the old list,
with local paths into the old policy's tree, until the next background
refresh, or for good when the new policy refused the gallery.

In cosignverify, a referrer the registry fails to serve now makes the
lookup an outage whatever other referrers failed and in any order, since
the unread one may be the valid signature. An invalid policy (Validate in
NewVerifier, an unparseable not_before) is ErrPolicyRejected, because no
fetch can make it usable.

The docs say that only an oci:// gallery with a verification block skips
non-OCI mirrors, and list an unusable policy as a refusal.

Assisted-by: Claude:claude-opus-5-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>

core/application/config_file_watcher.go

@@ -11,6 +11,7 @@ import (
 	"dario.cat/mergo"
 	"github.com/fsnotify/fsnotify"
 	"github.com/mudler/LocalAI/core/config"
+	"github.com/mudler/LocalAI/core/gallery"
 	"github.com/mudler/xlog"
 )
 
@@ -201,7 +202,9 @@ func readRuntimeSettingsJson(startupAppConfig config.ApplicationConfig) fileHand
 		// field previously changed via the API looks env-set to the
 		// baseline comparison, so a manual file edit of that field lands
 		// on the next restart instead of hot-applying.
+		prevGalleries, prevBackendGalleries := appConfig.Galleries, appConfig.BackendGalleries
 		appConfig.ApplyRuntimeSettingsAtStartup(&settings)
+		gallery.ResetGalleryModelCacheIfChanged(prevGalleries, prevBackendGalleries, appConfig)
 		if settings.ApiKeys != nil {
 			appConfig.ApiKeys = config.MergeAPIKeys(startupAppConfig.ApiKeys, *settings.ApiKeys)
 		}

core/application/runtime_settings_galleries_test.go

@@ -0,0 +1,51 @@
+package application
+
+import (
+	"encoding/json"
+	"net/http"
+	"net/http/httptest"
+
+	"github.com/mudler/LocalAI/core/config"
+	"github.com/mudler/LocalAI/core/gallery"
+	"github.com/mudler/LocalAI/pkg/system"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+// A manual edit of runtime_settings.json reaches the live config through the
+// file watcher, not the settings endpoint, and must drop the cached model
+// listing the same way: the UI lists from that cache.
+var _ = Describe("file watcher: galleries", func() {
+	It("drops the cached model listing when the galleries change", func() {
+		srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+			_, _ = w.Write([]byte("- name: acme-model\n"))
+		}))
+		DeferCleanup(srv.Close)
+		gallery.ResetGalleryModelCache()
+		DeferCleanup(gallery.ResetGalleryModelCache)
+
+		st, err := system.GetSystemState(system.WithModelPath(GinkgoT().TempDir()))
+		Expect(err).ToNot(HaveOccurred())
+		live := config.NewApplicationConfig()
+		live.SystemState = st
+		// At the default list, so the file value is not taken as env-set.
+		live.Galleries = config.DefaultRuntimeBaseline().Galleries
+
+		policy := config.GalleryVerification{Issuer: "https://token.actions.githubusercontent.com", IdentityRegex: "^https://github.com/acme/.*$"}
+		first, err := gallery.AvailableGalleryModelsCached([]config.Gallery{{Name: "old", URL: srv.URL}}, st)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(first).To(HaveLen(1))
+
+		body, err := json.Marshal(map[string]any{
+			"galleries": []config.Gallery{{Name: "acme", URL: srv.URL, Verification: &policy}},
+		})
+		Expect(err).ToNot(HaveOccurred())
+		Expect(readRuntimeSettingsJson(config.ApplicationConfig{})(body, live)).To(Succeed())
+		Expect(live.Galleries).To(HaveLen(1), "precondition: the file value was applied")
+
+		models, err := gallery.AvailableGalleryModelsCached(live.Galleries, st)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(models).To(HaveLen(1))
+		Expect(models[0].Gallery.Name).To(Equal("acme"), "the listing cached under the old gallery configuration was served")
+	})
+})

core/gallery/backends.go

@@ -25,6 +25,15 @@ import (
 // ErrBackendNotFound is returned when a backend is not found in the system.
 var ErrBackendNotFound = errors.New("backend not found")
 
+// lookupImageDigest asks a registry for the digest of an image reference. It
+// is what install and upgrade record for upgrade detection.
+//
+// A var so specs can see which references reach a registry client without
+// running one.
+var lookupImageDigest = func(ref string) (string, error) {
+	return oci.GetImageDigest(ref, "", nil, nil)
+}
+
 const (
 	metadataFile = "metadata.json"
 	runFile      = "run.sh"
@@ -184,7 +193,8 @@ func newGalleryVerifier(p *config.GalleryVerification) (*cosignverify.Verifier,
 	if p.NotBefore != "" {
 		t, err := time.Parse(time.RFC3339, p.NotBefore)
 		if err != nil {
-			return nil, fmt.Errorf("not_before %q: %w", p.NotBefore, err)
+			// A refusal, not an outage: no fetch can make this policy usable.
+			return nil, fmt.Errorf("%w: not_before %q: %w", cosignverify.ErrPolicyRejected, p.NotBefore, err)
 		}
 		pol.NotBefore = t
 	}
@@ -435,8 +445,8 @@ func InstallBackend(ctx context.Context, systemState *system.SystemState, modelL
 	}
 
 	// Record the OCI digest for upgrade detection (non-fatal on failure)
-	if uri.LooksLikeOCI() {
-		digest, digestErr := oci.GetImageDigest(uri.OCIReference(), "", nil, nil)
+	if uri.LooksLikeRegistryOCI() {
+		digest, digestErr := lookupImageDigest(uri.OCIReference())
 		if digestErr != nil {
 			xlog.Warn("Failed to get OCI image digest for backend", "uri", string(uri), "error", digestErr)
 		} else {

core/gallery/backends_oci_uri_test.go

@@ -11,6 +11,7 @@ import (
 	"github.com/google/go-containerregistry/pkg/v1/empty"
 	"github.com/google/go-containerregistry/pkg/v1/mutate"
 	"github.com/google/go-containerregistry/pkg/v1/remote"
+	"github.com/google/go-containerregistry/pkg/v1/tarball"
 	"github.com/mudler/LocalAI/core/config"
 	"github.com/mudler/LocalAI/pkg/model"
 	"github.com/mudler/LocalAI/pkg/system"
@@ -119,3 +120,75 @@ var _ = Describe("backends installed from an oci:// URI", func() {
 		Expect(installedDigest("acme-backend")).To(Equal(newDigest))
 	})
 })
+
+// ocifile:// and ollama:// look like OCI to the downloader but name no
+// registry image, so the digest lookup that follows an install or backs an
+// upgrade check can only fail against them, with a warning every time.
+var _ = Describe("registry digest lookups", func() {
+	var (
+		systemState *system.SystemState
+		ml          *model.ModelLoader
+		backendsDir string
+		looked      []string
+	)
+
+	BeforeEach(func() {
+		backendsDir = GinkgoT().TempDir()
+		var err error
+		systemState, err = system.GetSystemState(system.WithBackendPath(backendsDir))
+		Expect(err).ToNot(HaveOccurred())
+		ml = model.NewModelLoader(systemState)
+
+		looked = nil
+		original := lookupImageDigest
+		lookupImageDigest = func(ref string) (string, error) {
+			looked = append(looked, ref)
+			return original(ref)
+		}
+		DeferCleanup(func() { lookupImageDigest = original })
+	})
+
+	// ociTarball writes a minimal backend image as a local OCI tarball, the
+	// form an ocifile:// URI names.
+	ociTarball := func() string {
+		GinkgoHelper()
+		layer, err := crane.Layer(map[string][]byte{"run.sh": []byte("#!/bin/sh\necho local\n")})
+		Expect(err).ToNot(HaveOccurred())
+		img, err := mutate.AppendLayers(empty.Image, layer)
+		Expect(err).ToNot(HaveOccurred())
+		ref, err := name.ParseReference("local/backend:v1")
+		Expect(err).ToNot(HaveOccurred())
+		path := filepath.Join(GinkgoT().TempDir(), "backend.tar")
+		Expect(tarball.WriteToFile(path, ref, img)).To(Succeed())
+		return path
+	}
+
+	It("does not ask a registry for the digest of a backend installed from ocifile://", func() {
+		uri := "ocifile://" + ociTarball()
+
+		Expect(InstallBackend(context.Background(), systemState, ml, &GalleryBackend{
+			Metadata: Metadata{Name: "local-backend"},
+			URI:      uri,
+		}, nil, false)).To(Succeed())
+		Expect(filepath.Join(backendsDir, "local-backend", "run.sh")).To(BeARegularFile())
+		Expect(looked).To(BeEmpty(), "a local tarball was looked up in a registry")
+	})
+
+	It("does not ask a registry for the digest of an ocifile:// gallery entry in an upgrade check", func() {
+		uri := "ocifile://" + ociTarball()
+		Expect(InstallBackend(context.Background(), systemState, ml, &GalleryBackend{
+			Metadata: Metadata{Name: "local-backend"},
+			URI:      uri,
+		}, nil, false)).To(Succeed())
+
+		galleryFile := filepath.Join(backendsDir, "gallery.yaml")
+		data, err := yaml.Marshal([]GalleryBackend{{Metadata: Metadata{Name: "local-backend"}, URI: uri}})
+		Expect(err).ToNot(HaveOccurred())
+		Expect(os.WriteFile(galleryFile, data, 0o644)).To(Succeed())
+
+		_, err = CheckBackendUpgrades(context.Background(),
+			[]config.Gallery{{Name: "local", URL: "file://" + galleryFile}}, systemState)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(looked).To(BeEmpty(), "a local tarball was looked up in a registry")
+	})
+})

core/gallery/gallery.go

@@ -378,11 +378,11 @@ func GalleryGeneration() uint64 { return galleryGeneration.Load() }
 // ResetGalleryModelCache drops the cached model list, once any background
 // refresh already in flight has finished writing to it.
 //
-// It exists for tests. The cache is a package global keyed by nothing, which is
-// right for a process serving one gallery configuration and wrong for a suite
-// where each spec stands up its own: a refresh one spec triggered can land in
-// the middle of the next and answer it with the previous spec's entries, so
-// whichever assertion happens to straddle it fails at random.
+// The cache is a package global keyed by nothing, which is right for a process
+// serving one gallery configuration and wrong once that configuration changes:
+// see ResetGalleryModelCacheIfChanged. Suites use it too, because each spec
+// stands up its own configuration, and a refresh one spec triggered can land in
+// the middle of the next and answer it with the previous spec's entries.
 //
 // Waiting for the in-flight refresh rather than only clearing is the point. The
 // refresh publishes its result after this call would otherwise have returned,
@@ -400,6 +400,23 @@ func ResetGalleryModelCache() {
 	lastRefreshUnixNano.Store(0)
 }
 
+// ResetGalleryModelCacheIfChanged drops the cached model list when the model or
+// backend gallery configuration differs from what it was before a settings
+// change.
+//
+// The UI lists from that cache, and a gallery edit at runtime (a tightened
+// verification policy, a new mirror, another URL) must show at once. Kept
+// until the next background refresh, the old list would point relative entries
+// into a tree the new policy has not produced; and when the new policy refuses
+// the gallery, no refresh ever replaces it.
+func ResetGalleryModelCacheIfChanged(prevGalleries, prevBackendGalleries []config.Gallery, cfg *config.ApplicationConfig) {
+	if config.GalleriesEqual(prevGalleries, cfg.Galleries) &&
+		config.GalleriesEqual(prevBackendGalleries, cfg.BackendGalleries) {
+		return
+	}
+	ResetGalleryModelCache()
+}
+
 // AvailableGalleryModelsCached returns gallery models from an in-memory cache.
 // Local-only fields (installed status) are refreshed on every call. A background
 // goroutine is triggered to re-fetch the full model list (including network
@@ -609,6 +626,17 @@ func (entry galleryCacheEntry) hasExpired() bool {
 
 var galleryCache = xsync.NewSyncedMap[string, galleryCacheEntry]()
 
+// galleryIndexCacheKey names a gallery's entry in the in-memory index cache.
+//
+// The verification policy is part of it for the same reason it is part of the
+// on-disk name: the gallery settings can change at runtime, and a listing that
+// an older policy admitted must not keep being served under a new one. It
+// would also point relative entry urls at an unpacked tree the new policy has
+// not produced yet, so they could not be installed.
+func galleryIndexCacheKey(g config.Gallery) string {
+	return g.Name + "-" + galleryCacheName(g.URL, g.Verification)
+}
+
 func getGalleryElements[T GalleryElement](gallery config.Gallery, basePath string, requireIntegrity bool, isInstalledCallback func(T) bool) ([]T, error) {
 	var models []T = []T{}
 
@@ -620,7 +648,7 @@ func getGalleryElements[T GalleryElement](gallery config.Gallery, basePath strin
 		}
 	}
 
-	cacheKey := fmt.Sprintf("%s-%s", gallery.Name, gallery.URL)
+	cacheKey := galleryIndexCacheKey(gallery)
 	if galleryCache.Exists(cacheKey) {
 		entry := galleryCache.Get(cacheKey)
 		// refresh if last updated is more than 1 hour ago

core/gallery/gallery_mirrors.go

@@ -133,6 +133,44 @@ func galleryCacheName(url string, policy *config.GalleryVerification) string {
 	return hex.EncodeToString(sum[:])
 }
 
+// indexCachePolicy is the policy a gallery's index is actually checked
+// against, which is what its cached copy may be named after.
+//
+// Only an oci:// gallery has its index verified. An HTTP gallery can carry a
+// verification block too, for the backend images it lists, but nothing checks
+// the index it serves, so naming that copy after the policy would claim a
+// check that never happened.
+func indexCachePolicy(g config.Gallery) *config.GalleryVerification {
+	if !looksLikeOCIGallery(g.URL) {
+		return nil
+	}
+	return g.Verification
+}
+
+// verifiableCandidates drops the candidates that cannot answer for a signed
+// gallery.
+//
+// A gallery whose index is signature-checked, or would have to be under strict
+// integrity, can only be served by sources the check applies to. An https://,
+// github: or file:// mirror of it would hand back an index no policy looked
+// at, and after a refusal it would turn "this artifact is not trusted" into
+// "use this other, unchecked copy instead".
+func verifiableCandidates(g config.Gallery, candidates []string, requireIntegrity bool) []string {
+	if !looksLikeOCIGallery(g.URL) || (g.Verification == nil && !requireIntegrity) {
+		return candidates
+	}
+	out := make([]string, 0, len(candidates))
+	for _, c := range candidates {
+		if !looksLikeOCIGallery(c) {
+			xlog.Warn("ignoring a gallery mirror that cannot be signature-checked: a signed oci:// gallery is only served from oci:// sources",
+				"gallery", g.Name, "url", c)
+			continue
+		}
+		out = append(out, c)
+	}
+	return out
+}
+
 // isUsableGalleryIndex reports whether body is worth keeping as the last known
 // good copy.
 //
@@ -222,7 +260,7 @@ func persistGalleryIndex(basePath, url string, policy *config.GalleryVerificatio
 // path is returned as the source. Nothing else in the chain helps a machine
 // that has no network at all.
 func fetchGalleryIndex(ctx context.Context, g config.Gallery, basePath string, requireIntegrity bool) ([]byte, string, error) {
-	candidates := galleryCandidates(g)
+	candidates := verifiableCandidates(g, galleryCandidates(g), requireIntegrity)
 	if len(candidates) == 0 {
 		return nil, "", fmt.Errorf("gallery %q has no URL", g.Name)
 	}
@@ -237,7 +275,8 @@ func fetchGalleryIndex(ctx context.Context, g config.Gallery, basePath string, r
 		attempt = candidates
 	}
 
-	var lastErr, refused error
+	var lastErr error
+	var refused *galleryVerificationError
 	for _, candidate := range attempt {
 		attemptCtx, cancel := context.WithTimeout(ctx, galleryFetchTimeout)
 
@@ -248,7 +287,8 @@ func fetchGalleryIndex(ctx context.Context, g config.Gallery, basePath string, r
 		// rest of this loop does not care which it was, so mirrors, cooldown,
 		// the per-candidate timeout and the last known good copy all work the
 		// same for both schemes, and a gallery can even mirror an OCI primary
-		// with an HTTP fallback.
+		// with an HTTP fallback, unless its index must be signature-checked
+		// (see verifiableCandidates).
 		if looksLikeOCIGallery(candidate) {
 			body, err = fetchOCIGalleryIndex(attemptCtx, g, candidate, basePath, requireIntegrity)
 		} else {
@@ -268,14 +308,14 @@ func fetchGalleryIndex(ctx context.Context, g config.Gallery, basePath string, r
 			// Keyed on the gallery's own URL rather than the candidate that
 			// answered: a mirror serves the same index, so a mirror-served
 			// fetch must refresh the copy an offline run will look for.
-			persistGalleryIndex(basePath, g.URL, g.Verification, body)
+			persistGalleryIndex(basePath, g.URL, indexCachePolicy(g), body)
 			return body, candidate, nil
 		}
 
 		lastErr = err
 		var notVerified *galleryVerificationError
 		if errors.As(err, &notVerified) {
-			refused = err
+			refused = notVerified
 		}
 		// Only blame the source for its own failures. If the caller gave up —
 		// a browser disconnecting mid-listing, once a request context is wired
@@ -293,14 +333,18 @@ func fetchGalleryIndex(ctx context.Context, g config.Gallery, basePath string, r
 	// this machine must not trust. Falling back to an older copy there would
 	// turn a refusal into a silent downgrade, so it is reported instead.
 	if refused != nil {
-		return nil, "", fmt.Errorf("gallery %q was refused by its verification policy and no cached copy is served: %w", g.Name, refused)
+		by := "its verification policy"
+		if refused.strict {
+			by = "strict integrity (--require-backend-integrity)"
+		}
+		return nil, "", fmt.Errorf("gallery %q was refused by %s and no cached copy is served: %w", g.Name, by, refused)
 	}
 
 	// Every source failed. A copy from a previous run is much better than no
 	// gallery at all — this is what lets an offline or airgapped machine still
 	// list what it already knows about. The copy is looked up under the
 	// current policy, so it is one that policy admitted.
-	cachePath := galleryCachePath(basePath, g.URL, g.Verification)
+	cachePath := galleryCachePath(basePath, g.URL, indexCachePolicy(g))
 	if cachePath != "" {
 		// #nosec G304 -- cachePath is galleryCachePath's own construction: a
 		// hex sha256 of the URL under the fixed <basePath>/../cache/gallery

core/gallery/gallery_mirrors_test.go

@@ -350,7 +350,7 @@ var _ = Describe("getGalleryElements", func() {
 		mirror, _ := countingServer(http.StatusOK, "- name: mirror-model\n  description: served by a mirror\n")
 
 		g := config.Gallery{Name: "mirror-fallback-spec", URL: primary.URL, Mirrors: []string{mirror.URL}}
-		DeferCleanup(func() { galleryCache.Delete(g.Name + "-" + g.URL) })
+		DeferCleanup(func() { galleryCache.Delete(galleryIndexCacheKey(g)) })
 
 		models, err := getGalleryElements(g, tempModelsDir(), false, func(*GalleryModel) bool { return false })
 		Expect(err).ToNot(HaveOccurred())
@@ -359,7 +359,7 @@ var _ = Describe("getGalleryElements", func() {
 
 		// The cache identifies the gallery, not whichever source answered, so a
 		// mirror-served fetch must populate the entry the primary URL would hit.
-		Expect(galleryCache.Exists(g.Name + "-" + g.URL)).To(BeTrue(),
+		Expect(galleryCache.Exists(galleryIndexCacheKey(g))).To(BeTrue(),
 			"mirror-served index was not cached under the gallery's own key")
 	})
 })

core/gallery/gallery_oci.go

@@ -2,6 +2,7 @@ package gallery
 
 import (
 	"context"
+	"errors"
 	"fmt"
 	"os"
 	"path/filepath"
@@ -11,6 +12,7 @@ import (
 	"github.com/mudler/LocalAI/core/config"
 	"github.com/mudler/LocalAI/pkg/downloader"
 	"github.com/mudler/LocalAI/pkg/oci"
+	"github.com/mudler/LocalAI/pkg/oci/cosignverify"
 	"github.com/mudler/xlog"
 )
 
@@ -36,7 +38,13 @@ const (
 // refused by the verification policy (or by strict integrity), as opposed to
 // one that could not reach it. The caller must not answer a refusal with an
 // older cached copy.
-type galleryVerificationError struct{ err error }
+//
+// strict tells the two refusals apart, so the message names the setting the
+// operator has to change rather than a policy that may not even exist.
+type galleryVerificationError struct {
+	err    error
+	strict bool
+}
 
 func (e *galleryVerificationError) Error() string { return e.err.Error() }
 func (e *galleryVerificationError) Unwrap() error { return e.err }
@@ -147,7 +155,10 @@ func fetchOCIGalleryIndex(ctx context.Context, g config.Gallery, candidate, base
 	// off was never verified, and turning strict integrity on must not keep
 	// serving it for the rest of its TTL.
 	if g.Verification == nil && requireIntegrity {
-		return nil, &galleryVerificationError{fmt.Errorf("strict integrity: gallery %q has no verification policy for %q (set verification: in the gallery configuration or disable --require-backend-integrity)", g.Name, candidate)}
+		return nil, &galleryVerificationError{
+			strict: true,
+			err:    fmt.Errorf("no verification policy is set for %q (set verification: in the gallery configuration or disable --require-backend-integrity)", candidate),
+		}
 	}
 
 	cacheDir := ociGalleryCacheDir(basePath, candidate, g.Verification)
@@ -169,7 +180,15 @@ func fetchOCIGalleryIndex(ctx context.Context, g config.Gallery, candidate, base
 			return nil, err
 		}
 		if err := verifyGalleryArtifact(ctx, g.Verification, digestRef); err != nil {
-			return nil, &galleryVerificationError{fmt.Errorf("gallery %q failed signature verification: %w", g.Name, err)}
+			// Only a decision about the artifact is a refusal. The
+			// verifier also reaches the Sigstore TUF mirror and the
+			// registry, and a timeout or a 5xx there says nothing about
+			// the gallery: it is an outage, and the caller may serve the
+			// copy this same policy verified before.
+			if errors.Is(err, cosignverify.ErrPolicyRejected) {
+				return nil, &galleryVerificationError{err: fmt.Errorf("signature verification of %q failed: %w", candidate, err)}
+			}
+			return nil, fmt.Errorf("could not verify the signature of %q: %w", candidate, err)
 		}
 		pullRef = digestRef
 	} else {

core/gallery/gallery_oci_outcomes_test.go

@@ -0,0 +1,307 @@
+package gallery
+
+import (
+	"context"
+	"crypto/sha256"
+	"encoding/hex"
+	"errors"
+	"fmt"
+	"net"
+	"net/http"
+	"os"
+	"path/filepath"
+	"reflect"
+	"strings"
+	"sync"
+	"sync/atomic"
+
+	"github.com/google/go-containerregistry/pkg/v1/remote/transport"
+	"github.com/mudler/LocalAI/core/config"
+	"github.com/mudler/LocalAI/pkg/oci/cosignverify"
+	"github.com/mudler/LocalAI/pkg/system"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+// verifierVerdict is what a stubbed verifier answers, changeable mid-spec.
+// atomic.Value would do, except that it cannot hold a nil error.
+type verifierVerdict struct {
+	mu  sync.Mutex
+	err error
+}
+
+func (v *verifierVerdict) Store(err error) {
+	v.mu.Lock()
+	defer v.mu.Unlock()
+	v.err = err
+}
+
+func (v *verifierVerdict) Load() error {
+	v.mu.Lock()
+	defer v.mu.Unlock()
+	return v.err
+}
+
+var _ = Describe("oci:// gallery verification outcomes", func() {
+	const index = "- name: acme-model\n"
+
+	policy := &config.GalleryVerification{
+		Issuer:        "https://token.actions.githubusercontent.com",
+		IdentityRegex: "^https://github.com/acme/.*$",
+	}
+
+	BeforeEach(resetGalleryFailures)
+
+	// signedGallery publishes a gallery, fetches it once under the policy so
+	// a verified copy is on disk, and expires the unpacked cache so the next
+	// fetch goes back to the registry.
+	signedGallery := func() (config.Gallery, string, *verifierVerdict) {
+		GinkgoHelper()
+		srv, _, _ := ociRegistry()
+		url := pushGalleryArtifact(srv.URL, "galleries/acme", galleryArtifactType, []ociGalleryFile{
+			{title: "index.yaml", body: index},
+		})
+		verdict := &verifierVerdict{}
+		stubGalleryVerifier(func(context.Context, *config.GalleryVerification, string) error {
+			return verdict.Load()
+		})
+		base := tempModelsDir()
+		g := config.Gallery{URL: url, Name: "acme", Verification: policy}
+		_, _, err := fetchGalleryIndex(context.Background(), g, base, false)
+		Expect(err).ToNot(HaveOccurred())
+		expireOCIGalleryCache()
+		return g, base, verdict
+	}
+
+	// The verifier fetches the Sigstore trusted root and the signature from
+	// the network. When that fails it has decided nothing about the gallery,
+	// so the copy verified under the same policy is served, as it is when
+	// the registry itself is down.
+	DescribeTable("falls back to the verified copy when verification cannot reach its sources",
+		func(failure error) {
+			g, base, verdict := signedGallery()
+			verdict.Store(failure)
+
+			body, served, err := fetchGalleryIndex(context.Background(), g, base, false)
+			Expect(err).ToNot(HaveOccurred())
+			Expect(string(body)).To(Equal(index))
+			Expect(served).To(Equal(galleryCachePath(base, g.URL, policy)))
+		},
+		Entry("a timeout", fmt.Errorf("cosignverify: fetching trusted_root.json: %w", context.DeadlineExceeded)),
+		Entry("a registry 5xx", fmt.Errorf("cosignverify: querying referrers: %w",
+			&transport.Error{StatusCode: http.StatusServiceUnavailable})),
+		Entry("a connection error", fmt.Errorf("cosignverify: querying referrers: %w",
+			&net.OpError{Op: "dial", Net: "tcp", Err: errors.New("connection refused")})),
+	)
+
+	It("does not fall back when the policy rejects the signature", func() {
+		g, base, verdict := signedGallery()
+		verdict.Store(fmt.Errorf("cosignverify: verification failed: %w", cosignverify.ErrPolicyRejected))
+
+		body, served, err := fetchGalleryIndex(context.Background(), g, base, false)
+		Expect(err).To(HaveOccurred(), "served %q from %q", string(body), served)
+		var refused *galleryVerificationError
+		Expect(errors.As(err, &refused)).To(BeTrue(), "not reported as a refusal: %v", err)
+		Expect(strings.Count(err.Error(), `"acme"`)).To(Equal(1), "gallery name repeated: %v", err)
+	})
+
+	// An unusable policy admits nothing, so it is reported as a refusal and
+	// not as an outage. The real verifier is used: it fails while building
+	// the policy, before it would reach the network.
+	It("reports a policy that cannot be used as a refusal", func() {
+		srv, _, _ := ociRegistry()
+		url := pushGalleryArtifact(srv.URL, "galleries/acme", galleryArtifactType, []ociGalleryFile{
+			{title: "index.yaml", body: index},
+		})
+		broken := *policy
+		broken.NotBefore = "last tuesday"
+
+		_, _, err := fetchGalleryIndex(context.Background(),
+			config.Gallery{URL: url, Name: "acme", Verification: &broken}, tempModelsDir(), false)
+		Expect(err).To(HaveOccurred())
+		var refused *galleryVerificationError
+		Expect(errors.As(err, &refused)).To(BeTrue(), "an unusable policy was reported as an outage: %v", err)
+		Expect(err.Error()).To(ContainSubstring("not_before"))
+	})
+
+	Context("with a mirror that is not an OCI artifact", func() {
+		It("does not let the mirror answer for a refused primary", func() {
+			g, base, verdict := signedGallery()
+			mirror, hits := countingServer(http.StatusOK, "- name: evil\n")
+			g.Mirrors = []string{mirror.URL}
+			Expect(galleryCachePath(base, g.URL, policy)).To(BeARegularFile())
+			verdict.Store(fmt.Errorf("cosignverify: %w", cosignverify.ErrPolicyRejected))
+
+			body, served, err := fetchGalleryIndex(context.Background(), g, base, false)
+			Expect(err).To(HaveOccurred(), "served %q from %q", string(body), served)
+			Expect(hits.Load()).To(BeZero(), "an unsigned mirror was asked for a signed gallery")
+			onDisk, readErr := os.ReadFile(galleryCachePath(base, g.URL, policy))
+			Expect(readErr).ToNot(HaveOccurred())
+			Expect(string(onDisk)).To(Equal(index), "the mirror's body replaced the verified copy")
+		})
+
+		It("does not let the mirror answer for an unreachable primary", func() {
+			srv, _, _ := ociRegistry()
+			url := pushGalleryArtifact(srv.URL, "galleries/acme", galleryArtifactType, []ociGalleryFile{
+				{title: "index.yaml", body: index},
+			})
+			stubGalleryVerifier(func(context.Context, *config.GalleryVerification, string) error { return nil })
+			srv.Close()
+			mirror, hits := countingServer(http.StatusOK, "- name: evil\n")
+			base := tempModelsDir()
+
+			body, served, err := fetchGalleryIndex(context.Background(),
+				config.Gallery{URL: url, Name: "acme", Mirrors: []string{mirror.URL}, Verification: policy}, base, false)
+			Expect(err).To(HaveOccurred(), "served %q from %q", string(body), served)
+			Expect(hits.Load()).To(BeZero())
+			Expect(galleryCachePath(base, url, policy)).ToNot(BeAnExistingFile())
+		})
+
+		It("does not let the mirror answer for a primary refused by strict integrity", func() {
+			srv, _, _ := ociRegistry()
+			url := pushGalleryArtifact(srv.URL, "galleries/acme", galleryArtifactType, []ociGalleryFile{
+				{title: "index.yaml", body: index},
+			})
+			mirror, hits := countingServer(http.StatusOK, "- name: evil\n")
+
+			body, served, err := fetchGalleryIndex(context.Background(),
+				config.Gallery{URL: url, Name: "acme", Mirrors: []string{mirror.URL}}, tempModelsDir(), true)
+			Expect(err).To(HaveOccurred(), "served %q from %q", string(body), served)
+			Expect(err.Error()).To(ContainSubstring("strict integrity"))
+			Expect(hits.Load()).To(BeZero())
+		})
+
+		It("still uses the mirror when the gallery has no policy", func() {
+			srv, _, _ := ociRegistry()
+			url := pushGalleryArtifact(srv.URL, "galleries/acme", galleryArtifactType, []ociGalleryFile{
+				{title: "index.yaml", body: index},
+			})
+			srv.Close()
+			mirror, _ := countingServer(http.StatusOK, "- name: mirrored\n")
+
+			body, served, err := fetchGalleryIndex(context.Background(),
+				config.Gallery{URL: url, Name: "acme", Mirrors: []string{mirror.URL}}, tempModelsDir(), false)
+			Expect(err).ToNot(HaveOccurred())
+			Expect(served).To(Equal(mirror.URL))
+			Expect(string(body)).To(Equal("- name: mirrored\n"))
+		})
+	})
+
+	// A backend gallery served over HTTPS carries a verification block for
+	// the backend images it lists. Nothing checks its index, so the index
+	// must not be stored under a name that claims a policy admitted it.
+	It("keeps the URL-only cache name for an HTTP gallery whose policy covers its backends", func() {
+		srv, _ := countingServer(http.StatusOK, index)
+		base := tempModelsDir()
+		g := config.Gallery{URL: srv.URL, Name: "backends", Verification: policy}
+
+		_, _, err := fetchGalleryIndex(context.Background(), g, base, false)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(galleryCachePath(base, srv.URL, policy)).ToNot(BeAnExistingFile())
+		Expect(galleryCachePath(base, srv.URL, nil)).To(BeARegularFile())
+
+		srv.Close()
+		_, served, err := fetchGalleryIndex(context.Background(), g, base, false)
+		Expect(err).ToNot(HaveOccurred())
+		Expect(served).To(Equal(galleryCachePath(base, srv.URL, nil)))
+	})
+
+	It("says strict integrity refused the gallery, not its verification policy", func() {
+		srv, _, _ := ociRegistry()
+		url := pushGalleryArtifact(srv.URL, "galleries/acme", galleryArtifactType, []ociGalleryFile{
+			{title: "index.yaml", body: index},
+		})
+
+		_, _, err := fetchGalleryIndex(context.Background(),
+			config.Gallery{URL: url, Name: "acme"}, tempModelsDir(), true)
+		Expect(err).To(HaveOccurred())
+		Expect(err.Error()).To(ContainSubstring("strict integrity"))
+		Expect(err.Error()).ToNot(ContainSubstring("refused by its verification policy"))
+		Expect(strings.Count(err.Error(), `"acme"`)).To(Equal(1), "gallery name repeated: %v", err)
+	})
+
+	// The listing cache sits in front of the on-disk one. Keyed without the
+	// policy, it kept listing what the old policy admitted after a runtime
+	// settings change, and a relative entry then pointed at an unpacked tree
+	// that the new policy had not produced.
+	It("lists and installs again after the policy changes", func() {
+		srv, _, _ := ociRegistry()
+		url := pushGalleryArtifact(srv.URL, "galleries/relative-policy", galleryArtifactType, []ociGalleryFile{
+			{title: "index.yaml", body: "- name: relative-entry\n  url: base/virtual.yaml\n"},
+			{title: "base/virtual.yaml", body: "name: virtual\nconfig_file: |\n  backend: llama\n"},
+		})
+		var calls atomic.Int64
+		stubGalleryVerifier(func(context.Context, *config.GalleryVerification, string) error {
+			calls.Add(1)
+			return nil
+		})
+		base := tempModelsDir()
+		systemState, err := system.GetSystemState(system.WithModelPath(base))
+		Expect(err).ToNot(HaveOccurred())
+
+		install := func(v *config.GalleryVerification) {
+			GinkgoHelper()
+			galleries := []config.Gallery{{Name: "relative-policy", URL: url, Verification: v}}
+			models, err := AvailableGalleryModels(galleries, systemState)
+			Expect(err).ToNot(HaveOccurred())
+			Expect(galleryModelNames(models)).To(ConsistOf("relative-entry"))
+			Expect(InstallModelFromGallery(context.Background(), galleries, nil, systemState, nil,
+				"relative-policy@relative-entry", GalleryModel{}, noProgress, false, false, false)).To(Succeed())
+		}
+
+		install(policy)
+		Expect(calls.Load()).To(Equal(int64(1)))
+
+		tightened := *policy
+		tightened.SourceRepository = "https://github.com/acme/gallery"
+		install(&tightened)
+		Expect(calls.Load()).To(Equal(int64(2)), "the listing verified under the old policy was reused")
+		Expect(filepath.Join(base, "relative-entry.yaml")).To(BeAnExistingFile())
+	})
+})
+
+// pinnedPolicyCacheName is galleryCacheName of the fixed policy below.
+const pinnedPolicyCacheName = "fe2f2092cdd99f3979c185bd3590873721e2c9394fd04411962d2e5d5b84333f"
+
+var _ = Describe("galleryCacheName", func() {
+	const url = "oci://registry.example.com/acme/gallery:latest"
+	full := config.GalleryVerification{
+		Issuer:           "https://token.actions.githubusercontent.com",
+		IssuerRegex:      "^https://token\\.actions\\.githubusercontent\\.com$",
+		Identity:         "https://github.com/acme/gallery/.github/workflows/publish.yml@refs/heads/main",
+		IdentityRegex:    "^https://github\\.com/acme/.*$",
+		SourceRepository: "https://github.com/acme/gallery",
+		NotBefore:        "2026-01-01T00:00:00Z",
+	}
+
+	// Copies cached before policies became part of the name must stay usable
+	// after an upgrade, so a gallery without a policy keeps the old name.
+	It("keeps the URL-only name for a gallery without a policy", func() {
+		sum := sha256.Sum256([]byte(url))
+		Expect(galleryCacheName(url, nil)).To(Equal(hex.EncodeToString(sum[:])))
+	})
+
+	// Pinned so a reordered struct or a field that loses omitempty, either
+	// of which renames every policy's cache and drops its offline copy on
+	// upgrade, is a visible change rather than a silent one.
+	It("keeps a stable name for a fixed policy", func() {
+		Expect(galleryCacheName(url, &full)).To(Equal(pinnedPolicyCacheName))
+	})
+
+	// A field left out of the key would let a copy admitted by one value of
+	// it be served under another. Walking the struct makes a newly added
+	// field fail here until it is part of the key.
+	It("changes the name when any policy field changes", func() {
+		base := galleryCacheName(url, &full)
+		v := reflect.ValueOf(&full).Elem()
+		for i := range v.NumField() {
+			changed := full
+			field := reflect.ValueOf(&changed).Elem().Field(i)
+			Expect(field.Kind()).To(Equal(reflect.String),
+				"GalleryVerification.%s is not a string: extend this spec and galleryCacheName for it", v.Type().Field(i).Name)
+			field.SetString(field.String() + "-changed")
+			Expect(galleryCacheName(url, &changed)).ToNot(Equal(base),
+				"GalleryVerification.%s is not part of the cache name", v.Type().Field(i).Name)
+		}
+	})
+})

core/gallery/gallery_oci_test.go

@@ -3,7 +3,7 @@ package gallery
 import (
 	"bytes"
 	"context"
-	"errors"
+	"fmt"
 	"net/http"
 	"net/http/httptest"
 	"os"
@@ -13,6 +13,7 @@ import (
 
 	"github.com/google/go-containerregistry/pkg/registry"
 	"github.com/mudler/LocalAI/core/config"
+	"github.com/mudler/LocalAI/pkg/oci/cosignverify"
 	. "github.com/onsi/ginkgo/v2"
 	. "github.com/onsi/gomega"
 	ocispec "github.com/opencontainers/image-spec/specs-go/v1"
@@ -24,7 +25,7 @@ import (
 
 // errNoGallerySignature stands in for what the real verifier reports when an
 // artifact has no signature attached.
-var errNoGallerySignature = errors.New("no signature found for the gallery artifact")
+var errNoGallerySignature = fmt.Errorf("no signature found for the gallery artifact: %w", cosignverify.ErrPolicyRejected)
 
 // stubGalleryVerifier replaces the signature check for the duration of a spec
 // and restores it afterwards.

core/gallery/upgrade.go

@@ -11,7 +11,6 @@ import (
 	"github.com/mudler/LocalAI/core/config"
 	"github.com/mudler/LocalAI/pkg/downloader"
 	"github.com/mudler/LocalAI/pkg/model"
-	"github.com/mudler/LocalAI/pkg/oci"
 	"github.com/mudler/LocalAI/pkg/system"
 	"github.com/mudler/xlog"
 	cp "github.com/otiai10/copy"
@@ -169,8 +168,8 @@ func CheckUpgradesAgainst(ctx context.Context, galleries []config.Gallery, syste
 		}
 
 		// Fall back to OCI digest comparison when versions are unavailable.
-		if galleryURI := downloader.URI(galleryEntry.URI); galleryURI.LooksLikeOCI() {
-			remoteDigest, err := oci.GetImageDigest(galleryURI.OCIReference(), "", nil, nil)
+		if galleryURI := downloader.URI(galleryEntry.URI); galleryURI.LooksLikeRegistryOCI() {
+			remoteDigest, err := lookupImageDigest(galleryURI.OCIReference())
 			if err != nil {
 				xlog.Warn("Failed to get remote OCI digest for upgrade check", "backend", installed.Metadata.Name, "error", err)
 				continue
@@ -353,8 +352,8 @@ func UpgradeBackend(ctx context.Context, systemState *system.SystemState, modelL
 	}
 
 	// Record OCI digest if applicable (non-fatal on failure)
-	if uri.LooksLikeOCI() {
-		digest, digestErr := oci.GetImageDigest(uri.OCIReference(), "", nil, nil)
+	if uri.LooksLikeRegistryOCI() {
+		digest, digestErr := lookupImageDigest(uri.OCIReference())
 		if digestErr != nil {
 			xlog.Warn("Failed to get OCI image digest after upgrade", "uri", galleryEntry.URI, "error", digestErr)
 		} else {

core/http/endpoints/localai/settings.go

@@ -10,6 +10,7 @@ import (
 	"github.com/labstack/echo/v4"
 	"github.com/mudler/LocalAI/core/application"
 	"github.com/mudler/LocalAI/core/config"
+	"github.com/mudler/LocalAI/core/gallery"
 	"github.com/mudler/LocalAI/core/http/endpoints/openresponses"
 	"github.com/mudler/LocalAI/core/p2p"
 	"github.com/mudler/LocalAI/core/schema"
@@ -186,7 +187,9 @@ func UpdateSettingsEndpoint(app *application.Application) echo.HandlerFunc {
 		}
 
 		// Apply settings using centralized method
+		prevGalleries, prevBackendGalleries := appConfig.Galleries, appConfig.BackendGalleries
 		watchdogChanged := appConfig.ApplyRuntimeSettings(&settings)
+		gallery.ResetGalleryModelCacheIfChanged(prevGalleries, prevBackendGalleries, appConfig)
 		if settings.VRAMPersistentCache != nil || settings.AutoloadGalleries != nil {
 			if appConfig.VRAMPersistentCache && appConfig.AutoloadGalleries {
 				vram.ConfigurePersistentCache(filepath.Join(appConfig.SystemState.Model.ModelsPath, "..", "cache", "vram"), 24*time.Hour)

core/http/endpoints/localai/settings_test.go

@@ -12,6 +12,7 @@ import (
 	"github.com/labstack/echo/v4"
 	"github.com/mudler/LocalAI/core/application"
 	"github.com/mudler/LocalAI/core/config"
+	"github.com/mudler/LocalAI/core/gallery"
 	. "github.com/mudler/LocalAI/core/http/endpoints/localai"
 	"github.com/mudler/LocalAI/pkg/system"
 	. "github.com/onsi/ginkgo/v2"
@@ -179,4 +180,43 @@ var _ = Describe("Settings endpoints", func() {
 		Expect(app.ModelLoader().GetWatchDog()).ToNot(BeNil(),
 			"watchdog should be running after a cold enable, without waiting for a restart")
 	})
+	// The UI lists from the cached model listing, which is keyed by nothing.
+	// A gallery change through this endpoint (for example a tightened
+	// verification policy) must drop it, or the UI keeps showing what the old
+	// configuration admitted until the next background refresh, or forever
+	// when the new policy refuses the gallery.
+	It("drops the cached model listing when the galleries change", func() {
+		srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+			_, _ = w.Write([]byte("- name: acme-model\n"))
+		}))
+		DeferCleanup(srv.Close)
+		gallery.ResetGalleryModelCache()
+		DeferCleanup(gallery.ResetGalleryModelCache)
+
+		loose := config.GalleryVerification{Issuer: "https://token.actions.githubusercontent.com", IdentityRegex: "^https://github.com/acme/.*$"}
+		appConfig := app.ApplicationConfig()
+		appConfig.Galleries = []config.Gallery{{Name: "acme", URL: srv.URL, Verification: &loose}}
+
+		listedPolicy := func() config.GalleryVerification {
+			GinkgoHelper()
+			models, err := gallery.AvailableGalleryModelsCached(appConfig.Galleries, appConfig.SystemState)
+			Expect(err).ToNot(HaveOccurred())
+			Expect(models).To(HaveLen(1))
+			Expect(models[0].Gallery.Verification).ToNot(BeNil())
+			return *models[0].Gallery.Verification
+		}
+		Expect(listedPolicy()).To(Equal(loose))
+
+		tightened := loose
+		tightened.SourceRepository = "https://github.com/acme/gallery"
+		body, err := json.Marshal(map[string]any{
+			"galleries": []config.Gallery{{Name: "acme", URL: srv.URL, Verification: &tightened}},
+		})
+		Expect(err).ToNot(HaveOccurred())
+		rec := post(string(body))
+		Expect(rec.Code).To(Equal(http.StatusOK), rec.Body.String())
+		Expect(appConfig.Galleries[0].Verification.SourceRepository).To(Equal(tightened.SourceRepository), "precondition: the setting was applied")
+
+		Expect(listedPolicy()).To(Equal(tightened), "the listing cached under the old gallery configuration was served")
+	})
 })

docs/content/features/model-gallery.md

@@ -136,7 +136,16 @@ GALLERIES=[{"name":"premium","url":"oci://quay.io/acme/gallery:latest","verifica
 
 The tag is resolved to a digest, the signature is checked against that digest, and the same digest is then pulled. A gallery that fails verification is never written to the cache, so no unverified file reaches your disk. The optional `not_before` RFC3339 value revokes signatures logged before that time, exactly as it does for backends.
 
-Cached copies of a gallery are kept per verification policy. When you change the `verification` block (for example, you add `source_repository` or move `not_before` forward), LocalAI fetches the gallery again and verifies it under the new policy. It does not serve a copy that an older policy admitted. If the registry is unreachable, LocalAI serves the last copy that was verified under the current policy. If the registry answers with an artifact that fails verification, LocalAI shows an error and does not serve a cached copy.
+Cached copies of a gallery are kept per verification policy on disk, and the in-memory listings are dropped when the gallery settings change. When you change the `verification` block (for example, you add `source_repository` or move `not_before` forward), in the configuration, at runtime through the settings API, or by editing `runtime_settings.json`, the next listing fetches the gallery again and verifies it under the new policy. LocalAI does not serve a copy that an older policy admitted.
+
+LocalAI tells a refusal apart from an outage:
+
+- **Refusal.** The artifact has no signature, or its signature does not match the issuer, identity, `source_repository` or `not_before` of the policy, or the policy itself cannot be used (for example, `not_before` is not an RFC3339 time). LocalAI shows an error and does not serve a cached copy.
+- **Outage.** The registry or the Sigstore trust root cannot be reached, answers with a server error, or the fetch times out. This includes failures during the signature check itself. LocalAI serves the last copy that was verified under the current policy. If no such copy exists, it shows an error. It never serves a copy that was verified under a different policy.
+
+An `oci://` gallery with a `verification` block is only served from `oci://` sources. LocalAI cannot check a signature on an `https://`, `github:` or `file://` mirror, so it ignores these mirrors for that gallery and logs a warning. List only `oci://` mirrors for a signed gallery.
+
+With strict integrity on (`--require-backend-integrity` or `LOCALAI_REQUIRE_BACKEND_INTEGRITY`), an `oci://` gallery without a `verification` block is refused. This is also true when a copy from an earlier fetch is in the cache, because that copy was never verified, and when the gallery has an `https://`, `github:` or `file://` mirror, because LocalAI ignores these mirrors in strict mode too.
 
 The optional `source_repository` value works the same for `oci://` galleries as it does for backends: it pins the repository the signature was made for when a shared reusable workflow does the signing. See [Verifying OCI Backends]({{%relref "features/backends#verifying-oci-backends" %}}).
 

pkg/downloader/uri.go

@@ -315,10 +315,26 @@ func (s URI) LooksLikeOCI() bool {
 		strings.HasPrefix(string(s), "docker.io")
 }
 
+// LooksLikeRegistryOCI reports whether the URI names an image in a registry.
+//
+// LooksLikeOCI also accepts ollama:// and ocifile://, which the downloader
+// pulls through its OCI path but which name no registry image: a caller about
+// to ask a registry about the URI must check this instead, or it sends
+// "ollama://..." or "ocifile:///path" to a registry client that can only fail.
+func (s URI) LooksLikeRegistryOCI() bool {
+	return s.LooksLikeOCI() &&
+		!strings.HasPrefix(string(s), OllamaPrefix) &&
+		!strings.HasPrefix(string(s), OCIFilePrefix)
+}
+
 // OCIReference returns the registry reference an OCI URI names, without the
 // oci:// scheme. The scheme is LocalAI's own marker for "this is an image":
 // registry clients do not know it and read "oci" as the registry host, so
 // every consumer that hands a URI to a registry client goes through here.
+//
+// It is only meaningful for a URI that LooksLikeRegistryOCI: ollama:// and
+// ocifile:// URIs come back unchanged, and no registry client can resolve
+// them.
 func (s URI) OCIReference() string {
 	return strings.TrimPrefix(string(s), OCIPrefix)
 }

pkg/downloader/uri_test.go

@@ -98,6 +98,19 @@ var _ = Describe("OCIReference", func() {
 	)
 })
 
+var _ = Describe("LooksLikeRegistryOCI", func() {
+	DescribeTable("accepts only URIs that name an image in a registry",
+		func(uri string, want bool) {
+			Expect(URI(uri).LooksLikeRegistryOCI()).To(Equal(want))
+		},
+		Entry("oci:// reference", "oci://registry.example.com/acme/backend:v1", true),
+		Entry("bare quay.io reference", "quay.io/acme/backend:latest", true),
+		Entry("ollama:// model", "ollama://gemma:2b", false),
+		Entry("ocifile:// tarball", "ocifile:///srv/backend.tar", false),
+		Entry("https URL", "https://example.com/backend.tar", false),
+	)
+})
+
 var _ = Describe("ContentLength", func() {
 	Context("local file", func() {
 		It("returns file size for existing file", func() {

pkg/oci/cosignverify/bundle.go

@@ -61,17 +61,26 @@ func bundleFromOCISignature(ref name.Reference, imageDigest v1.Hash, opts []remo
 	}
 
 	if len(manifest.Manifests) == 0 {
-		return nil, fmt.Errorf("cosignverify: no referrers found for %s", digestRef.Name())
+		return nil, fmt.Errorf("cosignverify: no referrers found for %s: %w", digestRef.Name(), ErrPolicyRejected)
 	}
 
-	var lastErr error
+	// outage remembers a referrer the registry failed to serve. That one may
+	// be the valid signature, so it decides the result whatever else failed
+	// and in whatever order the index lists them.
+	var lastErr, outage error
+	noteFailure := func(err error) {
+		lastErr = err
+		if !errors.Is(err, ErrPolicyRejected) {
+			outage = err
+		}
+	}
 	for _, desc := range manifest.Manifests {
 		if !isSigstoreBundleArtifactType(string(desc.ArtifactType)) {
 			continue
 		}
 		b, err := fetchBundleFromReferrer(ref, desc, opts)
 		if err != nil {
-			lastErr = err
+			noteFailure(err)
 			continue
 		}
 		return b, nil
@@ -90,21 +99,32 @@ func bundleFromOCISignature(ref name.Reference, imageDigest v1.Hash, opts []remo
 		if isSigstoreBundleArtifactType(string(desc.ArtifactType)) {
 			continue // already tried above
 		}
-		if !isBundleManifest(ref, desc, opts) {
+		isBundle, err := isBundleManifest(ref, desc, opts)
+		if err != nil {
+			// Unread is not unsigned: a referrer the registry failed to
+			// serve may be the signature, so an outage here must not end
+			// up reported as "no signature".
+			noteFailure(err)
+			continue
+		}
+		if !isBundle {
 			continue
 		}
 		b, err := fetchBundleFromReferrer(ref, desc, opts)
 		if err != nil {
-			lastErr = err
+			noteFailure(err)
 			continue
 		}
 		return b, nil
 	}
 
+	if outage != nil {
+		return nil, fmt.Errorf("cosignverify: could not read every referrer of %s: %w", digestRef.Name(), outage)
+	}
 	if lastErr != nil {
 		return nil, fmt.Errorf("cosignverify: no usable Sigstore bundle referrer for %s: %w", digestRef.Name(), lastErr)
 	}
-	return nil, fmt.Errorf("cosignverify: no Sigstore bundle referrer for %s (signed with --new-bundle-format?)", digestRef.Name())
+	return nil, fmt.Errorf("cosignverify: no Sigstore bundle referrer for %s (signed with --new-bundle-format?): %w", digestRef.Name(), ErrPolicyRejected)
 }
 
 // maxReferrersInspected bounds the second pass. Each step there is a manifest
@@ -117,20 +137,20 @@ const maxReferrersInspected = 16
 // artifactType and its first layer are checked: the layer is what actually
 // holds the bundle, and a manifest can reach a registry with neither field
 // copied onto the index.
-func isBundleManifest(ref name.Reference, desc v1.Descriptor, opts []remote.Option) bool {
+func isBundleManifest(ref name.Reference, desc v1.Descriptor, opts []remote.Option) (bool, error) {
 	artRef := ref.Context().Digest(desc.Digest.String())
 	img, err := remote.Image(artRef, opts...)
 	if err != nil {
-		return false
+		return false, fmt.Errorf("fetching referrer image %s: %w", artRef.Name(), err)
 	}
 	m, err := img.Manifest()
 	if err != nil {
-		return false
+		return false, fmt.Errorf("reading referrer manifest %s: %w", artRef.Name(), err)
 	}
 	if isSigstoreBundleArtifactType(m.ArtifactType) {
-		return true
+		return true, nil
 	}
-	return len(m.Layers) > 0 && isSigstoreBundleArtifactType(string(m.Layers[0].MediaType))
+	return len(m.Layers) > 0 && isSigstoreBundleArtifactType(string(m.Layers[0].MediaType)), nil
 }
 
 func fetchBundleFromReferrer(ref name.Reference, desc v1.Descriptor, opts []remote.Option) (*bundle.Bundle, error) {
@@ -144,7 +164,7 @@ func fetchBundleFromReferrer(ref name.Reference, desc v1.Descriptor, opts []remo
 		return nil, fmt.Errorf("reading referrer layers: %w", err)
 	}
 	if len(layers) == 0 {
-		return nil, errors.New("referrer artifact has no layers")
+		return nil, fmt.Errorf("referrer artifact has no layers: %w", ErrPolicyRejected)
 	}
 
 	rc, err := layers[0].Uncompressed()
@@ -160,7 +180,9 @@ func fetchBundleFromReferrer(ref name.Reference, desc v1.Descriptor, opts []remo
 
 	b := &bundle.Bundle{}
 	if err := b.UnmarshalJSON(data); err != nil {
-		return nil, fmt.Errorf("parsing bundle JSON: %w", err)
+		// The registry served this referrer in full; what it holds is not
+		// a signature, which is an answer about the image.
+		return nil, fmt.Errorf("parsing bundle JSON: %w: %w", ErrPolicyRejected, err)
 	}
 	return b, nil
 }

pkg/oci/cosignverify/policy_rejected_internal_test.go

@@ -0,0 +1,190 @@
+package cosignverify
+
+import (
+	"encoding/json"
+	"errors"
+	"net/http"
+	"net/http/httptest"
+	"net/url"
+	"strings"
+	"time"
+
+	"github.com/google/go-containerregistry/pkg/name"
+	v1 "github.com/google/go-containerregistry/pkg/v1"
+	"github.com/google/go-containerregistry/pkg/v1/remote"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+	"github.com/sigstore/sigstore-go/pkg/verify"
+)
+
+// A caller that keeps a copy verified earlier serves it through an outage and
+// refuses it over a policy decision, so the two must be told apart by the
+// error alone.
+var _ = Describe("ErrPolicyRejected", func() {
+	registryRef := func(h http.Handler) name.Reference {
+		GinkgoHelper()
+		srv := httptest.NewServer(h)
+		DeferCleanup(srv.Close)
+		u, err := url.Parse(srv.URL)
+		Expect(err).NotTo(HaveOccurred())
+		ref, err := name.ParseReference(u.Host+"/gallery:v1", name.Insecure)
+		Expect(err).NotTo(HaveOccurred())
+		return ref
+	}
+	// One attempt: the retry backoff would only slow the 5xx spec down.
+	noRetry := []remote.Option{remote.WithRetryBackoff(remote.Backoff{Steps: 1})}
+
+	It("marks a signature older than not_before as a policy decision", func() {
+		cutoff := time.Date(2026, 5, 14, 12, 0, 0, 0, time.UTC)
+		res := &verify.VerificationResult{VerifiedTimestamps: []verify.TimestampVerificationResult{{Timestamp: cutoff.Add(-time.Hour)}}}
+		Expect(errors.Is(enforceNotBefore(res, cutoff), ErrPolicyRejected)).To(BeTrue())
+	})
+
+	It("marks an image with no signature at all as a policy decision", func() {
+		reg, subjectDigest := signedByCosignWithoutReferrersAPI([]byte(`{}`))
+		// Drop the referrers-tag index: the image is now simply unsigned.
+		for k := range reg.manifests {
+			if k != subjectDigest && len(k) > 7 && k[:7] == "sha256-" {
+				delete(reg.manifests, k)
+			}
+		}
+		_, err := bundleFromOCISignature(registryRef(reg), mustHash(subjectDigest), noRetry)
+		Expect(err).To(HaveOccurred())
+		Expect(errors.Is(err, ErrPolicyRejected)).To(BeTrue(), "got %v", err)
+	})
+
+	It("marks a signature referrer that is not a valid bundle as a policy decision", func() {
+		reg, subjectDigest := signedByCosignWithoutReferrersAPI([]byte(`{"not":"a bundle"}`))
+		_, err := bundleFromOCISignature(registryRef(reg), mustHash(subjectDigest), noRetry)
+		Expect(err).To(HaveOccurred())
+		Expect(errors.Is(err, ErrPolicyRejected)).To(BeTrue(), "got %v", err)
+	})
+
+	It("does not mark a registry that answers 5xx as a policy decision", func() {
+		_, subjectDigest := signedByCosignWithoutReferrersAPI([]byte(`{}`))
+		down := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+			if r.URL.Path == "/v2/" {
+				w.WriteHeader(http.StatusOK)
+				return
+			}
+			http.Error(w, "upstream down", http.StatusServiceUnavailable)
+		})
+		_, err := bundleFromOCISignature(registryRef(down), mustHash(subjectDigest), noRetry)
+		Expect(err).To(HaveOccurred())
+		Expect(errors.Is(err, ErrPolicyRejected)).To(BeFalse(), "an outage was reported as a refusal: %v", err)
+	})
+})
+
+// referrer is one entry of a test referrers index.
+type referrer struct {
+	// advertised puts the Sigstore artifact type on the index entry, so the
+	// first pass picks it up; otherwise only the second pass, which asks the
+	// manifest itself, can find it.
+	advertised bool
+	// unavailable makes the registry answer 503 for the referrer manifest.
+	unavailable bool
+	// bundle is the referrer's payload.
+	bundle []byte
+}
+
+// registryWithReferrers serves a subject whose referrers-tag index lists the
+// given referrers, in order.
+func registryWithReferrers(refs ...referrer) (*fakeRegistry, string) {
+	reg := &fakeRegistry{manifests: map[string]manifestEntry{}, blobs: map[string][]byte{}, unavailable: map[string]bool{}}
+	emptyCfg := []byte("{}")
+	reg.blobs[digestOf(emptyCfg)] = emptyCfg
+	cfg := v1.Descriptor{MediaType: "application/vnd.oci.empty.v1+json", Size: int64(len(emptyCfg)), Digest: mustHash(digestOf(emptyCfg))}
+
+	subject, _ := json.Marshal(v1.Manifest{SchemaVersion: 2, MediaType: "application/vnd.oci.image.manifest.v1+json", Config: cfg})
+	subjectDigest := digestOf(subject)
+	reg.manifests[subjectDigest] = manifestEntry{mediaType: "application/vnd.oci.image.manifest.v1+json", body: subject}
+
+	entries := []v1.Descriptor{}
+	for _, r := range refs {
+		reg.blobs[digestOf(r.bundle)] = r.bundle
+		sig, _ := json.Marshal(v1.Manifest{
+			SchemaVersion: 2,
+			MediaType:     "application/vnd.oci.image.manifest.v1+json",
+			ArtifactType:  "application/vnd.dev.sigstore.bundle.v0.3+json",
+			Config:        cfg,
+			Layers: []v1.Descriptor{{
+				MediaType: "application/vnd.dev.sigstore.bundle.v0.3+json",
+				Size:      int64(len(r.bundle)),
+				Digest:    mustHash(digestOf(r.bundle)),
+			}},
+		})
+		sigDigest := digestOf(sig)
+		reg.manifests[sigDigest] = manifestEntry{mediaType: "application/vnd.oci.image.manifest.v1+json", body: sig}
+		if r.unavailable {
+			reg.unavailable[sigDigest] = true
+		}
+		artifactType := "application/vnd.oci.empty.v1+json"
+		if r.advertised {
+			artifactType = "application/vnd.dev.sigstore.bundle.v0.3+json"
+		}
+		entries = append(entries, v1.Descriptor{
+			MediaType:    "application/vnd.oci.image.manifest.v1+json",
+			Size:         int64(len(sig)),
+			Digest:       mustHash(sigDigest),
+			ArtifactType: artifactType,
+		})
+	}
+	idx, _ := json.Marshal(v1.IndexManifest{SchemaVersion: 2, MediaType: "application/vnd.oci.image.index.v1+json", Manifests: entries})
+	reg.manifests[strings.Replace(subjectDigest, ":", "-", 1)] = manifestEntry{mediaType: "application/vnd.oci.image.index.v1+json", body: idx}
+	return reg, subjectDigest
+}
+
+var _ = Describe("referrers the registry fails to serve", func() {
+	lookup := func(reg *fakeRegistry, subjectDigest string) error {
+		GinkgoHelper()
+		srv := httptest.NewServer(reg)
+		DeferCleanup(srv.Close)
+		u, err := url.Parse(srv.URL)
+		Expect(err).NotTo(HaveOccurred())
+		ref, err := name.ParseReference(u.Host+"/gallery:v1", name.Insecure)
+		Expect(err).NotTo(HaveOccurred())
+		_, err = bundleFromOCISignature(ref, mustHash(subjectDigest),
+			[]remote.Option{remote.WithRetryBackoff(remote.Backoff{Steps: 1})})
+		Expect(err).To(HaveOccurred())
+		return err
+	}
+	notABundle := []byte(`{"not":"a bundle"}`)
+
+	// The index is served but the only referrer is not: that referrer may be
+	// the signature, so this is an outage and not an unsigned image.
+	It("reports an outage when an unadvertised referrer returns 503", func() {
+		err := lookup(registryWithReferrers(referrer{unavailable: true, bundle: notABundle}))
+		Expect(errors.Is(err, ErrPolicyRejected)).To(BeFalse(), "an unread referrer was reported as no signature: %v", err)
+	})
+
+	// Which failure comes last in the index is an accident of the registry.
+	// An unread referrer may be the valid signature, so any outage makes the
+	// whole lookup an outage, whatever the order.
+	DescribeTable("reports an outage when one referrer is unreadable and another is not a bundle",
+		func(refs ...referrer) {
+			err := lookup(registryWithReferrers(refs...))
+			Expect(errors.Is(err, ErrPolicyRejected)).To(BeFalse(), "reported as a refusal: %v", err)
+		},
+		Entry("unreadable first",
+			referrer{advertised: true, unavailable: true, bundle: []byte(`{"a":1}`)},
+			referrer{advertised: true, bundle: notABundle}),
+		Entry("unreadable last",
+			referrer{advertised: true, bundle: notABundle},
+			referrer{advertised: true, unavailable: true, bundle: []byte(`{"a":1}`)}),
+	)
+
+	It("still reports a refusal when every referrer was read and none is a bundle", func() {
+		err := lookup(registryWithReferrers(referrer{advertised: true, bundle: notABundle}))
+		Expect(errors.Is(err, ErrPolicyRejected)).To(BeTrue(), "got %v", err)
+	})
+})
+
+// A policy that cannot be built admits nothing, whatever the network does,
+// so it must not read as an outage that a cached copy could cover.
+var _ = Describe("an invalid policy", func() {
+	It("is a policy decision", func() {
+		_, err := NewVerifier(Policy{Issuer: "https://token.actions.githubusercontent.com"}, nil, nil)
+		Expect(err).To(HaveOccurred())
+		Expect(errors.Is(err, ErrPolicyRejected)).To(BeTrue(), "got %v", err)
+	})
+})

pkg/oci/cosignverify/referrers_internal_test.go

@@ -22,8 +22,9 @@ import (
 // does not register the route at all, so every client falls back to the
 // referrers-tag scheme.
 type fakeRegistry struct {
-	manifests map[string]manifestEntry // by tag and by digest
-	blobs     map[string][]byte        // by digest
+	manifests   map[string]manifestEntry // by tag and by digest
+	blobs       map[string][]byte        // by digest
+	unavailable map[string]bool          // manifest digests answered with a 503
 }
 
 type manifestEntry struct {
@@ -43,6 +44,10 @@ func (f *fakeRegistry) ServeHTTP(w http.ResponseWriter, r *http.Request) {
 		http.NotFound(w, r)
 	case strings.Contains(p, "/manifests/"):
 		ref := p[strings.LastIndex(p, "/manifests/")+len("/manifests/"):]
+		if f.unavailable[ref] {
+			http.Error(w, "upstream down", http.StatusServiceUnavailable)
+			return
+		}
 		m, ok := f.manifests[ref]
 		if !ok {
 			http.Error(w, "unknown manifest", http.StatusNotFound)

pkg/oci/cosignverify/verify.go

@@ -40,6 +40,15 @@ import (
 	"github.com/sigstore/sigstore-go/pkg/verify"
 )
 
+// ErrPolicyRejected marks a verification that reached a decision: the image
+// carries no signature, or its signature does not satisfy the policy.
+//
+// Callers need the distinction because every other failure here (the TUF
+// root or the registry being unreachable, a timeout, a 5xx) says nothing
+// about the image, and a caller that keeps a copy verified earlier may serve
+// it through an outage but must never serve it over a refusal.
+var ErrPolicyRejected = errors.New("rejected by the signature policy")
+
 // Policy is the verification policy a backend image must satisfy.
 //
 // At least one of Issuer / IssuerRegex must be set, and at least one of
@@ -153,7 +162,9 @@ type Verifier struct {
 // it is loaded on the first call to VerifyImage. auth and t may be nil.
 func NewVerifier(p Policy, auth *registrytypes.AuthConfig, t http.RoundTripper) (*Verifier, error) {
 	if err := p.Validate(); err != nil {
-		return nil, err
+		// A policy that cannot be used admits nothing, so a caller must
+		// treat it as a refusal and not as an outage a cached copy covers.
+		return nil, fmt.Errorf("%w: %w", ErrPolicyRejected, err)
 	}
 	return &Verifier{policy: p, auth: auth, transport: t}, nil
 }
@@ -266,7 +277,9 @@ func (v *Verifier) VerifyImage(ctx context.Context, imageRef string) error {
 
 	certID, err := v.policy.certificateIdentity()
 	if err != nil {
-		return fmt.Errorf("cosignverify: building identity policy: %w", err)
+		// A policy that cannot be built admits nothing, whatever the
+		// network does, so this is a decision rather than an outage.
+		return fmt.Errorf("cosignverify: building identity policy: %w: %w", ErrPolicyRejected, err)
 	}
 
 	sev, err := verify.NewVerifier(trusted, verifierOpts...)
@@ -282,7 +295,7 @@ func (v *Verifier) VerifyImage(ctx context.Context, imageRef string) error {
 
 	result, err := sev.Verify(bun, verify.NewPolicy(artifactPolicy, verify.WithCertificateIdentity(certID)))
 	if err != nil {
-		return fmt.Errorf("cosignverify: verification failed for %s: %w", imageRef, err)
+		return fmt.Errorf("cosignverify: verification failed for %s: %w: %w", imageRef, ErrPolicyRejected, err)
 	}
 
 	if !v.policy.NotBefore.IsZero() {
@@ -302,7 +315,7 @@ func enforceNotBefore(result *verify.VerificationResult, cutoff time.Time) error
 		// timestamp, so this branch is only reachable if a caller set
 		// RequireTLog=false. Treat as a hard error: if you opted into
 		// NotBefore, you implicitly opted into needing a timestamp.
-		return errors.New("signature has no verified timestamp; cannot enforce NotBefore")
+		return fmt.Errorf("%w: signature has no verified timestamp; cannot enforce NotBefore", ErrPolicyRejected)
 	}
 	earliest := result.VerifiedTimestamps[0].Timestamp
 	for _, ts := range result.VerifiedTimestamps[1:] {
@@ -311,8 +324,8 @@ func enforceNotBefore(result *verify.VerificationResult, cutoff time.Time) error
 		}
 	}
 	if earliest.Before(cutoff) {
-		return fmt.Errorf("signature integrated time %s is before NotBefore cutoff %s",
-			earliest.Format(time.RFC3339), cutoff.Format(time.RFC3339))
+		return fmt.Errorf("%w: signature integrated time %s is before NotBefore cutoff %s",
+			ErrPolicyRejected, earliest.Format(time.RFC3339), cutoff.Format(time.RFC3339))
 	}
 	return nil
 }