From a05a7900214f1f0bd198fc9b7cfea6326d16e202 Mon Sep 17 00:00:00 2001 From: localai-org-maint-bot Date: Wed, 5 Aug 2026 09:39:35 +0200 Subject: [PATCH] fix(ci): emit verifiable backend signature bundles (#11366) Cosign v2.4.1 does not select the Sigstore bundle format by default, while LocalAI's verifier only consumes OCI bundle referrers. Request the format explicitly for both registries and guard the producer contract with a shell regression test. Document strict backend integrity configuration and release-tag identities for operators. Assisted-by: Codex:gpt-5 Co-authored-by: localai-org-maint-bot <306269227+localai-org-maint-bot@users.noreply.github.com> --- .agents/backend-signing.md | 10 +++---- .github/workflows/backend_merge.yml | 6 +++-- docs/content/features/backends.md | 38 +++++++++++++++++++++++++++ scripts/build/backend-signing_test.sh | 13 +++++++++ 4 files changed, 60 insertions(+), 7 deletions(-) create mode 100755 scripts/build/backend-signing_test.sh diff --git a/.agents/backend-signing.md b/.agents/backend-signing.md index 3abb31d7a..98c32d3e9 100644 --- a/.agents/backend-signing.md +++ b/.agents/backend-signing.md @@ -16,8 +16,7 @@ side (`pkg/oci/cosignverify` plus the gallery YAML). per-arch manifest before checking signatures. - **Storage:** Signatures are written as OCI 1.1 referrers (`--registry-referrers-mode=oci-1-1`) in the new Sigstore bundle format - (current cosign releases do this by default; no `--new-bundle-format` - flag). No `:sha256-.sig` tag clutter. + (`--new-bundle-format`). No `:sha256-.sig` tag clutter. - **Consumer:** `pkg/oci/cosignverify` discovers the bundle via the referrers API, hands it to `sigstore-go`, and verifies it against the policy declared in the gallery YAML (`Gallery.Verification`). @@ -34,14 +33,15 @@ to sign. The job needs: - `permissions: { id-token: write, contents: read }` at the job level so the runner can exchange its GitHub OIDC token for a Fulcio cert. -- `sigstore/cosign-installer@v3` step (current cosign releases already - default to the new bundle format). +- `sigstore/cosign-installer@v3` step (the pinned cosign v2 release needs + `--new-bundle-format` explicitly). - After each `docker buildx imagetools create`, resolve the resulting list digest with `docker buildx imagetools inspect --format '{{.Manifest.Digest}}'` and sign: ```sh cosign sign --yes --recursive \ + --new-bundle-format \ --registry-referrers-mode=oci-1-1 \ "${REGISTRY_REPO}@${DIGEST}" ``` @@ -70,7 +70,7 @@ entry (`backend/index.yaml`): url: github:mudler/LocalAI/backend/index.yaml@master verification: issuer: "https://token.actions.githubusercontent.com" - identity_regex: "^https://github\\.com/mudler/LocalAI/\\.github/workflows/backend_merge\\.yml@refs/heads/master$" + identity_regex: "^https://github\\.com/mudler/LocalAI/\\.github/workflows/backend_merge\\.yml@refs/(heads/master|tags/.+)$" # Optional revocation cutoff; advance during incident response. # not_before: "2026-06-01T00:00:00Z" ``` diff --git a/.github/workflows/backend_merge.yml b/.github/workflows/backend_merge.yml index 37f606aa9..25fc93c75 100644 --- a/.github/workflows/backend_merge.yml +++ b/.github/workflows/backend_merge.yml @@ -71,8 +71,8 @@ jobs: # cosign signs each pushed manifest list with --recursive so the # index and every per-arch entry get an attached Sigstore bundle. - # Recent cosign releases always emit the new bundle format, so - # there's no extra CLI flag to opt into it. + # The pinned cosign v2 release needs --new-bundle-format explicitly; + # the verifier only consumes OCI 1.1 Sigstore bundle referrers. - name: Install cosign if: github.event_name != 'pull_request' uses: sigstore/cosign-installer@v3 @@ -159,6 +159,7 @@ jobs: # manifest before checking signatures need the per-arch # signatures, not just the list-level one. cosign sign --yes --recursive \ + --new-bundle-format \ --registry-referrers-mode=oci-1-1 \ "quay.io/go-skynet/local-ai-backends@${digest}" @@ -185,6 +186,7 @@ jobs: ' <<< "$DOCKER_METADATA_OUTPUT_JSON") digest=$(docker buildx imagetools inspect "$first_tag" --format '{{.Manifest.Digest}}') cosign sign --yes --recursive \ + --new-bundle-format \ --registry-referrers-mode=oci-1-1 \ "localai/localai-backends@${digest}" diff --git a/docs/content/features/backends.md b/docs/content/features/backends.md index 0e7c960a6..5eed109d0 100644 --- a/docs/content/features/backends.md +++ b/docs/content/features/backends.md @@ -72,6 +72,44 @@ tags: - "text-generation" ``` +### Verifying OCI Backends + +Backend galleries can require keyless Sigstore signatures for every OCI image +they provide. Add a `verification` policy to the gallery configuration, then +enable strict integrity mode: + +```bash +export LOCALAI_BACKEND_GALLERIES='[{"name":"localai","url":"github:mudler/LocalAI/backend/index.yaml@master","verification":{"issuer":"https://token.actions.githubusercontent.com","identity_regex":"^https://github\\.com/mudler/LocalAI/\\.github/workflows/backend_merge\\.yml@refs/(heads/master|tags/.+)$"}}]' +export LOCALAI_REQUIRE_BACKEND_INTEGRITY=1 +local-ai run +``` + +The policy pins the Fulcio issuer and the GitHub Actions workflow identity that +signed the image. The identity expression covers development images produced +from `master` and release images produced from tags. Use a narrower expression +if your deployment only accepts one release channel. + +Without strict mode, an OCI gallery without a verification policy installs +with a warning. With strict mode, LocalAI refuses galleries without a policy, +images without a compatible Sigstore bundle, and signatures that do not match +the configured identity. Existing images published before bundle signing was +enabled must be rebuilt or re-signed before strict deployments can install +them. + +An optional `not_before` RFC3339 value revokes signatures logged before that +time. Advance it after a signing-workflow compromise, then rebuild or re-sign +the trusted images: + +```json +{ + "verification": { + "issuer": "https://token.actions.githubusercontent.com", + "identity_regex": "^https://github\\.com/mudler/LocalAI/\\.github/workflows/backend_merge\\.yml@refs/(heads/master|tags/.+)$", + "not_before": "2026-08-05T00:00:00Z" + } +} +``` + ## Pre-installing Backends You can pre-install backends when starting LocalAI using the `LOCALAI_EXTERNAL_BACKENDS` environment variable: diff --git a/scripts/build/backend-signing_test.sh b/scripts/build/backend-signing_test.sh new file mode 100755 index 000000000..0453f4f26 --- /dev/null +++ b/scripts/build/backend-signing_test.sh @@ -0,0 +1,13 @@ +#!/usr/bin/env bash +set -euo pipefail + +WORKFLOW="$(dirname "$(realpath "$0")")/../../.github/workflows/backend_merge.yml" + +sign_commands=$(grep -Ec -- '^[[:space:]]+cosign sign([[:space:]]|$)' "$WORKFLOW" || true) +bundle_flags=$(grep -Ec -- '^[[:space:]]+--new-bundle-format([[:space:]]|$)' "$WORKFLOW" || true) +if [ "$sign_commands" -ne 2 ] || [ "$bundle_flags" -ne "$sign_commands" ]; then + echo "FAIL: every backend signing command must request the new bundle format (commands=$sign_commands flags=$bundle_flags)" + exit 1 +fi + +echo "PASS: backend signing emits Sigstore bundles for both registries"