ci(gallery): have the index bots write the model library (#12849)

* ci(gallery): have the checksum bot write the model library

The checksum bot edited gallery/index.yaml, which is a generated copy
of gallery/models/index.yaml. The model library check fails any PR
where the copy differs from the generated one, and an edit made to the
copy never reaches the source.

Point the checker at the library, read the files under .model.files as
the library keeps them, and run modelcatalog after the checker so the
PR carries both files.

Assisted-by: Claude:claude-sonnet-5-5 [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* ci(gallery): have the other index bots write the model library

The gallery agent and the variant proposer still read and wrote the
generated gallery copy, so their pull requests failed the model
library check and their changes never reached the source.

The gallery agent now appends library entries to
gallery/models/index.yaml, converting its entries with the gallery
package and skipping ids the library already has. Its duplicate check
reads the same file.

The variant proposer decodes the library shape (the body under
model:) and splices variants into the model body through a library
layout in galleryedit. The anchor refusal goes: the library has no
anchors. Its tests use library entries.

Both workflows run modelcatalog after the bot and before the pull
request step, and use the Go version from go.mod.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>

.github/workflows/checksum_checker.yaml

@@ -16,6 +16,19 @@ jobs:
           sudo apt-get update
           sudo apt-get install -y pip wget
           pip install huggingface_hub
+      - uses: actions/setup-go@v6
+        with:
+          go-version-file: go.mod
+      - name: Generate the gRPC stubs the Go packages import
+        # modelcatalog imports them and they are not in the tree. Same pinned
+        # tools as the model library workflow.
+        run: |
+          curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v26.1/protoc-26.1-linux-x86_64.zip -o protoc.zip
+          sudo unzip -j -d /usr/local/bin protoc.zip bin/protoc
+          rm protoc.zip
+          go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.34.2
+          go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@1958fcbe2ca8bd93af633f11e97d44e567e945af
+          PATH="$PATH:$HOME/go/bin" make protogen-go
       - name: 'Setup yq'
         uses: dcarbone/install-yq-action@v1.3.1
         with:
@@ -28,14 +41,19 @@ jobs:
           export HF_HOME=/hf_cache
           sudo mkdir /hf_cache
           sudo chmod 777 /hf_cache
-          bash .github/checksum_checker.sh gallery/index.yaml
+          # The model library is the source. The checker edits it.
+          bash .github/checksum_checker.sh gallery/models/index.yaml
+      - name: Regenerate the gallery copy from the library
+        # gallery/index.yaml is generated; the model library check fails the
+        # PR when it differs from the generated copy.
+        run: go run ./scripts/models/cmd/modelcatalog --library gallery/models/index.yaml --out gallery/index.yaml
       - name: Create Pull Request
         uses: peter-evans/create-pull-request@v8
         with:
           token: ${{ secrets.UPDATE_BOT_TOKEN }}
           push-to-fork: ci-forks/LocalAI
-          commit-message: ':arrow_up: Checksum updates in gallery/index.yaml'
+          commit-message: ':arrow_up: Checksum updates in the model library'
           title: 'chore(model-gallery): :arrow_up: update checksum'
           branch: "update/checksum"
-          body: Updating checksums in gallery/index.yaml
+          body: Updating checksums in gallery/models/index.yaml and regenerating gallery/index.yaml
           signoff: true

.github/workflows/gallery-agent.yaml

@@ -38,7 +38,7 @@ jobs:
       - name: Set up Go
         uses: actions/setup-go@v5
         with:
-          go-version: '1.21'
+          go-version-file: go.mod
       - name: Proto Dependencies
         run: |
           # Install protoc
@@ -131,19 +131,25 @@ jobs:
           MAX_MODELS: ${{ github.event.inputs.max_models || '1' }}
           EXTRA_SKIP_URLS: ${{ steps.open_prs.outputs.urls }}
         run: |
-          export GALLERY_INDEX_PATH=$PWD/gallery/index.yaml
+          # The agent reads and appends to the model library, the source.
+          export GALLERY_INDEX_PATH=$PWD/gallery/models/index.yaml
           go run ./.github/gallery-agent
 
+      - name: Regenerate the gallery copy from the library
+        # gallery/index.yaml is generated from the library; the model library
+        # check fails the PR when they disagree.
+        run: go run ./scripts/models/cmd/modelcatalog --library gallery/models/index.yaml --out gallery/index.yaml
+
       - name: Check for changes
         id: check_changes
         run: |
-          if git diff --quiet gallery/index.yaml; then
+          if git diff --quiet gallery/models/index.yaml gallery/index.yaml; then
             echo "changes=false" >> $GITHUB_OUTPUT
-            echo "No changes detected in gallery/index.yaml"
+            echo "No changes detected in the model library"
           else
             echo "changes=true" >> $GITHUB_OUTPUT
-            echo "Changes detected in gallery/index.yaml"
-            git diff gallery/index.yaml
+            echo "Changes detected in the model library"
+            git diff gallery/models/index.yaml gallery/index.yaml
           fi
 
       - name: Read gallery agent summary

.github/workflows/gallery_variant_proposals.yaml

@@ -19,12 +19,24 @@ jobs:
       - name: Test the proposer
         run: go test ./.github/ci/variantproposals/
 
+      # The proposer calls no gallery code, but modelcatalog imports packages
+      # that need the generated gRPC stubs. Same pinned tools as the model
+      # library workflow.
+      - name: Generate the gRPC stubs the Go packages import
+        run: |
+          curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v26.1/protoc-26.1-linux-x86_64.zip -o protoc.zip
+          sudo unzip -j -d /usr/local/bin protoc.zip bin/protoc
+          rm protoc.zip
+          go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.34.2
+          go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@1958fcbe2ca8bd93af633f11e97d44e567e945af
+          PATH="$PATH:$HOME/go/bin" make protogen-go
+
       - name: Propose groupings 🔧
         id: propose
         run: |
           rm -f /tmp/variant-proposals-body.md
           go run ./.github/ci/variantproposals \
-            -index gallery/index.yaml \
+            -index gallery/models/index.yaml \
             -ledger gallery/variant-exclusions.yaml \
             -body-out /tmp/variant-proposals-body.md \
             -apply
@@ -39,6 +51,13 @@ jobs:
             echo "have_proposals=false" >> "$GITHUB_OUTPUT"
           fi
 
+      # The proposer edits the model library, the source. The gallery copy is
+      # generated from it, and the model library check fails a PR where they
+      # disagree. Only regenerate when the library changed.
+      - name: Regenerate the gallery copy from the library
+        if: steps.propose.outputs.have_proposals == 'true'
+        run: go run ./scripts/models/cmd/modelcatalog --library gallery/models/index.yaml --out gallery/index.yaml
+
       # No body file means the proposer found nothing. Opening an empty pull
       # request every run is how a proposal job gets muted by its reviewers.
       - name: Create Pull Request