feat(ci): allow routing apt traffic through an alternate Ubuntu mirror (#9650)

* feat(ci): allow routing apt traffic through an alternate Ubuntu mirror

Adds opt-in APT_MIRROR / APT_PORTS_MIRROR knobs to all Dockerfiles, the
Makefile, and CI workflows so we can fail over to a non-canonical Ubuntu
mirror when archive.ubuntu.com / security.ubuntu.com / ports.ubuntu.com
are degraded (recently observed: multi-day DDoS against the default pool).

Defaults are empty everywhere — behavior is unchanged unless a mirror is
configured. To enable in CI, set the repo-level GitHub Actions variables
APT_MIRROR (and APT_PORTS_MIRROR for arm64 builds). Locally:
    make docker APT_MIRROR=http://azure.archive.ubuntu.com

A small POSIX-sh helper in .docker/apt-mirror.sh rewrites both DEB822
(/etc/apt/sources.list.d/ubuntu.sources, Ubuntu 24.04+) and the legacy
/etc/apt/sources.list before the first apt-get update. Dockerfile stages
load it via RUN --mount=type=bind, so there is no extra layer and no
cache invalidation when the script is unchanged. Reusable workflows also
rewrite the runner's own /etc/apt sources before any sudo apt-get call.

Assisted-by: Claude:claude-opus-4-7[1m] [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* ci(apt-mirror): default to the Azure mirror, visible in the workflow source

Bakes Azure (http://azure.archive.ubuntu.com / http://azure.ports.ubuntu.com)
in as the default for both Docker builds and runner-side apt — rather than
hiding the URL behind a GitHub Actions repo variable that's not visible
from the source tree.

A new composite action at .github/actions/configure-apt-mirror is the
single source of truth for runner-side rewrites. Five standalone
workflows (build-test, release, tests-e2e, tests-ui-e2e, update_swagger)
just `uses: ./.github/actions/configure-apt-mirror`.

Three workflows (image_build, backend_build, checksum_checker) keep an
inline bash rewrite, because they install/upgrade git via apt *before*
the checkout step (so the local composite action isn't loadable yet).
The Azure URL is visible in those files too.

The `apt-mirror` / `apt-ports-mirror` inputs of the reusable workflows
keep their now-Azure defaults — they still feed the Docker build-args
block in addition to the inline runner-side rewrite. Callers (image.yml,
image-pr.yml, backend.yml, backend_pr.yml) drop the previous
`vars.APT_MIRROR` plumbing and rely on those defaults.

Assisted-by: Claude:claude-opus-4-7[1m] [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* ci(apt-mirror): drop Force Install GIT, consolidate on the composite action

The PPA git upgrade ran add-apt-repository ppa:git-core/ppa, which talks
to api.launchpad.net — also part of Canonical's infrastructure and
currently returning HTTP 504. The Azure mirror only covers
archive.ubuntu.com / security.ubuntu.com / ports.ubuntu.com, not PPAs.

The system git that ubuntu-latest already ships is sufficient for
actions/checkout and the build pipeline, so just drop the upgrade. With
that gone, the apt-before-checkout constraint disappears too — all three
holdouts (image_build, backend_build, checksum_checker) can now switch
to ./.github/actions/configure-apt-mirror like the other five.

Net: 0 inline apt-mirror blocks, all 8 workflows route through the
composite action.

Assisted-by: Claude:claude-opus-4-7[1m] [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

.github/actions/configure-apt-mirror/action.yml

@@ -0,0 +1,49 @@
+name: 'Configure apt mirror'
+description: |
+  Reconfigure the GitHub Actions runner's Ubuntu apt sources to use an
+  alternate mirror. Defaults to the Azure-hosted Ubuntu mirror, which lives
+  on the same network as the runners and is independent of the canonical
+  Ubuntu pool — useful as a fallback when archive.ubuntu.com /
+  security.ubuntu.com / ports.ubuntu.com are degraded.
+
+  Pass mirror: '' (empty string) to skip the rewrite and keep upstream.
+
+inputs:
+  mirror:
+    description: 'Replacement URL for archive.ubuntu.com / security.ubuntu.com (empty = keep upstream)'
+    required: false
+    default: 'http://azure.archive.ubuntu.com'
+  ports-mirror:
+    description: 'Replacement URL for ports.ubuntu.com on arm64 (empty = keep upstream)'
+    required: false
+    default: 'http://azure.ports.ubuntu.com'
+
+runs:
+  using: 'composite'
+  steps:
+    - name: Rewrite apt sources
+      shell: bash
+      env:
+        APT_MIRROR: ${{ inputs.mirror }}
+        APT_PORTS_MIRROR: ${{ inputs.ports-mirror }}
+      run: |
+        set -e
+        if [ -z "${APT_MIRROR}" ] && [ -z "${APT_PORTS_MIRROR}" ]; then
+          echo "configure-apt-mirror: both inputs empty, leaving upstream sources untouched"
+          exit 0
+        fi
+        # Ubuntu 24.04 (noble) ships DEB822 sources at
+        # /etc/apt/sources.list.d/ubuntu.sources; older releases use
+        # /etc/apt/sources.list. Rewrite whichever exists.
+        for f in /etc/apt/sources.list.d/ubuntu.sources /etc/apt/sources.list; do
+          sudo test -f "$f" || continue
+          if [ -n "${APT_MIRROR}" ]; then
+            # Comma delimiter so the alternation pipe in the regex is not
+            # interpreted as the s/// separator.
+            sudo sed -i -E "s,https?://(archive\.ubuntu\.com|security\.ubuntu\.com),${APT_MIRROR},g" "$f"
+          fi
+          if [ -n "${APT_PORTS_MIRROR}" ]; then
+            sudo sed -i -E "s,https?://ports\.ubuntu\.com,${APT_PORTS_MIRROR},g" "$f"
+          fi
+        done
+        echo "configure-apt-mirror: APT_MIRROR='${APT_MIRROR}' APT_PORTS_MIRROR='${APT_PORTS_MIRROR}'"

.github/workflows/backend_build.yml

@@ -63,6 +63,16 @@ on:
         required: false
         default: ''
         type: string
+      apt-mirror:
+        description: 'Replacement URL for archive.ubuntu.com / security.ubuntu.com (empty = use upstream)'
+        required: false
+        default: 'http://azure.archive.ubuntu.com'
+        type: string
+      apt-ports-mirror:
+        description: 'Replacement URL for ports.ubuntu.com (arm64 etc., empty = use upstream)'
+        required: false
+        default: 'http://azure.ports.ubuntu.com'
+        type: string
     secrets:
       dockerUsername:
         required: false
@@ -80,6 +90,16 @@ jobs:
         quay_username: ${{ secrets.quayUsername }}
     steps:
 
+      - name: Checkout
+        uses: actions/checkout@v6
+        with:
+          submodules: true
+
+      - name: Configure apt mirror on runner
+        uses: ./.github/actions/configure-apt-mirror
+        with:
+          mirror: ${{ inputs.apt-mirror }}
+          ports-mirror: ${{ inputs.apt-ports-mirror }}
 
       - name: Free Disk Space (Ubuntu)
         if: inputs.runs-on == 'ubuntu-latest'
@@ -97,20 +117,6 @@ jobs:
           docker-images: true
           swap-storage: true
 
-      - name: Force Install GIT latest
-        run: |
-          sudo apt-get update \
-          && sudo apt-get install -y software-properties-common \
-          && sudo apt-get update \
-          && sudo add-apt-repository -y ppa:git-core/ppa \
-          && sudo apt-get update \
-          && sudo apt-get install -y git
-
-      - name: Checkout
-        uses: actions/checkout@v6
-        with:
-          submodules: true
-
       - name: Release space from worker
         if: inputs.runs-on == 'ubuntu-latest'
         run: |
@@ -231,6 +237,8 @@ jobs:
             BACKEND=${{ inputs.backend }}
             UBUNTU_VERSION=${{ inputs.ubuntu-version }}
             AMDGPU_TARGETS=${{ inputs.amdgpu-targets }}
+            APT_MIRROR=${{ inputs.apt-mirror }}
+            APT_PORTS_MIRROR=${{ inputs.apt-ports-mirror }}
             DEPS_REFRESH=${{ steps.deps_refresh.outputs.key }}
           context: ${{ inputs.context }}
           file: ${{ inputs.dockerfile }}
@@ -255,6 +263,8 @@ jobs:
             BACKEND=${{ inputs.backend }}
             UBUNTU_VERSION=${{ inputs.ubuntu-version }}
             AMDGPU_TARGETS=${{ inputs.amdgpu-targets }}
+            APT_MIRROR=${{ inputs.apt-mirror }}
+            APT_PORTS_MIRROR=${{ inputs.apt-ports-mirror }}
             DEPS_REFRESH=${{ steps.deps_refresh.outputs.key }}
           context: ${{ inputs.context }}
           file: ${{ inputs.dockerfile }}

.github/workflows/build-test.yaml

@@ -50,6 +50,8 @@ jobs:
         uses: actions/checkout@v6
         with:
           fetch-depth: 0
+      - name: Configure apt mirror on runner
+        uses: ./.github/actions/configure-apt-mirror
       - name: Set up Go
         uses: actions/setup-go@v5
         with:

.github/workflows/checksum_checker.yaml

@@ -8,15 +8,9 @@ jobs:
     if: github.repository == 'mudler/LocalAI'
     runs-on: ubuntu-latest
     steps:
-      - name: Force Install GIT latest
-        run: |
-          sudo apt-get update \
-          && sudo apt-get install -y software-properties-common \
-          && sudo apt-get update \
-          && sudo add-apt-repository -y ppa:git-core/ppa \
-          && sudo apt-get update \
-          && sudo apt-get install -y git
       - uses: actions/checkout@v6
+      - name: Configure apt mirror on runner
+        uses: ./.github/actions/configure-apt-mirror
       - name: Install dependencies
         run: |
           sudo apt-get update

.github/workflows/image_build.yml

@@ -56,6 +56,16 @@ on:
         required: false
         default: 'noble'
         type: string
+      apt-mirror:
+        description: 'Replacement URL for archive.ubuntu.com / security.ubuntu.com (empty = use upstream)'
+        required: false
+        default: 'http://azure.archive.ubuntu.com'
+        type: string
+      apt-ports-mirror:
+        description: 'Replacement URL for ports.ubuntu.com (arm64 etc., empty = use upstream)'
+        required: false
+        default: 'http://azure.ports.ubuntu.com'
+        type: string
     secrets:
       dockerUsername:
         required: true
@@ -70,6 +80,15 @@ jobs:
     runs-on: ${{ inputs.runs-on }}
     steps:
 
+      - name: Checkout
+        uses: actions/checkout@v6
+
+      - name: Configure apt mirror on runner
+        uses: ./.github/actions/configure-apt-mirror
+        with:
+          mirror: ${{ inputs.apt-mirror }}
+          ports-mirror: ${{ inputs.apt-ports-mirror }}
+
       - name: Free Disk Space (Ubuntu)
         if: inputs.runs-on == 'ubuntu-latest'
         uses: jlumbroso/free-disk-space@main
@@ -85,16 +104,6 @@ jobs:
           large-packages: true
           docker-images: true
           swap-storage: true
-      - name: Force Install GIT latest
-        run: |
-          sudo apt-get update \
-          && sudo apt-get install -y software-properties-common \
-          && sudo apt-get update \
-          && sudo add-apt-repository -y ppa:git-core/ppa \
-          && sudo apt-get update \
-          && sudo apt-get install -y git
-      - name: Checkout
-        uses: actions/checkout@v6
 
       - name: Release space from worker
         if: inputs.runs-on == 'ubuntu-latest'
@@ -205,6 +214,8 @@ jobs:
             SKIP_DRIVERS=${{ inputs.skip-drivers }}
             UBUNTU_VERSION=${{ inputs.ubuntu-version }}
             UBUNTU_CODENAME=${{ inputs.ubuntu-codename }}
+            APT_MIRROR=${{ inputs.apt-mirror }}
+            APT_PORTS_MIRROR=${{ inputs.apt-ports-mirror }}
           context: .
           file: ./Dockerfile
           cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache-localai${{ inputs.tag-suffix }}
@@ -228,6 +239,8 @@ jobs:
             SKIP_DRIVERS=${{ inputs.skip-drivers }}
             UBUNTU_VERSION=${{ inputs.ubuntu-version }}
             UBUNTU_CODENAME=${{ inputs.ubuntu-codename }}
+            APT_MIRROR=${{ inputs.apt-mirror }}
+            APT_PORTS_MIRROR=${{ inputs.apt-ports-mirror }}
           context: .
           file: ./Dockerfile
           cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache-localai${{ inputs.tag-suffix }}

.github/workflows/release.yaml

@@ -49,6 +49,8 @@ jobs:
         uses: actions/checkout@v6
         with:
           fetch-depth: 0
+      - name: Configure apt mirror on runner
+        uses: ./.github/actions/configure-apt-mirror
       - name: Set up Go
         uses: actions/setup-go@v5
         with:

.github/workflows/tests-e2e.yml

@@ -30,6 +30,8 @@ jobs:
         uses: actions/checkout@v6
         with:
           submodules: true
+      - name: Configure apt mirror on runner
+        uses: ./.github/actions/configure-apt-mirror
       - name: Setup Go ${{ matrix.go-version }}
         uses: actions/setup-go@v5
         with:

.github/workflows/tests-ui-e2e.yml

@@ -26,6 +26,8 @@ jobs:
         uses: actions/checkout@v6
         with:
           submodules: true
+      - name: Configure apt mirror on runner
+        uses: ./.github/actions/configure-apt-mirror
       - name: Setup Go ${{ matrix.go-version }}
         uses: actions/setup-go@v5
         with:

.github/workflows/update_swagger.yaml

@@ -11,6 +11,8 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v6
+      - name: Configure apt mirror on runner
+        uses: ./.github/actions/configure-apt-mirror
       - uses: actions/setup-go@v5
         with:
           go-version: 'stable'