feat(ci): allow routing apt traffic through an alternate Ubuntu mirror (#9650)

* feat(ci): allow routing apt traffic through an alternate Ubuntu mirror

Adds opt-in APT_MIRROR / APT_PORTS_MIRROR knobs to all Dockerfiles, the
Makefile, and CI workflows so we can fail over to a non-canonical Ubuntu
mirror when archive.ubuntu.com / security.ubuntu.com / ports.ubuntu.com
are degraded (recently observed: multi-day DDoS against the default pool).

Defaults are empty everywhere — behavior is unchanged unless a mirror is
configured. To enable in CI, set the repo-level GitHub Actions variables
APT_MIRROR (and APT_PORTS_MIRROR for arm64 builds). Locally:
    make docker APT_MIRROR=http://azure.archive.ubuntu.com

A small POSIX-sh helper in .docker/apt-mirror.sh rewrites both DEB822
(/etc/apt/sources.list.d/ubuntu.sources, Ubuntu 24.04+) and the legacy
/etc/apt/sources.list before the first apt-get update. Dockerfile stages
load it via RUN --mount=type=bind, so there is no extra layer and no
cache invalidation when the script is unchanged. Reusable workflows also
rewrite the runner's own /etc/apt sources before any sudo apt-get call.

Assisted-by: Claude:claude-opus-4-7[1m] [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* ci(apt-mirror): default to the Azure mirror, visible in the workflow source

Bakes Azure (http://azure.archive.ubuntu.com / http://azure.ports.ubuntu.com)
in as the default for both Docker builds and runner-side apt — rather than
hiding the URL behind a GitHub Actions repo variable that's not visible
from the source tree.

A new composite action at .github/actions/configure-apt-mirror is the
single source of truth for runner-side rewrites. Five standalone
workflows (build-test, release, tests-e2e, tests-ui-e2e, update_swagger)
just `uses: ./.github/actions/configure-apt-mirror`.

Three workflows (image_build, backend_build, checksum_checker) keep an
inline bash rewrite, because they install/upgrade git via apt *before*
the checkout step (so the local composite action isn't loadable yet).
The Azure URL is visible in those files too.

The `apt-mirror` / `apt-ports-mirror` inputs of the reusable workflows
keep their now-Azure defaults — they still feed the Docker build-args
block in addition to the inline runner-side rewrite. Callers (image.yml,
image-pr.yml, backend.yml, backend_pr.yml) drop the previous
`vars.APT_MIRROR` plumbing and rely on those defaults.

Assisted-by: Claude:claude-opus-4-7[1m] [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

* ci(apt-mirror): drop Force Install GIT, consolidate on the composite action

The PPA git upgrade ran add-apt-repository ppa:git-core/ppa, which talks
to api.launchpad.net — also part of Canonical's infrastructure and
currently returning HTTP 504. The Azure mirror only covers
archive.ubuntu.com / security.ubuntu.com / ports.ubuntu.com, not PPAs.

The system git that ubuntu-latest already ships is sufficient for
actions/checkout and the build pipeline, so just drop the upgrade. With
that gone, the apt-before-checkout constraint disappears too — all three
holdouts (image_build, backend_build, checksum_checker) can now switch
to ./.github/actions/configure-apt-mirror like the other five.

Net: 0 inline apt-mirror blocks, all 8 workflows route through the
composite action.

Assisted-by: Claude:claude-opus-4-7[1m] [Claude Code]
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>

.docker/apt-mirror.sh

@@ -0,0 +1,39 @@
+#!/bin/sh
+# Reconfigure Ubuntu apt sources to point at an alternate mirror.
+#
+# Used by Dockerfiles via `RUN --mount=type=bind,source=.docker/apt-mirror.sh,...`
+# and by CI workflows on the runner to mitigate outages of the default
+# archive.ubuntu.com / security.ubuntu.com / ports.ubuntu.com pool.
+#
+# Inputs (env):
+#   APT_MIRROR        Replacement for archive.ubuntu.com and security.ubuntu.com
+#                     (e.g. "http://azure.archive.ubuntu.com" or
+#                      "https://mirrors.edge.kernel.org").
+#                     Leave empty to keep upstream. The trailing "/ubuntu/..."
+#                     path is preserved by the rewrite.
+#   APT_PORTS_MIRROR  Replacement for ports.ubuntu.com (arm64/ppc64el/...).
+#                     Leave empty to keep upstream.
+#
+# Both default to empty, in which case the script is a no-op.
+
+set -e
+
+if [ -z "${APT_MIRROR}" ] && [ -z "${APT_PORTS_MIRROR}" ]; then
+    exit 0
+fi
+
+# Ubuntu 24.04 (noble) ships DEB822 sources at /etc/apt/sources.list.d/ubuntu.sources;
+# older releases use /etc/apt/sources.list. We rewrite whichever exists.
+for f in /etc/apt/sources.list.d/ubuntu.sources /etc/apt/sources.list; do
+    [ -f "$f" ] || continue
+    if [ -n "${APT_MIRROR}" ]; then
+        # Use a comma delimiter so the alternation pipe in the regex
+        # is not interpreted as the s/// separator.
+        sed -i -E "s,https?://(archive\.ubuntu\.com|security\.ubuntu\.com),${APT_MIRROR},g" "$f"
+    fi
+    if [ -n "${APT_PORTS_MIRROR}" ]; then
+        sed -i -E "s,https?://ports\.ubuntu\.com,${APT_PORTS_MIRROR},g" "$f"
+    fi
+done
+
+echo "apt-mirror: rewrote sources (APT_MIRROR='${APT_MIRROR}', APT_PORTS_MIRROR='${APT_PORTS_MIRROR}')"

.github/actions/configure-apt-mirror/action.yml

@@ -0,0 +1,49 @@
+name: 'Configure apt mirror'
+description: |
+  Reconfigure the GitHub Actions runner's Ubuntu apt sources to use an
+  alternate mirror. Defaults to the Azure-hosted Ubuntu mirror, which lives
+  on the same network as the runners and is independent of the canonical
+  Ubuntu pool — useful as a fallback when archive.ubuntu.com /
+  security.ubuntu.com / ports.ubuntu.com are degraded.
+
+  Pass mirror: '' (empty string) to skip the rewrite and keep upstream.
+
+inputs:
+  mirror:
+    description: 'Replacement URL for archive.ubuntu.com / security.ubuntu.com (empty = keep upstream)'
+    required: false
+    default: 'http://azure.archive.ubuntu.com'
+  ports-mirror:
+    description: 'Replacement URL for ports.ubuntu.com on arm64 (empty = keep upstream)'
+    required: false
+    default: 'http://azure.ports.ubuntu.com'
+
+runs:
+  using: 'composite'
+  steps:
+    - name: Rewrite apt sources
+      shell: bash
+      env:
+        APT_MIRROR: ${{ inputs.mirror }}
+        APT_PORTS_MIRROR: ${{ inputs.ports-mirror }}
+      run: |
+        set -e
+        if [ -z "${APT_MIRROR}" ] && [ -z "${APT_PORTS_MIRROR}" ]; then
+          echo "configure-apt-mirror: both inputs empty, leaving upstream sources untouched"
+          exit 0
+        fi
+        # Ubuntu 24.04 (noble) ships DEB822 sources at
+        # /etc/apt/sources.list.d/ubuntu.sources; older releases use
+        # /etc/apt/sources.list. Rewrite whichever exists.
+        for f in /etc/apt/sources.list.d/ubuntu.sources /etc/apt/sources.list; do
+          sudo test -f "$f" || continue
+          if [ -n "${APT_MIRROR}" ]; then
+            # Comma delimiter so the alternation pipe in the regex is not
+            # interpreted as the s/// separator.
+            sudo sed -i -E "s,https?://(archive\.ubuntu\.com|security\.ubuntu\.com),${APT_MIRROR},g" "$f"
+          fi
+          if [ -n "${APT_PORTS_MIRROR}" ]; then
+            sudo sed -i -E "s,https?://ports\.ubuntu\.com,${APT_PORTS_MIRROR},g" "$f"
+          fi
+        done
+        echo "configure-apt-mirror: APT_MIRROR='${APT_MIRROR}' APT_PORTS_MIRROR='${APT_PORTS_MIRROR}'"

.github/workflows/backend_build.yml

@@ -63,6 +63,16 @@ on:
         required: false
         default: ''
         type: string
+      apt-mirror:
+        description: 'Replacement URL for archive.ubuntu.com / security.ubuntu.com (empty = use upstream)'
+        required: false
+        default: 'http://azure.archive.ubuntu.com'
+        type: string
+      apt-ports-mirror:
+        description: 'Replacement URL for ports.ubuntu.com (arm64 etc., empty = use upstream)'
+        required: false
+        default: 'http://azure.ports.ubuntu.com'
+        type: string
     secrets:
       dockerUsername:
         required: false
@@ -80,6 +90,16 @@ jobs:
         quay_username: ${{ secrets.quayUsername }}
     steps:
 
+      - name: Checkout
+        uses: actions/checkout@v6
+        with:
+          submodules: true
+
+      - name: Configure apt mirror on runner
+        uses: ./.github/actions/configure-apt-mirror
+        with:
+          mirror: ${{ inputs.apt-mirror }}
+          ports-mirror: ${{ inputs.apt-ports-mirror }}
 
       - name: Free Disk Space (Ubuntu)
         if: inputs.runs-on == 'ubuntu-latest'
@@ -97,20 +117,6 @@ jobs:
           docker-images: true
           swap-storage: true
 
-      - name: Force Install GIT latest
-        run: |
-          sudo apt-get update \
-          && sudo apt-get install -y software-properties-common \
-          && sudo apt-get update \
-          && sudo add-apt-repository -y ppa:git-core/ppa \
-          && sudo apt-get update \
-          && sudo apt-get install -y git
-
-      - name: Checkout
-        uses: actions/checkout@v6
-        with:
-          submodules: true
-
       - name: Release space from worker
         if: inputs.runs-on == 'ubuntu-latest'
         run: |
@@ -231,6 +237,8 @@ jobs:
             BACKEND=${{ inputs.backend }}
             UBUNTU_VERSION=${{ inputs.ubuntu-version }}
             AMDGPU_TARGETS=${{ inputs.amdgpu-targets }}
+            APT_MIRROR=${{ inputs.apt-mirror }}
+            APT_PORTS_MIRROR=${{ inputs.apt-ports-mirror }}
             DEPS_REFRESH=${{ steps.deps_refresh.outputs.key }}
           context: ${{ inputs.context }}
           file: ${{ inputs.dockerfile }}
@@ -255,6 +263,8 @@ jobs:
             BACKEND=${{ inputs.backend }}
             UBUNTU_VERSION=${{ inputs.ubuntu-version }}
             AMDGPU_TARGETS=${{ inputs.amdgpu-targets }}
+            APT_MIRROR=${{ inputs.apt-mirror }}
+            APT_PORTS_MIRROR=${{ inputs.apt-ports-mirror }}
             DEPS_REFRESH=${{ steps.deps_refresh.outputs.key }}
           context: ${{ inputs.context }}
           file: ${{ inputs.dockerfile }}

.github/workflows/build-test.yaml

@@ -50,6 +50,8 @@ jobs:
         uses: actions/checkout@v6
         with:
           fetch-depth: 0
+      - name: Configure apt mirror on runner
+        uses: ./.github/actions/configure-apt-mirror
       - name: Set up Go
         uses: actions/setup-go@v5
         with:

.github/workflows/checksum_checker.yaml

@@ -8,15 +8,9 @@ jobs:
     if: github.repository == 'mudler/LocalAI'
     runs-on: ubuntu-latest
     steps:
-      - name: Force Install GIT latest
-        run: |
-          sudo apt-get update \
-          && sudo apt-get install -y software-properties-common \
-          && sudo apt-get update \
-          && sudo add-apt-repository -y ppa:git-core/ppa \
-          && sudo apt-get update \
-          && sudo apt-get install -y git
       - uses: actions/checkout@v6
+      - name: Configure apt mirror on runner
+        uses: ./.github/actions/configure-apt-mirror
       - name: Install dependencies
         run: |
           sudo apt-get update

.github/workflows/image_build.yml

@@ -56,6 +56,16 @@ on:
         required: false
         default: 'noble'
         type: string
+      apt-mirror:
+        description: 'Replacement URL for archive.ubuntu.com / security.ubuntu.com (empty = use upstream)'
+        required: false
+        default: 'http://azure.archive.ubuntu.com'
+        type: string
+      apt-ports-mirror:
+        description: 'Replacement URL for ports.ubuntu.com (arm64 etc., empty = use upstream)'
+        required: false
+        default: 'http://azure.ports.ubuntu.com'
+        type: string
     secrets:
       dockerUsername:
         required: true
@@ -70,6 +80,15 @@ jobs:
     runs-on: ${{ inputs.runs-on }}
     steps:
 
+      - name: Checkout
+        uses: actions/checkout@v6
+
+      - name: Configure apt mirror on runner
+        uses: ./.github/actions/configure-apt-mirror
+        with:
+          mirror: ${{ inputs.apt-mirror }}
+          ports-mirror: ${{ inputs.apt-ports-mirror }}
+
       - name: Free Disk Space (Ubuntu)
         if: inputs.runs-on == 'ubuntu-latest'
         uses: jlumbroso/free-disk-space@main
@@ -85,16 +104,6 @@ jobs:
           large-packages: true
           docker-images: true
           swap-storage: true
-      - name: Force Install GIT latest
-        run: |
-          sudo apt-get update \
-          && sudo apt-get install -y software-properties-common \
-          && sudo apt-get update \
-          && sudo add-apt-repository -y ppa:git-core/ppa \
-          && sudo apt-get update \
-          && sudo apt-get install -y git
-      - name: Checkout
-        uses: actions/checkout@v6
 
       - name: Release space from worker
         if: inputs.runs-on == 'ubuntu-latest'
@@ -205,6 +214,8 @@ jobs:
             SKIP_DRIVERS=${{ inputs.skip-drivers }}
             UBUNTU_VERSION=${{ inputs.ubuntu-version }}
             UBUNTU_CODENAME=${{ inputs.ubuntu-codename }}
+            APT_MIRROR=${{ inputs.apt-mirror }}
+            APT_PORTS_MIRROR=${{ inputs.apt-ports-mirror }}
           context: .
           file: ./Dockerfile
           cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache-localai${{ inputs.tag-suffix }}
@@ -228,6 +239,8 @@ jobs:
             SKIP_DRIVERS=${{ inputs.skip-drivers }}
             UBUNTU_VERSION=${{ inputs.ubuntu-version }}
             UBUNTU_CODENAME=${{ inputs.ubuntu-codename }}
+            APT_MIRROR=${{ inputs.apt-mirror }}
+            APT_PORTS_MIRROR=${{ inputs.apt-ports-mirror }}
           context: .
           file: ./Dockerfile
           cache-from: type=registry,ref=quay.io/go-skynet/ci-cache:cache-localai${{ inputs.tag-suffix }}

.github/workflows/release.yaml

@@ -49,6 +49,8 @@ jobs:
         uses: actions/checkout@v6
         with:
           fetch-depth: 0
+      - name: Configure apt mirror on runner
+        uses: ./.github/actions/configure-apt-mirror
       - name: Set up Go
         uses: actions/setup-go@v5
         with:

.github/workflows/tests-e2e.yml

@@ -30,6 +30,8 @@ jobs:
         uses: actions/checkout@v6
         with:
           submodules: true
+      - name: Configure apt mirror on runner
+        uses: ./.github/actions/configure-apt-mirror
       - name: Setup Go ${{ matrix.go-version }}
         uses: actions/setup-go@v5
         with:

.github/workflows/tests-ui-e2e.yml

@@ -26,6 +26,8 @@ jobs:
         uses: actions/checkout@v6
         with:
           submodules: true
+      - name: Configure apt mirror on runner
+        uses: ./.github/actions/configure-apt-mirror
       - name: Setup Go ${{ matrix.go-version }}
         uses: actions/setup-go@v5
         with:

.github/workflows/update_swagger.yaml

@@ -11,6 +11,8 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v6
+      - name: Configure apt mirror on runner
+        uses: ./.github/actions/configure-apt-mirror
       - uses: actions/setup-go@v5
         with:
           go-version: 'stable'

Dockerfile

@@ -1,12 +1,20 @@
 ARG BASE_IMAGE=ubuntu:24.04
 ARG INTEL_BASE_IMAGE=${BASE_IMAGE}
 ARG UBUNTU_CODENAME=noble
+# Optional alternate Ubuntu apt mirror(s). Empty = use upstream.
+# See .docker/apt-mirror.sh for accepted values.
+ARG APT_MIRROR=""
+ARG APT_PORTS_MIRROR=""
 
 FROM ${BASE_IMAGE} AS requirements
 
+ARG APT_MIRROR
+ARG APT_PORTS_MIRROR
 ENV DEBIAN_FRONTEND=noninteractive
 
-RUN apt-get update && \
+RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
+    APT_MIRROR="${APT_MIRROR}" APT_PORTS_MIRROR="${APT_PORTS_MIRROR}" sh /usr/local/sbin/apt-mirror && \
+    apt-get update && \
     apt-get install -y --no-install-recommends \
         ca-certificates curl wget espeak-ng libgomp1 \
         ffmpeg libopenblas0 libopenblas-dev libopus0 sox && \
@@ -240,10 +248,14 @@ WORKDIR /build
 # This is a temporary workaround until Intel fixes their repository
 FROM ${INTEL_BASE_IMAGE} AS intel
 ARG UBUNTU_CODENAME=noble
+ARG APT_MIRROR
+ARG APT_PORTS_MIRROR
 RUN wget -qO - https://repositories.intel.com/gpu/intel-graphics.key | \
 gpg --yes --dearmor --output /usr/share/keyrings/intel-graphics.gpg
 RUN echo "deb [arch=amd64 signed-by=/usr/share/keyrings/intel-graphics.gpg] https://repositories.intel.com/gpu/ubuntu ${UBUNTU_CODENAME}/lts/2350 unified" > /etc/apt/sources.list.d/intel-graphics.list
-RUN apt-get update && \
+RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
+    APT_MIRROR="${APT_MIRROR}" APT_PORTS_MIRROR="${APT_PORTS_MIRROR}" sh /usr/local/sbin/apt-mirror && \
+    apt-get update && \
     apt-get install -y --no-install-recommends \
         intel-oneapi-runtime-libs && \
     apt-get clean && \

Makefile

@@ -10,6 +10,13 @@ LAUNCHER_BINARY_NAME=local-ai-launcher
 UBUNTU_VERSION?=2404
 UBUNTU_CODENAME?=noble
 
+# Optional Ubuntu apt mirror overrides forwarded to docker builds.
+# Empty = use upstream archive.ubuntu.com / security.ubuntu.com / ports.ubuntu.com.
+# Set e.g. APT_MIRROR=http://azure.archive.ubuntu.com to route apt traffic
+# during outages of the default Ubuntu pool.
+APT_MIRROR?=
+APT_PORTS_MIRROR?=
+
 GORELEASER?=
 
 export BUILD_TYPE?=
@@ -209,6 +216,8 @@ docker-build-e2e:
 		--build-arg CUDA_MINOR_VERSION=$(CUDA_MINOR_VERSION) \
 		--build-arg UBUNTU_VERSION=$(UBUNTU_VERSION) \
 		--build-arg UBUNTU_CODENAME=$(UBUNTU_CODENAME) \
+		--build-arg APT_MIRROR=$(APT_MIRROR) \
+		--build-arg APT_PORTS_MIRROR=$(APT_PORTS_MIRROR) \
 		--build-arg GO_TAGS="$(GO_TAGS)" \
 		-t local-ai:tests -f Dockerfile .
 
@@ -236,6 +245,8 @@ prepare-e2e:
 		--build-arg CUDA_MINOR_VERSION=$(CUDA_MINOR_VERSION) \
 		--build-arg UBUNTU_VERSION=$(UBUNTU_VERSION) \
 		--build-arg UBUNTU_CODENAME=$(UBUNTU_CODENAME) \
+		--build-arg APT_MIRROR=$(APT_MIRROR) \
+		--build-arg APT_PORTS_MIRROR=$(APT_PORTS_MIRROR) \
 		--build-arg GO_TAGS="$(GO_TAGS)" \
 		--build-arg MAKEFLAGS="$(DOCKER_MAKEFLAGS)" \
 		-t localai-tests .
@@ -277,6 +288,8 @@ test-opus-docker:
 	docker build --target builder \
 	  --build-arg BUILD_TYPE=$(or $(BUILD_TYPE),) \
 	  --build-arg BASE_IMAGE=$(or $(BASE_IMAGE),ubuntu:24.04) \
+	  --build-arg APT_MIRROR=$(APT_MIRROR) \
+	  --build-arg APT_PORTS_MIRROR=$(APT_PORTS_MIRROR) \
 	  --build-arg BACKEND=opus \
 	  -t localai-opus-test -f backend/Dockerfile.golang .
 	docker run --rm localai-opus-test \
@@ -318,6 +331,8 @@ test-realtime-models-docker: build-mock-backend
 	  --build-arg BUILD_TYPE=$(or $(BUILD_TYPE),cublas) \
 	  --build-arg CUDA_MAJOR_VERSION=$(or $(CUDA_MAJOR_VERSION),13) \
 	  --build-arg CUDA_MINOR_VERSION=$(or $(CUDA_MINOR_VERSION),0) \
+	  --build-arg APT_MIRROR=$(APT_MIRROR) \
+	  --build-arg APT_PORTS_MIRROR=$(APT_PORTS_MIRROR) \
 	  -t localai-test-runner .
 	docker run --rm \
 	  $(REALTIME_DOCKER_FLAGS) \
@@ -901,6 +916,8 @@ docker:
 		--build-arg CUDA_MINOR_VERSION=$(CUDA_MINOR_VERSION) \
 		--build-arg UBUNTU_VERSION=$(UBUNTU_VERSION) \
 		--build-arg UBUNTU_CODENAME=$(UBUNTU_CODENAME) \
+		--build-arg APT_MIRROR=$(APT_MIRROR) \
+		--build-arg APT_PORTS_MIRROR=$(APT_PORTS_MIRROR) \
 		-t $(DOCKER_IMAGE) .
 
 docker-cuda12:
@@ -914,6 +931,8 @@ docker-cuda12:
 		--build-arg BUILD_TYPE=$(BUILD_TYPE) \
 		--build-arg UBUNTU_VERSION=$(UBUNTU_VERSION) \
 		--build-arg UBUNTU_CODENAME=$(UBUNTU_CODENAME) \
+		--build-arg APT_MIRROR=$(APT_MIRROR) \
+		--build-arg APT_PORTS_MIRROR=$(APT_PORTS_MIRROR) \
 		-t $(DOCKER_IMAGE)-cuda-12 .
 
 docker-image-intel:
@@ -927,6 +946,8 @@ docker-image-intel:
 		--build-arg CUDA_MINOR_VERSION=$(CUDA_MINOR_VERSION) \
 		--build-arg UBUNTU_VERSION=$(UBUNTU_VERSION) \
 		--build-arg UBUNTU_CODENAME=$(UBUNTU_CODENAME) \
+		--build-arg APT_MIRROR=$(APT_MIRROR) \
+		--build-arg APT_PORTS_MIRROR=$(APT_PORTS_MIRROR) \
 		-t $(DOCKER_IMAGE) .
 
 ########################################################
@@ -1050,6 +1071,8 @@ define docker-build-backend
 		--build-arg CUDA_MINOR_VERSION=$(CUDA_MINOR_VERSION) \
 		--build-arg UBUNTU_VERSION=$(UBUNTU_VERSION) \
 		--build-arg UBUNTU_CODENAME=$(UBUNTU_CODENAME) \
+		--build-arg APT_MIRROR=$(APT_MIRROR) \
+		--build-arg APT_PORTS_MIRROR=$(APT_PORTS_MIRROR) \
 		$(if $(FROM_SOURCE),--build-arg FROM_SOURCE=$(FROM_SOURCE)) \
 		$(if $(AMDGPU_TARGETS),--build-arg AMDGPU_TARGETS=$(AMDGPU_TARGETS)) \
 		$(if $(filter true,$(5)),--build-arg BACKEND=$(1)) \

backend/Dockerfile.golang

@@ -1,4 +1,6 @@
 ARG BASE_IMAGE=ubuntu:24.04
+ARG APT_MIRROR=""
+ARG APT_PORTS_MIRROR=""
 
 FROM ${BASE_IMAGE} AS builder
 ARG BACKEND=rerankers
@@ -16,8 +18,12 @@ ARG GO_VERSION=1.25.4
 ARG UBUNTU_VERSION=2404
 ARG AMDGPU_TARGETS
 ENV AMDGPU_TARGETS=${AMDGPU_TARGETS}
+ARG APT_MIRROR
+ARG APT_PORTS_MIRROR
 
-RUN apt-get update && \
+RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
+    APT_MIRROR="${APT_MIRROR}" APT_PORTS_MIRROR="${APT_PORTS_MIRROR}" sh /usr/local/sbin/apt-mirror && \
+    apt-get update && \
     apt-get install -y --no-install-recommends \
         build-essential \
         git ccache \

backend/Dockerfile.ik-llama-cpp

@@ -1,5 +1,7 @@
 ARG BASE_IMAGE=ubuntu:24.04
 ARG GRPC_BASE_IMAGE=${BASE_IMAGE}
+ARG APT_MIRROR=""
+ARG APT_PORTS_MIRROR=""
 
 
 # The grpc target does one thing, it builds and installs GRPC.  This is in it's own layer so that it can be effectively cached by CI.
@@ -12,12 +14,16 @@ ARG GRPC_VERSION=v1.65.0
 ARG CMAKE_FROM_SOURCE=false
 # CUDA Toolkit 13.x compatibility: CMake 3.31.9+ fixes toolchain detection/arch table issues
 ARG CMAKE_VERSION=3.31.10
+ARG APT_MIRROR
+ARG APT_PORTS_MIRROR
 
 ENV MAKEFLAGS=${GRPC_MAKEFLAGS}
 
 WORKDIR /build
 
-RUN apt-get update && \
+RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
+    APT_MIRROR="${APT_MIRROR}" APT_PORTS_MIRROR="${APT_PORTS_MIRROR}" sh /usr/local/sbin/apt-mirror && \
+    apt-get update && \
     apt-get install -y --no-install-recommends \
         ca-certificates \
         build-essential curl libssl-dev \
@@ -71,8 +77,12 @@ ARG TARGETARCH
 ARG TARGETVARIANT
 ARG GO_VERSION=1.25.4
 ARG UBUNTU_VERSION=2404
+ARG APT_MIRROR
+ARG APT_PORTS_MIRROR
 
-RUN apt-get update && \
+RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
+    APT_MIRROR="${APT_MIRROR}" APT_PORTS_MIRROR="${APT_PORTS_MIRROR}" sh /usr/local/sbin/apt-mirror && \
+    apt-get update && \
     apt-get install -y --no-install-recommends \
         build-essential \
         ccache git \

backend/Dockerfile.llama-cpp

@@ -1,5 +1,7 @@
 ARG BASE_IMAGE=ubuntu:24.04
 ARG GRPC_BASE_IMAGE=${BASE_IMAGE}
+ARG APT_MIRROR=""
+ARG APT_PORTS_MIRROR=""
 
 
 # The grpc target does one thing, it builds and installs GRPC.  This is in it's own layer so that it can be effectively cached by CI.
@@ -12,12 +14,16 @@ ARG GRPC_VERSION=v1.65.0
 ARG CMAKE_FROM_SOURCE=false
 # CUDA Toolkit 13.x compatibility: CMake 3.31.9+ fixes toolchain detection/arch table issues
 ARG CMAKE_VERSION=3.31.10
+ARG APT_MIRROR
+ARG APT_PORTS_MIRROR
 
 ENV MAKEFLAGS=${GRPC_MAKEFLAGS}
 
 WORKDIR /build
 
-RUN apt-get update && \
+RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
+    APT_MIRROR="${APT_MIRROR}" APT_PORTS_MIRROR="${APT_PORTS_MIRROR}" sh /usr/local/sbin/apt-mirror && \
+    apt-get update && \
     apt-get install -y --no-install-recommends \
         ca-certificates \
         build-essential curl libssl-dev \
@@ -73,8 +79,12 @@ ARG TARGETARCH
 ARG TARGETVARIANT
 ARG GO_VERSION=1.25.4
 ARG UBUNTU_VERSION=2404
+ARG APT_MIRROR
+ARG APT_PORTS_MIRROR
 
-RUN apt-get update && \
+RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
+    APT_MIRROR="${APT_MIRROR}" APT_PORTS_MIRROR="${APT_PORTS_MIRROR}" sh /usr/local/sbin/apt-mirror && \
+    apt-get update && \
     apt-get install -y --no-install-recommends \
         build-essential \
         ccache git \

backend/Dockerfile.python

@@ -1,4 +1,6 @@
 ARG BASE_IMAGE=ubuntu:24.04
+ARG APT_MIRROR=""
+ARG APT_PORTS_MIRROR=""
 
 FROM ${BASE_IMAGE} AS builder
 ARG BACKEND=rerankers
@@ -13,8 +15,12 @@ ENV DEBIAN_FRONTEND=noninteractive
 ARG TARGETARCH
 ARG TARGETVARIANT
 ARG UBUNTU_VERSION=2404
+ARG APT_MIRROR
+ARG APT_PORTS_MIRROR
 
-RUN apt-get update && \
+RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
+    APT_MIRROR="${APT_MIRROR}" APT_PORTS_MIRROR="${APT_PORTS_MIRROR}" sh /usr/local/sbin/apt-mirror && \
+    apt-get update && \
     apt-get install -y --no-install-recommends \
         build-essential \
         ccache \

backend/Dockerfile.rust

@@ -1,12 +1,18 @@
 ARG BASE_IMAGE=ubuntu:24.04
+ARG APT_MIRROR=""
+ARG APT_PORTS_MIRROR=""
 
 FROM ${BASE_IMAGE} AS builder
 ARG BACKEND=kokoros
 ENV DEBIAN_FRONTEND=noninteractive
 ARG TARGETARCH
 ARG TARGETVARIANT
+ARG APT_MIRROR
+ARG APT_PORTS_MIRROR
 
-RUN apt-get update && \
+RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
+    APT_MIRROR="${APT_MIRROR}" APT_PORTS_MIRROR="${APT_PORTS_MIRROR}" sh /usr/local/sbin/apt-mirror && \
+    apt-get update && \
     apt-get install -y --no-install-recommends \
         build-essential \
         git ccache \

backend/Dockerfile.turboquant

@@ -1,5 +1,7 @@
 ARG BASE_IMAGE=ubuntu:24.04
 ARG GRPC_BASE_IMAGE=${BASE_IMAGE}
+ARG APT_MIRROR=""
+ARG APT_PORTS_MIRROR=""
 
 
 # The grpc target does one thing, it builds and installs GRPC.  This is in it's own layer so that it can be effectively cached by CI.
@@ -12,12 +14,16 @@ ARG GRPC_VERSION=v1.65.0
 ARG CMAKE_FROM_SOURCE=false
 # CUDA Toolkit 13.x compatibility: CMake 3.31.9+ fixes toolchain detection/arch table issues
 ARG CMAKE_VERSION=3.31.10
+ARG APT_MIRROR
+ARG APT_PORTS_MIRROR
 
 ENV MAKEFLAGS=${GRPC_MAKEFLAGS}
 
 WORKDIR /build
 
-RUN apt-get update && \
+RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
+    APT_MIRROR="${APT_MIRROR}" APT_PORTS_MIRROR="${APT_PORTS_MIRROR}" sh /usr/local/sbin/apt-mirror && \
+    apt-get update && \
     apt-get install -y --no-install-recommends \
         ca-certificates \
         build-essential curl libssl-dev \
@@ -71,8 +77,12 @@ ARG TARGETARCH
 ARG TARGETVARIANT
 ARG GO_VERSION=1.25.4
 ARG UBUNTU_VERSION=2404
+ARG APT_MIRROR
+ARG APT_PORTS_MIRROR
 
-RUN apt-get update && \
+RUN --mount=type=bind,source=.docker/apt-mirror.sh,target=/usr/local/sbin/apt-mirror \
+    APT_MIRROR="${APT_MIRROR}" APT_PORTS_MIRROR="${APT_PORTS_MIRROR}" sh /usr/local/sbin/apt-mirror && \
+    apt-get update && \
     apt-get install -y --no-install-recommends \
         build-essential \
         ccache git \