fix(ci): build the native engine in a layer the registry cache can restore (#11221)

Dockerfile.golang builds 215 of the 434 matrix entries: every ggml/C++ engine
wrapped in Go. Each of those Makefiles clones an upstream repo at a pinned SHA
and compiles it once per SIMD variant (depth-anything-cpp builds four: avx,
avx2, avx512, fallback), and those variant targets depend only on the clone.
They cannot observe a change anywhere else in the LocalAI tree.

The compile sat below `COPY . /LocalAI`, so any edit anywhere invalidated it and
recompiled C++ that had not changed. Move it above that COPY, behind a copy of
only the backend's own directory.

This lands the compile in the part of the image the registry cache already
restores. Measured on two real CI builds of this Dockerfile (jobs 90551029008
and 90551028904, both fresh runners): 13 of 17 layers CACHED from
quay.io/go-skynet/ci-cache. The uncached tail is exactly `COPY . /LocalAI`, the
git-config RUN and the build RUN. Putting the engine above the COPY moves it
from the uncached tail into the cached region.

Local measurement, depth-anything-cpp CPU, rebuild after editing a Go file
outside the backend:

  master          78s, 216 C++ objects compiled
  this change     25s,   0 C++ objects compiled   (67% faster)

Note what this deliberately is not. An earlier attempt wired a
--mount=type=cache ccache into the same RUN. BuildKit does not export cache
mounts to a registry cache, so that measured well locally and is a no-op in CI
(see the ccache section of .agents/ci-caching.md). This change relies only on
ordinary layer caching, which the 13-of-17 figure above shows already works
here.

The layer copies the backend's whole directory rather than just the Makefile:
the CMake targets also need CMakeLists.txt and the file list differs per
backend. The cost is that editing a backend's own Go sources invalidates its
engine layer. The expensive cases are unaffected, since a shared-build-input or
backend.proto change, the weekly full-matrix cron and a tag push all rebuild
every backend while touching none of their directories.

Scoped to one backend for now: only depth-anything-cpp gains the `engine`
target. The other 27 fall through the `make -n engine` guard and build exactly
as before, verified against local-store and silero-vad. Rolling the target out
to the remaining 12 backends that define VARIANT_TARGETS is mechanical once this
is confirmed against the registry cache on master.

One caveat on merge: inserting layers shifts the cache keys, so the first build
of each entry after this lands is a full miss. It pays for itself on the second.


Assisted-by: Claude:opus-5 [claude-code]

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>

backend/Dockerfile.golang

@@ -248,10 +248,48 @@ RUN <<EOT bash
     fi
 EOT
 
-COPY . /LocalAI
-
 RUN git config --global --add safe.directory /LocalAI
 
+# Prebuild the native engine from a layer that depends on this backend's own
+# directory and nothing else.
+#
+# The expensive part of a C++ backend build is the engine: each of these
+# Makefiles clones an upstream repo at a pinned SHA and compiles it once per
+# SIMD variant (depth-anything-cpp builds four: avx, avx2, avx512, fallback),
+# and those variant targets depend only on the clone. They cannot observe a
+# change anywhere else in the LocalAI tree. Building them below `COPY . /LocalAI`
+# threw that away: any Go-side edit invalidated the layer and recompiled C++ that
+# had not changed. Measured on 2026-07-30, that is a 100+ minute rebuild for the
+# larger engines.
+#
+# Copying only this backend's directory first keeps the compile in a layer that
+# survives any change elsewhere in the tree, so `cache-from: type=registry`
+# restores it. That covers the expensive cases directly: a shared-build-input or
+# backend.proto change, the weekly full-matrix cron and a tag push all rebuild
+# every backend while touching none of their directories. This is the mechanism
+# behind base-grpc-* applied one level down; unlike a --mount=type=cache it is a
+# real layer, which is what actually survives to the registry.
+#
+# The whole directory rather than just the Makefile: the CMake targets also need
+# CMakeLists.txt, and the file list differs per backend. The cost is that editing
+# this backend's Go sources also invalidates the engine layer.
+#
+# Backends whose Makefile has no `engine` target are unaffected: the guard skips
+# the prebuild and their engine still compiles in the `build` step below.
+COPY backend/go/${BACKEND}/ /LocalAI/backend/go/${BACKEND}/
+RUN cd /LocalAI/backend/go/${BACKEND} && \
+    if make -n engine >/dev/null 2>&1; then \
+        echo "==> prebuilding engine for ${BACKEND} (cacheable layer)" && \
+        make engine; \
+    else \
+        echo "==> ${BACKEND} has no engine target; it builds with the backend"; \
+    fi
+
+COPY . /LocalAI
+
+# The engine variants built above survive this COPY (they are build outputs, not
+# tracked files) and are newer than the pinned clone, so make treats them as up
+# to date and goes straight to the Go binary.
 RUN cd /LocalAI && make protogen-go && make -C /LocalAI/backend/go/${BACKEND} build
 
 FROM scratch

backend/go/depth-anything-cpp/Makefile

@@ -82,6 +82,16 @@ else
 	VARIANT_TARGETS = libdepthanythingcpp-fallback.dylib
 endif
 
+## Builds the native engine variants and stops short of the Go binary. The
+## variants depend only on sources/depth-anything.cpp, a clone pinned by
+## DEPTHANYTHING_VERSION, so nothing in this target can observe a change
+## elsewhere in the LocalAI tree. Dockerfile.golang calls it from a layer that
+## copies in this Makefile and nothing else, which keeps the multi-minute ggml
+## compile in the registry layer cache across builds whose only change is on the
+## Go side. See .agents/ci-caching.md.
+.PHONY: engine
+engine: $(VARIANT_TARGETS)
+
 depth-anything-cpp: main.go godepthanythingcpp.go $(VARIANT_TARGETS)
 	CGO_ENABLED=0 $(GOCMD) build -tags "$(GO_TAGS)" -o depth-anything-cpp ./