Files
localai-org-maint-botandEttore Di Giacinto b4e16b963f fix(ci): stop the e2e teardown from failing a green suite (#11816)
* fix(ci): remove the e2e container before removing its image

`docker stop` returns as soon as the container exits, but the daemon
reaps a `--rm` container asynchronously after that. The `docker rmi
localai-tests` that follows teardown-e2e then loses the race against the
reaper and fails with "conflict: ... is using its referenced image", so
make exits 1 and the job goes red after every spec has passed.

This is why the E2E Backend Tests job fails at random across pull
requests. Runs 33435319093, 33435332991, 33412165884 and 33444669207 all
report "SUCCESS! -- 235 Passed | 0 Failed" and then die in teardown.

`docker rm -f` is synchronous, so the image reference is gone before
teardown-e2e returns. It also covers the case where no container is
running, which `docker stop` could not because it rejects an empty
argument list.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]

* fix(ci): open a tmate session only when a PR asks for one

The tmate step runs on every failure and then holds the runner until
GitHub cancels the job at the 6 hour limit. A one second cleanup race in
the e2e teardown therefore costs a whole ubuntu-latest slot. The recent
run list is full of 6h, 7h and 12h cancelled runs for that reason.

The step now needs the `ci-debug` label on the pull request, so a
session opens when somebody wants to debug and never otherwise. The
30 minute step timeout caps the cost when the label is left behind.

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude:claude-opus-5 [Claude Code]

---------

Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
2026-09-01 08:55:35 +02:00

92 lines
3.9 KiB
YAML

---
name: 'tests-aio'
# Runs the all-in-one (AIO) Docker image with real backends + real models.
# Heavy: builds llama-cpp/whisper/piper/silero-vad/stablediffusion-ggml/local-store
# and exercises end-to-end inference inside the container. Moved out of test.yml
# (which used to run on every PR) so PR CI no longer pays this cost.
#
# Triggers:
# - schedule (nightly @ 04:00 UTC) — catches packaging/image regressions within 24h
# - workflow_dispatch — manual run on-demand
# - push to master/tags — sanity check after merge / before release
on:
schedule:
- cron: '0 4 * * *'
workflow_dispatch:
push:
branches:
- master
tags:
- '*'
concurrency:
group: ci-tests-aio-${{ github.event.pull_request.number || github.sha }}-${{ github.repository }}
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
jobs:
tests-aio:
runs-on: ubuntu-latest
steps:
- name: Release space from worker
run: |
echo "Listing top largest packages"
pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
head -n 30 <<< "${pkgs}"
echo
df -h
echo
sudo apt-get remove -y '^llvm-.*|^libllvm.*' || true
sudo apt-get remove --auto-remove android-sdk-platform-tools || true
sudo apt-get purge --auto-remove android-sdk-platform-tools || true
sudo rm -rf /usr/local/lib/android
sudo apt-get remove -y '^dotnet-.*|^aspnetcore-.*' || true
sudo rm -rf /usr/share/dotnet
sudo apt-get remove -y '^mono-.*' || true
sudo apt-get remove -y '^ghc-.*' || true
sudo apt-get remove -y '.*jdk.*|.*jre.*' || true
sudo apt-get remove -y 'php.*' || true
sudo apt-get remove -y hhvm powershell firefox monodoc-manual msbuild || true
sudo apt-get remove -y '^google-.*' || true
sudo apt-get remove -y azure-cli || true
sudo apt-get remove -y '^mongo.*-.*|^postgresql-.*|^mysql-.*|^mssql-.*' || true
sudo apt-get remove -y '^gfortran-.*' || true
sudo apt-get autoremove -y
sudo apt-get clean
echo
echo "Listing top largest packages"
pkgs=$(dpkg-query -Wf '${Installed-Size}\t${Package}\t${Status}\n' | awk '$NF == "installed"{print $1 "\t" $2}' | sort -nr)
head -n 30 <<< "${pkgs}"
echo
sudo rm -rfv build || true
df -h
- name: Clone
uses: actions/checkout@v7
with:
submodules: true
- name: Dependencies
run: |
# Install protoc
curl -L -s https://github.com/protocolbuffers/protobuf/releases/download/v26.1/protoc-26.1-linux-x86_64.zip -o protoc.zip && \
unzip -j -d /usr/local/bin protoc.zip bin/protoc && \
rm protoc.zip
go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.34.2
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@1958fcbe2ca8bd93af633f11e97d44e567e945af
PATH="$PATH:$HOME/go/bin" make protogen-go
- name: Test
run: |
PATH="$PATH:$HOME/go/bin" make backends/local-store backends/silero-vad backends/llama-cpp backends/whisper backends/piper backends/stablediffusion-ggml docker-build-e2e e2e-aio
# tmate keeps the runner busy until the 6 hour job limit, so a single
# failure costs a whole runner slot. Only open a session when someone
# asked for one by labelling the pull request `ci-debug`, and cap the
# session so a forgotten label cannot idle a runner either.
- name: Setup tmate session if tests fail
if: ${{ failure() && contains(github.event.pull_request.labels.*.name, 'ci-debug') }}
timeout-minutes: 30
uses: mxschmitt/action-tmate@v3.23
with:
detached: true
connect-timeout-seconds: 180
limit-access-to-actor: true