something is wonky with some server side cache that refuses to give the
latest ref. clone and build manually instead so we can be sure it is the
latest ref we are on
Integrates OpenQA image testing into CI.
Builds images once in the imaging job, then passes the resulting image and
update channel to the OpenQA pipeline in kde-linux/os-autoinst-distri-kdelinux.
For regular test builds, upload the built ISO and a temporary sysupdate tree
to ci-artifacts on storage.kde.org. The resulting IMAGE_URL, STAGING_CHANNEL_URL
and ephemeral update signing key are exported through the imaging job dotenv
artifact so OpenQA can boot the image and run upgrade tests against the build
under test.
For protected default-branch builds, stage the release artifacts under a
per-job staging prefix in the kde-linux bucket. OpenQA tests the staged ISO
and sysupdate channel before the publish job is allowed to run.
Publishing downloads the staged artifacts, uploads the public ISO/torrent and
sysupdate assets to files.kde.org, chunks and uploads the root to the desync
chunk store, then merges the staged S3 tree into the live tree on the bucket
and regenerates the published SHA256SUMS.
Changes:
- Included necessary services, binaries, and udev rules in the initrd for ISO9660 support
- systemd-boot boot counter in live.efi is being disabled (read-only ESP = scary warnings)
- Increased ESP size
- Created a hybrid ISO (ISO + valid GPT disk)
- Upload "only" the .iso (friendlier for less experienced users)
- Resolved my will to live
Signed-off-by: Hadi Chokr <hadichokr@icloud.com>
Resolves#287
Integrates the OpenQA pipelines from kde-linux/os-autoinst-distri-kdelinux into CI.
Splits the image upload logic into separate staging and publish steps, where staged builds are not exposed for public consumption until OpenQA tests pass.
this is a bit dangerous but necessary for more efficient delta updates.
we must be very careful to avoid a bug in systemd caused by putting
foo.erofs.caibx into the sha256sums. so instead we have foo.caibx that
gets installed with the correct name by sysupdate
we still have the foo.erofs.caibx on the server for backwards compat,
but continue to not put it into the sha256sums
this is a bit of a shot in the dark, but I believe we may have
unnecessary delta in our images caused by the rebuilding of software on
a daily basis. this would result in mtimes changing when the files
actually do not.
a tiny mtimer tool is meant to work around that by consuming an input
json file of mtimes+checksums and if mtimes change it will checksum the
affected file to verify it actually has changed in content as well.
assuming reproducible builds this should result in far less delta in the
erofs and by extension the delta download
this should make things more amazing. from my testing it looks like
aligning the sizes improves compression and improves caibx generation by
aligning chunk boundaries better.
in a test scenario of adding a single 128M random data file to /usr/lib
this brought the fragmentation from a couple thousand segments down to 8
(of which 5 are in the superblock and the new file appears as large
contiguous chunk delta). the actual download size is a 135M delta
in the hopes of getting upgrades listed again
the trouble appears to be that since we no longer create the tar or
erofs addon systemd-sysupdate considers the candidates not viable and
just ignores the new versions. by having mock artifacts I hope things
will work again
we'll need to invent some tech to eventually rotate the old artifacts
out of /system though
