mirror/kde-linux
1
0
Fork 0
mirror of https://github.com/KDE/kde-linux.git synced 2026-08-04 03:02:45 -04:00
Code Issues Packages Projects Releases Wiki Activity
1,235 Commits 111 Branches 0 Tags
buildstream
Commit Graph

63 Commits

Author SHA1 Message Date
Thomas Duckworth
51c0706ceb Stage to ci-artifacts, again 
Should work now that we have the permissions.

See 46aee53994
 2026-07-18 21:28:56 +10:00
Thomas Duckworth
630ff954a7 Revert "Stage to ci-artifacts per-pipeline" 
This reverts commit b4b388cbfd
 2026-07-07 19:19:40 +10:00
Thomas Duckworth
b4b388cbfd Stage to ci-artifacts per-pipeline 
This should be authenticated, and leverages a 3-day lifecycle rule that
deletes stale staged images.
 2026-07-07 00:59:12 +10:00
Thomas Duckworth
c736a64956 Stage to the kde-linux bucket instead 
It doesn't seem to have permissions to download
from the ci-artifacts bucket.
 2026-07-04 23:42:58 +10:00
Thomas Duckworth
57cb1b403f Stage to ci-artifacts bucket rather than kde-linux 
Clean up the paths a bit and leverage the pre-existing 3 day
lifecycle rule in ci-artifacts that will delete stale artifacts.

Implement support for transfers between buckets in publisher, with
explicit source and destination URLs.
 2026-07-04 22:06:07 +10:00
Bhushan Shah
9a46a0ec07 upload: fix permissions for uploaded files 2026-07-03 19:51:15 +05:30
Thomas Duckworth
8748d74db0 Ensure publishing is safe 
Ensures we can't have two publish jobs running at the same time that
point to the same staging directory. Also ensures that publish will
immediately exit if there's nothing to stage, so we aren't vacuuming
prematurely.
 2026-07-03 13:14:37 +00:00
Thomas Duckworth
63a164f975 Explicitly set an empty sysupdate pubkey in staging 
Otherwise we end up passing the literal string $SYSUPDATE_PUBKEY_B64, the openQA worker tries to decode it, and it dies.
 2026-07-03 10:57:40 +00:00
Thomas Duckworth
df6899e791 Fix secure file download for image staging 
Download secure files before staging builds, so mkosi can embed the
public key into the image again, and so it can import the public key for
us. Also explicitly create the GPG homedir again and import the public
key in the publish step because mkosi won't have created it there.
 2026-07-03 10:29:33 +00:00
Thomas Duckworth
cfc528f6d0 Reintroduce OpenQA testing 
Integrates OpenQA image testing into CI.

Builds images once in the imaging job, then passes the resulting image and
update channel to the OpenQA pipeline in kde-linux/os-autoinst-distri-kdelinux.

For regular test builds, upload the built ISO and a temporary sysupdate tree
to ci-artifacts on storage.kde.org. The resulting IMAGE_URL, STAGING_CHANNEL_URL
and ephemeral update signing key are exported through the imaging job dotenv
artifact so OpenQA can boot the image and run upgrade tests against the build
under test.

For protected default-branch builds, stage the release artifacts under a
per-job staging prefix in the kde-linux bucket. OpenQA tests the staged ISO
and sysupdate channel before the publish job is allowed to run.

Publishing downloads the staged artifacts, uploads the public ISO/torrent and
sysupdate assets to files.kde.org, chunks and uploads the root to the desync
chunk store, then merges the staged S3 tree into the live tree on the bucket
and regenerates the published SHA256SUMS.
 2026-07-03 04:57:34 +00:00
Hadi Chokr
db0bb13913 Make our Images comply with ISO9660 (.iso) 
Changes:
- Included necessary services, binaries, and udev rules in the initrd for ISO9660 support
- systemd-boot boot counter in live.efi is being disabled (read-only ESP = scary warnings)
- Increased ESP size
- Created a hybrid ISO (ISO + valid GPT disk)
- Upload "only" the .iso (friendlier for less experienced users)
- Resolved my will to live

Signed-off-by: Hadi Chokr <hadichokr@icloud.com>

Resolves #287
 2026-06-25 15:08:32 +02:00
Thomas Duckworth
071af6560f Revert the introduction of OpenQA testing 
CI needs a bit of work before we can ship this properly. Revert it for the time being.

This reverts commit 164c0c67b4.
 2026-06-22 22:48:27 +10:00
Thomas Duckworth
164c0c67b4 Introduce OpenQA testing 
Integrates the OpenQA pipelines from kde-linux/os-autoinst-distri-kdelinux into CI. 

Splits the image upload logic into separate staging and publish steps, where staged builds are not exposed for public consumption until OpenQA tests pass.
 2026-06-21 12:05:32 +02:00
Harald Sitter
3ca74d152d upload: move raw and torrent to correct subdir 2026-05-08 23:20:13 +02:00
Harald Sitter
d6992c65d2 upload: rejigger upload-tree 
this should bring it in alignment with the vacuum-v3 handling and fix
the sha256sums location coming out of that to match our expectation.
hopefully making the pipeline green again
 2026-05-08 22:47:02 +02:00
Harald Sitter
a533d8b954 upload: put desync into path 
should make it also available for the vacuum helper
 2026-04-22 15:12:04 +02:00
Harald Sitter
df8c400a29 upload: build and run vacuum in one command 
less verbose this way
 2026-04-22 13:18:28 +02:00
Harald Sitter
08948c85e9 upload: prime vacuuming 
everything seems to be in order now. let's clean up!
 2026-04-15 21:42:01 +02:00
Harald Sitter
23ef25928d upload: make sure we have an updated token before vacuuming 
more ideally we should refresh the token on-demand but that is a larger
refactor
 2026-04-15 14:51:40 +02:00
Harald Sitter
ef363735c4 add upload-vacuum-v3 
cleans up the s3 and prepares for the repo v3 (which accounts for
editions in the repo path)

Closes #590
 2026-04-15 14:04:27 +02:00
Hadi Chokr
4b43c460bc Revert "Final MR Cleanup" 
This reverts commit 2df0b40a9f
 2026-02-25 12:59:29 +00:00
Hadi Chokr
2df0b40a9f Final MR Cleanup 2026-02-25 11:12:43 +00:00
Hadi Chokr
be432ce532 Hotfix: Fix Relative Path 2026-02-25 10:37:02 +00:00
Hadi Chokr
acf2a82511 Hotfix a final time before reverting. 2026-02-25 10:19:00 +00:00
Hadi Chokr
c6b874ae3e Hotfix: New Output Dir in upload.sh 2026-02-25 09:33:55 +00:00
Harald Sitter
dd93564cfb upload: make all dirs 2026-02-20 04:18:36 +01:00
Harald Sitter
b03438d9c1 upload: activate desync chopping again 
this time it should hopefully work :|
 2026-02-19 13:18:10 +01:00
Harald Sitter
4344b8529e upload: isolate upload testing into its own little job 
it continues to not want to work :(
 2026-02-18 23:12:04 +01:00
Harald Sitter
581ae8e722 debug-- 2026-02-18 22:24:28 +01:00
Harald Sitter
9de6060960 debug++ 2026-02-18 22:22:46 +01:00
Harald Sitter
339fdf783f upload: syntax fix 2026-02-18 21:50:22 +01:00
Harald Sitter
faed0b8797 build,upload: chop in upload not build 
for practical reasons it seems more useful to chop during the upload
stage, as we can directly feed into the s3 from there
 2026-02-18 21:20:42 +01:00
Harald Sitter
59d8be68fb upload: remove upload to storage for now 
doesn't quite work yet
 2026-02-17 06:14:12 +01:00
Harald Sitter
7c9d595c5f upload: moving chopping to build stage 
it's where we make the caibx already and have desync installed
 2026-02-17 03:04:23 +01:00
Harald Sitter
9309fbc2b1 upload: upload a desync store in addition to existing artifacts 
this is a prototype and may explode. the idea here is that we'll have
one large chunk store from which we can feed all update needs
 2026-02-17 02:38:16 +01:00
Harald Sitter
24eda82e5f consume caibx file via sysupdate 
this is a bit dangerous but necessary for more efficient delta updates.

we must be very careful to avoid a bug in systemd caused by putting
foo.erofs.caibx into the sha256sums. so instead we have foo.caibx that
gets installed with the correct name by sysupdate

we still have the foo.erofs.caibx on the server for backwards compat,
but continue to not put it into the sha256sums
 2025-09-17 14:17:24 +02:00
Harald Sitter
2d8dca8329 move sha256sums to sysupdate dir 
systemd doesn't like subdir references in the sha256sums so we can't
have a single sums file in the root but rather need to have multiple
 2025-08-21 01:15:10 +02:00
Harald Sitter
5653cd0db6 try to invoke scp correctly 2025-08-20 20:31:09 +02:00
Harald Sitter
d532605483 comment++ 2025-08-20 19:51:07 +02:00
Harald Sitter
8c51beb068 shell glob please 2025-08-20 19:49:15 +02:00
Harald Sitter
c5be947571 upload sysupdate artifacts to a versioned directory 
helps with implementation of a banana split

https://community.kde.org/KDE_Linux/Banana_Split
 2025-08-20 18:54:27 +02:00
Harald Sitter
5ed5b5d752 yet more caibx hotfixing 2025-08-17 01:30:40 +02:00
Harald Sitter
c21666a893 don't put the caibx into the shasums 
it breaks matching on the erofs somehow such that systemd downloads the
caibx thinking it is the erofs
 2025-08-14 22:09:09 +02:00
Harald Sitter
3f455168d7 generate a desync index for the erofs 
allows to desync... in theory anyway
 2025-08-14 20:47:46 +02:00
Harald Sitter
ce0649c129 make upload more verbose 
it gets stuck on the VM based CI and it's not clear why
 2025-07-30 18:02:57 +02:00
Harald Sitter
64841b3ebb files are owned by root because we sudo. take them back 2025-07-30 12:20:03 +02:00
Harald Sitter
d1a627607c try to make sure the uploader has access 
not sure why that is being weird after the VM transition. let's see if a chown fixes it
 2025-07-30 11:50:04 +02:00
Harald Sitter
20bda00747 create an erofs of the rootfs as well 
we might end up using it instead of a btrfs subvolume. first we need one
to experiment with though
 2025-03-20 00:57:33 +00:00
Ian Monroe
1921a0140c Revert "Use mkosi.output for build artifacts" 
This reverts commit ea7ca55cd2.
 2024-12-16 11:34:28 -08:00
Bruno Pajdek
ea7ca55cd2 Use mkosi.output for build artifacts 
mkosi already provides us with a nice default directory it puts artifacts in called `mkosi.output` if `--output-directory` isn't explicitly set. For some reason, it _is_ explicitly set to `.`, so it puts them in the source tree itself which pollutes it and gets messy real quick.

After build, the contents of the source tree are now
```
drwxrwxrwx 1 root root  100 Dec 15 15:18 .git
-rw-rw-rw- 1 root root  226 Dec 15 15:18 .gitignore
-rw-rw-rw- 1 root root 1.1K Dec 15 15:18 .gitlab-ci.yml
drwxrwxrwx 1 root root  370 Dec 15 15:18 LICENSES
-rw-rw-rw- 1 root root  337 Dec 15 15:18 README.md
-rw-rw-rw- 1 root root  128 Dec 15 15:18 REUSE.toml
-rwxrwxrwx 1 root root  824 Dec 15 15:18 addons.sh
-rwxrwxrwx 1 root root 1.7K Dec 15 15:18 bootstrap.sh
-rwxrwxrwx 1 root root 6.1K Dec 15 15:18 build.sh
-rwxrwxrwx 1 root root 2.9K Dec 15 15:18 build_docker.sh
-rwxrwxrwx 1 root root  427 Dec 15 15:18 in_docker.sh
drwxrwxrwx 1 root root   96 Dec 15 15:25 kde-linux.cache
drwxrwxrwx 1 root root   20 Dec 15 15:18 mkosi.cache
-rw-rw-rw- 1 root root 1.8K Dec 15 15:18 mkosi.conf
drwxrwxrwx 1 root root  420 Dec 15 15:18 mkosi.conf.d
drwxrwxrwx 1 root root   20 Dec 15 15:18 mkosi.extra
-rwxrwxrwx 1 root root 3.6K Dec 15 15:18 mkosi.finalize.chroot
drwxr-xr-x 1 root root  500 Dec 15 15:37 mkosi.output
-rwxrwxrwx 1 root root 5.2K Dec 15 15:18 mkosi.postinst.chroot
-rwxrwxrwx 1 root root 1.3K Dec 15 15:18 mkosi.prepare.chroot
drwxrwxrwx 1 root root   46 Dec 15 15:18 mkosi.repart
drwxrwxrwx 1 root root    6 Dec 15 15:18 mkosi.skeleton
-rwxrwxrwx 1 root root  742 Dec 15 15:18 permission-fix.sh
drwxrwxrwx 1 root root   30 Dec 15 15:18 strip
-rwxrwxrwx 1 root root 1.4K Dec 15 15:18 torrent-create.rb
-rwxrwxrwx 1 root root  796 Dec 15 15:18 update.sh
drwxrwxrwx 1 root root   94 Dec 15 15:18 upload-vacuum
-rwxrwxrwx 1 root root 1.6K Dec 15 15:18 upload.sh
```

And the contents of `mkosi.output` are
```
drwxr-xr-x 1 root root  206 Dec 15 15:28 kde-linux_202412151518
-rw-r--r-- 1 root root 173M Dec 15 15:25 kde-linux_202412151518.efi
-rw-r--r-- 1 root root 5.2G Dec 15 15:36 kde-linux_202412151518.raw
-rw------- 1 root root  55K Dec 15 15:37 kde-linux_202412151518.torrent
-rw-r--r-- 1 root root  14M Dec 15 15:25 kde-linux_202412151518.vmlinuz
-rw-r--r-- 1 root root 1.6G Dec 15 15:23 kde-linux_202412151518_debug-x86-64.tar.zst
-rw-r--r-- 1 root root 173M Dec 15 15:24 kde-linux_202412151518_live.efi
-rw-r--r-- 1 root root 3.0G Dec 15 15:36 kde-linux_202412151518_root-x86-64.tar.zst
```
 2024-12-15 17:55:10 +00:00