mirror of
https://github.com/KDE/kde-linux.git
synced 2025-12-24 00:18:23 -05:00
Improve image populating/shrinking/building process.
Alright, this is the main set of changes done in `build.sh`. After this, as far as `build.sh` is concerned, it's mostly style and documentation changes I will write MRs for. There was no way for me to split this into smaller independent changes without leaving the tree in an inconsistent state, this is really the most "atomic" I can personally do. The changes may look scary at first, but I promise you they're not that major, breaking or hard to review. The MR focuses on how the image is created, populated and shrunk from the rootfs generated by mkosi. Firstly, the variables that are set at the start of the program were simplified and refactored so they can be used in (in my opinion) a more elegant and useful way with the rest of the now-modified build process. They now focus more on simply the paths/filenames of the outputted artifacts, so they can be changed conveniently, with the internal naming now still staying a static `kde-linux`. As a result of this, the image ID (`kde-linux`) is now in `mkosi.conf`. `OUTPUT_IS_BTRFS_SUBVOLUME` was dropped. This is a remnant of when we could populate the images without a btrfs'd build environment, but those times are now gone and if this variable was determined to be false, the build would fail pretty quickly anyways, so really, it should always be true in a properly configured build environment. Because of it being used as a condition in if-else statements (if $OUTPUT_IS_BTRFS_SUBVOLUME is true... else...), those statements were dropped and only their main branch was kept with their "else" branch (which, again, should never even run on a working build environment) discarded. One script that was only ever called from one of those discarded else-branches was `btrfs-copy.sh`, so it was deleted as it is now fully unused. The way everything used to work after this was fairly convoluted, which is probably the result of piling new shrinking methods and the likes onto old existing code. I will drop some details for the sake of brevity. After mkosi finished, three btrfs subvolumes were created and the files from the mkosi-generated rootfs were copied to those (`/var/lib/flatpak` was copied to a flatpak subvolume, `/live` was copied to a live subvolume, and the rest to a kde-linux subvolume). These subvolumes were then all packed into three respective files. Using `systemd-repart` and the definitions inside `mkosi.repart`, a full disk image containing both a 260M ESP populated with files from `/efi-template` and a btrfs partition of a size large enough to house all of those subvolumes was created. Using `systemd-dissect`, only the btrfs partition from the full disk image was mounted, and the script `btrfs-send-receive.sh` was called inside of the mounted partition, which unpacked all of the previously packed subvolumes onto the filesystem. It then called `btrfs-shrink.py` which would try and shrink the filesystem to the minimum possible size and write that size into a file for future usage by `part-rebuild.py` after it's done. `part-rebuild.py` got called which extracted the btrfs partition and ESP from the full disk image separately using `dd` and shrunk the btrfs part of the image to the size that the actual inner filesystem was shrunk to by `btrfs-shrink.py`. Then `systemd-repart` was called AGAIN, this time with `--size=auto` instead of a fixed size and using definitions inside `mkosi.repart-rebuild`, which would merge the two part-rebuild.py-dd-separated partition images of the ESP and shrunk btrfs filesystem back into one full disk image. The way it now works is the ESP (of 260M) and btrfs filesystem (of a size large enough, 8G) are created inside of two files using native tools (`fallocate` and `mkfs.btrfs`/`fat`). The ESP file is mounted, files from `/efi-template` are copied to it, and then it's unmounted. Then, the btrfs file is mounted, files from the mkosi-generated rootfs are copied to it (what `btrfs-send-receive.sh` effectively did), the filesystem is shrunk to the minimum possible size (what `btrfs-shrink.py` effectively did), and then it's unmounted. Since the underlying btrfs filesystem inside of the btrfs file was shrunk but the actual file is still 8G, the file is, using `truncate`, shrunk to the size of the actual btrfs filesystem inside (what `part-rebuild.py` effectively did). The two files are now ready to be merged into one full bootable disk image, so `systemd-repart` is called a single time right away with `--size=auto` using the definitions in `mkosi.repart` to do so. Everything is done in `build.sh`. As a result, the files no longer necessary were deleted from the tree - `btrfs-copy.sh`, `btrfs-send-receive.sh`, `btrfs-shrink.py`, `part-rebuild.py`, and the definitions inside `mkosi.repart` were simplified to conform to the new direct disk image build method. I'm pretty sure it's faster (30min build time down to 20min) because it's less convoluted, and generates smaller images (6ish G down to 5ish G). I'm not sure what the reasoning is behind the latter, but it's also a welcome change :) !91 will also be updated to refer to this MR instead of the now-closed !90 and can then be merged to fully get rid of Python for now.
This commit is contained in:
Bruno Pajdek
committed by
Lasath Fernando
Lasath Fernando
parent
cb7d2e2c4a
commit
2be5c59c92
@@ -1,32 +0,0 @@
|
||||
#!/bin/sh
|
||||
# SPDX-License-Identifier: GPL-2.0-only OR GPL-3.0-only OR LicenseRef-KDE-Accepted-GPL
|
||||
# SPDX-FileCopyrightText: 2023 Harald Sitter <sitter@kde.org>
|
||||
|
||||
# Executed inside an image to populate the subvolume via plain old copy
|
||||
|
||||
set -ex
|
||||
|
||||
OUTPUT_ABS=$1
|
||||
OUTPUT=$2
|
||||
|
||||
[ "$OUTPUT_ABS" = "" ] && exit 1
|
||||
[ "$OUTPUT" = "" ] && exit 1
|
||||
|
||||
cleanup() {
|
||||
[ -d @import ] && btrfs subvolume delete @import
|
||||
return 0
|
||||
}
|
||||
trap cleanup INT TERM EXIT
|
||||
cleanup
|
||||
|
||||
mount -o remount,compress-force=zstd "$PWD"
|
||||
btrfs property set . compression zstd
|
||||
btrfs balance start --force -mconvert=single -dconvert=single .
|
||||
|
||||
btrfs subvolume create @import
|
||||
cp -ra "$OUTPUT_ABS/." @import/
|
||||
btrfs subvolume snapshot -r @import "@$OUTPUT"
|
||||
btrfs subvolume delete @import
|
||||
|
||||
ln -svf "@$OUTPUT" "$ID"
|
||||
btrfs subvolume set-default "$ID"
|
||||
@@ -1,67 +0,0 @@
|
||||
#!/bin/sh
|
||||
# SPDX-License-Identifier: GPL-2.0-only OR GPL-3.0-only OR LicenseRef-KDE-Accepted-GPL
|
||||
# SPDX-FileCopyrightText: 2023 Harald Sitter <sitter@kde.org>
|
||||
|
||||
# Executed inside an image to populate the subvolume via btrfs snapshot magic
|
||||
|
||||
set -ex
|
||||
|
||||
OUTPUT_ABS=$1
|
||||
OUTPUT=$2
|
||||
ID=$3
|
||||
|
||||
[ "$OUTPUT_ABS" = "" ] && exit 1
|
||||
[ "$OUTPUT" = "" ] && exit 1
|
||||
[ "$ID" = "" ] && exit 1
|
||||
|
||||
OUTPUT_DIR=$(dirname "$OUTPUT_ABS")
|
||||
EXPORT="$OUTPUT.export"
|
||||
|
||||
cleanup() {
|
||||
[ -d "$EXPORT" ] && btrfs subvolume delete "$EXPORT"
|
||||
btrfs filesystem show . || true
|
||||
btrfs filesystem df . || true
|
||||
dmesg > "$OUTPUT_DIR/dmesg.log"
|
||||
return 0
|
||||
}
|
||||
trap cleanup INT TERM EXIT
|
||||
cleanup
|
||||
|
||||
mount -o remount,compress-force=zstd:15 "$PWD"
|
||||
btrfs property set . compression zstd
|
||||
btrfs balance start --force -mconvert=single -dconvert=single .
|
||||
|
||||
btrfs receive -f "$OUTPUT_ABS.btrfs" .
|
||||
btrfs subvolume snapshot -r "$EXPORT" "@$OUTPUT"
|
||||
btrfs subvolume delete "$EXPORT"
|
||||
rm -f "$OUTPUT_ABS.btrfs"
|
||||
|
||||
btrfs receive -f "$OUTPUT_ABS.btrfs.flatpak" .
|
||||
btrfs subvolume snapshot -r "$EXPORT.flatpak" "@flatpak"
|
||||
btrfs subvolume delete "$EXPORT.flatpak"
|
||||
rm -f "$OUTPUT_ABS.btrfs.flatpak"
|
||||
|
||||
btrfs receive -f "$OUTPUT_ABS.btrfs.live" .
|
||||
btrfs subvolume snapshot -r "$EXPORT.live" "@live"
|
||||
btrfs subvolume delete "$EXPORT.live"
|
||||
rm -f "$OUTPUT_ABS.btrfs.live"
|
||||
|
||||
# Sync before we start to mess with the filesystem again. Otherwise we might make it fall over.
|
||||
btrfs subvolume sync .
|
||||
btrfs filesystem sync .
|
||||
|
||||
# Finally let's condense the data.
|
||||
btrfs filesystem usage .
|
||||
## Use duperemove to deduplicate files.
|
||||
## I would also love to use bees here as it works on extents but we don't know when it is done :( https://github.com/Zygo/bees/issues/279
|
||||
duperemove -dr . > /dev/null
|
||||
btrfs subvolume sync .
|
||||
btrfs filesystem sync .
|
||||
## And to finish things off we shrink the filesystem to the minimum size.
|
||||
$OUTPUT_DIR/btrfs-shrink.py
|
||||
## Sync changes to disk.
|
||||
btrfs filesystem sync .
|
||||
# Final report.
|
||||
btrfs filesystem usage .
|
||||
|
||||
ln -svf "@$OUTPUT" "$ID"
|
||||
@@ -1,44 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
|
||||
# SPDX-License-Identifier: GPL-2.0-only OR GPL-3.0-only OR LicenseRef-KDE-Accepted-GPL
|
||||
# SPDX-FileCopyrightText: 2024 Harald Sitter <sitter@kde.org>
|
||||
|
||||
# Shrink btrfs. It's a bit awkward because we don't really have a reliable way
|
||||
# to calculate how much space we actually need. So we first chop off a dynamic
|
||||
# portion but leave a bit of a buffer behind. Then we keep resizing until the
|
||||
# resize starts failing.
|
||||
|
||||
import json
|
||||
import os
|
||||
import math
|
||||
import subprocess
|
||||
from subprocess import check_output
|
||||
|
||||
out = check_output(["btrfs", "--format", "json", "filesystem", "df", "."])
|
||||
data = json.loads(out)
|
||||
df = data["filesystem-df"]
|
||||
|
||||
size = 0
|
||||
for block_group in df:
|
||||
size += block_group["total"]
|
||||
|
||||
# Give 10% buffer space. We'll shrink from there in smaller steps.
|
||||
size = max(512 * 1024 * 1024, math.ceil(size * 1.1))
|
||||
|
||||
subprocess.run(["btrfs", "filesystem", "resize", str(size), "."], check=True)
|
||||
|
||||
# With compression one extent is always 128KiB as per btrfs documentation.
|
||||
extent_size = 128 * 1024
|
||||
while True:
|
||||
try:
|
||||
subprocess.run(["btrfs", "filesystem", "resize", f"-{extent_size}", "."], stdout=subprocess.DEVNULL, stdin=subprocess.DEVNULL, check=True)
|
||||
subprocess.run(["btrfs", "filesystem", "sync", "."], check=True)
|
||||
size -= extent_size
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(e)
|
||||
break
|
||||
|
||||
script_dir = os.path.dirname(os.path.realpath(__file__))
|
||||
with open(f"{script_dir}/btrfs.json", "w") as file:
|
||||
# Writing data to a file
|
||||
file.write(json.dumps({"size": size}))
|
||||
222
build.sh
222
build.sh
@@ -9,16 +9,22 @@
|
||||
|
||||
set -ex
|
||||
|
||||
NAME=kde-linux
|
||||
_EPOCH=$(date +%s)
|
||||
_DATE=$(date --date="@$_EPOCH" +%Y%m%d)
|
||||
_TIME=$(date --date="@$_EPOCH" +%H%M)
|
||||
DATETIME="${_DATE}${_TIME}"
|
||||
VERSION="$DATETIME"
|
||||
OUTPUT=${NAME}_$VERSION
|
||||
EFI=$OUTPUT+3.efi # +3 is the default boot number for systemd-boot
|
||||
TAR=${OUTPUT}_root-x86-64.tar
|
||||
IMG=$OUTPUT.raw
|
||||
VERSION=$(date +%Y%m%d%H%M) # Build version, will just be YYYYmmddHHMM for now
|
||||
OUTPUT=kde-linux_$VERSION # Built rootfs path (mkosi uses this directory by default)
|
||||
|
||||
# Canonicalize the path in $OUTPUT to avoid any possible path issues.
|
||||
OUTPUT="$(readlink --canonicalize-missing "$OUTPUT")"
|
||||
|
||||
MAIN_UKI=${OUTPUT}.efi # Output main UKI path
|
||||
LIVE_UKI=${OUTPUT}_live.efi # Output live UKI path
|
||||
DEBUG_TAR=${OUTPUT}_debug-x86-64.tar.zst # Output debug archive path
|
||||
ROOTFS_TAR=${OUTPUT}_root-x86-64.tar # Output rootfs tarball path (.zst will be added)
|
||||
IMG=${OUTPUT}.raw # Output raw image path
|
||||
|
||||
EFI=kde-linux_${VERSION}+3.efi # Name of primary UKI in the image's ESP
|
||||
|
||||
# Clean up old build artifacts.
|
||||
rm --recursive --force kde-linux.cache/*.raw kde-linux.cache/*.mnt
|
||||
|
||||
export SYSTEMD_LOG_LEVEL=debug
|
||||
|
||||
@@ -29,7 +35,6 @@ mkosi \
|
||||
--environment="CI_COMMIT_SHORT_SHA=${CI_COMMIT_SHORT_SHA:-unknownSHA}" \
|
||||
--environment="CI_COMMIT_SHA=${CI_COMMIT_SHA:-unknownSHA}" \
|
||||
--environment="CI_PIPELINE_URL=${CI_PIPELINE_URL:-https://invent.kde.org}" \
|
||||
--image-id="$NAME" \
|
||||
--image-version="$VERSION" \
|
||||
--output-directory=. \
|
||||
"$@"
|
||||
@@ -40,88 +45,133 @@ rm -rfv "${OUTPUT}/efi"
|
||||
[ -d "${OUTPUT}/efi-template" ] || mkdir --mode 0700 "${OUTPUT}/efi-template"
|
||||
[ -d "${OUTPUT}/efi-template/EFI" ] || mkdir --mode 0700 "${OUTPUT}/efi-template/EFI"
|
||||
[ -d "${OUTPUT}/efi-template/EFI/Linux" ] || mkdir --mode 0700 "${OUTPUT}/efi-template/EFI/Linux"
|
||||
cp -v "${OUTPUT}"/${NAME}*.efi "$OUTPUT.efi"
|
||||
mv -v "${OUTPUT}"/${NAME}*.efi "${OUTPUT}/efi-template/EFI/Linux/$EFI"
|
||||
mv -v "${OUTPUT}"/live.efi "${OUTPUT}_live.efi"
|
||||
|
||||
OUTPUT_IS_BTRFS_SUBVOLUME=false
|
||||
if [ "$(stat --file-system --format %T "$OUTPUT")" = "btrfs" ] && [ "$(stat --format %i "$OUTPUT")" = "256" ]; then
|
||||
OUTPUT_IS_BTRFS_SUBVOLUME=true
|
||||
fi
|
||||
cp -v "${OUTPUT}"/kde-linux.efi "$MAIN_UKI"
|
||||
mv -v "${OUTPUT}"/kde-linux.efi "${OUTPUT}/efi-template/EFI/Linux/$EFI"
|
||||
mv -v "${OUTPUT}"/live.efi "$LIVE_UKI"
|
||||
|
||||
# Move debug tarball out of the tree
|
||||
mv -v "$OUTPUT/debug.tar.zst" "${OUTPUT}_debug-x86-64.tar.zst"
|
||||
mv -v "$OUTPUT/debug.tar.zst" "$DEBUG_TAR"
|
||||
|
||||
FLATPAK_SIZE=""
|
||||
# Move /flatpak out of the tree and into subvolume
|
||||
if $OUTPUT_IS_BTRFS_SUBVOLUME; then
|
||||
btrfs subvolume create "$OUTPUT.flatpak"
|
||||
cp -rf --reflink=always "$OUTPUT/flatpak/." "$OUTPUT.flatpak"
|
||||
rm -rf "$OUTPUT/flatpak"
|
||||
# Note that compression is applied on a mount-level via the host system.
|
||||
compsize "$OUTPUT.flatpak"
|
||||
btrfs subvolume snapshot -r "$OUTPUT.flatpak" "$OUTPUT.export.flatpak"
|
||||
btrfs send --compressed-data -f "$OUTPUT.btrfs.flatpak" "$OUTPUT.export.flatpak"
|
||||
btrfs subvolume delete "$OUTPUT.export.flatpak"
|
||||
FLATPAK_SIZE=$(stat --format %s "$OUTPUT.btrfs.flatpak") # the actual size of all data
|
||||
fi
|
||||
# Now let's actually build a live raw image. First, the ESP.
|
||||
# We use kde-linux.cache instead of /tmp as usual because we'll probably run out of space there.
|
||||
|
||||
LIVE_SIZE=""
|
||||
# Move /live out of the tree and into subvolume
|
||||
if $OUTPUT_IS_BTRFS_SUBVOLUME; then
|
||||
btrfs subvolume create "$OUTPUT.live"
|
||||
cp -rf --reflink=always "$OUTPUT/live/." "$OUTPUT.live"
|
||||
rm -rf "$OUTPUT/live"
|
||||
# Note that compression is applied on a mount-level via the host system.
|
||||
compsize "$OUTPUT.live"
|
||||
btrfs subvolume snapshot -r "$OUTPUT.live" "$OUTPUT.export.live"
|
||||
btrfs send --compressed-data -f "$OUTPUT.btrfs.live" "$OUTPUT.export.live"
|
||||
btrfs subvolume delete "$OUTPUT.export.live"
|
||||
LIVE_SIZE=$(stat --format %s "$OUTPUT.btrfs.live") # the actual size of all data
|
||||
fi
|
||||
# Since we're building a live image, replace the main UKI with the live one.
|
||||
cp "$LIVE_UKI" "${OUTPUT}/efi-template/EFI/Linux/$EFI"
|
||||
|
||||
# Change to kde-linux.cache since we'll be working there.
|
||||
cd kde-linux.cache
|
||||
|
||||
# Create a 260M large FAT32 filesystem inside of esp.raw.
|
||||
fallocate -l 260M esp.raw
|
||||
mkfs.fat -F 32 esp.raw
|
||||
|
||||
# Mount it to esp.raw.mnt.
|
||||
mkdir esp.raw.mnt
|
||||
mount esp.raw esp.raw.mnt
|
||||
|
||||
# Copy everything from /efi-template into esp.raw.mnt.
|
||||
cp --archive --recursive "${OUTPUT}/efi-template/." esp.raw.mnt
|
||||
|
||||
# We're done, unmount esp.raw.mnt.
|
||||
umount esp.raw.mnt
|
||||
|
||||
# Now, the root.
|
||||
|
||||
# Copy back the main UKI for the root.
|
||||
cp "$MAIN_UKI" "${OUTPUT}/efi-template/EFI/Linux/$EFI"
|
||||
|
||||
# Create an 8G large btrfs filesystem inside of root.raw.
|
||||
# Don't fret, we'll shrink this down to however much we actually need later.
|
||||
fallocate -l 8G root.raw
|
||||
mkfs.btrfs -L KDELinuxLive root.raw
|
||||
|
||||
# Mount it to root.raw.mnt.
|
||||
mkdir root.raw.mnt
|
||||
mount -o compress-force=zstd:15 root.raw root.raw.mnt
|
||||
|
||||
# Change to root.raw.mnt since we'll be working there.
|
||||
cd root.raw.mnt
|
||||
|
||||
# Enable compression filesystem-wide.
|
||||
btrfs property set . compression zstd:15
|
||||
|
||||
# Store both data and metadata only once for more compactness.
|
||||
btrfs balance start --force -mconvert=single -dconvert=single .
|
||||
|
||||
# Create all the subvolumes we need.
|
||||
btrfs subvolume create \
|
||||
@home \
|
||||
@root \
|
||||
@locale \
|
||||
@snap \
|
||||
@etc-overlay \
|
||||
@var-overlay \
|
||||
@live \
|
||||
@flatpak \
|
||||
"@kde-linux_$VERSION"
|
||||
|
||||
mkdir @etc-overlay/upper \
|
||||
@etc-overlay/work \
|
||||
@var-overlay/upper \
|
||||
@var-overlay/work
|
||||
|
||||
# Create read-only subvolumes from /live, /flatpak and /.
|
||||
cp --archive --recursive "${OUTPUT}/live/." @live
|
||||
cp --archive --recursive "${OUTPUT}/flatpak/." @flatpak
|
||||
rm --recursive "${OUTPUT}/live"
|
||||
rm --recursive "${OUTPUT}/flatpak"
|
||||
cp --archive --recursive "${OUTPUT}/." "@kde-linux_$VERSION"
|
||||
btrfs property set @live ro true
|
||||
btrfs property set @flatpak ro true
|
||||
btrfs property set "@kde-linux_$VERSION" ro true
|
||||
|
||||
# Make a symlink called @kde-linux to the rootfs subvolume.
|
||||
ln --symbolic "@kde-linux_$VERSION" @kde-linux
|
||||
|
||||
# Make sure everything is written before we continue.
|
||||
btrfs filesystem sync .
|
||||
|
||||
# Optimize the filesystem for better shrinking/performance.
|
||||
btrfs filesystem defragment -r .
|
||||
btrfs filesystem sync .
|
||||
duperemove -rdq .
|
||||
btrfs filesystem sync .
|
||||
btrfs balance start --full-balance --enqueue .
|
||||
btrfs filesystem sync .
|
||||
|
||||
# How much we'll keep shrinking the filesystem by, in bytes.
|
||||
# Too large = too imprecise, too small = shrink is too slow.
|
||||
# One mebibyte seems good for now.
|
||||
SHRINK_AMOUNT=1048576
|
||||
|
||||
# Repeatedly shrink the filesystem by $SHRINK_AMOUNT until we get an error.
|
||||
# Store the size it has been successfully shrunk by in $SHRINK_SIZE.
|
||||
SHRINK_SIZE=0
|
||||
while true; do
|
||||
btrfs filesystem resize -$SHRINK_AMOUNT . || break
|
||||
SHRINK_SIZE=$((SHRINK_SIZE + SHRINK_AMOUNT))
|
||||
btrfs filesystem sync .
|
||||
done
|
||||
|
||||
# Back out to kde-linux.cache, then unmount root.raw.mnt.
|
||||
cd ..
|
||||
umount root.raw.mnt
|
||||
|
||||
# We shrunk the filesystem, but root.raw as a file itself is still 8G.
|
||||
# Let's safely truncate it by the previously stored $SHRINK_SIZE.
|
||||
truncate --size=-$SHRINK_SIZE root.raw
|
||||
|
||||
# We're done, back out of kde-linux.cache into the root.
|
||||
cd ..
|
||||
|
||||
# Create rootfs tarball for consumption by systemd-sysext (doesn't currently support consuming raw images :()
|
||||
rm -rf "$TAR" ./*.tar
|
||||
tar -C "${OUTPUT}"/ --xattrs --xattrs-include=*.* -cf "$TAR" .
|
||||
SIZE=$(stat --format %s "$TAR") # the apparent size of all data
|
||||
zstd -T0 --rm "$TAR"
|
||||
rm -rf "$ROOTFS_TAR" ./*.tar
|
||||
tar -C "${OUTPUT}"/ --xattrs --xattrs-include=*.* -cf "$ROOTFS_TAR" .
|
||||
zstd -T0 --rm "$ROOTFS_TAR"
|
||||
|
||||
# Accurate sizing is a bit of a challenge. In the most ideal scenario we'll be working on btrfs and are able to
|
||||
# compress the entire subvolume into a file. This file size will then be more or less the DATA size in the filesystem.
|
||||
# On top of that we have the btrfs meta data and system data, these are kind of dependent on the actual partition size
|
||||
# but will generally be ~768M (this value entirely depends on how many files we have) and <50M for partitions <50G.
|
||||
if $OUTPUT_IS_BTRFS_SUBVOLUME; then
|
||||
# Note that compression is applied on a mount-level via the host system.
|
||||
compsize "$OUTPUT"
|
||||
btrfs subvolume snapshot -r "$OUTPUT" "$OUTPUT.export"
|
||||
btrfs send --compressed-data -f "$OUTPUT.btrfs" "$OUTPUT.export"
|
||||
btrfs subvolume delete "$OUTPUT.export"
|
||||
SIZE=$(stat --format %s "$OUTPUT.btrfs") # the actual size of all data
|
||||
SIZE=$((SIZE+2147483648)) # 2G slack -- this needs to be sufficient for our deduplication and balancing run. We'll shrink this way down later.
|
||||
else
|
||||
SIZE=$((SIZE+4294967296)) # 4G slack (our guess is less precise without btrfs)
|
||||
fi
|
||||
SIZE=$((SIZE+FLATPAK_SIZE)) # however much we need for flatpak
|
||||
SIZE=$((SIZE+LIVE_SIZE)) # however much we need for live data
|
||||
SIZE=$((SIZE+851443712)) # 768M for btrfs metadata, 44M for system block
|
||||
SIZE=$((SIZE+536870912)) # 512M for ESP
|
||||
|
||||
rm -f "$IMG" ./*.raw
|
||||
# Now assemble the two generated images using systemd-repart and the definitions in mkosi.repart into $IMG.
|
||||
touch "$IMG"
|
||||
# The root partition contains the shipable efi image for use on the installed system.
|
||||
systemd-repart --no-pager --empty=allow --size="$SIZE" --dry-run=no --root="${OUTPUT}" --definitions=mkosi.repart --defer-partitions=esp "$IMG"
|
||||
if $OUTPUT_IS_BTRFS_SUBVOLUME; then # btrfs subvolume
|
||||
systemd-dissect --with "$IMG" "$(pwd)/btrfs-send-receive.sh" "$PWD/$OUTPUT" "$OUTPUT" "@$NAME"
|
||||
else # do a raw copy
|
||||
systemd-dissect --with "$IMG" "$(pwd)/btrfs-copy.sh" "$PWD/$OUTPUT" "$OUTPUT" "@$NAME"
|
||||
fi
|
||||
# The esp of the image contains the live efi image (divergent cmdline).
|
||||
# We copy into efi-template for convenience, it won't actually be used from there, just copied by systemd-repart.
|
||||
cp -v "${OUTPUT}_live.efi" "${OUTPUT}/efi-template/EFI/Linux/$EFI"
|
||||
systemd-repart --no-pager --empty=allow --size=auto --dry-run=no --root="${OUTPUT}" --definitions=mkosi.repart --defer-partitions=root "$IMG"
|
||||
|
||||
# Finally rebuild the actual image file with appropriate partition sizing. In particular with squeezed btrfs.
|
||||
./part-rebuild.py "$IMG"
|
||||
systemd-repart --no-pager --empty=allow --size=auto --dry-run=no --root=kde-linux.cache --definitions=mkosi.repart "$IMG"
|
||||
|
||||
# Create a torrent for the image
|
||||
./torrent-create.rb "$VERSION" "$OUTPUT" "$IMG"
|
||||
@@ -129,5 +179,5 @@ systemd-repart --no-pager --empty=allow --size=auto --dry-run=no --root="${OUTPU
|
||||
# TODO before accepting new uploads perform sanity checks on the artifacts (e.g. the tar being well formed)
|
||||
|
||||
# efi images and torrents are 700, make them readable so the server can serve them
|
||||
chmod go+r ./"$OUTPUT".* ./*.efi ./*.torrent
|
||||
chmod go+r "$OUTPUT".* ./*.efi ./*.torrent
|
||||
ls -lah
|
||||
|
||||
@@ -6,6 +6,7 @@ Distribution=arch
|
||||
Mirror=https://mirror.23m.com/archlinux/
|
||||
|
||||
[Output]
|
||||
ImageId=kde-linux
|
||||
Format=directory
|
||||
SplitArtifacts=yes
|
||||
|
||||
|
||||
@@ -1,7 +0,0 @@
|
||||
# SPDX-License-Identifier: GPL-2.0-only OR GPL-3.0-only OR LicenseRef-KDE-Accepted-GPL
|
||||
# SPDX-FileCopyrightText: 2023 Harald Sitter <sitter@kde.org>
|
||||
|
||||
[Partition]
|
||||
Type=esp
|
||||
SplitName=esp
|
||||
CopyBlocks=/esp.raw
|
||||
@@ -1,9 +0,0 @@
|
||||
<!--
|
||||
SPDX-License-Identifier: CC0-1.0
|
||||
SPDX-FileCopyrightText: 2024 Harald Sitter <sitter@kde.org>
|
||||
-->
|
||||
|
||||
This isn't actually used by mkosi but named like it so it is next to the actual mkosi directory.
|
||||
The configs in here simply pick up split partitions from the original repart run into a rebuild run.
|
||||
|
||||
Also see part-rebuild.py
|
||||
@@ -3,8 +3,4 @@
|
||||
|
||||
[Partition]
|
||||
Type=esp
|
||||
Format=vfat
|
||||
# Would be better if we calculated this from the actual artifact size
|
||||
SizeMaxBytes=260M
|
||||
CopyFiles=/efi-template:/
|
||||
SplitName=esp
|
||||
CopyBlocks=/esp.raw
|
||||
|
||||
@@ -4,5 +4,4 @@
|
||||
[Partition]
|
||||
Type=root
|
||||
Label=KDELinuxLive
|
||||
SplitName=root
|
||||
CopyBlocks=/KDELinuxLive.raw
|
||||
CopyBlocks=/root.raw
|
||||
@@ -1,32 +0,0 @@
|
||||
# SPDX-License-Identifier: GPL-2.0-only OR GPL-3.0-only OR LicenseRef-KDE-Accepted-GPL
|
||||
# SPDX-FileCopyrightText: 2023 Harald Sitter <sitter@kde.org>
|
||||
|
||||
[Partition]
|
||||
Type=root
|
||||
Format=btrfs
|
||||
SizeMinBytes=1G
|
||||
# Do not use spaces. We need to extract this later as a file path.
|
||||
Label=KDELinuxLive
|
||||
SplitName=root
|
||||
# NOTE: the FS isn't read only because we have mutable subvolumes. Individual subvolumes may be readonly though.
|
||||
# ReadOnly=true
|
||||
|
||||
Subvolumes=/@home
|
||||
MakeDirectories=/@home
|
||||
|
||||
Subvolumes=/@root
|
||||
MakeDirectories=/@root
|
||||
|
||||
Subvolumes=/@locale
|
||||
MakeDirectories=/@locale
|
||||
|
||||
Subvolumes=/@snap
|
||||
MakeDirectories=/@snap
|
||||
|
||||
Subvolumes=/@etc-overlay
|
||||
MakeDirectories=/@etc-overlay /@etc-overlay/upper /@etc-overlay/work
|
||||
|
||||
Subvolumes=/@var-overlay
|
||||
MakeDirectories=/@var-overlay /@var-overlay/upper /@var-overlay/work
|
||||
|
||||
# @flatpak is created by build.sh
|
||||
@@ -1,44 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
|
||||
# SPDX-License-Identifier: GPL-2.0-only OR GPL-3.0-only OR LicenseRef-KDE-Accepted-GPL
|
||||
# SPDX-FileCopyrightText: 2024 Harald Sitter <sitter@kde.org>
|
||||
|
||||
# Rebuilds the image using appropriate partition sizes.
|
||||
|
||||
import json
|
||||
import math
|
||||
import subprocess
|
||||
from subprocess import check_output
|
||||
import sys
|
||||
import os
|
||||
|
||||
RAW = sys.argv[1]
|
||||
|
||||
# Load the partition table
|
||||
out = check_output(["sfdisk", "--json", RAW])
|
||||
data = json.loads(out)
|
||||
partitiontable = data["partitiontable"]
|
||||
sectorsize = partitiontable["sectorsize"]
|
||||
partitions = partitiontable["partitions"]
|
||||
|
||||
# Find the root partition
|
||||
root_partition = None
|
||||
for index, partition in enumerate(partitions):
|
||||
if partition["name"] == "KDELinuxLive":
|
||||
root_partition = partition
|
||||
|
||||
# Update its size to what it actually is on the filesystem level
|
||||
btrfs = json.loads(open("btrfs.json").read())
|
||||
root_partition["size"] = math.ceil(btrfs['size'] / sectorsize)
|
||||
|
||||
# Fish out the partitions with appropriate skip and count values (i.e. the actual filesystem sizes)
|
||||
for index, partition in enumerate(partitions):
|
||||
subprocess.run(["dd", f"if={RAW}", f"of={partition["name"]}.raw", f"bs={sectorsize}", f"skip={partition["start"]}", f"count={partition["size"]}"], check=True)
|
||||
|
||||
# Then reassemble them into a new image using those squeezed partition images.
|
||||
open(RAW, "w").close()
|
||||
subprocess.run(["systemd-repart", "--size=auto", "--empty=allow", "--dry-run=no", "--definitions=mkosi.repart-rebuild", "--root", os.getcwd(), RAW], check=True)
|
||||
|
||||
# Clean up the partitions again
|
||||
for partition in partitions:
|
||||
os.remove(f"{partition["name"]}.raw")
|
||||
Reference in New Issue
Block a user