Currently Flatpak only supports extensions which come from the same
remote as the thing being extended; for discussion on this see
https://github.com/flatpak/flatpak/issues/861
However in general it isn't clear from the metadata what remote provides
an extension. For example com.endlessm.apps.Platform//5 defines the
extension org.freedesktop.Platform.VAAPI.Intel (inherited from
org.freedesktop.Platform) which can be found on flathub not eos-sdk. So
we don't want to add an extension refspec to the transaction if the
current remote is not the one that actually provides it. In
flatpak_dir_find_remote_related_for_metadata() this invariant is
maintained because we check if the ref exists in the remote before
adding it with add_related(). However in
flatpak_dir_find_local_related_for_metadata() we check for existing
deploy data but omit checking that the deploy origin matches the origin
passed in, and in that case can accidentally add an incorrect refspec to
a transaction. So this commit adds the missing origin check.
One way to reproduce this issue is with this command, having both the
Endless platform and the VAAPI extension already installed:
$ flatpak update --no-pull com.endlessm.apps.Platform//5
Looking for updates…
error: Refspec 'eos-sdk:runtime/org.freedesktop.Platform.VAAPI.Intel/x86_64/1.6' not found
This issue also affects GNOME Software which does a no-deploy
transaction followed by a no-pull transaction (this issue occurs in the
latter) and is treated as fatal causing updates not to be applied.
However it only happens in some niche circumstances, because
flatpak_transaction_add_op() will only use the first remote passed to it
if called more than once for the same ref from different remotes. This
is normally fine; refs generally only come from one remote. But it does
mean this issue only occurs if the extension in question was not already
added to the transaction with a correct origin.
We only want the run dir to be overridable in unit tests because we
depend on it being /run/flatpak in flatpak-create-sideload-symlinks.sh,
so don't mention it in the flatpak man page.
Currently we only support links in /var/lib/flatpak/sideload-repos,
/run/flatpak/sideload-repos, etc. to be actual ostree repos, but this
commit makes it so you can also link to the root directory of a USB,
and Flatpak will check the subpaths "ostree/repo", ".ostree/repo", and
".ostree/repos.d" for compatibility with "flatpak create-usb". This will
allow the logic in the following commit to be much simpler, where we're
linking to hot-plugged drives in a script run by systemd.
Note that we still only allow actual repos in the other places where a
sideload path can be specified, such as the --sideload-repo CLI option.
If the magical io.github.containers.DeltaUrl label is set in the
index, then try to download this to use as the delta manifest for the
image. This allows servers to store deltas outside the registry
itself. The label is propagated to the xa.delta-url metadata in the
generated "fake summary" for the remote, and read back on pull.
Note that the delta manifest layers descriptor will need to have a
"urls" key where it references the blobs if the blobs are also not
stored on the registry.
If the specified manifest doesn't exist or doesn't apply to the target
image we fall back to resolving via the _deltaindex tag.
Call flatpak_remote_state_ensure_summary() before dereferencing
RemoteState->summary to return an error instead of crashing if
downloading the summary failed.
When mirroring to a local OCI dir we apply deltas and generate
uncompressed layer blobs instead of regular blobs. Then we pick this
up on the system-helper side.
When we create a system child registry we also set the current token on
it. This is not used directly in the client, however its saved in a
file called .token and re-read in the system-helper, allowing it to
also do the remote registry operations it needs to verify the child
registry.
This downloads a manifest by digest and reconstructs the commit except
for the tree/metadata reference. This allows things like flatpak remote-info
and flatpak update --commit=xxx to work.
Only have the single function flatpak_remote_state_load_ref_commit()
that loads the commit objects (and don't support loading other types
of objects as that is not used).
This is a minor cleanup, but it also will be useful later when
we want to be able to "load" commits from OCI repositories.
Re-implement flatpak_installation_list_installed_refs_for_update() using
a FlatpakTransaction, so we can guarantee it always gives the same set
of things to update as the update command. This API is used by GNOME
Software and many times in the past g-s has not shown the same list of
apps to be updated as the flatpak CLI. See:
- https://gitlab.gnome.org/GNOME/gnome-software/issues/539
- https://gitlab.gnome.org/GNOME/gnome-software/-/merge_requests/430
This commit also expands the unit tests for this API, which were already
quite good. Now we test that missing subpaths of locale extensions show
up as updates, and updates that have been pulled but not deployed show
up as well. The latter is a break from how this function used to behave,
but it seems unlikely to break any application.
We used to always create a commitpartial file in child repo, because
ostree doesn't follow parent repos when loading commitpartial state,
and when the commit was in the parent repo it would find the commit
but no commitpartial and assume the commit was complete and do nothing.
However, having a commitpartial file seems to break delta downloads in
ostree, as per: https://github.com/ostreedev/ostree/issues/2053
causing us to download too much data when using deltas.
So, we now only create a commitpartial if there is one in the parent
repo. This still means we will get the ostree problems in case there
is one, but in the much more common case we avoid the issue.
In order to "fix" the uncommon case we also (separately) cap the
reported progress at 100%. (We should probably also fix the upstream
ostree issue too.)
We need to get the commit object to setup the extra-data progress information,
and this is currently done using a complex pull operation to a temporary
repo. According to https://github.com/flatpak/flatpak/issues/3515 it
even causes an unecessary download of the summary in some cases.
Now that we don't need to support p2p we can instead directly download
the commit object using a simple http operation (or from the sideload
repos), as we know the commit id at this point anyway.
I noticed several places in flatpak-dir.c that didn't check for
NULL progress, so lets move the check inside the implementation so
we can ensure its always checked.
To avoid the complexities of passing (and chaining) OstreeAsyncProgress
objects around, we only create one just before calling to ostree.
The rest of flatpak only ever uses the new FlatpakProgress object.
Co-authored by: Philip Chimento <philip@endlessm.com>
Instead of having a global config option we scan a directory for
symlinks into the sideload repos. These come from
/var/lib/flatpak/sideload-repos and /run/flatpak/sideload-repos (for
default system installation).
This is much easier to update atomically, and the two different
options are useful for persistant (the first) or dynamic (the second)
usescase.
Fixes https://github.com/flatpak/flatpak/issues/3494
Instead of doing a lot of FlatpakInstallation calls we do lower level
FlatpakDir calls, sharing a single RemoteState per remote for the
entire operation. Also, some parts of the checks are moved to FlatpakDir
as flatpak_dir_check_if_installed_ref_needs_update()
This is similar to lookup_cache() but it also works for
sideloaded refs. Additionally it returns an allocated metadata
pointer rather than a pointer to the cache.
Also convert some callers to use this when it makes sense.
We use the localcache-repos option to ostree_repo_pull to make ostree
directly import any files that are locally available in the sideload
repo even when pulling the main commit from upstream.
This also adds a test that verifies that such files are not
pulled via http.
