Re-implement flatpak_installation_list_installed_refs_for_update() using
a FlatpakTransaction, so we can guarantee it always gives the same set
of things to update as the update command. This API is used by GNOME
Software and many times in the past g-s has not shown the same list of
apps to be updated as the flatpak CLI. See:
- https://gitlab.gnome.org/GNOME/gnome-software/issues/539
- https://gitlab.gnome.org/GNOME/gnome-software/-/merge_requests/430
This commit also expands the unit tests for this API, which were already
quite good. Now we test that missing subpaths of locale extensions show
up as updates, and updates that have been pulled but not deployed show
up as well. The latter is a break from how this function used to behave,
but it seems unlikely to break any application.
We used to always create a commitpartial file in child repo, because
ostree doesn't follow parent repos when loading commitpartial state,
and when the commit was in the parent repo it would find the commit
but no commitpartial and assume the commit was complete and do nothing.
However, having a commitpartial file seems to break delta downloads in
ostree, as per: https://github.com/ostreedev/ostree/issues/2053
causing us to download too much data when using deltas.
So, we now only create a commitpartial if there is one in the parent
repo. This still means we will get the ostree problems in case there
is one, but in the much more common case we avoid the issue.
In order to "fix" the uncommon case we also (separately) cap the
reported progress at 100%. (We should probably also fix the upstream
ostree issue too.)
We need to get the commit object to setup the extra-data progress information,
and this is currently done using a complex pull operation to a temporary
repo. According to https://github.com/flatpak/flatpak/issues/3515 it
even causes an unecessary download of the summary in some cases.
Now that we don't need to support p2p we can instead directly download
the commit object using a simple http operation (or from the sideload
repos), as we know the commit id at this point anyway.
I noticed several places in flatpak-dir.c that didn't check for
NULL progress, so lets move the check inside the implementation so
we can ensure its always checked.
To avoid the complexities of passing (and chaining) OstreeAsyncProgress
objects around, we only create one just before calling to ostree.
The rest of flatpak only ever uses the new FlatpakProgress object.
Co-authored by: Philip Chimento <philip@endlessm.com>
Instead of having a global config option we scan a directory for
symlinks into the sideload repos. These come from
/var/lib/flatpak/sideload-repos and /run/flatpak/sideload-repos (for
default system installation).
This is much easier to update atomically, and the two different
options are useful for persistant (the first) or dynamic (the second)
usescase.
Fixes https://github.com/flatpak/flatpak/issues/3494
Instead of doing a lot of FlatpakInstallation calls we do lower level
FlatpakDir calls, sharing a single RemoteState per remote for the
entire operation. Also, some parts of the checks are moved to FlatpakDir
as flatpak_dir_check_if_installed_ref_needs_update()
This is similar to lookup_cache() but it also works for
sideloaded refs. Additionally it returns an allocated metadata
pointer rather than a pointer to the cache.
Also convert some callers to use this when it makes sense.
We use the localcache-repos option to ostree_repo_pull to make ostree
directly import any files that are locally available in the sideload
repo even when pulling the main commit from upstream.
This also adds a test that verifies that such files are not
pulled via http.
This is a new version of --deploy-collection-id that only applies
the collection id update for new (1.7.x+) version of flatpak clients.
This allows you to enable collection ids for sideload use but not
affect older clients where the p2p codepaths are not as tested.
This used to not be set for collection-id remotes as we used the
ostree-metadata branch for resolving. However, we now use the summary
always when doing a remote install (and not ostree-metadata for local
sideloads), so we still want to verify summary.
The signature on the summary is a nice security feature, but it is also
a very efficient small file to download to verify that no new summary
needs to be downloaded in the no-op update case.
Nothing fundamentally happens differently in ostree if the collection-id
is set, as long as we don't call the p2p specific apis. So, lets keep
using it instead of adding our own special magic.
With the new sideload approach to collection ids it is fine to require
gpg signed summaries. (Not for the child repos or the sideload repos
though, but thoser are either trusted (sideload) or safe for other
reasons like ref-bindings and signed commits).
We used to to do a MIRROR pull and make a copy of the upstream summary
file in the child repo. However, now that we want to allow side-loading
from a repo with a partial summary we can no longer do that.
So, the new approach is that the child repo *always* contains
"remote:ref" style refs (never heads or mirror/collection-id refs),
but in order to not let you import a signel commit into the wrong ref
name we now require the commits to have a ref-binding (they all do
since a long time anyway).
Most code that looks for a regular collection id set on the remote is
removed, as these should never happen in flatpak repo setups now.
Some is replaces with looking at xa.sideload-collection-id:
* The libflatpak FlatpakRef::collection-id property now comes comes from the sideload id
* Various CLI commands showing or changing the collection-id for a remote now uses the sideload id
* Collection id deploy in update now sets the sideload-collection-id instead
* Setting the collection id for a remote in libflatpak now sets the sideload id
Additionally we now delete the code that allows unsigned summaries
when there is a collection id (because there is none).
create-usb now uses the sideload id as as collection id source when exporting.
The direct repo operations (export, bundle, commit-from) still support
collection ids, because on the server we do want to set it so that we
can sideload.
This adds a xa.sideload-collection-id option to the remote
configuration and a global xa.sideload-repos option (which is a list
of paths to local repos).
When resolving or listing refs, if we fail to download the real remote
summary (i.e. we're offline) then we instead look into the configured
sideloaded repos for refs that match ref and the sideloaded collection
id for the remote.
For the transaction to resolve the ref we need more metadata. In the
regular summary case we use the metadata from the summary, but that
is not available in the (partial) summary in the sideload repo, so
there we load the actual commit object and use the data from there.
(The ostree-metadata branch is not used/needed.)
This actually also fixes a longstanding issue when you "flatpak update
--checksum=XYZ" because we now handle this correctly by downloading
the commit object from the remote. Before we used the metadata in the
summary which is not right for non-HEAD commits.
To handle the sideloading we record the path to the sideload repo
when sideloading and pass the url to the repo as the remote name
when pulling, which will do a direct local pull.
We avoid using sideloaded refs when offline if the timestamp in the
commits is older than what is already installed locally.
This removes the most basic codepaths for p2p installation, as well
as the tests for it. There still remains various codepaths that
looks as the collection id, these will be removed later.
This is the first step in dropping the p2p code and replacing it with
a simpler approach that focuses on the sideloading case only.
We're using the metadata from the summary, ostree-metadata or available
commit when making security sensitive decisions, so lets verify this
matches what we get in the actual commit we pulled.
We already did check that this then actually also matches what gets deployed,
so the new check shares code with that.
Note, we don't do this for OCI installs, because it seems the current
fedora flatpaks don't have this set, and we don't want to break
existing remotes.
When storing the token-type in the commit and the summary cache we
hardcode it to little-endian.
In theory this breaks the "ABI", but in practice this change is a
no-op on little-endian systems which is what most are. Additionally as most
servers are little-endian this also fixes using big-endian clients with
such servers.
This fixes:
https://github.com/flatpak/flatpak/issues/3434
