We need to get the commit object to setup the extra-data progress information,
and this is currently done using a complex pull operation to a temporary
repo. According to https://github.com/flatpak/flatpak/issues/3515 it
even causes an unecessary download of the summary in some cases.
Now that we don't need to support p2p we can instead directly download
the commit object using a simple http operation (or from the sideload
repos), as we know the commit id at this point anyway.
I noticed several places in flatpak-dir.c that didn't check for
NULL progress, so lets move the check inside the implementation so
we can ensure its always checked.
To avoid the complexities of passing (and chaining) OstreeAsyncProgress
objects around, we only create one just before calling to ostree.
The rest of flatpak only ever uses the new FlatpakProgress object.
Co-authored by: Philip Chimento <philip@endlessm.com>
A make rule like
a b: x y
command
does not mean "run command to generate a and b from x and y". Instead,
it means "run command to generate a from x and y", and, separately,
"run command to generate b from x and y". In a parallel build this
could mean that we try to run the variant-schema-compiler twice, in
parallel, with the output from each run overwriting the other.
Signed-off-by: Simon McVittie <smcv@collabora.com>
This means it doesn't go into dist tarballs, and we don't need to add
it to BUILT_SOURCES and CLEANFILES separately because it's already
there.
Signed-off-by: Simon McVittie <smcv@collabora.com>
Instead of having a global config option we scan a directory for
symlinks into the sideload repos. These come from
/var/lib/flatpak/sideload-repos and /run/flatpak/sideload-repos (for
default system installation).
This is much easier to update atomically, and the two different
options are useful for persistant (the first) or dynamic (the second)
usescase.
Fixes https://github.com/flatpak/flatpak/issues/3494
With the new sideload support usb sideload works differently, they don't
appear like separate remotes. Its just the normal remote and installs
from it will automatically work. However, to list just the sideloadable
refs we point the docs to the new ONLY_SIDELOADED flag.
Instead of doing a lot of FlatpakInstallation calls we do lower level
FlatpakDir calls, sharing a single RemoteState per remote for the
entire operation. Also, some parts of the checks are moved to FlatpakDir
as flatpak_dir_check_if_installed_ref_needs_update()
This is similar to lookup_cache() but it also works for
sideloaded refs. Additionally it returns an allocated metadata
pointer rather than a pointer to the cache.
Also convert some callers to use this when it makes sense.
We use the localcache-repos option to ostree_repo_pull to make ostree
directly import any files that are locally available in the sideload
repo even when pulling the main commit from upstream.
This also adds a test that verifies that such files are not
pulled via http.
This is a new version of --deploy-collection-id that only applies
the collection id update for new (1.7.x+) version of flatpak clients.
This allows you to enable collection ids for sideload use but not
affect older clients where the p2p codepaths are not as tested.
This used to not be set for collection-id remotes as we used the
ostree-metadata branch for resolving. However, we now use the summary
always when doing a remote install (and not ostree-metadata for local
sideloads), so we still want to verify summary.
The signature on the summary is a nice security feature, but it is also
a very efficient small file to download to verify that no new summary
needs to be downloaded in the no-op update case.
Nothing fundamentally happens differently in ostree if the collection-id
is set, as long as we don't call the p2p specific apis. So, lets keep
using it instead of adding our own special magic.
With the new sideload approach to collection ids it is fine to require
gpg signed summaries. (Not for the child repos or the sideload repos
though, but thoser are either trusted (sideload) or safe for other
reasons like ref-bindings and signed commits).
We used to to do a MIRROR pull and make a copy of the upstream summary
file in the child repo. However, now that we want to allow side-loading
from a repo with a partial summary we can no longer do that.
So, the new approach is that the child repo *always* contains
"remote:ref" style refs (never heads or mirror/collection-id refs),
but in order to not let you import a signel commit into the wrong ref
name we now require the commits to have a ref-binding (they all do
since a long time anyway).
Most code that looks for a regular collection id set on the remote is
removed, as these should never happen in flatpak repo setups now.
Some is replaces with looking at xa.sideload-collection-id:
* The libflatpak FlatpakRef::collection-id property now comes comes from the sideload id
* Various CLI commands showing or changing the collection-id for a remote now uses the sideload id
* Collection id deploy in update now sets the sideload-collection-id instead
* Setting the collection id for a remote in libflatpak now sets the sideload id
Additionally we now delete the code that allows unsigned summaries
when there is a collection id (because there is none).
create-usb now uses the sideload id as as collection id source when exporting.
The direct repo operations (export, bundle, commit-from) still support
collection ids, because on the server we do want to set it so that we
can sideload.
This adds a xa.sideload-collection-id option to the remote
configuration and a global xa.sideload-repos option (which is a list
of paths to local repos).
When resolving or listing refs, if we fail to download the real remote
summary (i.e. we're offline) then we instead look into the configured
sideloaded repos for refs that match ref and the sideloaded collection
id for the remote.
For the transaction to resolve the ref we need more metadata. In the
regular summary case we use the metadata from the summary, but that
is not available in the (partial) summary in the sideload repo, so
there we load the actual commit object and use the data from there.
(The ostree-metadata branch is not used/needed.)
This actually also fixes a longstanding issue when you "flatpak update
--checksum=XYZ" because we now handle this correctly by downloading
the commit object from the remote. Before we used the metadata in the
summary which is not right for non-HEAD commits.
To handle the sideloading we record the path to the sideload repo
when sideloading and pass the url to the repo as the remote name
when pulling, which will do a direct local pull.
We avoid using sideloaded refs when offline if the timestamp in the
commits is older than what is already installed locally.
This removes the most basic codepaths for p2p installation, as well
as the tests for it. There still remains various codepaths that
looks as the collection id, these will be removed later.
This is the first step in dropping the p2p code and replacing it with
a simpler approach that focuses on the sideloading case only.
We're using the metadata from the summary, ostree-metadata or available
commit when making security sensitive decisions, so lets verify this
matches what we get in the actual commit we pulled.
We already did check that this then actually also matches what gets deployed,
so the new check shares code with that.
Note, we don't do this for OCI installs, because it seems the current
fedora flatpaks don't have this set, and we don't want to break
existing remotes.
Currently if flatpak is installing an extra data app such as Spotify and
the server with the .deb file fails to complete the request, the
installation fails with a message like "Connection terminated
unexpectedly". This commit makes flatpak instead try 5 times to download
a given URI if the error returned seems like a transient one (so not,
for example, 404 not found). This is analogous to what was done in
libostree in commit 938055392fd455027a69398c441b992ae521aa87, and we use
some code from there.
