This adds the nr of extra datas, as well as the total size in the sparse
cache for all refs that has them. This is what we need for in the download,
and having it in the summary means we don't have to separately download
the commit.
Additionally we add a cache version field to the summary so that we
can know if we can rely on the existance of the new data.
We used to always create a commitpartial file in child repo, because
ostree doesn't follow parent repos when loading commitpartial state,
and when the commit was in the parent repo it would find the commit
but no commitpartial and assume the commit was complete and do nothing.
However, having a commitpartial file seems to break delta downloads in
ostree, as per: https://github.com/ostreedev/ostree/issues/2053
causing us to download too much data when using deltas.
So, we now only create a commitpartial if there is one in the parent
repo. This still means we will get the ostree problems in case there
is one, but in the much more common case we avoid the issue.
In order to "fix" the uncommon case we also (separately) cap the
reported progress at 100%. (We should probably also fix the upstream
ostree issue too.)
Due to bugs in ostree (see
e.g. https://github.com/flatpak/flatpak/pull/3524) it sometimes
happens that we download more data than we expect to. This isn't
great, but when it happens its better to limit the progress to 100%
anyway to avoid breaking apps (for example, the flatpak CLI doesn't
really handle this very well).
We need to get the commit object to setup the extra-data progress information,
and this is currently done using a complex pull operation to a temporary
repo. According to https://github.com/flatpak/flatpak/issues/3515 it
even causes an unecessary download of the summary in some cases.
Now that we don't need to support p2p we can instead directly download
the commit object using a simple http operation (or from the sideload
repos), as we know the commit id at this point anyway.
I noticed several places in flatpak-dir.c that didn't check for
NULL progress, so lets move the check inside the implementation so
we can ensure its always checked.
To avoid the complexities of passing (and chaining) OstreeAsyncProgress
objects around, we only create one just before calling to ostree.
The rest of flatpak only ever uses the new FlatpakProgress object.
Co-authored by: Philip Chimento <philip@endlessm.com>
A make rule like
a b: x y
command
does not mean "run command to generate a and b from x and y". Instead,
it means "run command to generate a from x and y", and, separately,
"run command to generate b from x and y". In a parallel build this
could mean that we try to run the variant-schema-compiler twice, in
parallel, with the output from each run overwriting the other.
Signed-off-by: Simon McVittie <smcv@collabora.com>
This means it doesn't go into dist tarballs, and we don't need to add
it to BUILT_SOURCES and CLEANFILES separately because it's already
there.
Signed-off-by: Simon McVittie <smcv@collabora.com>
Instead of having a global config option we scan a directory for
symlinks into the sideload repos. These come from
/var/lib/flatpak/sideload-repos and /run/flatpak/sideload-repos (for
default system installation).
This is much easier to update atomically, and the two different
options are useful for persistant (the first) or dynamic (the second)
usescase.
Fixes https://github.com/flatpak/flatpak/issues/3494
With the new sideload support usb sideload works differently, they don't
appear like separate remotes. Its just the normal remote and installs
from it will automatically work. However, to list just the sideloadable
refs we point the docs to the new ONLY_SIDELOADED flag.
Instead of doing a lot of FlatpakInstallation calls we do lower level
FlatpakDir calls, sharing a single RemoteState per remote for the
entire operation. Also, some parts of the checks are moved to FlatpakDir
as flatpak_dir_check_if_installed_ref_needs_update()
This is similar to lookup_cache() but it also works for
sideloaded refs. Additionally it returns an allocated metadata
pointer rather than a pointer to the cache.
Also convert some callers to use this when it makes sense.
We use the localcache-repos option to ostree_repo_pull to make ostree
directly import any files that are locally available in the sideload
repo even when pulling the main commit from upstream.
This also adds a test that verifies that such files are not
pulled via http.
This is a new version of --deploy-collection-id that only applies
the collection id update for new (1.7.x+) version of flatpak clients.
This allows you to enable collection ids for sideload use but not
affect older clients where the p2p codepaths are not as tested.
This used to not be set for collection-id remotes as we used the
ostree-metadata branch for resolving. However, we now use the summary
always when doing a remote install (and not ostree-metadata for local
sideloads), so we still want to verify summary.
The signature on the summary is a nice security feature, but it is also
a very efficient small file to download to verify that no new summary
needs to be downloaded in the no-op update case.
Nothing fundamentally happens differently in ostree if the collection-id
is set, as long as we don't call the p2p specific apis. So, lets keep
using it instead of adding our own special magic.
