This adds the nr of extra datas, as well as the total size in the sparse
cache for all refs that has them. This is what we need for in the download,
and having it in the summary means we don't have to separately download
the commit.
Additionally we add a cache version field to the summary so that we
can know if we can rely on the existance of the new data.
To avoid the complexities of passing (and chaining) OstreeAsyncProgress
objects around, we only create one just before calling to ostree.
The rest of flatpak only ever uses the new FlatpakProgress object.
Co-authored by: Philip Chimento <philip@endlessm.com>
This is a new version of --deploy-collection-id that only applies
the collection id update for new (1.7.x+) version of flatpak clients.
This allows you to enable collection ids for sideload use but not
affect older clients where the p2p codepaths are not as tested.
With the new sideload approach to collection ids it is fine to require
gpg signed summaries. (Not for the child repos or the sideload repos
though, but thoser are either trusted (sideload) or safe for other
reasons like ref-bindings and signed commits).
Allow the app-id or the DConf path to finish with a digit and still be
considered similar enough for DConf migration purposes.
This allows the org.gnome.Rhythmbox3 app-id to migrate its
/org/gnome/rhythmbox DConf path.
See https://github.com/flathub/org.gnome.Rhythmbox3/pull/26
When storing the token-type in the commit and the summary cache we
hardcode it to little-endian.
In theory this breaks the "ABI", but in practice this change is a
no-op on little-endian systems which is what most are. Additionally as most
servers are little-endian this also fixes using big-endian clients with
such servers.
This fixes:
https://github.com/flatpak/flatpak/issues/3434
We can't use the built-in bsearch from the codegen because its an array
instead of a dict, so we have to keep that but its now not using
variant at least.
In flatpak-builtins-build-commit-from.c we call flatpak_repo_collect_sizes()
without initializing the passed in download size to zero, which mean
we sum with sizes with some random value as the start.
This is fixed by having flatpak_repo_collect_sizes() always initialize
the counters to 0 at the start.
Fixes https://github.com/flatpak/flatpak/issues/3362
flatpak build-update-repo now lets you modify the
autenticator-name/install/options keys, and these are migrated to
the summary/metadata during update.
With the latest ostree that enables the chaining of progress the
testsuite broke because we were not getting changed events. Looking
into this the reason seems to be that when we run the
ostree_async_progress_finish() on the chained progress it is marked
as dead, which causes ostree_async_progress_copy_state() to not copy
any data when called from handle_chained_progress().
The fix is to copy the content manually before calling the finish().
Also, the entire callback chaining system seems wildly
overcomplicated, so I simplified it by relying on the existing change
notification of OstreeAsyncProgress.
If we're using a chained progress, it will be unchained
in the destroy notifier. However, it was newly constructed so we
need to also unref it or we'll leak it.
This also makes some minor cleanups:
1) Centralize version checks to one place and replace users
with #ifdef FLATPAK_DO_CHAIN_PROGRESS which makes it
easier to read and to test the fallback.
2) Make flatpak_progress_chain return a FlatpakAsyncProgressChained
to make it clear the two needs to be paired.
This is a slightly incompatible change, as we now only support
oci images generated with (what was before) build-export --oci-use-labels.
However, there are not a lot of OCI implementations in the wild, and
we can modify the ones in use to ensure there are labels (and
annotations if needed for older flatpak clients).
This also removes the --oci-use-label option from build-bundle --oci as
this is now the default.
This is in the same order as the xa.cache array and contains the id of
the commit that the cached data is about. This is not necessary in the
non-p2p summary metadata, because in that we always have a matching
ref -> commit array.
However, in the p2p case this information can be useful.
These are explicitly made short to save space, so lets have defines
for them to make sure we don't mistype them, especially as we
will be adding new keys.
The previous code checked whether the progress object had
"outstanding-fetches" set in order to decide whether it had been
initialized enough to show progress information. However, if the
callback saw a progress object on which flatpak_dir_setup_extra_data()
had not yet been called, then it would crash.
Therefore it seems that we should additionally be checking for the
presence of "outstanding-extra-data" which is set in
flatpak_dir_setup_extra_data().
It's likely that this wasn't previously a problem because the callback
would never get called due to the progress object's associated main
context not being iterated. It crashes now because that problem was
fixed in a previous commit.
Previously, in flatpak_dir_setup_extra_data(), n_extra_data (a gsize
which is 8 bytes wide on x86_64) was passed in a varargs list where an
unsigned int (4 bytes wide) was expected due to the "u" variant type
specifier.
This doesn't seem to have directly caused any crashes for me, but it's
undefined behaviour.
Therefore, this changes the affected keys "outstanding-extra-data" and
"total-extra-data" to be guint64 types instead of unsigned ints. The
gsize returned from g_variant_n_children() is cast to guint64 by virtue
of being assigned to a guint64-typed variable, but should not lose any
bits on supported platforms.
It's a common idiom in this codebase to push a temporary GMainContext as
the thread default context in order to run an async operation as if it
were sync. If we are not expecting progress callbacks this isn't a
problem, but it becomes a problem if we pass in an OstreeAsyncProgress
object that was created under a different GMainContext. The reason for
this is that OstreeAsyncProgress creates an idle source and attaches it
to the thread default context, so if we are iterating a temporary
context then the OstreeAsyncProgress's context never gets iterated, and
so no progress signals are fired.
To fix this, we introduce flatpak_progress_chain() and a RAII helper
FlatpakAsyncProgressChained which creates a new OstreeAsyncProgress
under the temporary GMainContext, but forwards all its state and updates
to the previous OstreeAsyncProgress's callbacks.
This is documented in a comment in the code as well.
All known instances of this problem in the existing code are fixed in
this commit.
This uses new API in libostree which is proposed in
ostreedev/ostree#1968. In anticipation of it being included in libostree
version 2019.6, the bug fix is predicated on that version being present.
If compiling against an older version, the old buggy behaviour will be
the fallback.
This problem was solved conceptually by Philip Withnall, I only wrote
the code.
This will be used for parental controls, and potentially other reasons
for denying installation of an app.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
This means you can use "org.foo.bar//stable" instead of "org.foo.bar/*/stable"
which is similar to what other APIs do.
We want to use this for masking extensions too, thus the export.
Support a list of versions that are supported. This will be useful
for e.g. the extra_data for extensions once that is backported to
1.2, because that will require it to say that it is supported for
> 1.2.5 in the 1.2 series and > 1.4.2 otherwise.
Closes: #3112
Approved by: alexlarsson
flatpak_dir_collect_deployed_refs() has the order of its "branch" and "arch"
arguments wrong, as does its only caller flatpak_list_deployed_refs().
When flatpak_list_deployed_refs() is called by add_extension() the arch
is put in the branch argument and vice versa. But then in the
implementation the arch is used as if it's the branch and vice versa, so
there's no functional bug here. Fix the order for readability.
Similarly, flatpak_list_unmaintained_refs() has the order wrong, but the
confusion is only within that function, since the order is correct in
its caller add_extension() and in the function it uses,
flatpak_dir_collect_unmaintained_refs(). So there's no functional bug
there either, but fix the order.
Closes: #3067
Approved by: alexlarsson
When we export a manifest to the index, always pass the ref we're
targeting instead of relying on the org.opencontainers.image.ref.name
annotation, because that may not be set if we're using labels instead.
This is no big deal, because we know what ref we're handling anyway.
Closes: #2978
Approved by: alexlarsson
We now pull the image config as well as the manifest and fall
back on the labels field if the keys we're looking for are not
in the annotations field.
This lets us support docker manifests too, which don't have
annotations (but do have labels).
Closes: #2978
Approved by: alexlarsson
The org.freedesktop.Flatpak user D-Bus service isn't just used by
flatpak(1) or applications running as Flatpaks. It's also used by
toolbox(1) for similar reasons:
* To keep various configuration files inside the container
synchronized with the host
* To let the container request certain commands to be run on the host
The org.freedesktop.Flatpak D-Bus service itself doesn't need much in
the way of dependencies, but inherits a lot of unused shared library
linkages through the libflatpak-common.la convenience library. Removing
these unused shared libraries reduces the footprint of toolbox(1) for
those who care about such things. eg., Fedora CoreOS.
This commit brings down the number of shared libraries to 19 from 62.
Closes: #3052
Approved by: alexlarsson
The docs for flatpak_is_valid_branch() say branch names can't start with
a digit but the implementation doesn't enforce this, and we have lots of
branches out in the wild that start with a digit (e.g. "3.32" and
"5.12"). So make the docs imply that branches can start with a digit.
The implementation also disallows "." as the leading character for a
branch, so add that to the docs. I'm just guessing that "." was intended
to be disallowed but it makes sense; otherwise the file we create named
with the branch would be hidden.
Closes: #3023
Approved by: alexlarsson
This adds a new error to FlatpakError and uses it everywhere a ref is not
found, either locally or in a remote. This should hopefully be more useful than
the status quo of either returning FLATPAK_ERROR_INVALID_DATA or
G_IO_ERROR_NOT_FOUND or something else. Technically this is an API break but it
seems worth the risk. I checked gnome-software which does not seem affected by
this, and I checked eos-updater which does check for G_IO_ERROR_NOT_FOUND in
one place that will be affected by this but we can patch that.
Closes: #2895
Approved by: matthiasclasen
As described in the last commit message, it makes sense to move toward
using flatpak_repo_resolve_rev() rather than ostree_repo_resolve_rev()
for a few reasons:
1) It means we can use OSTREE_REPO_PULL_FLAGS_MIRROR which causes refs
to be pulled into repo/refs/mirrors/ rather than repo/refs/remotes/
which fixes a few edge cases of using collection IDs[1]
2) It falls back to using ostree_repo_resolve_rev() if
ostree_repo_resolve_collection_ref() fails so we can maintain backwards
compatibility for repo/refs/remotes/
3) It distinguishes between remote and local refs, and in the local case
uses OSTREE_REPO_RESOLVE_REV_EXT_LOCAL_ONLY and
ostree_repo_resolve_rev_ext() to make sure we don't for example
accidentally use a remote's repo metadata rather than the local repo's
metadata for the "flatpak repo" command.
So this commit changes every instance of ostree_repo_resolve_rev() in
the codebase to flatpak_repo_resolve_rev().
[1] https://github.com/flatpak/flatpak/issues/1832Closes: #2705
Approved by: alexlarsson
In flatpak_dir_do_resolve_p2p_refs() after pulling a ref we use
ostree_repo_resolve_rev() and pass a refspec with the remote from which
the ref originated. This has a couple side effects, one good and one
bad:
1) The good side effect is that the attack I speculated about in this
comment[1] is not exploitable. Because if the ref in question is pulled
from any remote other than its origin (or a LAN/USB source using another
remote's keyring) it will not be found by ostree_repo_resolve_rev() and
the malicious commit will not be used.
2) The bad side effect is that there are some legitimate reasons a ref
could be pulled from another remote (say, a configured mirror), and in
those cases the pulled ref will not be found. So if I have remote A and
remote B both configured with the same collection ID, a ref installed
from one could be pulled from the other. See this issue[2] for a
concrete example.
The solution is to use OSTREE_REPO_PULL_FLAGS_MIRROR for the pull and
use ostree_repo_resolve_collection_ref() to resolve the ref. This is
done in the caller as well for consistency
(flatpak_dir_resolve_p2p_refs()). This fixes the bad side effect
described above and brings us a step closer to fixing issue #1832. This
also means the attack from #1447 is exploitable, but that is addressed
in a subsequent commit.
This change is conditional on a version check for OSTree 2019.2 because
we need this bug fix[3].
Also, add a helper function flatpak_repo_resolve_rev() which falls back
to using ostree_repo_resolve_rev() when
ostree_repo_resolve_collection_ref() fails, so we start to move toward
using /refs/mirrors/ but maintain backwards compatibility for
/refs/remotes/. A subsequent commit will make wider use of
flatpak_repo_resolve_rev() across the codebase; for now just use it for
the case described above.
[1] https://github.com/flatpak/flatpak/issues/1447#issuecomment-445347590
[2] https://github.com/flatpak/flatpak/issues/1832
[3] https://github.com/ostreedev/ostree/pull/1821Closes: #2705
Approved by: alexlarsson
In general xa.filter being "" means don't filter, so that it can be
used to override-unset a previous filter. However, We canonicalize
to unset/NULL when comparing, returing, or setting the ostree config.
Closes: #2884
Approved by: alexlarsson
This is in preparation for writing some unit tests
for these functions. Apparently they are not too
trivial to get them wrong.
Closes: #2880
Approved by: alexlarsson
