This removes flatpak_dir_list_remote_refs() and
flatpak_dir_list_all_remote_refs() which have no more users, as everything
migrated to the _decomposed() versions.
This means we do a bit less string operations, but it also allows us
to pick up the collection-id when listing refs from a file: uri, fixing
the issues described in https://github.com/flatpak/flatpak/pull/3909
This slightly changes the behaviour of collection ids and remotes.
When listing regular remotes we only ever return the refs with the
collection id specified by the configuration of that remote. However,
when specifying a file: uri for the remote name we return all the refs
in the repo, with their corresponding collection id.
This means the test suite has to change a bit as before we returned
all the collrefs from the remote, not just the one matching the configured
collection id.
If you're listing a file: uri we (as before) add the collection-ref
tables, but we now also set the proper collection-id for the main ref
table if the ostree.summary.collection-id key is set.
If you are listing a remote named (e.g.) file:///some/path/to/repo then we
have no idea what the actual collection id is for that remote, so we can't
pick the right part. In practice, this is typically done when listing
a side-load repo, such as described in
https://blogs.gnome.org/mclasen/2018/08/26/about-flatpak-installations/
or by the endless software installer that lists available offline apps.
This kinda abuses the FlatpakDecomposed object to store the collection
id, even if that doesn't normally carry this. This is not optimal, but it
is kind of a weird case and there isn't really any other place for it that
makes sense either, and having it there is efficient and keeps the APIs.
getters now have three forms:
const char *foo_peek() returns pointer into data and length
const char* foo_get() returns zero terminated pointer into data
char *foo_dup() returns allocated zero terminated pointer
Not everything has all of these (as that is not always possible), but
it is clear what each one does.
We also now support storing a collection id for a
FlatpakDecomposed. This is not typically done (as we use the source
remote to define the collecion id and don't expose this concept to the
user). However, for the case when directly listing a local repo (such as
a side-load repo) we need this.
We now validate the arch string. The valid chars are a-z, A-Z, 0-9 and
_. This was't previously verified, but no arches in existance have other
chars.
There is a new flatpak_decomposed_new_from_decomposed() which lets you
modify part of another ref. Say replace the arch or the branch and
keep the rest the same.
We used to go:
* indexed summary memory cache
* indexed summary fetch
* compat summary memory cache
* compat summary fetch
But now we go:
* indexed summary memory cache
* compat summary memory cache
* indexed summary fetch
* compat summary fetch
This means in the compat case we don't constantly have to fail the
summary fetch before using the memory cache.
This moves a bunch of code from FlatpakDir to FlatpakTransaction which was
only used to find dependencies. This allows us to reuse the existing
cache of FlatpakRemoteState objects in the transaction.
This fixed some issues in the test where changing the url cause the
cache:ing to fail to update due to the mtimes being to new in combination
with the HTTP If-Modified-Since use.
For example, if the url changes we need to re-read the summary. This
was causing issues in a repo test where we re-wrote the url, and then
the new summary was not downloaded because of a st_mtime based
If-Modified-After check in the python httpd.
We store the cached subsummaries as `$remote-$arch-digest.sub`, this
way we can prune everything but the most recent (i.e. highest mtime)
rather than looking at the index. This way we don't prune old
subsummaries for other arches when we update one arch, which would not
let us use deltas for those.
Some details:
* If mtimes are identical (mtime precision issue) for several
subsummaries we keep all.
* If we just saved a subsummary in the cache we don't prune that
in case there is something wonky with mtimes.
The subsets for a remote are pruned every time we write a new cached
subsummary.
These now take (optional) string length args:
flatpak_id_has_subref_suffix()
flatpak_levenshtein_distance()
flatpak_is_valid_name()
flatpak_is_valid_branch()
We need to load the ensure the right arch whenever we need it.
Also this restructures the RemoteState handling a bit in general so that
we avoid loading the same remote state multiple times when converting
partial refs to full refs.
This drops the deltas from the summary and uses the per-commit
metadata field to add the cache data to avoid the need for the separate
xa.cache and xa.sparse-cache indexes. This way we avoid repeating the
refs in multiple places.
Nothing uses this format yet, but we still pass make check if we enable
it.
If accountsservice isn't available on the system bus, then we can't
ask it for the user's parental controls settings, and we also can't
ask it whether it even has the malcontent extension. Since this is
not a real security boundary, fail open.
This can be dropped if we depend on a version of libmalcontent that maps
these errors to MCT_APP_FILTER_ERROR_DISABLED.
Resolves: https://github.com/flatpak/flatpak/issues/3902
Bug-Debian: https://bugs.debian.org/972138
Signed-off-by: Simon McVittie <smcv@collabora.com>
This way we can get the proper eol status for the new to-be-installed
refs, rather than whatever was previously installed. This allows us to
detect when a runtime is updated and the new one is eol, and nothing
uses it, so it can be auto-uninstalled.
Rather than trying to figure out which runtimes are affected byt
the current setup of ops we run flatpak_dir_list_unused_refs() twice,
once with and once without the changes the transaction will cause.
Any unused refs after the transaction that were not unused before are
caused by the transaction and we start uninstall ops for those.
Also rename flatpak_dir_list_unused_refs_with_options() to
flatpak_dir_list_unused_refs() as it need not be so long.
If we're auto-installing dependencies we want to limit them to those
from the same remote. However you can still (manually) install such
dependencies, so when we're looking for things to possibly uninstall
we need to check dependencies from all remotes.
In a few places we are using
flatpak_installation_list_unused_refs() and then only
using the ref strings not the FlatpakInstalledRef objects, so the
resources used to construct those objects are wasted. Add a flatpak_dir_
function to be used internally instead. One day we will figure out how
to make flatpak-dir.c less of a wilderness.
This also adds the flatpak_installation_list_unused_refs_with_options()
verion that has extended features.
This is no longer needed since a FlatpakRemoteState is passed in, which
ensures the repo exists. The similar call was already removed from
flatpak_dir_install().
Avoid shadowing variables that are already declared in a previous scope,
and make such occurrences compile-time errors. These are not functional
changes.
In a few places do related code cleanup.
A similar ostree PR is here:
https://github.com/ostreedev/ostree/pull/2195
Without this patch, the remote-info command will sometimes emit a
critical error "g_utf8_strlen: assertion 'p != NULL || max == 0' failed"
and print (null) for the "Commit:" field, since the commit doesn't get
initialized properly.
If a runtime is installed explicitly rather than as a dependency, pin it
so it doesn't get automatically removed when unused runtimes are being
removed. We do this because the runtime might be installed for
development or other uses.
This commit also rearranges some code in the mask and pin commands, and
adds a unit test.
As discussed here [1], we want a way to mark runtimes to be kept even
when they are unused by any apps and we are removing such runtimes.
Currently this is a command that can be run manually; a subsequent
commit will pin runtimes automatically if they are installed
independently of any app.
A unit test is included.
[1] https://github.com/flatpak/flatpak/issues/2639#issuecomment-662311756
This allows OSTree to avoid re-checking the `summary.sig` for freshness
once per pull in a transaction, since flatpak has already done that.
This avoids latency in a transaction (especially on high latency or
lossy network connections) and avoids the potential race condition of
the `summary` file changing on the server part-way through a
transaction.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
`FlatpakRemoteState` already caches the parsed `GVariant` form of the
summary, but it will be useful (in the following commits) to also have a
pointer to the `GBytes` which backs that variant. The `GBytes` will be
passed into OSTree in the following commits, to allow it to avoid
unnecessarily re-downloading the `summary` file.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
`flatpak-dir.c` supports two caches for `summary{,.sig}` files: one in
memory (per `FlatpakDir` instance) and one on disk (shared between all
users of the repository). Clarify in this comment which one is being
referred to.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Adding brackets makes it a little more obvious that it’s 5 minutes, and
less likely that operator precedence where the macro is used will result
in the wrong value being calculated.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
Cache hits didn’t result in the cache entry’s timeout being extended,
and timeouts didn’t result in the entry being removed from the cache to
free up resources (even though it would never be returned as a cache hit
again).
Signed-off-by: Philip Withnall <withnall@endlessm.com>
This commit is a follow-up for the commit "dir: Validate locally
configured collection IDs". Whereas in that commit we validate the
collection ID in flatpak_remote_state_fetch_commit_object(), here we do
it in _flatpak_dir_get_remote_state(), since fetch_commit_object() is
not called on the code path normally taken for a transaction (only if
try_resolve_op_from_metadata() fails and in other niche scenarios).
This should ensure that if someone erroneously sets a collection ID on a
remote, the error will be caught quickly. It also helps the eos-updater
unit tests pass.
Some things could need to run e.g. i386 code in apply_extra (for
example #3742). In this very limited context (almost everything is
read-only) this seems pretty secure.
We could require the app to specify a multiarch pemission to allow
this, but such permissions only really make sense for an app, and
extra data is often used for other things like runtimes and
extensions, that seems a bit weird. Lets just enable it always.
These are system dbus calls that could potentially be somewhat slow
and its unlikely that the local config changes during runtime of the
session (because the desktop really needs a restart to pick up a new
locale). Also, if there are any issues with these (such as #3755) each
call will be the 25sec dbus timeout, which is not great.
Fixes https://github.com/flatpak/flatpak/issues/3792
Currently when searching for a remote to provide the runtime for an app,
we search remotes in priority order. This commit makes it so we search
the remote providing the app before others with the same priority, and
otherwise still search in priority order. This means for the common
case where every remote has the default priority of 1, the app's origin
will have the first chance to provide the runtime. This behavior seems
logical, but the impetus for this change was also to keep a unit test
passing in eos-updater[1] after a port to FlatpakTransaction.
Originally the eos-updater unit test was written to prioritize the
origin remote regardless of the priorities on any other remote, but
during code review it was decided to let higher priority remotes stay
above the app's origin.
In practice it's usually true that only one remote provides a runtime
and priorities aren't set at all, so this is an edge case that probably
doesn't come up much.
A unit test and documentation updates are included.
[1] eede0a8b9c/tests/test-update-install-flatpaks.c (L1919)
Currently if the user configures a collection ID on a remote which is
incorrect, either because it doesn't match the one configured server
side or because the server doesn't have one configured at all, Flatpak
will not notice the issue and happily still pull from the remote. This
is new since 1.7.1; before that such a problem would be caught because
the server wouldn't provide the ostree-metadata ref for the configured
collection ID.
This commit catches such errors by checking commit metadata, as we
already do for the ref binding metadata. Otherwise such a mismatch would
prevent successful offline distribution of the apps/runtimes provided by
the remote.
The impetus for this is to keep an eos-updater unit test passing with
Flatpak 1.8.x:
"/updater/install-flatpaks-pull-to-repo-error-if-collection-id-invalid"
Currently Flatpak only supports extensions which come from the same
remote as the thing being extended; for discussion on this see
https://github.com/flatpak/flatpak/issues/861
However in general it isn't clear from the metadata what remote provides
an extension. For example com.endlessm.apps.Platform//5 defines the
extension org.freedesktop.Platform.VAAPI.Intel (inherited from
org.freedesktop.Platform) which can be found on flathub not eos-sdk. So
we don't want to add an extension refspec to the transaction if the
current remote is not the one that actually provides it. In
flatpak_dir_find_remote_related_for_metadata() this invariant is
maintained because we check if the ref exists in the remote before
adding it with add_related(). However in
flatpak_dir_find_local_related_for_metadata() we check for existing
deploy data but omit checking that the deploy origin matches the origin
passed in, and in that case can accidentally add an incorrect refspec to
a transaction. So this commit adds the missing origin check.
One way to reproduce this issue is with this command, having both the
Endless platform and the VAAPI extension already installed:
$ flatpak update --no-pull com.endlessm.apps.Platform//5
Looking for updates…
error: Refspec 'eos-sdk:runtime/org.freedesktop.Platform.VAAPI.Intel/x86_64/1.6' not found
This issue also affects GNOME Software which does a no-deploy
transaction followed by a no-pull transaction (this issue occurs in the
latter) and is treated as fatal causing updates not to be applied.
However it only happens in some niche circumstances, because
flatpak_transaction_add_op() will only use the first remote passed to it
if called more than once for the same ref from different remotes. This
is normally fine; refs generally only come from one remote. But it does
mean this issue only occurs if the extension in question was not already
added to the transaction with a correct origin.
