This removes flatpak_dir_list_remote_refs() and
flatpak_dir_list_all_remote_refs() which have no more users, as everything
migrated to the _decomposed() versions.
This means we do a bit less string operations, but it also allows us
to pick up the collection-id when listing refs from a file: uri, fixing
the issues described in https://github.com/flatpak/flatpak/pull/3909
This slightly changes the behaviour of collection ids and remotes.
When listing regular remotes we only ever return the refs with the
collection id specified by the configuration of that remote. However,
when specifying a file: uri for the remote name we return all the refs
in the repo, with their corresponding collection id.
This means the test suite has to change a bit as before we returned
all the collrefs from the remote, not just the one matching the configured
collection id.
If you're listing a file: uri we (as before) add the collection-ref
tables, but we now also set the proper collection-id for the main ref
table if the ostree.summary.collection-id key is set.
flatpak-utils.c is getting large, and also I got some issues with
include order, so split out things that are purely handling ref stings
in various ways to its own file.
If you are listing a remote named (e.g.) file:///some/path/to/repo then we
have no idea what the actual collection id is for that remote, so we can't
pick the right part. In practice, this is typically done when listing
a side-load repo, such as described in
https://blogs.gnome.org/mclasen/2018/08/26/about-flatpak-installations/
or by the endless software installer that lists available offline apps.
This kinda abuses the FlatpakDecomposed object to store the collection
id, even if that doesn't normally carry this. This is not optimal, but it
is kind of a weird case and there isn't really any other place for it that
makes sense either, and having it there is efficient and keeps the APIs.
getters now have three forms:
const char *foo_peek() returns pointer into data and length
const char* foo_get() returns zero terminated pointer into data
char *foo_dup() returns allocated zero terminated pointer
Not everything has all of these (as that is not always possible), but
it is clear what each one does.
We also now support storing a collection id for a
FlatpakDecomposed. This is not typically done (as we use the source
remote to define the collecion id and don't expose this concept to the
user). However, for the case when directly listing a local repo (such as
a side-load repo) we need this.
We now validate the arch string. The valid chars are a-z, A-Z, 0-9 and
_. This was't previously verified, but no arches in existance have other
chars.
There is a new flatpak_decomposed_new_from_decomposed() which lets you
modify part of another ref. Say replace the arch or the branch and
keep the rest the same.
We used to go:
* indexed summary memory cache
* indexed summary fetch
* compat summary memory cache
* compat summary fetch
But now we go:
* indexed summary memory cache
* compat summary memory cache
* indexed summary fetch
* compat summary fetch
This means in the compat case we don't constantly have to fail the
summary fetch before using the memory cache.
This moves a bunch of code from FlatpakDir to FlatpakTransaction which was
only used to find dependencies. This allows us to reuse the existing
cache of FlatpakRemoteState objects in the transaction.
The st_mtime field is all we can guarantee to keep, so whenever we
write a new summary file for a repo we ensure it is newer than the old
by checking if st_mtime is equal (or less) and if so, use the old
st_mtime plus one.
This means sometimes the generated summary files will be in the future but
that only happens in the corner case of multiple changes inside one
second, and in that particular case we need this change.
This fixes some issues I had with the mtime chacking in the tests due to
the python httpd handling If-Modified-Since based on st_mtime.
This fixed some issues in the test where changing the url cause the
cache:ing to fail to update due to the mtimes being to new in combination
with the HTTP If-Modified-Since use.
For example, if the url changes we need to re-read the summary. This
was causing issues in a repo test where we re-wrote the url, and then
the new summary was not downloaded because of a st_mtime based
If-Modified-After check in the python httpd.
We store the cached subsummaries as `$remote-$arch-digest.sub`, this
way we can prune everything but the most recent (i.e. highest mtime)
rather than looking at the index. This way we don't prune old
subsummaries for other arches when we update one arch, which would not
let us use deltas for those.
Some details:
* If mtimes are identical (mtime precision issue) for several
subsummaries we keep all.
* If we just saved a subsummary in the cache we don't prune that
in case there is something wonky with mtimes.
The subsets for a remote are pruned every time we write a new cached
subsummary.
These now take (optional) string length args:
flatpak_id_has_subref_suffix()
flatpak_levenshtein_distance()
flatpak_is_valid_name()
flatpak_is_valid_branch()
We need to load the ensure the right arch whenever we need it.
Also this restructures the RemoteState handling a bit in general so that
we avoid loading the same remote state multiple times when converting
partial refs to full refs.
In addition to the old summary file we create a summary.idx and
a set of per-arch subsummaries (and extra copies of these if any commit
specify a subset). These are much smaller, and eventually we will also
get deltas for them.
We are not yet using these new formats, although the code is there
to use them once we start downloading them.
This drops the deltas from the summary and uses the per-commit
metadata field to add the cache data to avoid the need for the separate
xa.cache and xa.sparse-cache indexes. This way we avoid repeating the
refs in multiple places.
Nothing uses this format yet, but we still pass make check if we enable
it.
This moves the generation of the summary files completely into flatpak
allowing us to (later) customise what goes into it in more detail and
generate other forms of summaries.
If accountsservice isn't available on the system bus, then we can't
ask it for the user's parental controls settings, and we also can't
ask it whether it even has the malcontent extension. Since this is
not a real security boundary, fail open.
This can be dropped if we depend on a version of libmalcontent that maps
these errors to MCT_APP_FILTER_ERROR_DISABLED.
Resolves: https://github.com/flatpak/flatpak/issues/3902
Bug-Debian: https://bugs.debian.org/972138
Signed-off-by: Simon McVittie <smcv@collabora.com>
Sometimes a server might return a HTTP error 500 (this seems to happen
sometimes with Microsoft’s VSCode server, for example). Map this to
`G_IO_ERROR_HOST_UNREACHABLE` for now, which is a bit more specific than
returning `G_IO_ERROR_FAILED`, but without the hassle of introducing a
new public error domain which could give more detail.
In particular, this should allow gnome-software to show an error message
to the user for such failed downloads, rather than hiding the error and
logging the following:
```
not handling error failed for action download: While downloading http://packages.microsoft.com/repos/vscode/pool/main/c/code/code_1.45.1-1589445302_amd64.deb: Server returned status 500: Internal Server Error
```
Signed-off-by: Philip Withnall <pwithnall@endlessos.org>
