It's a good idea to NULL initialize g_autoptr/g_autofree variables, so
we can be sure uninitialized memory isn't passed to g_free or similar.
Closes: #1968
Approved by: alexlarsson
Sometimes (for example in some test-repo-collections.sh test that broke) we
update from a remote with an older ostree-metadata branch, and the
check for downgrades broke in this case.
Its unclear exactly what it the best solution here, maybe to silently
disallow the update. However, this change instead just re-allows the
downgrade for this particular case so we get the old behaviour.
Add:
* Testing of flatpakrefs and bundles pointing to OCI remotes
* Changing a remote from OCI to non-OCI, including during bundle
installation.
* Pruning origin remotes.
When we switch the remote type, we need to clean up cached files
(appstream, OCI index/summary) because they are stored differently
for the two types of remote.
The old pattern of using a separate 'OCI' flag was very ugly
internally in the code once it was extended to flatpak bundles and
flatpakrefs - using a different URI scheme means that the nature
of the remote can't be accidentally lost in some part of the code.
Probing would be possible as well, but would make it difficult to
add a remote when offline, and also doesn't deal well with the
fact that our data layout is different for the two types of remotes -
the type of remote could change at any point!
As a side effect this change enables flatpakrefs and flatpak bundles for OCI
registries.
* Restrict the queried images to the desired architecture
* Sort query parameters as the spec requests
* Allow a fragment on the remote URI to mean "tag to query for
in the registry"
* Tweak flatpak_oci_index_ensure_cached() not to return the
index URL in the normal error case.
The normal behavior where we only list already installed refs for
a noenumerate remote doesn't work for the case where flatpak-system-helper
verifies a ref on an OCI server during installation - in that case, the
ref being installed to does not *yet* exist locally.
In general Flatpak tries to prevent downgrades of anything: apps,
runtimes, repo metadata, etc. with some exceptions such as when the user
specifies a commit they want. However at the moment the detection of a
downgrade is broken if both of the following are true: (1) a collection
ID is enabled on the relevant remote, and (2) a per-user installation
is being used instead of the system-wide one (or the system-helper is
otherwise being circumvented, such as by running flatpak as root).
This bug is a security vulnerability, but it's one with limited impact
because very few people have collection IDs enabled yet, and the
downgrade attack would require either a MITM on the network connection
(which HTTPS should prevent) or a malicious USB drive or local network
peer.
This mirrors ostree commit --disable-fsync and is useful in some cases.
For instance, we'd like to use it when building the temporary import
repositories in flathub.
Closes: #1951
Approved by: alexlarsson
Now that appstream data and icons are retrievded from the index, the OCI
code is expected to be fully usable, and to work with registries as they
will actually be deployed.
Closes: #1910
Approved by: alexlarsson
We previously made a separate request to the registry index to see if
the manifest hash of an image was the hash of the image in the registry.
Since the summary is now downloaded by the system helper and trusted, just
check if the hash matches the hash in the summary data. This is as good,
and in is a lot more efficient if the index is statically generated,
and we can't get the index data for just one image.
Closes: #1910
Approved by: alexlarsson
The OCI index information should be highly compressable (especially if
icons are remote URI's rather than data URI's) so downloading it and
storing it compressed will provide sigificant efficiency gains.
Closes: #1910
Approved by: alexlarsson
Add a new flag for flatpak_cache_http_uri() that adds Accept-Encoding: gzip
to the request, and if the result is returned compressed, stores the data
compressed. If the data result is return uncompressed, it's compressed.
Closes: #1910
Approved by: alexlarsson
Add a new test case to test the OCI remote functionality. The tests talk to
a server that implements good-enough index generation and bits of the
docker registry protocol. Adding and remove remotes, summary and appstream
generation, and image installation are all tested.
Closes: #1910
Approved by: alexlarsson
Checking the registry against a previous etag is now handled inside
flatpak_cache_http_uri(), so remove the etag parameters that were
previously passed around in various places for simplicity.
Closes: #1910
Approved by: alexlarsson
Previously the code assumed that appstream data was stored in a separate
OCI image in the registry. Replace that with storing the appstream data
and icons as image annotations. When we download a new version of the
image index, the appstream data is combined, and icons are downloaded
as necessary.
Since there is no longer a content hash for the appstream data, it's
not practical for the user to download the appstream data and pass it
to the system helper, instead the system helper just downloads the
appstream data directly.
Closes: #1910
Approved by: alexlarsson
Redo the handling of generating summary information from an OCI registry
to be a two step process. First download the index, using the newly
added HTTP caching functionality. Then regenerate the summary from the
index, using mtimes to avoid duplicate work.
Closes: #1910
Approved by: alexlarsson
Add a new function, flatpak_cache_http_uri() that when passed an URL and
a local destination location, either a) downloads the content and stores
it at the destination location, storing HTTP cache header information
like Last-Modified, Etag into user xattrs (if available) or a separate
file or b) if the downloaded content is already present, checks the
header information to decide whether the downloaded content can be used
or needs to be revalidated witha conditional request.
Tests are added that use a special case test server that adds HTTP caching
headers and reacts to them based on query parameters. A small test binary
'httpcache' is added for the tests to use.
Closes: #1910
Approved by: alexlarsson
In preparation for extending the HTTP downloading function to include
caching, split HTTP related utilities into a separate file with a
separate header.
Closes: #1910
Approved by: alexlarsson
The directory where we export an OCI image to is not a "registry", even
if it's a FlatpakOciRegistry internally. So change export registry from
oci/registry to oci/image.
Closes: #1910
Approved by: alexlarsson
The code checked whether an OCI registry URI was an OCI image layout accessed
via HTTP by looking for /oci-layout, but distributing OCI images in this way
is not really a thing anybody does. It would be inefficient way to store
large numbers of images, since all versions need to be listed in index.json.
The code still uses OCI image layouts to represent "local registries" in
analogy to local ostree repositories.
Closes: #1910
Approved by: alexlarsson
In a couple different places, double slashes were inserted into the
generated OCI index URIs - e..g, index//static instead of index/static.
While most HTTP servers/applications will normalize double slashes,
this is not required, and such URIs are, in any case, ugly.
Closes: #1910
Approved by: alexlarsson
When Flatpak is fetching a locale extension it has to decide which
subpaths to fetch based on what language is being used on the computer.
This happens in flatpak_dir_get_locale_subpaths() which indirectly uses
the org.freedesktop.Accounts D-Bus object to check what language is
configured for each user. The problem is that if any user doesn't have a
language set, Flatpak falls back to pulling all languages, rather than
checking the system default using localed. The effect is that on Endless
OS systems, Flatpak is pulling entire locale extensions rather than just
the subset for the configured language, which is a significant waste of
bandwidth. In my testing, the "Language" property on the primary user
account is not set on Endless, but it is set on Fedora.
A side effect of this bug is to cause offline USB app installs to
sometimes fail, because if the USB only has a partial locale and you try
to pull the whole thing, the pull fails.
This commit fixes the issue by doing another D-Bus call to localed to get
the system default(s), then checking AccountsService as before, treating
an unset language for a user account as meaning "use the system
default". Then only if no languages are set for the users or the system,
fall back to pulling all languages. The code to communicate with localed
is based on the code in gnome-control-center in
panels/region/cc-region-panel.c
This extra synchronous D-Bus call adds some overhead which might be able
to be avoided; see https://github.com/flatpak/flatpak/issues/1938
Using this patch I can see that Flatpak is pulling partial locales now,
based on the output of `flatpak list -a | grep partial` after installing
Bijiben from Flathub.
Closes: #1937
Approved by: alexlarsson
Copying refs from the system repo into a repo on a USB drive requires
the summary in the system repo to be up to date (and similarly for other
flatpak installations like a per-user one). At the moment we expect the
user to run `sudo ostree summary -u` before `flatpak create-usb` which
is a bad user experience. Another option is to set
`core.auto-update-summary` to true on the ostree repo config, but there
are significant performance concerns with that: it involves updating the
summary after every transaction rather than only when we need it. So
this commit changes the create-usb command to use the "UpdateSummary"
system-helper method to update the summary in the source repo before
copying to the destination. This strategy allows us to continue to let
non-root users use `flatpak create-usb`. This commit also tries to
update the remote repo metadata and appstream data for each remote
before copying to the USB, because we can now do that without
invalidating the summary.
Closes: #1945
Approved by: alexlarsson
