Add a new test case to test the OCI remote functionality. The tests talk to
a server that implements good-enough index generation and bits of the
docker registry protocol. Adding and remove remotes, summary and appstream
generation, and image installation are all tested.
Closes: #1910
Approved by: alexlarsson
Checking the registry against a previous etag is now handled inside
flatpak_cache_http_uri(), so remove the etag parameters that were
previously passed around in various places for simplicity.
Closes: #1910
Approved by: alexlarsson
Previously the code assumed that appstream data was stored in a separate
OCI image in the registry. Replace that with storing the appstream data
and icons as image annotations. When we download a new version of the
image index, the appstream data is combined, and icons are downloaded
as necessary.
Since there is no longer a content hash for the appstream data, it's
not practical for the user to download the appstream data and pass it
to the system helper, instead the system helper just downloads the
appstream data directly.
Closes: #1910
Approved by: alexlarsson
Redo the handling of generating summary information from an OCI registry
to be a two step process. First download the index, using the newly
added HTTP caching functionality. Then regenerate the summary from the
index, using mtimes to avoid duplicate work.
Closes: #1910
Approved by: alexlarsson
Add a new function, flatpak_cache_http_uri() that when passed an URL and
a local destination location, either a) downloads the content and stores
it at the destination location, storing HTTP cache header information
like Last-Modified, Etag into user xattrs (if available) or a separate
file or b) if the downloaded content is already present, checks the
header information to decide whether the downloaded content can be used
or needs to be revalidated witha conditional request.
Tests are added that use a special case test server that adds HTTP caching
headers and reacts to them based on query parameters. A small test binary
'httpcache' is added for the tests to use.
Closes: #1910
Approved by: alexlarsson
In preparation for extending the HTTP downloading function to include
caching, split HTTP related utilities into a separate file with a
separate header.
Closes: #1910
Approved by: alexlarsson
The directory where we export an OCI image to is not a "registry", even
if it's a FlatpakOciRegistry internally. So change export registry from
oci/registry to oci/image.
Closes: #1910
Approved by: alexlarsson
The code checked whether an OCI registry URI was an OCI image layout accessed
via HTTP by looking for /oci-layout, but distributing OCI images in this way
is not really a thing anybody does. It would be inefficient way to store
large numbers of images, since all versions need to be listed in index.json.
The code still uses OCI image layouts to represent "local registries" in
analogy to local ostree repositories.
Closes: #1910
Approved by: alexlarsson
In a couple different places, double slashes were inserted into the
generated OCI index URIs - e..g, index//static instead of index/static.
While most HTTP servers/applications will normalize double slashes,
this is not required, and such URIs are, in any case, ugly.
Closes: #1910
Approved by: alexlarsson
When Flatpak is fetching a locale extension it has to decide which
subpaths to fetch based on what language is being used on the computer.
This happens in flatpak_dir_get_locale_subpaths() which indirectly uses
the org.freedesktop.Accounts D-Bus object to check what language is
configured for each user. The problem is that if any user doesn't have a
language set, Flatpak falls back to pulling all languages, rather than
checking the system default using localed. The effect is that on Endless
OS systems, Flatpak is pulling entire locale extensions rather than just
the subset for the configured language, which is a significant waste of
bandwidth. In my testing, the "Language" property on the primary user
account is not set on Endless, but it is set on Fedora.
A side effect of this bug is to cause offline USB app installs to
sometimes fail, because if the USB only has a partial locale and you try
to pull the whole thing, the pull fails.
This commit fixes the issue by doing another D-Bus call to localed to get
the system default(s), then checking AccountsService as before, treating
an unset language for a user account as meaning "use the system
default". Then only if no languages are set for the users or the system,
fall back to pulling all languages. The code to communicate with localed
is based on the code in gnome-control-center in
panels/region/cc-region-panel.c
This extra synchronous D-Bus call adds some overhead which might be able
to be avoided; see https://github.com/flatpak/flatpak/issues/1938
Using this patch I can see that Flatpak is pulling partial locales now,
based on the output of `flatpak list -a | grep partial` after installing
Bijiben from Flathub.
Closes: #1937
Approved by: alexlarsson
Copying refs from the system repo into a repo on a USB drive requires
the summary in the system repo to be up to date (and similarly for other
flatpak installations like a per-user one). At the moment we expect the
user to run `sudo ostree summary -u` before `flatpak create-usb` which
is a bad user experience. Another option is to set
`core.auto-update-summary` to true on the ostree repo config, but there
are significant performance concerns with that: it involves updating the
summary after every transaction rather than only when we need it. So
this commit changes the create-usb command to use the "UpdateSummary"
system-helper method to update the summary in the source repo before
copying to the destination. This strategy allows us to continue to let
non-root users use `flatpak create-usb`. This commit also tries to
update the remote repo metadata and appstream data for each remote
before copying to the USB, because we can now do that without
invalidating the summary.
Closes: #1945
Approved by: alexlarsson
This system-helper method will allow flatpak to update the ostree repo
summary while running as a non-root user that's in the "sudo" group.
This will allow the `flatpak create-usb` command to work without
requiring the user to first run `sudo ostree summary -u`, and without
requiring the system to have `core.auto-update-summary` set to true in
the ostree repo config. This is also much more efficient than
`core.auto-update-summary` because it allows us to only update the
summary when we need it rather than after every transaction.
Generating the summary basically just involves traversing the repo to
enumerate all the refs and putting this information into a file, so I
don't think there are security concerns with allowing it to happen
without authentication.
Closes: #1945
Approved by: alexlarsson
Use min-free-space-size as a replacement for min-free-space-percent.
Previously, flatpak used to disable min-free-space-percent by re-writing
the config with min-free-space-percent=0. As the new version of ostree (2018.7)
now supports min-free-space-size, we should use that and migrate from -percent
option in existing repos to -size=500MB.
Config is rewritten with min-free-space-size in case of:
1) It has min-free-space-percent=0 only. (That is probably from the
previous re-writes).
2) If there are no min-free-space-* options.
Other than that, the config remains unchanged and the co-existence(if any)
of these options is governed by ostree.
https://github.com/flatpak/flatpak/issues/1826Closes: #1912
Approved by: alexlarsson
The flatpak.1 man page serves as an overview page that lists
all the individual command man pages. Make it a complete
overview by listing the file format man pages as well.
Closes: #1944
Approved by: alexlarsson
When the system bus is not available, we set
system_helper_bus to (gpointer)1. And then we
segfault in finalize, trying to unref it.
Thats not nice, so avoid it.
Closes: #1940
Approved by: alexlarsson
When we fail to parse an installation, emit some
warnings that give the user a chance to figure out
what is going wrong. I ran into the missing quotes
issue myself when I first tried to create a custom
installation.
Closes: #1939
Approved by: alexlarsson
We need all_collection_ids to be initialized outside the loop that
iterates over all the refs; otherwise it only has the collection IDs
from the last ref (and related).
This fixes a bug where flatpak only includes the ostree-metadata and
appstream data for the collection IDs associated with the last ref given
on the command line and its runtime and related refs. So flatpak now
includes ostree-metadata and appstream data for collection IDs
associated with all the refs given (and their runtimes and related
refs).
Closes: #1930
Approved by: treitter
Currently the create-usb command looks for a runtime for every ref
passed to it, whether it's an app or runtime. This leads to an error
when a runtime was specified because runtimes don't have runtimes.
Normally you shouldn't need to pass a runtime to create-usb because it
finds dependencies on its own, but you conceivably could (and in fact
the eos-updater-prepare-volume tool in Endless does in order to make
sure an icon theme "runtime" is included with OS updates).
This commit fixes the issue by only checking for runtimes for apps. The
way it's written takes extra care to ensure the FlatpakKinds value used
is based on the ref value used, to protect against the case where one ID
exists in multiple installations with a different kind in each one, even
though it's hard to imagine that happening in actual use.
It's difficult to add a unit test for this at the moment since the unit
test for create-usb is blocked in
https://github.com/flatpak/flatpak/pull/1877Closes: #1924
Approved by: treitter
This pulls in the renameat2() fix that's required for building with
latest glibc (e.g. Fedora rawhide).
Closes: #1890Closes: #1914
Approved by: alexlarsson
This commit ensures that flatpak_remote_state_save_summary()
initializes the passed GError pointer when returning FALSE. I found this
when looking into https://github.com/flatpak/flatpak/issues/1255 because
at the time of that bug report flatpak_dir_update_appstream() had this
g_file_replace_contents() code in it, which would have caused a seg
fault in update_appstream() after an unsuccessful call to
flatpak_dir_update_appstream().
Fixes https://github.com/flatpak/flatpak/issues/1255Closes: #1893
Approved by: alexlarsson
