These days we don't really need xattrs for the tests to run,
so move the test runs to /tmp, which brings down make check time
from 5 to 3.5 minutes on my machine.
Closes: #2170
Approved by: alexlarsson
Several tests have alternative wrappers that just set some feature
flags. For instance to switch between user and system and with or
without deltas. This change makes those wrappers automatically
generated from a simple description of the features.
For example the test-run.sh test goes from:
- tests/test-run.sh \
- tests/test-run-system.sh \
- tests/test-run-deltas.sh \
- tests/test-run-system-deltas.sh \
to
+ tests/test-run.sh{{user+system},{nodeltas+deltas}} \
There is some complexity here because the TESTS variable in automake
can't be runtime generated, so we rewrite it during "make
update-test-matrix" and commit the result to git.
The way it works is that make update-test-matrix takes the
TEST_MATRIX_SOURCE list, passes it to tests/expand-test-matrix.sh
which generates all the combinations in tests/Makefile-test-matrix.am.inc
where a test like tests/test-foo.sh{user+system} will get
converted to tests/test-foo.sh@user.wrap and tests/test-foo.sh@system.wrap.
We then have a pattern rule matchin tests/*.wrap which will generate
a wrapper that sets the right environment flags based on $0 and then
spawns the wrapper.
Closes: #2170
Approved by: alexlarsson
If we rely on the background subshell to create the httpd-output file,
and we are unlucky, then the "cp" invocation in the loop might execute
before the file has been created, and fail. This appears to have
happened on Debian's arm64 autobuilder, which failed with:
Running web server: PYTHONUNBUFFERED=1 setsid python3 /<<PKGBUILDDIR>>/tests/http-utils-test-server.py 0
Web server pid: 13319
Waiting for web server (1/300)...
cp: cannot stat '/var/tmp/tap-test.p1cxRN/httpd-output': No such file or directory
Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #2098
Approved by: alexlarsson
Currently Flatpak reads a repo metadata key called "xa.collection-id" to
decide whether to configure a collection ID on a remote. This commit
drops support for xa.collection-id and replaces it with
ostree.deploy-collection-id, which is a key defined by OSTree but so far
only implemented here.
The reason for the change is so that collection IDs can only be deployed
to users running recent versions of Flatpak and OSTree. The current
situation is that Endless wants to deploy collection IDs (and therefore
support for doing USB updates) to users, but servers such as Flathub
can't safely set xa.collection-id in their metadata, because many users
are still using old versions of Flatpak and OSTree[1] which would hit
various bugs[2][3][4] on the P2P code paths that are enabled by
collection IDs. Defining a new key means that only users running
recent (as-yet-unreleased) versions of Flatpak and OSTree will pay
attention to it and deploy the collection ID, leaving the users on old
versions unaffected.
The reason this metadata key is being defined at the level of OSTree
instead of Flatpak, is that OSTree may want to implement it in the
future.
The functionality of deploying a collection ID by setting the
"deploy-collection-id" key in the config on the server side (which in
turn causes ostree.deploy-collection-id to be set) is already covered by
the unit tests in test-update-remote-configuration.sh; this commit just
tweaks them to use the new key.
Another solution I proposed to this problem was to have a key
"eos.collection-id" which would only be understood by the Endless fork
of Flatpak, and use that temporarily until enough people are running
recent versions of Flatpak, at which point "xa.collection-id" can be
used. But this solution (abandoning xa.collection-id upstream) allows us
to avoid that migration step and leave users on old versions of Flatpak
completely unaffected.
[1] https://ahayzen.com/direct/flathub.html#downloadsbyflatpakstacked
[2] https://github.com/ostreedev/ostree/commit/e4e6d85ea
[3] https://github.com/flatpak/flatpak/commit/5813639f
[4] https://github.com/flatpak/flatpak/commit/5b21a5b7
This is only used in the flatpak-builder tests now, not the main
flatpak tests.
Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #1990
Approved by: alexlarsson
For installed-tests, the installed test directory is not on the PATH.
To make this easier, put the uninstalled binary in tests/, so that
in both build-time and installed tests, it is in ${test_builddir}.
Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #1989
Approved by: alexlarsson
Add:
* Testing of flatpakrefs and bundles pointing to OCI remotes
* Changing a remote from OCI to non-OCI, including during bundle
installation.
* Pruning origin remotes.
The old pattern of using a separate 'OCI' flag was very ugly
internally in the code once it was extended to flatpak bundles and
flatpakrefs - using a different URI scheme means that the nature
of the remote can't be accidentally lost in some part of the code.
Probing would be possible as well, but would make it difficult to
add a remote when offline, and also doesn't deal well with the
fact that our data layout is different for the two types of remotes -
the type of remote could change at any point!
As a side effect this change enables flatpakrefs and flatpak bundles for OCI
registries.
Now that appstream data and icons are retrievded from the index, the OCI
code is expected to be fully usable, and to work with registries as they
will actually be deployed.
Closes: #1910
Approved by: alexlarsson
Add a new flag for flatpak_cache_http_uri() that adds Accept-Encoding: gzip
to the request, and if the result is returned compressed, stores the data
compressed. If the data result is return uncompressed, it's compressed.
Closes: #1910
Approved by: alexlarsson
Add a new test case to test the OCI remote functionality. The tests talk to
a server that implements good-enough index generation and bits of the
docker registry protocol. Adding and remove remotes, summary and appstream
generation, and image installation are all tested.
Closes: #1910
Approved by: alexlarsson
Add a new function, flatpak_cache_http_uri() that when passed an URL and
a local destination location, either a) downloads the content and stores
it at the destination location, storing HTTP cache header information
like Last-Modified, Etag into user xattrs (if available) or a separate
file or b) if the downloaded content is already present, checks the
header information to decide whether the downloaded content can be used
or needs to be revalidated witha conditional request.
Tests are added that use a special case test server that adds HTTP caching
headers and reacts to them based on query parameters. A small test binary
'httpcache' is added for the tests to use.
Closes: #1910
Approved by: alexlarsson
The directory where we export an OCI image to is not a "registry", even
if it's a FlatpakOciRegistry internally. So change export registry from
oci/registry to oci/image.
Closes: #1910
Approved by: alexlarsson
This is the equivalent of commit 86fd56dce for the commands added
in commit 2465cf4 to test #1587.
Signed-off-by: Simon McVittie <smcv@debian.org>
Closes: #1864
Approved by: alexlarsson
Include annotations:
org.freedesktop.appstream.appdata
org.freedesktop.appstream.icon-{64,128}
into OCI bundles. This not only makes the bundle self-describing, but also
if the bundle is imported into a registry, it becomes possible to browse
the registry and recreate an appstream by only retrieving the annotations
for relevant images, without having to download the actual images.
The icon annotations are formatted as data: URI's with base64 data. The idea
is that a server-side process to collect annotations could extract the icons
to separate storage and rewrite the URI's to remote URL's.
Closes: #1834
Approved by: alexlarsson
When under load (for example running parallel tests) it can take a
while to get the web server ready.
Signed-off-by: Simon McVittie <smcv@collabora.com>
Closes: #1760
Approved by: alexlarsson
This always printed "setting HOME=(null)" because we didn't set
datadir yet, but it should have been looking at homedir anyway.
Signed-off-by: Simon McVittie <smcv@collabora.com>
Closes: #1760
Approved by: alexlarsson
In the no-pull case and when uninstalling, we never want to do any network
i/o for e.g. detecting depenedencies.
Closes: #1744
Approved by: alexlarsson
In the Fedora 28 base container, `coreutils-single` is used and so
`/usr/bin/ls` is actually a "script":
```
$ file /usr/bin/ls
/usr/bin/ls: a /usr/bin/coreutils --coreutils-prog-shebang=ls script, ASCII text executable
```
We handle this by detecting shebangs in dependencies and recursively adding them.
Closes: #1741
Approved by: alexlarsson
This allows one extension point to match multiple versions of an extension.
In particular, this is useful for OpenGL so that we can match the version matching the current
runtime, as well as the "magic" 1.4 version for nvidia.
Closes: #1722
Approved by: alexlarsson
