This is an optimized version of ostree_repo_prune() specialized for
archive mode repos. It is faster and uses less memory so that we can
prune larger repos (like flathub) in a realistic timeframe.
The primary reason it is faster is that it creates and uses a
`.commitmeta2` file for each commit, containing information about what
objects are reachable from that commit. This means incremental prunes
need only traverse over newly created commits.
Secondly, it uses the variant parser compiled accessors for the
various GVariants that are involved in the prune which is quite a bit
faster, especially if the repo is very large.
It also merges the scan-for-all-objects and prune-unreachable objects
phases, which means that we don't have to allocate a hashtable for
all the objects in the entire repo saving a lot of memory.
To save memory the hashtable of reachable objects, which can be quite
big on a big repo, points to a custom, very compact format for object
names.
Additionally it does the scanning for reachable objects twice, first
with a shared lock and then again (if anything changed) it with an
exclusive lock. This allows us to avoid using an exclusive lock during
the slowest part of the prune.
Unfortunately there are currently no public APIs for the ostree repo
locks. We really need to take an exclusive lock during the whole prune
or we parallel modifications (say a commit) might get their newly
written objects deleted. To work around this we have a minimal custom
implementation of an exclusive lock. Once the public API is available
we can start using that.
I created a repo with a lot of small commits to test this. It has 9M,
and pruning with depth=10 deletes 2M of them.
The original performance looks like:
Finding reachable objects: 287 seconds
Pruning unreachable: 69 seconds
Just using the pregenerated reachable data:
Finding reachable objects: 15 seconds
Pruning unreachable: 69 seconds
The final optimized prune (using pregenerated data):
Finding reachable objects: 12 seconds
Pruning unreachable: 51 seconds
The above are with the page caches cleaned, on a second run the performance
increase is even more noticeable.
As a comparison to the above, finding the reachable objects in the
actual flathub repo took 22 hours, but with the pregenerated reachable data
only 39 minutes.
scan-build complained that rest_argv_start could be used uninitialized,
because it can't see that rest_argc >= 2 implies that rest_argv_start
got initialized at the same time rest_argc was set. Make this easier
to understand.
Signed-off-by: Simon McVittie <smcv@collabora.com>
This hasn't done anything useful since 0978826c: it just takes a
new ref to the installation, and then releases that ref without doing
anything with it. Detected by scan-build.
Signed-off-by: Simon McVittie <smcv@collabora.com>
scan-build has a lot of false positives for this codebase because it
doesn't understand __attribute__((__cleanup__)) or GLib's GError
convention, but it seems to have been right about these.
Signed-off-by: Simon McVittie <smcv@collabora.com>
Like $XDG_RUNTIME_DIR/app/$FLATPAK_ID, this is shared between all
instances of the app, except for subsandboxed instances created by
flatpak-spawn --sandbox or equivalent. Unlike
$XDG_RUNTIME_DIR/app/$FLATPAK_ID, it does not exist at an equivalent
path on the host and in the sandboxed app.
Resolves: https://github.com/flatpak/flatpak/issues/4120
Signed-off-by: Simon McVittie <smcv@collabora.com>
If XDG_RUNTIME_DIR is under app control, as it will be with #4120, we
don't want to be mounting pieces of filesystem directly into it, because
that will mean that the app could create a symlink that will cause us
to create a mount point for it at the target of the symlink, potentially
elsewhere in the host filesystem.
Instead, we mount them in /run/flatpak, which is a per-instance
directory entirely controlled by Flatpak; and then create (relative)
symlinks in XDG_RUNTIME_DIR, pointing into /run/flatpak.
In this commit, we still know that the XDG_RUNTIME_DIR is a
per-instance tmpfs, so we can safely create the symlinks using
the --symlink option. In a subsequent commit this will change to
creating them in a shared XDG_RUNTIME_DIR, if any.
Signed-off-by: Simon McVittie <smcv@collabora.com>
This allows apps that use /tmp as an IPC rendezvous point, such as those
that embed Chromium-derived browsers, to communicate between instances;
this would not previously have worked without --filesystem=/tmp, which
is a significant weakening of the sandbox.
It also allows /tmp to be shared with subsandboxes (if they are not
sandboxed more strictly).
The temporary directory is actually created in XDG_RUNTIME_DIR,
to avoid it becoming visible to unrelated apps that happen to have
--filesystem=/tmp.
Signed-off-by: Simon McVittie <smcv@collabora.com>
The pressure-vessel container tool in Steam will want to use this, to
replace /usr with a Steam Runtime container supplied by the Steam CDN,
instead of using the same Flatpak runtime that is used to run the Steam
client and non-containerized games.
If a custom /usr is used, the "official" Flatpak runtime is still the
one reflected in the metadata. It is also mounted at /run/parent,
with all its extensions, so that pressure-vessel has the option of using
its graphics drivers (by populating the custom /usr with symlinks into
/run/parent and/or /run/host).
When doing this, we need to put an empty directory on /app, because
the real /app expects to be run on top of the real runtime. It would
also be reasonable to substitute a custom replacement for /app, so
I've included support for that too.
Partially addresses #3797.
Signed-off-by: Simon McVittie <smcv@collabora.com>
When we add a way to specify a different /usr for a subsandbox, we'll
want to mount the "official" runtime elsewhere and avoid adding it
to the LD_LIBRARY_PATH.
Signed-off-by: Simon McVittie <smcv@collabora.com>
This checks that the value of the REF field is not an empty string
before calling `flatpak_decomposed_new_from_ref`. Attempting to
decompose an empty string leads to a validation error and prevents
any history from being printed.
Fixes https://github.com/flatpak/flatpak/issues/4121
Currently the create-usb command copies extra-data flatpaks such as the
org.freedesktop.Platform.openh264 runtime extension along with normal
non-extra-data content. This is an issue because on the computer
installing from the USB it will attempt to get the extra-data content
from the URI specified in the commit metadata, which will fail if the
URI is a resource on the Internet and the computer is offline.
Instead, have create-usb omit such refs. If it was specified explicitly
this is treated as a hard error and if it's a dependency it's treated as
a warning, so that e.g. org.freedesktop.Platform can still be copied to
a USB and installed/updated offline. On the receiving end the absence of
org.freedesktop.Platform.openh264 will correctly not be treated as fatal
to the installation of org.freedesktop.Platform.
This issue was affecting offline updates of Endless OS, since such OS
updates include flatpaks along with the OS content.
For the (legally questionable) topic of properly supporting extra-data
distribution offline, see https://github.com/flatpak/flatpak/issues/969
The "new way" is supported since at least 2010 (!).
app/parse-datetime.y:302.1-12: warning: deprecated directive: ‘%pure-parser’, use ‘%define api.pure’ [-Wdeprecated]
302 | %pure-parser
| ^~~~~~~~~~~~
| %define api.pure
As it uses directives that POSIX yacc doesn't support:
app/parse-datetime.y:302.1-12: warning: POSIX Yacc does not support %pure-parser [-Wyacc]
302 | %pure-parser
| ^~~~~~~~~~~~
app/parse-datetime.y:307.1-7: warning: POSIX Yacc does not support %expect [-Wyacc]
307 | %expect 31
| ^~~~~~~
Use present tense consistently rather than gerunds for some headings in
"flatpak --help". Also tweak flatpak(1) to use similar wording but with
gerunds.
This means we can systematically pass the environment variables
through bwrap(1), even if it is setuid and thus is filtering out
security-sensitive environment variables. bwrap itself ends up being
run with an empty environment instead.
This fixes a regression when CVE-2021-21261 was fixed: before the
CVE fixes, LD_LIBRARY_PATH would have been passed through like this
and appeared in the `flatpak build` shell, but during the CVE fixes,
the special case that protected LD_LIBRARY_PATH was removed in favour
of the more general flatpak_bwrap_envp_to_args(). That reasoning only
works if we use flatpak_bwrap_envp_to_args(), consistently, everywhere
that we run the potentially-setuid bwrap.
Fixes: 6d1773d2 "run: Convert all environment variables into bwrap arguments"
Resolves: https://github.com/flatpak/flatpak/issues/4080
Bug-Debian: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=980323
Signed-off-by: Simon McVittie <smcv@collabora.com>
As with flatpak run --parent-expose-pids, this will only work if we have
a working, non-setuid bwrap. Systems where user namespace creation is
restricted and bwrap needs to be setuid (Debian 10, RHEL/CentOS 7,
Arch Linux linux-hardened kernel) will have degraded functionality.
This option is similar to --expose-pids, except that instead of making
the subsandbox use a nested pid namespace inside the parent's, it makes
the subsandbox share the parent's pid namespace as-is, so that process
IDs in the parent and the subsandbox are interchangeable. This will
be useful if the parent and the subsandbox communicate via protocols
that assume a global view of the process ID namespace, for example
passing process IDs across an AF_UNIX socket or in shared memory.
In particular, this will be useful for Steam's pressure-vessel container
tool: the IPC between the Steam client and the "game overlay" loaded into
Steam games uses process IDs, and becomes confused if they don't match up.
This weakens the security boundary between a subsandbox and the parent,
but that's OK in some cases, especially if the subsandbox is being used
as a way to get a different runtime /usr (flatpak-spawn --latest-version
or #4018) rather than as a security boundary.
Signed-off-by: Simon McVittie <smcv@collabora.com>
This is the environment needed to use flatpaks; a following commit will
hook this up to the systemd environment generator.
Signed-off-by: Philip Withnall <pwithnall@endlessos.org>
Gracefully handle flatpak_instance_get_app() returning NULL, which
happens for example when there is a shell in a runtime, e.g. flatpak run
--command=sh org.gnome.Platform
When installing to an installation we need to find the runtime to use
for the apply-extra-data script from the installation we're targeting,
because that is the one that FlatpakTransaction guaranteed has the
required dependencies (although its possible they came from the
default dependency source too, i.e. the system repos).
In particular, we run into this issue if nothing is installed
anywhere, and then we install an extra-data app into a custom
directory. The transaction will download the runtime, and it
will not be anywhere else. Without this change flatpak only
looked for the dependency in the systam an regular user dirs
where it isn't.
In case the second of these two fails, the first will still have been
added to the transaction. And since it's better to install the renamed
app but not uninstall the old one, than to uninstall the old one but not
install the new one, swap the order.
See also https://github.com/flatpak/flatpak/issues/3991
We remember what action we took for EOLs, and for sub-refs (ie .Locale)
we reuse that.
Also, we show if eol:ed refs are pinned (as that makes them not be
auto-uninstalled), and we list the apps that use the eol:ed runtime
ref.
Example run:
```
Looking for updates…
Info: (pinned) org.gnome.Sdk.Compat.i386 is end-of-life, with reason:
The GNOME 3.34 runtime is no longer supported as of 14th August 2020. Please ask your application developer to migrate to a supported platform.
Info: org.gnome.Platform is end-of-life, with reason:
The GNOME 3.32 runtime is no longer supported as of 11th March 2020. Please ask your application developer to migrate to a supported platform.
Applications using this runtime:
org.gnome.HexGL
```
The subset column is typically short, while for eol options the
options column can be very long. So, it makes more sense to have the
options at the end.
We used to match the ref id with .Locale and print "partial" in the
transaction status, but that is not right. Sometimes we do other refs
partially and sometimes locales are not partial. We now use the status
of the subpaths to do the right thing here.
