0cd4656ea1
This is an optimized version of ostree_repo_prune() specialized for archive mode repos. It is faster and uses less memory so that we can prune larger repos (like flathub) in a realistic timeframe. The primary reason it is faster is that it creates and uses a `.commitmeta2` file for each commit, containing information about what objects are reachable from that commit. This means incremental prunes need only traverse over newly created commits. Secondly, it uses the variant parser compiled accessors for the various GVariants that are involved in the prune which is quite a bit faster, especially if the repo is very large. It also merges the scan-for-all-objects and prune-unreachable objects phases, which means that we don't have to allocate a hashtable for all the objects in the entire repo saving a lot of memory. To save memory the hashtable of reachable objects, which can be quite big on a big repo, points to a custom, very compact format for object names. Additionally it does the scanning for reachable objects twice, first with a shared lock and then again (if anything changed) it with an exclusive lock. This allows us to avoid using an exclusive lock during the slowest part of the prune. Unfortunately there are currently no public APIs for the ostree repo locks. We really need to take an exclusive lock during the whole prune or we parallel modifications (say a commit) might get their newly written objects deleted. To work around this we have a minimal custom implementation of an exclusive lock. Once the public API is available we can start using that. I created a repo with a lot of small commits to test this. It has 9M, and pruning with depth=10 deletes 2M of them. The original performance looks like: Finding reachable objects: 287 seconds Pruning unreachable: 69 seconds Just using the pregenerated reachable data: Finding reachable objects: 15 seconds Pruning unreachable: 69 seconds The final optimized prune (using pregenerated data): Finding reachable objects: 12 seconds Pruning unreachable: 51 seconds The above are with the page caches cleaned, on a second run the performance increase is even more noticeable. As a comparison to the above, finding the reachable objects in the actual flathub repo took 22 hours, but with the pregenerated reachable data only 39 minutes.
Flatpak is a system for building, distributing, and running sandboxed desktop applications on Linux.
See https://flatpak.org/ for more information.
Community discussion happens in #flatpak on Freenode, on the mailing list, and on the Flathub Discourse.
Read documentation for Flatpak here.
Contributing
Flatpak welcomes contributions from anyone! Here are some ways you can help:
- Fix one of the issues and submit a PR
- Update flatpak's translations and submit a PR
- Improve flatpak's documentation, hosted at http://docs.flatpak.org and developed over in flatpak-docs
- Find a bug and submit a detailed report including your OS, flatpak version, and the steps to reproduce
- Add your favorite application to Flathub by writing a flatpak-builder manifest and submitting it
- Improve the Flatpak support in your favorite Linux distribution
Hacking
Flatpak uses a traditional autoconf-style build mechanism. To build just do
./autogen.sh
./configure [args]
make
make install
To automatically install dependencies on apt-based distributions you can try
running apt build-dep flatpak and on dnf ones try dnf builddep flatpak.
Dependencies you will need include: autoconf, automake, libtool, bison,
gettext, gtk-doc, gobject-introspection, libcap, libarchive, libxml2, libsoup,
gpgme, polkit, libXau, ostree, json-glib, appstream, libseccomp (or their devel
packages).
Most configure arguments are documented in ./configure --help. However,
there are some options that are a bit more complicated.
Flatpak relies on a project called Bubblewrap for the
low-level sandboxing. By default, an in-tree copy of this is built
(distributed in the tarball or using git submodules in the git
tree). This will build a helper called flatpak-bwrap. If your system
has a recent enough version of Bubblewrap already, you can use
--with-system-bubblewrap to use that instead.
Bubblewrap can run in two modes, either using unprivileged user
namespaces or setuid mode. This requires that the kernel supports this,
which some distributions disable. For instance, Debian and Arch
(linux kernel v4.14.5
or later), support user namespaces with the kernel.unprivileged_userns_clone
sysctl enabled.
If unprivileged user namespaces are not available, then Bubblewrap must be built as setuid root. This is believed to be safe, as it is designed to do this. Any build of Bubblewrap supports both unprivileged and setuid mode, you just need to set the setuid bit for it to change mode.
However, this does complicate the installation a bit. If you pass
--with-priv-mode=setuid to configure (of Flatpak or Bubblewrap) then
make install will try to set the setuid bit. However that means you
have to run make install as root. Alternatively, you can pass
--enable-sudo to configure and it will call sudo when setting the
setuid bit. Alternatively you can enable setuid completely outside of
the installation, which is common for example when packaging Bubblewrap
in a .deb or .rpm.
There are some complications when building Flatpak to a different
prefix than the system-installed version. First of all, the newly
built Flatpak will look for system-installed flatpaks in
$PREFIX/var/lib/flatpak, which will not match existing installations.
You can use --with-system-install-dir=/var/lib/flatpak to make both
installations use the same location.
Secondly, Flatpak ships with a root-privileged PolicyKit helper for
system-wide installation, called flatpak-system-helper. It is D-Bus
activated (on the system bus) and if you install in a non-standard
location it is likely that D-Bus will not find it and PolicyKit
integration will not work. However, if the system installation is
synchronized, you can often use the system installed helper instead—
at least if the two versions are close enough.
This repository
The Flatpak project consists of multiple pieces, and it can be a bit challenging to find your way around at first. Here is a quick intro to the major components of the flatpak repo:
common: contains the library, libflatpak. It also contains various pieces of code that are shared between the library, the client and the services. Non-public code can be recognized by having a-private.hheader file.app: the commandline client. Each command has aflatpak-builtins-source filedata: D-Bus interface definition filessession-helper: The flatpak-session-helper service, which provides various helpers for the sandbox setup at runtimesystem-helper: The flatpak-system-helper service, which runs as root on the system bus and allows non-root users to modify system installationsportal: The Flatpak portal service, which lets sandboxed apps request the creation of new sandboxesdoc: The sources for the documentation, both man pages and library documentationtests: The testsuitebubblewrap: Flatpak's unprivileged sandboxing tool which is developed separately and exists here as a submodulelibglnx: a small utility library for projects that use GLib on Linux, as a submoduledbus-proxy: a filtering proxy for D-Bus connections, as a submoduleicon-validator: A small utility that is used to validate iconsrevokefs: A fuse filesystem that is used to transfer files to the system-helper without copying
Related Projects
Here are some notable projects in the Flatpak ecosystem:
- Flatseal: An app for managing permissions of Flatpak apps without using the CLI
- Souk: A Flatpak-only app store
- Flat-manager: A tool for managing Flatpak repositories