Commit Graph

6139 Commits

Author SHA1 Message Date
Alexander Larsson
7483a98769 utils: Allow passing sizes to more utils
These now take (optional) string length args:
  flatpak_id_has_subref_suffix()
  flatpak_levenshtein_distance()
  flatpak_is_valid_name()
  flatpak_is_valid_branch()
2020-10-29 15:30:35 +01:00
Alexander Larsson
f147df908a repo: Support indexed summaries in existing code and add new support
If indexes are available we now use those. Also we now allow listing
subsummaries and filtering by subset prefix.
2020-10-29 15:30:35 +01:00
Alexander Larsson
5b52b64cd1 Export flatpak_repo_load_summary_index() and flatpak_repo_load_digested_summary() 2020-10-29 15:30:35 +01:00
Alexander Larsson
caa1c8c839 table printer: Add some helpers
Allow looking up pre-existing rows (by key) and appending to existing
cells.
2020-10-29 15:30:35 +01:00
Alexander Larsson
98e20c18f9 summaries: Support using summary deltas on client
Also prune old unreferenced subsummaries on each transaction.
2020-10-29 15:30:35 +01:00
Alexander Larsson
2c6fec556f summaries: Generate summary deltas when updating repo 2020-10-29 15:30:35 +01:00
Alexander Larsson
0221f5a1fa summaries: Ensure we can support non-default arches W/ indexed summaries
We need to load the ensure the right arch whenever we need it.

Also this restructures the RemoteState handling a bit in general so that
we avoid loading the same remote state multiple times when converting
partial refs to full refs.
2020-10-29 15:30:35 +01:00
Alexander Larsson
dce6f748bb summaries: Rework the debug spew for summary/cache loading
This way its easier to tell exactly what happens.
2020-10-29 15:30:35 +01:00
Alexander Larsson
5821f79d88 dir: Make flatpak_transaction_ensure_remote_state non-static
This moves it to fatpak-transaction-private.h so that we can use it
internally to be able to re-use the remote-states from the transaction
outside it.
2020-10-29 15:30:35 +01:00
Alexander Larsson
12fd8332da tests: Test stuff with the old (non-indexed) summary format too 2020-10-29 15:30:35 +01:00
Alexander Larsson
0d2cf085af Add option to disable generating summary index
This is mainly useful for the tests
2020-10-29 15:30:35 +01:00
Alexander Larsson
57954e2efc summary: Enable use of indexed summaries in the client
We first try to download the index, then fall back to the old format.
2020-10-29 15:30:35 +01:00
Alexander Larsson
0597f246c8 build-update-repo: Generate new summary format
In addition to the old summary file we create a summary.idx and
a set of per-arch subsummaries (and extra copies of these if any commit
specify a subset). These are much smaller, and eventually we will also
get deltas for them.

We are not yet using these new formats, although the code is there
to use them once we start downloading them.
2020-10-29 15:30:35 +01:00
Alexander Larsson
096daf91f2 summary: Implement a new, more efficient, summary format
This drops the deltas from the summary and uses the per-commit
metadata field to add the cache data to avoid the need for the separate
xa.cache and xa.sparse-cache indexes. This way we avoid repeating the
refs in multiple places.

Nothing uses this format yet, but we still pass make check if we enable
it.
2020-10-29 15:30:35 +01:00
Alexander Larsson
b094585545 utils: Expose get_compat_arch_reverse() and get_arch_for_ref()
We need to use these from some other files too.
2020-10-29 15:30:35 +01:00
Alexander Larsson
c5d76e5a85 summary: Extract the summary generation code to helper 2020-10-29 15:30:35 +01:00
Alexander Larsson
b7e90d3809 summaries: Add support for limiting which arches end up in summary
This is in preparation for adding other formats for summary and we
might not want all the arches in the fallback summary format.
2020-10-29 15:30:35 +01:00
Alexander Larsson
f84b48ac58 utils: Reimplement summary generation
This moves the generation of the summary files completely into flatpak
allowing us to (later) customise what goes into it in more detail and
generate other forms of summaries.
2020-10-29 15:30:35 +01:00
Alexander Larsson
6fd2ea749c Bump ostree requirement to 2020.8 for the new summary features 2020-10-29 15:30:35 +01:00
Alexander Larsson
387bb8c64e Add flatpak_dir_get_remote_subset()
Reads the subset config for the remote, if any.
2020-10-29 15:30:35 +01:00
Alexander Larsson
97fa217072 build-commit-from: Add --subset option
This sets the xa.subsets property on the commit, which we will later
use to create subsets of the repo.
2020-10-29 15:30:35 +01:00
Alexander Larsson
a32fb1e333 tests: Silence assert_remote_has_no_config
This is meant to fail, so redirect its stderr to get less confusing test
logs.
2020-10-29 15:30:35 +01:00
Alexander Larsson
67d3443999 tests: Fix unnecessary rebuilds of the test platform
The code to use the pre-generated test platform was buggy so it wasn't
used. Also, generate a "stable" branch of it too as that is used
by test-run.sh.
2020-10-29 15:30:35 +01:00
Alexander Larsson
95ba6c77b4 tests: Better logging of which tests are succeeding
This makes it easier to spot where things go wrong in the logs.
2020-10-29 15:30:35 +01:00
Alexander Larsson
d98943b400 tests: Use "flatpak build-update-repo" instead of ostree summary -u
As we tweak the summary generation code we want to use our code, not
the ostree one.
2020-10-29 15:30:35 +01:00
Alexander Larsson
3169fa58ed tests: Convert some flatpak calls to $FLATPAK
This allows these to run under valgrind when testing in valgrind.
2020-10-29 15:30:35 +01:00
Alexander Larsson
3d65276f79 Add --enable-internal-checks option and use in CI
This enables the internal checks in the generated variant parser.
2020-10-29 15:30:35 +01:00
Alexander Larsson
fb0691e895 Update to latest variant-schema-compiler with some fixes 2020-10-29 15:30:35 +01:00
Philip Withnall
7d30de344c Merge pull request #3906 from smcv/wip/tolerate-no-accountsservice
parental controls: Fail open if accountsservice is missing
2020-10-15 14:57:36 +01:00
Simon McVittie
1a58a86676 Skip parental controls checks on ServiceUnknown or NameHasNoOwner
If accountsservice isn't available on the system bus, then we can't
ask it for the user's parental controls settings, and we also can't
ask it whether it even has the malcontent extension. Since this is
not a real security boundary, fail open.

This can be dropped if we depend on a version of libmalcontent that maps
these errors to MCT_APP_FILTER_ERROR_DISABLED.

Resolves: https://github.com/flatpak/flatpak/issues/3902
Bug-Debian: https://bugs.debian.org/972138
Signed-off-by: Simon McVittie <smcv@collabora.com>
2020-10-15 09:24:14 +01:00
Philip Withnall
6c79a57114 http: Return HOST_UNREACHABLE on error 500
Sometimes a server might return a HTTP error 500 (this seems to happen
sometimes with Microsoft’s VSCode server, for example). Map this to
`G_IO_ERROR_HOST_UNREACHABLE` for now, which is a bit more specific than
returning `G_IO_ERROR_FAILED`, but without the hassle of introducing a
new public error domain which could give more detail.

In particular, this should allow gnome-software to show an error message
to the user for such failed downloads, rather than hiding the error and
logging the following:
```
not handling error failed for action download: While downloading http://packages.microsoft.com/repos/vscode/pool/main/c/code/code_1.45.1-1589445302_amd64.deb: Server returned status 500: Internal Server Error
```

Signed-off-by: Philip Withnall <pwithnall@endlessos.org>
2020-10-12 11:04:40 +02:00
Yuri Chornoivan
0200348cf7 Update Ukrainian translation 2020-10-12 10:39:57 +02:00
Alexander Larsson
180d807d2a oci-authenticator: Fix crash if anon auth fails and no_interaction is set
We were clearing the error from the anon test, and then not doing any
non-anon auth, so error was NULL, causing a crash when returning an
error message.
2020-10-09 16:57:57 +02:00
Alexander Larsson
33d1d7b1a5 transaction: Fix error handling in authentication failures
We were not correctly returning in case of error, which caused
a cascading "owerwrite gerror" problem.
2020-10-09 16:57:57 +02:00
Alexander Larsson
aec1f62f8f CLI: Ask for transaction confirmation before authentication
If possible, ask for confirmation of the entire transaction before
starting to do the per-ref authentication. We do this by splitting
the current "ready" callback into two parts, one in ready-pre-auth
and the rest in ready.

There is some complexity added because if we do any authentication
between the two signals we can't back-patch the transaction list we
already printed. To handle this we detect this case and re-print the
table if this happens.
2020-10-09 16:57:57 +02:00
Alexander Larsson
d7716f41ac libflatpak: Report available updates for password-protected refs too.
flatpak_installation_list_installed_refs_for_update() now uses the
ready-pre-auth signal instead of the ready signal. This means we will
report updates even for refs that require authentication to install.
2020-10-09 16:57:57 +02:00
Alexander Larsson
bf8ac3cc1c Transaction: Add ready-pre-auth signal
This is similar to the ready signal, except it is called before
per-ref authentication. Apps can use this if they want to be able
to ask for user input on progress before asking for authentication.

This is nice to be able to do in general, but it is also required for
the implementation of
flatpak_installation_list_installed_refs_for_update(), as it doesn't
install any authentication handler, so it will never report updates
for protected refs if using the ready signal.

Note: In special cases we will require authentication even earlier
if authenticating is needed during the resolve operation. This happens
for instance if you are doing a update to a particular commit (rather
than the lastest commit) where we need to get the commit object directly.
2020-10-09 16:57:57 +02:00
Ryan Gonzalez
ff2e7d6e90 transaction: Don't crash on invalid ref names
It is very much possible for an invalid ref name to occur, either due to
lack of validation on Flatpak creation like #3887, or just any
manually-written ref name due to skipping Flatpak tooling or malicious
intent. Regardless, this shouldn't crash, so check the names before
creating the transaction ops.

Fixes #3887.
2020-10-08 09:46:49 +02:00
Ryan Gonzalez
719eeee461 build-init/finish: Validate extension names
Otherwise, it becomes easy to accidentally create extensions with
invalid names. Ref #3887.
2020-10-08 09:46:49 +02:00
Ryan Gonzalez
9959fd80e9 repair: Add basic and "fancy" verify progress
This change makes repair print the current ref number being verified
and the total number, that way the user can observe the repair progress.
In addition, if fancy output is not disabled, the progress will be
cleanly printed on a single line.

On a dry run, this also will still print the delete messages without
actually performing the operations, to show what actions would be
taken on a non-dry-run.
2020-10-08 09:17:28 +02:00
Phaedrus Leeds
036aba8cb8 common: Fix docs for FLATPAK_QUERY_FLAGS_ONLY_SIDELOADED 2020-10-08 09:15:20 +02:00
Alexander Larsson
3001cb3174 Fix some clang errors 2020-10-08 09:14:51 +02:00
Alexander Larsson
a5d86b8acb transaction: inject eol status to flatpak_dir_list_unused_refs()
This way we can get the proper eol status for the new to-be-installed
refs, rather than whatever was previously installed. This allows us to
detect when a runtime is updated and the new one is eol, and nothing
uses it, so it can be auto-uninstalled.
2020-10-08 09:14:51 +02:00
Alexander Larsson
05a7bda140 transaction: Simplify add_uninstall_unused_ops()
Rather than trying to figure out which runtimes are affected byt
the current setup of ops we run flatpak_dir_list_unused_refs() twice,
once with and once without the changes the transaction will cause.

Any unused refs after the transaction that were not unused before are
caused by the transaction and we start uninstall ops for those.

Also rename flatpak_dir_list_unused_refs_with_options() to
flatpak_dir_list_unused_refs() as it need not be so long.
2020-10-08 09:14:51 +02:00
Alexander Larsson
6bdc364e95 find-unused: Don't care about origin when finding used deps
If we're auto-installing dependencies we want to limit them to those
from the same remote. However you can still (manually) install such
dependencies, so when we're looking for things to possibly uninstall
we need to check dependencies from all remotes.
2020-10-08 09:14:51 +02:00
Alexander Larsson
3642f59ea8 Fix some clang warnings for FlatpakRefPrivate atomic use 2020-10-08 09:14:51 +02:00
Alexander Larsson
478bcc0b12 Rewrite flatpak_dir_list_unused_refs_with_options()
This simplifies the implementation and makes it more effective in
avoiding to re-scan the same things several times.
2020-10-08 09:14:51 +02:00
Alexander Larsson
356b7fb346 tests: Add basic test for flatpak_dir_list_unused_refs_with_options()
This is currently failing, but I believe that is caused by actual
bugs.
2020-10-08 09:14:51 +02:00
Alexander Larsson
e83180e040 install: Add --no-auto-pin option
We want to use this for the test when we test the list unused refs
functions.
2020-10-08 09:14:51 +02:00
Alexander Larsson
84da902443 transaction: Add flatpak_transaction_set_disable_auto_pin()
This disables the automatic pinning of explicitly installed runtimes.
2020-10-08 09:14:51 +02:00