Commit Graph

6156 Commits

Author SHA1 Message Date
Yuri Chornoivan
df0362f012 Update Ukrainian translation 2020-11-02 08:54:45 +01:00
Phaedrus Leeds
08f3f1efb7 Merge pull request #3911 from flatpak/txn-auth-docs
transaction: Fix docs around op authentication
2020-10-30 10:25:07 -07:00
Phaedrus Leeds
08e55a36aa transaction: Fix docs around op authentication 2020-10-29 20:26:08 -07:00
Alexander Larsson
a0dd41d8c4 docs: Add initial document describing the flatpak summary files. 2020-10-29 17:19:47 +01:00
Alexander Larsson
7b5a0a5c86 Fix extra-data progress reporting
We were never calling the progress callback for extra-data downloaded
bytes.
2020-10-29 17:19:38 +01:00
Mejans
2c22604f9b Adds Occitan 2020-10-29 16:12:08 +01:00
Mejans
fa8f3735eb First commit for Occitan language 2020-10-29 16:12:08 +01:00
Phaedrus Leeds
835d2c78e6 testlibrary: Tweak one of the tests to avoid CLI
It seems cleaner to use library API than the CLI here.
2020-10-29 16:08:39 +01:00
Alexander Larsson
308e30c8f0 transaction: Remove transfer annotation for boolean
This doesn't do anything (and was causing warnings).
2020-10-29 16:08:12 +01:00
Alexander Larsson
c04df80ee1 Drop some spew from flatpak_dir_list_unused_refs()
This was showing up in every flatpak update, making it hard
to see the important stuff.
2020-10-29 16:08:12 +01:00
Phaedrus Leeds
ea7b262049 CI: Ignore some paths that don't affect build
No need to waste resources running every test after the README
changes.
2020-10-29 16:07:56 +01:00
Alexander Larsson
b8db46d0e2 tests: Add some tests for subsummaries 2020-10-29 15:30:35 +01:00
Alexander Larsson
e6d7a1148e summaries: Cache subsummaries per-arch
We store the cached subsummaries as `$remote-$arch-digest.sub`, this
way we can prune everything but the most recent (i.e. highest mtime)
rather than looking at the index. This way we don't prune old
subsummaries for other arches when we update one arch, which would not
let us use deltas for those.

Some details:
 * If mtimes are identical (mtime precision issue) for several
   subsummaries we keep all.
 * If we just saved a subsummary in the cache we don't prune that
   in case there is something wonky with mtimes.

The subsets for a remote are pruned every time we write a new cached
subsummary.
2020-10-29 15:30:35 +01:00
Alexander Larsson
97099bde37 CI: Build ostree from git (needed for new summary format) 2020-10-29 15:30:35 +01:00
Alexander Larsson
a60e2cbe2d Remove some unused variables 2020-10-29 15:30:35 +01:00
Alexander Larsson
2c8fe5b94c Use the new FlatpakDecomposed when mathing refs
This avoids lots of re-splitting and re-validation as well as strdups.
2020-10-29 15:30:35 +01:00
Alexander Larsson
472939f538 utils: Add FlatpakDecomposed helper
This is a ref-counted version of a ref string that is guaranteed
to be in the proper form, and is pre-decomposed for efficient partial
matches.
2020-10-29 15:30:35 +01:00
Alexander Larsson
7483a98769 utils: Allow passing sizes to more utils
These now take (optional) string length args:
  flatpak_id_has_subref_suffix()
  flatpak_levenshtein_distance()
  flatpak_is_valid_name()
  flatpak_is_valid_branch()
2020-10-29 15:30:35 +01:00
Alexander Larsson
f147df908a repo: Support indexed summaries in existing code and add new support
If indexes are available we now use those. Also we now allow listing
subsummaries and filtering by subset prefix.
2020-10-29 15:30:35 +01:00
Alexander Larsson
5b52b64cd1 Export flatpak_repo_load_summary_index() and flatpak_repo_load_digested_summary() 2020-10-29 15:30:35 +01:00
Alexander Larsson
caa1c8c839 table printer: Add some helpers
Allow looking up pre-existing rows (by key) and appending to existing
cells.
2020-10-29 15:30:35 +01:00
Alexander Larsson
98e20c18f9 summaries: Support using summary deltas on client
Also prune old unreferenced subsummaries on each transaction.
2020-10-29 15:30:35 +01:00
Alexander Larsson
2c6fec556f summaries: Generate summary deltas when updating repo 2020-10-29 15:30:35 +01:00
Alexander Larsson
0221f5a1fa summaries: Ensure we can support non-default arches W/ indexed summaries
We need to load the ensure the right arch whenever we need it.

Also this restructures the RemoteState handling a bit in general so that
we avoid loading the same remote state multiple times when converting
partial refs to full refs.
2020-10-29 15:30:35 +01:00
Alexander Larsson
dce6f748bb summaries: Rework the debug spew for summary/cache loading
This way its easier to tell exactly what happens.
2020-10-29 15:30:35 +01:00
Alexander Larsson
5821f79d88 dir: Make flatpak_transaction_ensure_remote_state non-static
This moves it to fatpak-transaction-private.h so that we can use it
internally to be able to re-use the remote-states from the transaction
outside it.
2020-10-29 15:30:35 +01:00
Alexander Larsson
12fd8332da tests: Test stuff with the old (non-indexed) summary format too 2020-10-29 15:30:35 +01:00
Alexander Larsson
0d2cf085af Add option to disable generating summary index
This is mainly useful for the tests
2020-10-29 15:30:35 +01:00
Alexander Larsson
57954e2efc summary: Enable use of indexed summaries in the client
We first try to download the index, then fall back to the old format.
2020-10-29 15:30:35 +01:00
Alexander Larsson
0597f246c8 build-update-repo: Generate new summary format
In addition to the old summary file we create a summary.idx and
a set of per-arch subsummaries (and extra copies of these if any commit
specify a subset). These are much smaller, and eventually we will also
get deltas for them.

We are not yet using these new formats, although the code is there
to use them once we start downloading them.
2020-10-29 15:30:35 +01:00
Alexander Larsson
096daf91f2 summary: Implement a new, more efficient, summary format
This drops the deltas from the summary and uses the per-commit
metadata field to add the cache data to avoid the need for the separate
xa.cache and xa.sparse-cache indexes. This way we avoid repeating the
refs in multiple places.

Nothing uses this format yet, but we still pass make check if we enable
it.
2020-10-29 15:30:35 +01:00
Alexander Larsson
b094585545 utils: Expose get_compat_arch_reverse() and get_arch_for_ref()
We need to use these from some other files too.
2020-10-29 15:30:35 +01:00
Alexander Larsson
c5d76e5a85 summary: Extract the summary generation code to helper 2020-10-29 15:30:35 +01:00
Alexander Larsson
b7e90d3809 summaries: Add support for limiting which arches end up in summary
This is in preparation for adding other formats for summary and we
might not want all the arches in the fallback summary format.
2020-10-29 15:30:35 +01:00
Alexander Larsson
f84b48ac58 utils: Reimplement summary generation
This moves the generation of the summary files completely into flatpak
allowing us to (later) customise what goes into it in more detail and
generate other forms of summaries.
2020-10-29 15:30:35 +01:00
Alexander Larsson
6fd2ea749c Bump ostree requirement to 2020.8 for the new summary features 2020-10-29 15:30:35 +01:00
Alexander Larsson
387bb8c64e Add flatpak_dir_get_remote_subset()
Reads the subset config for the remote, if any.
2020-10-29 15:30:35 +01:00
Alexander Larsson
97fa217072 build-commit-from: Add --subset option
This sets the xa.subsets property on the commit, which we will later
use to create subsets of the repo.
2020-10-29 15:30:35 +01:00
Alexander Larsson
a32fb1e333 tests: Silence assert_remote_has_no_config
This is meant to fail, so redirect its stderr to get less confusing test
logs.
2020-10-29 15:30:35 +01:00
Alexander Larsson
67d3443999 tests: Fix unnecessary rebuilds of the test platform
The code to use the pre-generated test platform was buggy so it wasn't
used. Also, generate a "stable" branch of it too as that is used
by test-run.sh.
2020-10-29 15:30:35 +01:00
Alexander Larsson
95ba6c77b4 tests: Better logging of which tests are succeeding
This makes it easier to spot where things go wrong in the logs.
2020-10-29 15:30:35 +01:00
Alexander Larsson
d98943b400 tests: Use "flatpak build-update-repo" instead of ostree summary -u
As we tweak the summary generation code we want to use our code, not
the ostree one.
2020-10-29 15:30:35 +01:00
Alexander Larsson
3169fa58ed tests: Convert some flatpak calls to $FLATPAK
This allows these to run under valgrind when testing in valgrind.
2020-10-29 15:30:35 +01:00
Alexander Larsson
3d65276f79 Add --enable-internal-checks option and use in CI
This enables the internal checks in the generated variant parser.
2020-10-29 15:30:35 +01:00
Alexander Larsson
fb0691e895 Update to latest variant-schema-compiler with some fixes 2020-10-29 15:30:35 +01:00
Philip Withnall
7d30de344c Merge pull request #3906 from smcv/wip/tolerate-no-accountsservice
parental controls: Fail open if accountsservice is missing
2020-10-15 14:57:36 +01:00
Simon McVittie
1a58a86676 Skip parental controls checks on ServiceUnknown or NameHasNoOwner
If accountsservice isn't available on the system bus, then we can't
ask it for the user's parental controls settings, and we also can't
ask it whether it even has the malcontent extension. Since this is
not a real security boundary, fail open.

This can be dropped if we depend on a version of libmalcontent that maps
these errors to MCT_APP_FILTER_ERROR_DISABLED.

Resolves: https://github.com/flatpak/flatpak/issues/3902
Bug-Debian: https://bugs.debian.org/972138
Signed-off-by: Simon McVittie <smcv@collabora.com>
2020-10-15 09:24:14 +01:00
Philip Withnall
6c79a57114 http: Return HOST_UNREACHABLE on error 500
Sometimes a server might return a HTTP error 500 (this seems to happen
sometimes with Microsoft’s VSCode server, for example). Map this to
`G_IO_ERROR_HOST_UNREACHABLE` for now, which is a bit more specific than
returning `G_IO_ERROR_FAILED`, but without the hassle of introducing a
new public error domain which could give more detail.

In particular, this should allow gnome-software to show an error message
to the user for such failed downloads, rather than hiding the error and
logging the following:
```
not handling error failed for action download: While downloading http://packages.microsoft.com/repos/vscode/pool/main/c/code/code_1.45.1-1589445302_amd64.deb: Server returned status 500: Internal Server Error
```

Signed-off-by: Philip Withnall <pwithnall@endlessos.org>
2020-10-12 11:04:40 +02:00
Yuri Chornoivan
0200348cf7 Update Ukrainian translation 2020-10-12 10:39:57 +02:00
Alexander Larsson
180d807d2a oci-authenticator: Fix crash if anon auth fails and no_interaction is set
We were clearing the error from the anon test, and then not doing any
non-anon auth, so error was NULL, causing a crash when returning an
error message.
2020-10-09 16:57:57 +02:00