FICLONE is the new alias for the formerly btrfs-specific ioctl; XFS
has experimental patches to support it.
Further, we should use copy_file_range() for the case where we're only doing a
limited copy. Both NFS and XFS (with reflink enabled) understand it.
Part of the reason I'm doing this is so that ostree's `/etc` merge will start
using XFS reflinks. But another major reason is to take the next step after and
copy this code into GLib as well, so that all of the general GLib users will
benefit; e.g. Nautilus will transparently do server copy offloads with NFS home
directories.
See also this coreutils thread about `copy_file_range()`:
<https://debbugs.gnu.org/cgi/bugreport.cgi?bug=24399>. I don't care about file
holes for our use cases, so it's fine.
Other changes while I'm here:
- Tweak the sendfile() case to match the newly inlined logic for cfr
- Add a TEMP_FAILURE_RETRY() around the read()
We should be able to rely upstream on everything *except* `glnx_unref_object`
which requires the library itself to depend on a newer glib, which isn't true
for e.g. RHEL7 libsoup.
libostree was almost ready for this; just a few patches to push
it to completion in
https://github.com/ostreedev/ostree/pull/1042
systemd does this by default. I think we should treat this as a fatal error
since it can cause really painful-to-debug problems if we don't just get
EBADF but actually close something else's fd due to a race.
Minor tweak to the new `GLNX_AUTO_PREFIX_ERROR`. Since the common case
is that there's no errors, let's bring down the same check that
`g_prefix_error` does to avoid a function call most of the time.
Another one where we have a lot of inlines in ostree at least. Not the same as
`glnx_shutil_mkdir_p_at()` since in these cases we don't want automatic
intermediate dirs, and it's cheaper to just call `mkdirat()` and handle `EEXIST`
rather than do a `stat()` first.
This is kind of long overdue. Reasons are the same as the other wrappers. I
debated adding `O_NOFOLLOW` support but the use cases for that are pretty
obscure, callers who want that can just use the syscall directly for now.
In a lot of places in ostree, we end up prefixing errors in the *caller*.
Often we only have 1-2 callers, and doing the error prefixing isn't
too duplicative. But there are definitely cases where it's cleaner
to do the prefixing in the callee. We have functions that aren't
ported to new style for this reason (they still do the prefixing
in `out:`).
Introduce a cleanup-oriented version of error prefixing so we can port those
functions too.
Mostly in ostree/rpm-ostree, we work in either raw `int fd`, or
`G{Input,Output}Stream`. One exception is the rpm-ostree `/etc/passwd`
handling, which uses `FILE*` since that's what glibc exposes.
And in general, there are use cases for `FILE*`; the raw `GUnixOutputStream` for
example isn't buffered, and doing so via e.g. `GBufferedOutputStream` means
allocating *two* GObjects and even worse going through multiple vfuncs for every
write.
`FILE*` is used heavily in systemd, and provides buffering. It is a bit cheaper
than gobjects, but has its own trap; by default every operation locks a mutex.
For more information on that, see `unlocked_stdio(3)`. However, callers can
avoid that by using e.g. `fwrite_unlocked`, which I plan to do for most users of
`FILE*` that aren't writing to one of the standard streams like `stdout` etc.
I'm not aware of a problem in practice here, but we should do this on general
principle. Writing this patch now because I hit a fd leak in the ostree static
delta processing that was introduced in the tmpfile prep code, but fixed in the
final port.
Looking at converting the ostree codebase, iterating over only the
values of a hash table (while ignoring the key) is actually a more
common pattern than I thought. So let's give it its own macro as well so
users don't have to resort to the _KV variant.
These macros make it much easier to iterate over a GHashTable. It takes
care of initializing an iterator and casting keys and values to their
proper types.
See the example usage in the docstring for more info.
I originally tried to get this into GLib:
https://bugzilla.gnome.org/show_bug.cgi?id=783751
But that looks like it's going to fail due to MSVC. Let's add it here at least
so I can start using it tomorrow and not wait for the MSVC team to catch up.
I renamed `glnx-alloca.h` to `glnx-macros.h` as a more natural collective
home for things from systemd's `macro.h`.
Finally, I used a Coccinelle spatch similar to the one referenced
in the above BZ to patch our uses.
This avoids callers having to use `glnx_steal_fd()` on their own; in general, I
think we should implement move semantics like this at the callee level.
Another reason to do this is there's a subtle problem with doing:
```
somefunction (steal_value (&v), ..., error);
```
in that if `somefunction` throws, it may not have taken ownership of the value.
At least `glnx_dirfd_iterator_init_take_fd()` didn't.
Add an `initialized` member which means we work by default
in structs allocated with `g_new0` etc. and don't need
a special initializer. This also fixes a bug where
we need to support `src_dfd == -1` or `AT_FDCWD`.
This fixes flatpak which uses AT_FDCWD.
Modified-by: Colin Walters <walters@verbum.org>
The core problem with the previous tmpfile code
is we don't have an autocleanup that calls `unlinkat`
in the non-`O_TMPFILE` case. And even if we did, it'd
be awkward still since the `glnx_link_tmpfile_at()` call
*consumes* the tmpfile.
Fix this by introducing a struct with a cleanup macro. This simplifies a number
of the callers in libostree - a notable case is where we had two arrays, one of
fds, one of paths. It makes other places in libostree a bit more complex, but
that's because some of the commit code paths want to deal with temporary
*symlinks* too.
Most callers are better though - in libglnx itself, `glnx_file_copy_at()` now
correctly unlinks on failure for example.
NOTE: This changes the error handling API of `glnx_loop_write()` to be
"old school POSIX" instead of "systemd".
In ostree in a few places we use `g_output_stream_splice()`. I
thought this would use `splice()`, but actually it doesn't today.
They also, if a cancellable is provided, end up dropping into `poll()` for every
read and write. (In addition to copying data to/from userspace).
My opinion on this is - for *local files* that's dumb. In the big picture, you
really only need cancellation when copying gigabytes. Down the line, we could
perhaps add a `glnx_copy_bytes_cancellable()` that only did that check e.g.
every gigabyte of copied data. And when we do that we should use
`g_cancellable_set_error_if_cancelled()` rather than a `poll()` with the regular
file FD, since regular files are *always* readable and writable.
For my use case with rpm-ostree though, we don't have gigabyte sized files, and
seeing all of the `poll()` calls in strace is annoying. So let's have the
non-cancellable file copying API that's modern and uses both reflink and
`sendfile()` if available, in that order.
My plan at some point once this is tested more is to migrate this code
into GLib.
Note that in order to keep our APIs consistent, I switched the systemd-imported
code to "old school POSIX" error conventions. Otherwise we'd have *3* (POSIX,
systemd, and GError) and particularly given the first two are easily confused,
it'd be a recipe for bugs.
There's one function that did `unlinkat()` in the cleanup section,
not doing that yet.
Note I uncovered a few bugs in a few places where we didn't preserve errno
before doing an `unlinkat()` in error paths in a few cases.
I also tried to prefix a few more error cases with the system call name.
Add two inline wrappers around fstat() and fstatat() which handle
retrying on EINTR and return other errors using GError, to be consistent
with other glnx functions.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
At the moment, it’s not possible for them to do this race-free (since
openat(O_DIRECTORY | O_CREAT | O_EXCL) doesn’t work), but in future this
could be possible. In any case, it’s a useful thing to want to do.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
This is a variant of glnx_shutil_mkdir_p_at() which opens the given
directory and returns a dirfd to it. Currently, the implementation
cannot be race-free (due to a kernel bug), but it could eventually be
made race-free.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
In general, all FDs < 0 are invalid (and should not have close() called
on them), so check that. This could have caused problems if a function
returned an error value < -1.
Signed-off-by: Philip Withnall <withnall@endlessm.com>
We were missing the previous automatic `: ` addition; noticed in
a failing ostree test.
Fix this by just calling the new API as the non-prefix case does too.
These are equivalent to the non-null throw, except that the returned
value is a NULL pointer. They can be used in functions where one wants
to return a pointer. E.g.:
GKeyFile *foo(GError **error) {
return glnx_null_throw (error, "foobar");
}
The function call redirections are wrapped around a compound statement
expression[1] so that they represent a single top-level expression. This
allows us to avoid -Wunused-value warnings vs using a comma operator if
the return value isn't used.
I made the 'args...' absorb the fmt argument as well so that callers can
still use it without always having to specify at least one additional
variadic argument. I had to check to be sure that the expansion is all
done by the preprocessor, so we don't need to worry about stack
intricacies.
[1] https://gcc.gnu.org/onlinedocs/gcc/Statement-Exprs.html
Following up to the previous commit, also shorten our use of
`g_set_error (..., G_IO_ERROR_FAILED, ...)`. There's a lot of
this in libostree at least.
See also https://bugzilla.gnome.org/show_bug.cgi?id=774061
We have a *lot* of code of the form:
```
if (unlinkat (fd, pathname) < 0)
{
glnx_set_error_from_errno (error);
goto out;
}
```
After conversion to `return FALSE style` which is in progress, it's way shorter,
and clearer like this:
```
if (unlinkat (fd, pathname) < 0)
return glnx_throw_errno (error);
```
I want the `RENAME_EXCHANGE` version for rpm-ostree, to atomically
swap `/usr/share/rpm` (a directory) with a new verison. While
we're here we might as well expose `RENAME_NOREPLACE` in case
something else wants it.
These both have fallbacks to the non-atomic version.
Closes: https://github.com/GNOME/libglnx/pull/36
