Some compression improvements.

The compression level of the first file in the transfer no longer sets
the level for all files that follow it.  Document that per-file level
switching has no current effect (except for a global "dont compress = *"
rule in the daemon).
This commit is contained in:
Wayne Davison
2021-12-31 10:58:19 -08:00
parent 13cfe6406f
commit c11467af36
4 changed files with 37 additions and 24 deletions

View File

@@ -17,7 +17,7 @@
export LC_ALL=C.UTF-8
```
or maybe:
or if iconv translations are needed:
```shell
if [ "${LC_ALL:-}" ]; then
@@ -60,6 +60,11 @@
- Avoid a weird failure if you run a local copy with a (useless) `--rsh`
option that contains a `V`.
- Fixed a long-standing compression bug where the compression level of the
first file transferred affected the level for all future files. Also, the
per-file compression skipping has apparently not worked in a very long time
(I checked back to 2.6.4), so it is now documented as being ineffective.
### ENHANCEMENTS:
- Use openssl's `-verify_hostname` option in the rsync-ssl script.

View File

@@ -2395,9 +2395,6 @@ your home directory (remove the '=' for that).
ignore this weirdness unless the rsync server complains and tells you to
specify `-zz`.
See also the `--skip-compress` option for the default list of file suffixes
that will be transferred with no (or minimal) compression.
0. `--compress-choice=STR`, `--zc=STR`
This option can be used to override the automatic negotiation of the
@@ -2442,8 +2439,8 @@ your home directory (remove the '=' for that).
> rsync -aiv --zc=zstd --zl=22 host:src/ dest/
For zlib & zlibx compression the valid values are from 1 to 9 with 6 being
the default. Specifying 0 turns compression off, and specifying -1 chooses
the default of 6.
the default. Specifying `--zl=0` turns compression off, and specifying
`--zl=-1` chooses the default level of 6.
For zstd compression the valid values are from -131072 to 22 with 3 being
the default. Specifying 0 chooses the default of 3.
@@ -2462,14 +2459,15 @@ your home directory (remove the '=' for that).
0. `--skip-compress=LIST`
**NOTE:** no compression method currently supports per-file compression
changes, so this option has no effect.
Override the list of file suffixes that will be compressed as little as
possible. Rsync sets the compression level on a per-file basis based on
the file's suffix. If the compression algorithm has an "off" level (such
as zlib/zlibx) then no compression occurs for those files. Other
algorithms that support changing the streaming level on-the-fly will have
the level minimized to reduces the CPU usage as much as possible for a
matching file. At this time, only zlib & zlibx compression support this
changing of levels on a per-file basis.
the file's suffix. If the compression algorithm has an "off" level, then
no compression occurs for those files. Other algorithms that support
changing the streaming level on-the-fly will have the level minimized to
reduces the CPU usage as much as possible for a matching file.
The **LIST** should be one or more file suffixes (without the dot) separated
by slashes (`/`). You may specify an empty string to indicate that no files

View File

@@ -922,13 +922,14 @@ the values of parameters. See the GLOBAL PARAMETERS section for more details.
> refuse options = * !a !delete* delete-after
A note on refusing "compress" -- it is better to set the "dont compress"
daemon parameter to "`*`" because that disables compression silently
A note on refusing "compress": it may be better to set the "dont compress"
daemon parameter to "`*`" and ensure that `RSYNC_COMPRESS_LIST=zlib` is set
in the environment of the daemon in order to disable compression silently
instead of returning an error that forces the client to remove the `-z`
option.
If you are un-refusing the compress option, you probably want to match
"`!compress*`" so that you also accept the `--compress-level` option.
If you are un-refusing the compress option, you may want to match
"`!compress*`" if you also want to allow the `--compress-level` option.
Note that the "write-devices" option is refused by default, but can be
explicitly accepted with "`!write-devices`". The options "log-file" and
@@ -954,6 +955,10 @@ the values of parameters. See the GLOBAL PARAMETERS section for more details.
0. `dont compress`
**NOTE:** This parameter currently has no effect except in one instance: if
it is set to "`*`" then it minimizes or disables compression for all files
(for those that don't want to refuse the `--compress` option completely).
This parameter allows you to select filenames based on wildcard patterns
that should not be compressed when pulling files from the daemon (no
analogous parameter exists to govern the pushing of files to a daemon).
@@ -964,14 +969,14 @@ the values of parameters. See the GLOBAL PARAMETERS section for more details.
The "dont compress" parameter takes a space-separated list of
case-insensitive wildcard patterns. Any source filename matching one of the
patterns will be compressed as little as possible during the transfer. If
the compression algorithm has an "off" level (such as zlib/zlibx) then no
compression occurs for those files. Other algorithms have the level
minimized to reduces the CPU usage as much as possible.
the compression algorithm has an "off" level, then no compression occurs
for those files. If an algorithms has the ability to change the level in
mid-stream, it will be minimized to reduce the CPU usage as much as
possible.
See the `--skip-compress` parameter in the **rsync**(1) manpage for the
list of file suffixes that are not compressed by default. Specifying a
value for the "dont compress" parameter changes the default when the daemon
is the sender.
list of file suffixes that are skipped by default if this parameter is not
set.
0. `early exec`, `pre-xfer exec`, `post-xfer exec`

View File

@@ -39,7 +39,6 @@ extern char *skip_compress;
#define Z_INSERT_ONLY Z_SYNC_FLUSH
#endif
static int compression_level; /* The compression level for the current file. */
static int skip_compression_level; /* The least possible compressing for handling skip-compress files. */
static int per_file_default_level; /* The default level that each new file gets prior to checking its suffix. */
@@ -224,9 +223,11 @@ static void init_set_compression(void)
/* determine the compression level based on a wildcard filename list */
void set_compression(const char *fname)
{
#if 0 /* No compression algorithms currently allow mid-stream changing of the level. */
const struct suffix_tree *node;
const char *s;
char ltr;
#endif
if (!do_compression)
return;
@@ -234,6 +235,7 @@ void set_compression(const char *fname)
if (!match_list)
init_set_compression();
#if 0
compression_level = per_file_default_level;
if (!*match_list && !suftree)
@@ -270,6 +272,9 @@ void set_compression(const char *fname)
if (!(node = node->child))
return;
}
#else
(void)fname;
#endif
}
/* non-compressing recv token */
@@ -361,7 +366,7 @@ send_deflated_token(int f, int32 token, struct map_struct *buf, OFF_T offset, in
tx_strm.next_in = NULL;
tx_strm.zalloc = NULL;
tx_strm.zfree = NULL;
if (deflateInit2(&tx_strm, compression_level,
if (deflateInit2(&tx_strm, per_file_default_level,
Z_DEFLATED, -15, 8,
Z_DEFAULT_STRATEGY) != Z_OK) {
rprintf(FERROR, "compression init failed\n");