Preparing for release of 2.6.4

did in older versions.
- Added a comment for log_format_has().

NEWS

@@ -1,4 +1,4 @@
-NEWS for rsync 2.6.4 (UNRELEASED)
+NEWS for rsync 2.6.4 (30 March 2005)
 Protocol: 29 (changed)
 Changes since 2.6.3:
 
@@ -21,12 +21,12 @@ Changes since 2.6.3:
       avoids outputting the name of the file twice in most circumstances.
       As long as the --log-format item does not refer to any post-transfer
       items (such as %b or %c), the --log-format message is output prior to
-      the transfer with --verbose being the equivalent of a --log-format of
-      '%n%L' (which outputs the name and any symlink info).  If the log
-      output must occur after the transfer to be complete, the only time
-      the name is also output prior to the transfer is when --progress was
-      specified (so that the name will precede the progress stats, and the
-      full --log-format output will come after).
+      the transfer, so --verbose is now the equivalent of a --log-format of
+      '%n%L' (which outputs the name and any link info).  If the log output
+      must occur after the transfer to be complete, the only time the name
+      is also output prior to the transfer is when --progress was specified
+      (so that the name will precede the progress stats, and the full
+      --log-format output will come after).
   
   BUG FIXES:
 
@@ -47,16 +47,24 @@ Changes since 2.6.3:
     - Fixed a potential hang when verbosity is high, the client side is
       the sender, and the file-list is large.
 
-    - We now check if the OS doesn't support using mknod() for creating
-      FIFOs and sockets, and compile-in using mkfifo() and socket() when
-      necessary.
+    - Fixed a potential protocol-corrupting bug where the generator could
+      merge a message from the receiver into the middle of a multiplexed
+      packet of data if only part of that data had been written out to the
+      socket when the message from the generator arrived.
 
-    - Fixed an off-by-one error in the handling of --max-delete=N.
+    - We now check if the OS doesn't support using mknod() for creating
+      FIFOs and sockets, and compile-in some compatibility code using
+      mkfifo() and socket() when necessary.
+
+    - Fixed an off-by-one error in the handling of --max-delete=N.  Also,
+      if the --max-delete limit is exceeded during a run, we now output a
+      warning about this at the end of the run and exit with a new error
+      code (25).
 
     - One place in the code wasn't checking if fork() failed.
 
     - The "ignore nonreadable" daemon parameter used to erroneously affect
-      symlinks that pointed to a non-existent file.  This has been fixed.
+      readable symlinks that pointed to a non-existent file.
 
     - If the OS does not have lchown() and a chown() of a symlink will
       affect the referent of a symlink (as it should), we no longer try
@@ -65,8 +73,8 @@ Changes since 2.6.3:
     - The generator now properly runs the hard-link loop and the dir-time
       rewriting loop after we're sure that the redo phase is complete.
 
-    - When --backup was specified with --partial-dir=DIR (where DIR is a
-      relative path), the backup code was erroneously trying to backup a
+    - When --backup was specified with --partial-dir=DIR, where DIR is a
+      relative path, the backup code was erroneously trying to backup a
       file that was put into the partial-dir.
 
     - If a file gets resent in a single transfer and the --backup option is
@@ -93,7 +101,7 @@ Changes since 2.6.3:
     - If a daemon can't open the specified log file (i.e. syslog is not
       being used), die without crashing.  We also output an error about
       the failure on stderr (which will only be seen if --no-detach was
-      specified).
+      specified) and exit with a new error code (6).
 
     - A local transfer no longer duplicates all its include/exclude options
       (since the forked process already has a copy of the exclude list,
@@ -111,12 +119,19 @@ Changes since 2.6.3:
       make progress.  (Requires protocol 29.)
 
     - The stat size of a device is not added to the total file size of the
-      items in the transfer since the size might be undefined on some OSes.
+      items in the transfer (the size might be undefined on some OSes).
 
     - Fixed a problem with refused-option messages sometimes not making it
       back to the client side when a remote --files-from was in effect and
       the daemon was the receiver.
 
+    - The --compare-dest option was not updating a file that differred in
+      (the preserved) attributes from the version in the compare-dest DIR.
+
+    - When rsync is copying files into a write-protected directory, fixed
+      the change-report output for the directory so that we don't report
+      an identical directory as changed.
+
   ENHANCEMENTS:
 
     - Rsync now supports popt's option aliases, which means that you can
@@ -125,44 +140,49 @@ Changes since 2.6.3:
     - Added the --delete-during (--del) option which will delete files
       from the receiving side incrementally as each directory in the
       transfer is being processed.  This makes it more efficient than the
-      default, before-the-transfer behavior, which is now available as
+      default, before-the-transfer behavior, which is now also available as
       --delete-before (and is still the default --delete-WHEN option that
       will be chosen if --delete or --delete-excluded is specified without
       a --delete-WHEN choice).  All the --del* options infer --delete, so
       an rsync daemon that refuses "delete" will still refuse to allow any
-      file-deleting options.
+      file-deleting options (including the new --remove-sent-files option).
 
     - All the --delete-WHEN options are now more memory efficient:
-      Previously an entire duplicate set of file-list objects was created
-      on the receiving side for the entire destination hierarchy.  The new
+      Previously an duplicate set of file-list objects was created on the
+      receiving side for the entire destination hierarchy.  The new
       algorithm only creates one directory of objects at a time (for files
       inside the transfer).
 
-    - Added support for specifying multiple --compare-dest or --link-dest
-      options, but only of a single type. (Promoted from the patches dir
-      and enhanced.)
+    - Added the --copy-dest option, which works like --link-dest except
+      that it locally copies identical files instead of hard-linking them.
+
+    - Added support for specifying multiple --compare-dest, --copy-dest, or
+      --link-dest options, but only of a single type. (Promoted from the
+      patches dir and enhanced.) (Requires protocol 29.)
 
     - Added the --max-size option. (Promoted from the patches dir.)
 
-    - The daemon-mode options were separated from the normal rsync options
-      so that they can't be mixed together.  This makes it impossible to
-      start a daemon that had improper default option values that could
-      cause problems when a client connects (e.g. a hang or an abort).
+    - The daemon-mode options are now separated from the normal rsync
+      options so that they can't be mixed together.  This makes it
+      impossible to start a daemon that has improper default option values
+      (which could cause problems when a client connects, such as hanging
+      or crashing).
 
     - The --bwlimit option may now be used in combination with --daemon
       to specify both a default value for the daemon side and a value
       that cannot be exceeded by a user-specified --bwlimit option.
 
     - Added the "port" parameter to the rsyncd.conf file. (Promoted from
-      the patches dir.)  Also added "address".  A command-line option
-      will take precedence over a config-file option, as expected.
+      the patches dir.)  Also added "address".  The command-line options
+      take precedence over a config-file option, as expected.
 
     - In _exit_cleanup(): when we are exiting with a partially-received
       file, we now flush any data in the write-cache before closing the
       partial file.
 
-    - The --inplace support was enhanced to work with --compare-dest and
-      --link-dest. (Requires protocol 29.)
+    - The --inplace support was enhanced to work with --compare-dest,
+      --link-dest, and (the new) --copy-dest options. (Requires protocol
+      29.)
 
     - Added the --dirs (-d) option for an easier way to copy directories
       without recursion.
@@ -178,7 +198,7 @@ Changes since 2.6.3:
     - Added the --omit-dir-times (-O) option, which will avoid updating
       the modified time for directories when --times was specified.  This
       option will avoid an extra pass through the file-list at the end of
-      the transfer (to tweak all the directory times), which can result in
+      the transfer (to tweak all the directory times), which may provide
       an appreciable speedup for a really large transfer. (Promoted from
       the patches dir.)
 
@@ -209,18 +229,18 @@ Changes since 2.6.3:
       very wrong).
 
     - Added the --itemize-changes (-i) option, which is a way to output a
-      more detailed list of what files changed in any way and how they
-      changed.  The effect is the same as specifying a --log-format of
-      "%i %n%L" (see the rsyncd.conf manpage).  Works with --dry-run too.
+      more detailed list of what files changed and in what way.  The effect
+      is the same as specifying a --log-format of "%i %n%L" (see both the
+      rsync and rsyncd.conf manpages).  Works with --dry-run too.
 
-    - Added the --fuzzy option, which attempts to find a basis file for a
-      file that is being created from scratch.  The current algorithm
-      only looks in the destination directory for the created file, but
-      it does attempt to find a match based on size/mod-time (in case the
-      file was renamed with no other changes) as well as based on a fuzzy
-      name-matching algorithm.  This option requires protocol 29 because
-      it needs the new file-sorting order.  (Promoted from patches dir
-      and enhanced.)
+    - Added the --fuzzy (-y) option, which attempts to find a basis file
+      for a file that is being created from scratch.  The current algorithm
+      only looks in the destination directory for the created file, but it
+      does attempt to find a match based on size/mod-time (in case the file
+      was renamed with no other changes) as well as based on a fuzzy
+      name-matching algorithm.  This option requires protocol 29 because it
+      needs the new file-sorting order.  (Promoted from patches dir and
+      enhanced.) (Requires protocol 29.)
 
     - Added the --remove-sent-files option, which lets you move files
       between systems.
@@ -229,10 +249,24 @@ Changes since 2.6.3:
       enclosed in '[' and ']' (e.g. "[::1]").  (We already allowed IPv6
       literals in the rsync://HOST:PORT/PATH format.)
 
+    - When rsync recurses to build the file list, it no longer keeps open
+      one or more directory handles from the dir's parent dirs.
+
     - When building under windows, the default for --daemon is now to
       avoid detaching, requiring the new --detach option to force rsync
       to detach.
 
+    - The --dry-run option can now be combined with either --write-batch or
+      --read-batch, allowing you to run a do-nothing test command to see
+      what would happen without --dry-run.
+
+    - The daemon's "read only" config item now sets an internal read_only
+      variable that makes extra sure that no write/delete calls on the
+      read-only side can succeed.
+
+    - The log-format % escapes can now have a numeric field width in
+      between the % and the escape letter (e.g. "%-40n %08p").
+
     - Improved the option descriptions in the --help text.
 
   SUPPORT FILES:
@@ -252,11 +286,11 @@ Changes since 2.6.3:
     - Added savetransfer.c to the support dir: a C program that can make
       a copy of all the data that flows over the wire.  This lets you test
       for data corruption (by saving the data on both the sending side and
-      the receiving side) or provides a way to help debug a protocol error.
+      the receiving side) and provides one way to debug a protocol error.
 
-    - Added rrsync to the support dir: this is my version of Joe Smith's
-      restricted rsync perl script.  This helps to ensure that only certain
-      rsync commands can be run by an ssh invocation.
+    - Added rrsync to the support dir: this is an updated version of Joe
+      Smith's restricted rsync perl script.  This helps to ensure that only
+      certain rsync commands can be run by an ssh invocation.
 
   INTERNAL:
 
@@ -276,10 +310,6 @@ Changes since 2.6.3:
     - Defined int32 in a way that ensures that the build dies if we can't
       find a variable with at least 32 bits.
 
-    - The daemon's "read only" config item now sets an internal read_only
-      variable that makes extra sure that no write/delete calls on the
-      read-only side can succeed.
-
   PROTOCOL DIFFERENCES FOR VERSION 29:
 
     - A 16-bit flag-word is transmitted after every file-list index.  This
@@ -287,12 +317,23 @@ Changes since 2.6.3:
       generator now transmits an index and a flag-word to indicate when
       dirs and symlinks have changed (instead of producing a message),
       which makes the outputting of the information more consistent and
-      less prone to screen corruption (because either the receiver or the
-      sender is now outputting all the file-change info).
+      less prone to screen corruption (because the local receiver/sender is
+      now outputting all the file-change info messages).
 
-    - If --inplace is specified, the generator flags any transfer that is
-      using an alternate basis file so that the sender can use the entire
-      basis file in the rsync algorithm (unlike a normal --inplace update).
+    - If a file is being hard-linked, the ITEM_XNAME_FOLLOWS bit is enabled
+      in the flag-word and the name of the file that was linked immediately
+      follows in vstring format (see below).
+
+    - If a file is being transferred with an alternate-basis file, the
+      ITEM_BASIS_TYPE_FOLLOWS bit is enabled in the flag-word and a single
+      byte follows, indicating what type of basis file was chosen.  If that
+      indicates that a fuzzy-match was selected, the ITEM_XNAME_FOLLOWS bit
+      is set in the flag-word and the name of the match in vstring format
+      follows the basis byte.  A vstring is a variable length string that
+      has its size written prior to the string, and no terminating null.
+      If the string is from 1-127 bytes, the length is a single byte.  If
+      it is from 128-32767 bytes, the length is written as ((len >> 8) |
+      0x80) followed by (len % 0x100).
 
     - The sending of exclude names is done using filter-rule syntax.  This
       means that all names have a prefixed rule indicator, even excludes
@@ -316,18 +357,14 @@ Changes since 2.6.3:
       build the file-list, and one for how long it took to send it over the
       wire (each expressed in thousandths of a second).
 
-    - When --delete-excluded is specified with some filter excludes, a
-      client sender will now initiate a send of the filter rules to the
-      receiver (older protocols used to omit the sending of excludes in
+    - When --delete-excluded is specified with some filter rules (AKA
+      excludes), a client sender will now initiate a send of the rules to
+      the receiver (older protocols used to omit the sending of excludes in
       this situation since there were no receiver-specific rules that
       survived --delete-excluded back then).  Note that, as with all the
       filter-list sending, only items that are significant to the other
       side will actually be sent over the wire, so the filter-rule list
-      is often empty in this scenario.
-
-    - A protocol-29 batch file includes a bit for the setting of the --dirs
-      option.  Also, the shell script created by --write-batch will use the
-      --filter option instead of --exclude-from to capture any filter rules.
+      that is sent in this scenario is often empty.
 
     - An index equal to the file-list count is sent as a keep-alive packet
       from the generator to the sender, which then forwards it on to the
@@ -335,6 +372,11 @@ Changes since 2.6.3:
       packet if the 16-bit flag-word that follows it contains a single bit
       (ITEM_IS_NEW, which is normally an illegal flag to appear alone).
 
+    - A protocol-29 batch file includes a bit for the setting of the --dirs
+      option and for the setting of the --compress option.  Also, the shell
+      script created by --write-batch will use the --filter option instead
+      of --exclude-from to capture any filter rules.
+
   BUILD CHANGES:
 
     - Handle an operating system that use mkdev() in place of makedev().

OLDNEWS

@@ -1042,7 +1042,8 @@ Changes since 2.4.6:
       build farm.
 
 Partial Protocol History
-	RELEASE DATE	VER.	DATE OF COMMIT	PROTOCOL
+	RELEASE DATE	VER.	DATE OF COMMIT*	PROTOCOL
+	30 Mar 2005	2.6.4	17 Jan 2005	29
 	30 Sep 2004	2.6.3			28
 	30 Apr 2004	2.6.2			28
 	26 Apr 2004	2.6.1	08 Jan 2004	28

batch.c

@@ -18,21 +18,22 @@ extern int preserve_devices;
 extern int preserve_uid;
 extern int preserve_gid;
 extern int always_checksum;
+extern int do_compression;
 extern int protocol_version;
 extern char *batch_name;
 
 extern struct filter_list_struct filter_list;
 
-static int fudged_recurse;
-
 static int *flag_ptr[] = {
-	&fudged_recurse,
-	&preserve_uid,
-	&preserve_gid,
-	&preserve_links,
-	&preserve_devices,
-	&preserve_hard_links,
-	&always_checksum,
+	&recurse,		/* 0 */
+	&preserve_uid,		/* 1 */
+	&preserve_gid,		/* 2 */
+	&preserve_links,	/* 3 */
+	&preserve_devices,	/* 4 */
+	&preserve_hard_links,	/* 5 */
+	&always_checksum,	/* 6 */
+	&xfer_dirs,		/* 7 (protocol 29) */
+	&do_compression,	/* 8 (protocol 29) */
 	NULL
 };
 
@@ -45,6 +46,7 @@ static char *flag_name[] = {
 	"--hard-links (-H)",
 	"--checksum (-c)",
 	"--dirs (-d)",
+	"--compress (-z)",
 	NULL
 };
 
@@ -54,7 +56,8 @@ void write_stream_flags(int fd)
 
 	/* Start the batch file with a bitmap of data-stream-affecting
 	 * flags. */
-	fudged_recurse = recurse < 0;
+	if (protocol_version < 29)
+		flag_ptr[7] = NULL;
 	for (i = 0, flags = 0; flag_ptr[i]; i++) {
 		if (*flag_ptr[i])
 			flags |= 1 << i;
@@ -66,9 +69,8 @@ void read_stream_flags(int fd)
 {
 	int i, flags;
 
-	fudged_recurse = recurse < 0;
 	if (protocol_version < 29)
-		xfer_dirs = 0;
+		flag_ptr[7] = NULL;
 	for (i = 0, flags = read_int(fd); flag_ptr[i]; i++) {
 		int set = flags & (1 << i) ? 1 : 0;
 		if (*flag_ptr[i] != set) {
@@ -80,9 +82,12 @@ void read_stream_flags(int fd)
 			*flag_ptr[i] = set;
 		}
 	}
-	recurse = fudged_recurse ? -1 : 0;
-	if (protocol_version < 29)
-		xfer_dirs = recurse ? 1 : 0;
+	if (protocol_version < 29) {
+		if (recurse)
+			xfer_dirs |= 1;
+		else if (xfer_dirs < 2)
+			xfer_dirs = 0;
+	}
 }
 
 static void write_arg(int fd, char *arg)

cleanup.c

@@ -136,10 +136,12 @@ void _exit_cleanup(int code, const char *file, int line)
 	}
 
 	if (code == 0) {
-		if ((io_error & ~IOERR_VANISHED) || log_got_error)
-			code = RERR_PARTIAL;
-		else if (io_error)
+		if (io_error & IOERR_DEL_LIMIT)
+			code = RERR_DEL_LIMIT;
+		if (io_error & IOERR_VANISHED)
 			code = RERR_VANISHED;
+		if (io_error & IOERR_GENERAL || log_got_error)
+			code = RERR_PARTIAL;
 	}
 
 	if (code)

clientserver.c

@@ -283,10 +283,10 @@ static int rsync_module(int f_in, int f_out, int i)
 		read_only = 1;
 
 	if (lp_transfer_logging(i)) {
-		if (strstr(lp_log_format(i), "%i") != NULL)
+		if (log_format_has(lp_log_format(i), 'i'))
 			daemon_log_format_has_i = 1;
 		if (daemon_log_format_has_i
-		    || strstr(lp_log_format(i), "%o") != NULL)
+		    || log_format_has(lp_log_format(i), 'o'))
 			daemon_log_format_has_o_or_i = 1;
 	}
 

compat.c

@@ -30,10 +30,13 @@ int remote_protocol = 0;
 extern int verbose;
 extern int am_server;
 extern int am_sender;
+extern int inplace;
 extern int fuzzy_basis;
 extern int read_batch;
 extern int checksum_seed;
+extern int basis_dir_cnt;
 extern int protocol_version;
+extern char *dest_option;
 
 void setup_protocol(int f_out,int f_in)
 {
@@ -77,11 +80,25 @@ void setup_protocol(int f_out,int f_in)
 
 	if (fuzzy_basis && protocol_version < 29) {
 		rprintf(FERROR,
-			"--fuzzy requres protocol 29 or higher (negotiated %d).\n",
+			"--fuzzy requires protocol 29 or higher (negotiated %d).\n",
 			protocol_version);
 		exit_cleanup(RERR_PROTOCOL);
 	}
 
+	if (basis_dir_cnt && inplace && protocol_version < 29) {
+		rprintf(FERROR,
+			"%s with --inplace requires protocol 29 or higher (negotiated %d).\n",
+			dest_option, protocol_version);
+		exit_cleanup(RERR_PROTOCOL);
+	}
+
+	if (basis_dir_cnt > 1 && protocol_version < 29) {
+		rprintf(FERROR,
+			"Multiple %s options requires protocol 29 or higher (negotiated %d).\n",
+			dest_option, protocol_version);
+		exit_cleanup(RERR_PROTOCOL);
+	}
+
 	if (am_server) {
 		if (!checksum_seed)
 			checksum_seed = time(NULL);

configure.in

@@ -5,7 +5,7 @@ AC_CONFIG_SRCDIR([byteorder.h])
 AC_CONFIG_HEADER(config.h)
 AC_PREREQ(2.59)
 
-RSYNC_VERSION=2.6.4pre2
+RSYNC_VERSION=2.6.4
 AC_SUBST(RSYNC_VERSION)
 AC_MSG_NOTICE([Configuring rsync $RSYNC_VERSION])
 

errcode.h

@@ -28,6 +28,7 @@
 #define RERR_FILESELECT 3       /* errors selecting input/output files, dirs */
 #define RERR_UNSUPPORTED 4      /* requested action not supported */
 #define RERR_STARTCLIENT 5      /* error starting client-server protocol */
+#define RERR_LOG_FAILURE 6      /* daemon unable to append to log-file */
 
 #define RERR_SOCKETIO   10      /* error in socket IO */
 #define RERR_FILEIO     11      /* error in file IO */
@@ -40,6 +41,7 @@
 #define RERR_MALLOC     22      /* error allocating core memory buffers */
 #define RERR_PARTIAL    23      /* partial transfer */
 #define RERR_VANISHED   24      /* file(s) vanished on sender side */
+#define RERR_DEL_LIMIT  25      /* skipped some deletes due to --max-delete */
 
 #define RERR_TIMEOUT    30      /* timeout in data send/receive */
 

flist.c

@@ -27,8 +27,6 @@
 
 #include "rsync.h"
 
-extern struct stats stats;
-
 extern int verbose;
 extern int dry_run;
 extern int list_only;
@@ -59,6 +57,8 @@ extern int copy_unsafe_links;
 extern int protocol_version;
 extern int sanitize_paths;
 extern int orig_umask;
+extern struct stats stats;
+extern struct file_list *the_file_list;
 
 extern char curr_dir[MAXPATHLEN];
 
@@ -69,8 +69,9 @@ int io_error;
 dev_t filesystem_dev; /* used to implement -x */
 
 static char empty_sum[MD4_SUM_LENGTH];
+static int flist_count_offset;
 static unsigned int file_struct_len;
-static struct file_list *received_flist, *sorting_flist;
+static struct file_list *sorting_flist;
 
 static void clean_flist(struct file_list *flist, int strip_root, int no_dups);
 static void output_flist(struct file_list *flist);
@@ -98,16 +99,16 @@ static void start_filelist_progress(char *kind)
 }
 
 
-static void emit_filelist_progress(const struct file_list *flist)
+static void emit_filelist_progress(int count)
 {
-	rprintf(FINFO, " %d files...\r", flist->count);
+	rprintf(FINFO, " %d files...\r", count);
 }
 
 
-static void maybe_emit_filelist_progress(const struct file_list *flist)
+static void maybe_emit_filelist_progress(int count)
 {
-	if (do_progress && show_filelist_p() && (flist->count % 100) == 0)
-		emit_filelist_progress(flist);
+	if (do_progress && show_filelist_p() && (count % 100) == 0)
+		emit_filelist_progress(count);
 }
 
 
@@ -943,7 +944,7 @@ skip_filters:
 		STRUCT_STAT st2;
 		int save_mode = file->mode;
 		file->mode = S_IFDIR; /* find a directory w/our name */
-		if (flist_find(received_flist, file) >= 0
+		if (flist_find(the_file_list, file) >= 0
 		    && do_stat(thisname, &st2) == 0 && S_ISDIR(st2.st_mode)) {
 			file->modtime = st2.st_mtime;
 			file->length = st2.st_size;
@@ -962,17 +963,16 @@ skip_filters:
 }
 
 
-void send_file_name(int f, struct file_list *flist, char *fname,
-		    int recursive, unsigned short base_flags)
+static struct file_struct *send_file_name(int f, struct file_list *flist,
+					  char *fname, unsigned short base_flags)
 {
 	struct file_struct *file;
-	char fbuf[MAXPATHLEN];
 
 	file = make_file(fname, flist, f == -2 ? SERVER_FILTERS : ALL_FILTERS);
 	if (!file)
-		return;
+		return NULL;
 
-	maybe_emit_filelist_progress(flist);
+	maybe_emit_filelist_progress(flist->count + flist_count_offset);
 
 	flist_expand(flist);
 
@@ -980,8 +980,15 @@ void send_file_name(int f, struct file_list *flist, char *fname,
 		flist->files[flist->count++] = file;
 		send_file_entry(file, f, base_flags);
 	}
+	return file;
+}
 
-	if (recursive && S_ISDIR(file->mode)
+static void send_if_directory(int f, struct file_list *flist,
+			      struct file_struct *file)
+{
+	char fbuf[MAXPATHLEN];
+
+	if (S_ISDIR(file->mode)
 	    && !(file->flags & FLAG_MOUNT_POINT) && f_name_to(file, fbuf)) {
 		void *save_filters;
 		unsigned int len = strlen(fbuf);
@@ -1000,13 +1007,11 @@ void send_file_name(int f, struct file_list *flist, char *fname,
 }
 
 
-/* Note that the "recurse" value either contains -1, for infinite recursion, or
- * a number >= 0 indicating how many levels of recursion we will allow.  This
- * function is normally called by the sender, but the receiving side also calls
- * it from delete_in_dir() with f set to -1 so that we just construct the file
- * list in memory without sending it over the wire.  Also, get_dirlist() might
- * call this with f set to -2, which indicates that local filter rules should
- * be ignored. */
+/* This function is normally called by the sender, but the receiving side also
+ * calls it from get_dirlist() with f set to -1 so that we just construct the
+ * file list in memory without sending it over the wire.  Also, get_dirlist()
+ * might call this with f set to -2, which also indicates that local filter
+ * rules should be ignored. */
 static void send_directory(int f, struct file_list *flist,
 			   char *fbuf, int len)
 {
@@ -1014,6 +1019,7 @@ static void send_directory(int f, struct file_list *flist,
 	unsigned remainder;
 	char *p;
 	DIR *d;
+	int start = flist->count;
 
 	if (!(d = opendir(fbuf))) {
 		io_error |= IOERR_GENERAL;
@@ -1032,10 +1038,9 @@ static void send_directory(int f, struct file_list *flist,
 		if (dname[0] == '.' && (dname[1] == '\0'
 		    || (dname[1] == '.' && dname[2] == '\0')))
 			continue;
-		if (strlcpy(p, dname, remainder) < remainder) {
-			int do_subdirs = recurse >= 1 ? recurse-- : recurse;
-			send_file_name(f, flist, fbuf, do_subdirs, 0);
-		} else {
+		if (strlcpy(p, dname, remainder) < remainder)
+			send_file_name(f, flist, fbuf, 0);
+		else {
 			io_error |= IOERR_GENERAL;
 			rprintf(FINFO,
 				"cannot send long-named file %s\n",
@@ -1051,6 +1056,12 @@ static void send_directory(int f, struct file_list *flist,
 	}
 
 	closedir(d);
+
+	if (recurse) {
+		int i, end = flist->count - 1;
+		for (i = start; i <= end; i++)
+			send_if_directory(f, flist, flist->files[i]);
+	}
 }
 
 
@@ -1084,9 +1095,10 @@ struct file_list *send_file_list(int f, int argc, char *argv[])
 	}
 
 	while (1) {
+		struct file_struct *file;
 		char fname2[MAXPATHLEN];
 		char *fname = fname2;
-		int do_subdirs;
+		int is_dot_dir;
 
 		if (use_ff_fd) {
 			if (read_filesfrom_line(filesfrom_fd, fname) == 0)
@@ -1109,12 +1121,11 @@ struct file_list *send_file_list(int f, int argc, char *argv[])
 				fname[l++] = '.';
 				fname[l] = '\0';
 			}
+			is_dot_dir = 1;
+		} else {
+			is_dot_dir = fname[l-1] == '.'
+				   && (l == 1 || fname[l-2] == '/');
 		}
-		if (fname[l-1] == '.' && (l == 1 || fname[l-2] == '/')) {
-			if (!recurse && xfer_dirs)
-				recurse = 1; /* allow one level */
-		} else if (recurse > 0)
-			recurse = 0;
 
 		if (link_stat(fname, &st, keep_dirlinks) != 0) {
 			io_error |= IOERR_GENERAL;
@@ -1162,7 +1173,7 @@ struct file_list *send_file_list(int f, int argc, char *argv[])
 				xfer_dirs = 1;
 				while ((slash = strchr(slash+1, '/')) != 0) {
 					*slash = 0;
-					send_file_name(f, flist, fname, 0, 0);
+					send_file_name(f, flist, fname, 0);
 					*slash = '/';
 				}
 				copy_links = save_copy_links;
@@ -1201,8 +1212,10 @@ struct file_list *send_file_list(int f, int argc, char *argv[])
 		if (one_file_system)
 			filesystem_dev = st.st_dev;
 
-		do_subdirs = recurse >= 1 ? recurse-- : recurse;
-		send_file_name(f, flist, fname, do_subdirs, XMIT_TOP_DIR);
+		if ((file = send_file_name(f, flist, fname, XMIT_TOP_DIR))) {
+			if (recurse || (xfer_dirs && is_dot_dir))
+				send_if_directory(f, flist, file);
+		}
 
 		if (olddir[0]) {
 			flist_dir = NULL;
@@ -1275,7 +1288,6 @@ struct file_list *recv_file_list(int f)
 	start_read = stats.total_read;
 
 	flist = flist_new(WITH_HLINK, "recv_file_list");
-	received_flist = flist;
 
 	flist->count = 0;
 	flist->malloced = 1000;
@@ -1298,7 +1310,7 @@ struct file_list *recv_file_list(int f)
 
 		flist->files[flist->count++] = file;
 
-		maybe_emit_filelist_progress(flist);
+		maybe_emit_filelist_progress(flist->count);
 
 		if (verbose > 2) {
 			rprintf(FINFO, "recv_file_name(%s)\n",
@@ -1675,18 +1687,6 @@ int f_name_cmp(struct file_struct *f1, struct file_struct *f2)
 		if (!*c2) {
 			switch (state2) {
 			case s_DIR:
-				if (state1 == s_SLASH && sorting_flist) {
-					int j;
-					/* Optimize for future comparisons. */
-					for (j = 0;
-					     j < sorting_flist->count;
-					     j++) {
-						struct file_struct *fp
-						    = sorting_flist->files[j];
-						if (fp->dirname == f2->dirname)
-							fp->dirname = f1->dirname;
-					}
-				}
 				state2 = s_SLASH;
 				c2 = (uchar*)"/";
 				break;
@@ -1773,6 +1773,8 @@ struct file_list *get_dirlist(char *dirname, int dlen,
 	recurse = 0;
 	send_directory(ignore_filter_rules ? -2 : -1, dirlist, dirname, dlen);
 	recurse = save_recurse;
+	if (do_progress)
+		flist_count_offset += dirlist->count;
 
 	clean_flist(dirlist, 0, 0);
 

generator.c

@@ -31,6 +31,7 @@ extern int daemon_log_format_has_i;
 extern int am_root;
 extern int am_server;
 extern int am_daemon;
+extern int do_progress;
 extern int recurse;
 extern int relative_paths;
 extern int keep_dirlinks;
@@ -48,6 +49,7 @@ extern int delete_after;
 extern int module_id;
 extern int ignore_errors;
 extern int remove_sent_files;
+extern int delay_updates;
 extern int update_only;
 extern int opt_ignore_existing;
 extern int inplace;
@@ -58,6 +60,7 @@ extern int size_only;
 extern OFF_T max_size;
 extern int io_timeout;
 extern int io_error;
+extern int sock_f_out;
 extern int ignore_timeout;
 extern int protocol_version;
 extern int fuzzy_basis;
@@ -65,9 +68,9 @@ extern int always_checksum;
 extern char *partial_dir;
 extern char *basis_dir[];
 extern int compare_dest;
+extern int copy_dest;
 extern int link_dest;
 extern int whole_file;
-extern int local_server;
 extern int list_only;
 extern int read_batch;
 extern int only_existing;
@@ -82,9 +85,11 @@ extern dev_t filesystem_dev;
 extern char *backup_dir;
 extern char *backup_suffix;
 extern int backup_suffix_len;
-
+extern struct file_list *the_file_list;
 extern struct filter_list_struct server_filter_list;
 
+int allowed_lull = 0;
+
 static int deletion_count = 0; /* used to implement --max-delete */
 
 
@@ -105,10 +110,9 @@ static int delete_item(char *fname, int mode, int flags)
 	int j, dlen, zap_dir, ok;
 	void *save_filters;
 
-	if (max_delete && deletion_count >= max_delete)
-		return -1;
-
 	if (!S_ISDIR(mode)) {
+		if (max_delete && ++deletion_count > max_delete)
+			return 0;
 		if (make_backups && (backup_dir || !is_backup_file(fname)))
 			ok = make_backup(fname);
 		else
@@ -116,11 +120,12 @@ static int delete_item(char *fname, int mode, int flags)
 		if (ok) {
 			if (!(flags & DEL_TERSE))
 				log_delete(fname, mode);
-			deletion_count++;
 			return 0;
 		}
-		if (errno == ENOENT)
+		if (errno == ENOENT) {
+			deletion_count--;
 			return 0;
+		}
 		rsyserr(FERROR, errno, "delete_file: unlink %s failed",
 			full_fname(fname));
 		return -1;
@@ -128,7 +133,8 @@ static int delete_item(char *fname, int mode, int flags)
 
 	zap_dir = (flags & DEL_FORCE_RECURSE || (force_delete && recurse))
 		&& !(flags & DEL_NO_RECURSE);
-	if (dry_run && zap_dir) {
+	if ((max_delete && ++deletion_count > max_delete)
+	    || (dry_run && zap_dir)) {
 		ok = 0;
 		errno = ENOTEMPTY;
 	} else if (make_backups && !backup_dir && !is_backup_file(fname)
@@ -139,17 +145,19 @@ static int delete_item(char *fname, int mode, int flags)
 	if (ok) {
 		if (!(flags & DEL_TERSE))
 			log_delete(fname, mode);
-		deletion_count++;
 		return 0;
 	}
-	if (errno == ENOENT)
+	if (errno == ENOENT) {
+		deletion_count--;
 		return 0;
+	}
 	if (!zap_dir || (errno != ENOTEMPTY && errno != EEXIST)) {
 		rsyserr(FERROR, errno, "delete_file: rmdir %s failed",
 			full_fname(fname));
 		return -1;
 	}
-	flags |= DEL_FORCE_RECURSE;
+	flags |= DEL_FORCE_RECURSE; /* mark subdir dels as not "in the way" */
+	deletion_count--;
 
 	dlen = strlcpy(buf, fname, MAXPATHLEN);
 	save_filters = push_local_filters(buf, dlen);
@@ -171,13 +179,12 @@ static int delete_item(char *fname, int mode, int flags)
 
 	pop_local_filters(save_filters);
 
-	if (max_delete && deletion_count >= max_delete)
-		return -1;
+	if (max_delete && ++deletion_count > max_delete)
+		return 0;
 
 	if (do_rmdir(fname) == 0) {
 		if (!(flags & DEL_TERSE))
 			log_delete(fname, mode);
-		deletion_count++;
 	} else if (errno != ENOTEMPTY && errno != ENOENT) {
 		rsyserr(FERROR, errno, "delete_file: rmdir %s failed",
 			full_fname(fname));
@@ -194,10 +201,11 @@ static int delete_item(char *fname, int mode, int flags)
  * call will append names onto the end, but the old dir value will be restored
  * on exit). */
 static void delete_in_dir(struct file_list *flist, char *fbuf,
-			  struct file_struct *file, int allowed_lull)
+			  struct file_struct *file)
 {
 	static int min_depth = MAXPATHLEN, cur_depth = -1;
 	static void *filt_array[MAXPATHLEN/2+1];
+	static int already_warned = 0;
 	struct file_list *dirlist;
 	char delbuf[MAXPATHLEN];
 	STRUCT_STAT st;
@@ -215,18 +223,17 @@ static void delete_in_dir(struct file_list *flist, char *fbuf,
 		rprintf(FINFO, "delete_in_dir(%s)\n", safe_fname(fbuf));
 
 	if (allowed_lull)
-		maybe_send_keepalive(allowed_lull, flist->count);
+		maybe_send_keepalive();
 
 	if (file->dir.depth >= MAXPATHLEN/2+1)
 		return; /* Impossible... */
 
-	if (max_delete && deletion_count >= max_delete)
-		return;
-
 	if (io_error && !(lp_ignore_errors(module_id) || ignore_errors)) {
+		if (already_warned)
+			return;
 		rprintf(FINFO,
 			"IO error encountered -- skipping file deletion\n");
-		max_delete = -1; /* avoid duplicating the above warning */
+		already_warned = 1;
 		return;
 	}
 
@@ -264,11 +271,14 @@ static void delete_in_dir(struct file_list *flist, char *fbuf,
 
 /* This deletes any files on the receiving side that are not present on the
  * sending side.  This is used by --delete-before and --delete-after. */
-static void do_delete_pass(struct file_list *flist, int allowed_lull)
+static void do_delete_pass(struct file_list *flist)
 {
 	char fbuf[MAXPATHLEN];
 	int j;
 
+	if (dry_run > 1) /* destination doesn't exist yet */
+		return;
+
 	for (j = 0; j < flist->count; j++) {
 		struct file_struct *file = flist->files[j];
 
@@ -279,7 +289,7 @@ static void do_delete_pass(struct file_list *flist, int allowed_lull)
 		if (verbose > 1 && file->flags & FLAG_TOP_DIR)
 			rprintf(FINFO, "deleting in %s\n", safe_fname(fbuf));
 
-		delete_in_dir(flist, fbuf, file, allowed_lull);
+		delete_in_dir(flist, fbuf, file);
 	}
 }
 
@@ -299,26 +309,18 @@ static int unchanged_attrs(struct file_struct *file, STRUCT_STAT *st)
 }
 
 
-#define SID_UPDATING		  ITEM_UPDATING
-#define SID_REPORT_CHECKSUM	  ITEM_REPORT_CHECKSUM
-#define SID_USING_ALT_BASIS	  ITEM_USING_ALT_BASIS
-/* This flag doesn't get sent, so it must be outside 0xffff. */
-#define SID_NO_DEST_AND_NO_UPDATE (1<<16)
-
-static void itemize(struct file_struct *file, int statret, STRUCT_STAT *st,
-		    int32 sflags, int f_out, int ndx)
+void itemize(struct file_struct *file, int ndx, int statret, STRUCT_STAT *st,
+	     int32 iflags, uchar fnamecmp_type, char *xname)
 {
-	int iflags = sflags & 0xffff;
-
-	if (statret >= 0) {
+	if (statret == 0) {
 		if (S_ISREG(file->mode) && file->length != st->st_size)
 			iflags |= ITEM_REPORT_SIZE;
-		if (!(sflags & SID_NO_DEST_AND_NO_UPDATE)) {
+		if (!(iflags & ITEM_NO_DEST_AND_NO_UPDATE)) {
 			int keep_time = !preserve_times ? 0
 			    : S_ISDIR(file->mode) ? !omit_dir_times
 			    : !S_ISLNK(file->mode);
 
-			if ((iflags & ITEM_UPDATING && !keep_time)
+			if ((iflags & (ITEM_TRANSFER|ITEM_LOCAL_CHANGE) && !keep_time)
 			    || (keep_time && file->modtime != st->st_mtime))
 				iflags |= ITEM_REPORT_TIME;
 			if (preserve_perms && file->mode != st->st_mode)
@@ -330,15 +332,21 @@ static void itemize(struct file_struct *file, int statret, STRUCT_STAT *st,
 				iflags |= ITEM_REPORT_GROUP;
 		}
 	} else
-		iflags |= ITEM_IS_NEW | ITEM_UPDATING;
+		iflags |= ITEM_IS_NEW;
 
-	if ((iflags || verbose > 1) && !read_batch) {
+	iflags &= 0xffff;
+	if ((iflags & SIGNIFICANT_ITEM_FLAGS || verbose > 1
+	  || (xname && *xname)) && !read_batch) {
 		if (protocol_version >= 29) {
 			if (ndx >= 0)
-				write_int(f_out, ndx);
-			write_shortint(f_out, iflags);
+				write_int(sock_f_out, ndx);
+			write_shortint(sock_f_out, iflags);
+			if (iflags & ITEM_BASIS_TYPE_FOLLOWS)
+				write_byte(sock_f_out, fnamecmp_type);
+			if (iflags & ITEM_XNAME_FOLLOWS)
+				write_vstring(sock_f_out, xname, strlen(xname));
 		} else if (ndx >= 0)
-			log_recv(file, &stats, iflags);
+			log_item(file, &stats, iflags, xname);
 	}
 }
 
@@ -378,7 +386,7 @@ static int unchanged_file(char *fn, struct file_struct *file, STRUCT_STAT *st)
  * The block size is a rounded square root of file length.
  *
  * The checksum size is determined according to:
- *     blocksum_bits = BLOCKSUM_EXP + 2*log2(file_len) - log2(block_len)
+ *     blocksum_bits = BLOCKSUM_BIAS + 2*log2(file_len) - log2(block_len)
  * provided by Donovan Baarda which gives a probability of rsync
  * algorithm corrupting data and falling back using the whole md4
  * checksums.
@@ -498,7 +506,7 @@ static int find_fuzzy(struct file_struct *file, struct file_list *dirlist)
 {
 	int fname_len, fname_suf_len;
 	const char *fname_suf, *fname = file->basename;
-	uint32 lowest_dist = 0x7FFFFFFF;
+	uint32 lowest_dist = 25 << 16; /* ignore a distance greater than 25 */
 	int j, lowest_j = -1;
 
 	fname_len = strlen(fname);
@@ -546,8 +554,26 @@ static int find_fuzzy(struct file_struct *file, struct file_list *dirlist)
 	return lowest_j;
 }
 
+void check_for_finished_hlinks(int itemizing, enum logcode code)
+{
+	struct file_struct *file;
+	int ndx;
 
-/* Acts on flist->file's ndx'th item, whose name is fname.  If a directory,
+	while ((ndx = get_hlink_num()) != -1) {
+		if (ndx < 0 || ndx >= the_file_list->count)
+			continue;
+
+		file = the_file_list->files[ndx];
+		if (!file->link_u.links)
+			continue;
+
+		hard_link_cluster(file, ndx, itemizing, code);
+	}
+}
+
+static int phase = 0;
+
+/* Acts on the_file_list->file's ndx'th item, whose name is fname.  If a dir,
  * make sure it exists, and has the right permissions/timestamp info.  For
  * all other non-regular files (symlinks, etc.) we create them here.  For
  * regular files that have changed, we try to find a basis file and then
@@ -555,20 +581,18 @@ static int find_fuzzy(struct file_struct *file, struct file_list *dirlist)
  *
  * Note that f_out is set to -1 when doing final directory-permission and
  * modification-time repair. */
-static void recv_generator(char *fname, struct file_list *flist,
-			   struct file_struct *file, int ndx,
+static void recv_generator(char *fname, struct file_struct *file, int ndx,
 			   int itemizing, int maybe_PERMS_REPORT,
-			   enum logcode code, int allowed_lull,
-			   int f_out, int f_out_name)
+			   enum logcode code, int f_out)
 {
 	static int missing_below = -1, excluded_below = -1;
-	static char *fuzzy_dirname = NULL;
+	static char *fuzzy_dirname = "";
 	static struct file_list *fuzzy_dirlist = NULL;
 	struct file_struct *fuzzy_file = NULL;
 	int fd = -1, f_copy = -1;
-	STRUCT_STAT st, partial_st;
+	STRUCT_STAT st, real_st, partial_st;
 	struct file_struct *back_file = NULL;
-	int statret, stat_errno;
+	int statret, real_ret, stat_errno;
 	char *fnamecmp, *partialptr, *backupptr = NULL;
 	char fnamecmpbuf[MAXPATHLEN];
 	uchar fnamecmp_type;
@@ -580,7 +604,7 @@ static void recv_generator(char *fname, struct file_list *flist,
 		if (fuzzy_dirlist) {
 			flist_free(fuzzy_dirlist);
 			fuzzy_dirlist = NULL;
-			fuzzy_dirname = NULL;
+			fuzzy_dirname = "";
 		}
 		if (missing_below >= 0) {
 			dry_run--;
@@ -624,15 +648,13 @@ static void recv_generator(char *fname, struct file_list *flist,
 	} else {
 		if (fuzzy_basis && S_ISREG(file->mode)) {
 			char *dn = file->dirname ? file->dirname : ".";
-			/* Yes, identical dirnames are guaranteed to have
-			 * identical pointers at this point. */
-			if (fuzzy_dirname != dn) {
+			if (fuzzy_dirname != dn
+			    && strcmp(fuzzy_dirname, dn) != 0) {
 				if (fuzzy_dirlist)
 					flist_free(fuzzy_dirlist);
-				fuzzy_dirname = dn;
-				fuzzy_dirlist = get_dirlist(fuzzy_dirname, -1,
-							    1);
+				fuzzy_dirlist = get_dirlist(dn, -1, 1);
 			}
+			fuzzy_dirname = dn;
 		}
 
 		statret = link_stat(fname, &st,
@@ -665,15 +687,18 @@ static void recv_generator(char *fname, struct file_list *flist,
 		 * we need to delete it.  If it doesn't exist, then
 		 * (perhaps recursively) create it. */
 		if (statret == 0 && !S_ISDIR(st.st_mode)) {
-			delete_item(fname, st.st_mode, DEL_TERSE);
+			if (delete_item(fname, st.st_mode, DEL_TERSE) < 0)
+				return;
 			statret = -1;
 		}
 		if (dry_run && statret != 0 && missing_below < 0) {
 			missing_below = file->dir.depth;
 			dry_run++;
 		}
-		if (itemizing && f_out != -1)
-			itemize(file, statret, &st, 0, f_out, ndx);
+		if (itemizing && f_out != -1) {
+			itemize(file, ndx, statret, &st,
+				statret ? ITEM_LOCAL_CHANGE : 0, 0, NULL);
+		}
 		if (statret != 0 && do_mkdir(fname,file->mode) != 0 && errno != EEXIST) {
 			if (!relative_paths || errno != ENOENT
 			    || create_directory_path(fname, orig_umask) < 0
@@ -686,9 +711,9 @@ static void recv_generator(char *fname, struct file_list *flist,
 		if (set_perms(fname, file, statret ? NULL : &st, 0)
 		    && verbose && code && f_out != -1)
 			rprintf(code, "%s/\n", safe_fname(fname));
-		if (delete_during && f_out != -1 && csum_length != SUM_LENGTH
+		if (delete_during && f_out != -1 && !phase && dry_run < 2
 		    && (file->flags & FLAG_DEL_HERE))
-			delete_in_dir(flist, fname, file, allowed_lull);
+			delete_in_dir(the_file_list, fname, file);
 		return;
 	}
 
@@ -723,8 +748,8 @@ static void recv_generator(char *fname, struct file_list *flist,
 				 * required. */
 				if (strcmp(lnk, file->u.link) == 0) {
 					if (itemizing) {
-						itemize(file, 0, &st, 0,
-							f_out, ndx);
+						itemize(file, ndx, 0, &st, 0,
+							0, NULL);
 					}
 					set_perms(fname, file, &st,
 						  maybe_PERMS_REPORT);
@@ -733,12 +758,10 @@ static void recv_generator(char *fname, struct file_list *flist,
 			}
 			/* Not the right symlink (or not a symlink), so
 			 * delete it. */
-			if (S_ISLNK(st.st_mode))
-				delete_item(fname, st.st_mode, DEL_TERSE);
-			else {
-				delete_item(fname, st.st_mode, DEL_TERSE);
+			if (delete_item(fname, st.st_mode, DEL_TERSE) < 0)
+				return;
+			if (!S_ISLNK(st.st_mode))
 				statret = -1;
-			}
 		}
 		if (do_symlink(file->u.link,fname) != 0) {
 			rsyserr(FERROR, errno, "symlink %s -> \"%s\" failed",
@@ -746,8 +769,8 @@ static void recv_generator(char *fname, struct file_list *flist,
 		} else {
 			set_perms(fname,file,NULL,0);
 			if (itemizing) {
-				itemize(file, statret, &st, SID_UPDATING,
-					f_out, ndx);
+				itemize(file, ndx, statret, &st,
+					ITEM_LOCAL_CHANGE, 0, NULL);
 			}
 			if (code && verbose) {
 				rprintf(code, "%s -> %s\n", safe_fname(fname),
@@ -767,7 +790,8 @@ static void recv_generator(char *fname, struct file_list *flist,
 		if (statret != 0 ||
 		    st.st_mode != file->mode ||
 		    st.st_rdev != file->u.rdev) {
-			delete_item(fname, st.st_mode, DEL_TERSE);
+			if (delete_item(fname, st.st_mode, DEL_TERSE) < 0)
+				return;
 			if (!IS_DEVICE(st.st_mode))
 				statret = -1;
 			if (verbose > 2) {
@@ -781,8 +805,8 @@ static void recv_generator(char *fname, struct file_list *flist,
 			} else {
 				set_perms(fname,file,NULL,0);
 				if (itemizing) {
-					itemize(file, statret, &st, SID_UPDATING,
-						f_out, ndx);
+					itemize(file, ndx, statret, &st,
+						ITEM_LOCAL_CHANGE, 0, NULL);
 				}
 				if (code && verbose) {
 					rprintf(code, "%s\n",
@@ -790,16 +814,14 @@ static void recv_generator(char *fname, struct file_list *flist,
 				}
 			}
 		} else {
-			if (itemizing) {
-				itemize(file, statret, &st, 0,
-					f_out, ndx);
-			}
+			if (itemizing)
+				itemize(file, ndx, statret, &st, 0, 0, NULL);
 			set_perms(fname, file, &st, maybe_PERMS_REPORT);
 		}
 		return;
 	}
 
-	if (preserve_hard_links && hard_link_check(file, HL_CHECK_MASTER))
+	if (preserve_hard_links && hard_link_check(file, ndx, HL_CHECK_MASTER))
 		return;
 
 	if (!S_ISREG(file->mode)) {
@@ -808,64 +830,22 @@ static void recv_generator(char *fname, struct file_list *flist,
 		return;
 	}
 
+	if (opt_ignore_existing && statret == 0) {
+		if (verbose > 1)
+			rprintf(FINFO, "%s exists\n", safe_fname(fname));
+		return;
+	}
+
+	if (update_only && statret == 0
+	    && cmp_modtime(st.st_mtime, file->modtime) > 0) {
+		if (verbose > 1)
+			rprintf(FINFO, "%s is newer\n", safe_fname(fname));
+		return;
+	}
+
 	fnamecmp = fname;
 	fnamecmp_type = FNAMECMP_FNAME;
 
-	if (statret != 0 && basis_dir[0] != NULL) {
-		int fallback_match = -1;
-		int match_level = 0;
-		int i = 0;
-		do {
-			pathjoin(fnamecmpbuf, sizeof fnamecmpbuf,
-				 basis_dir[i], fname);
-			if (link_stat(fnamecmpbuf, &st, 0) == 0
-			    && S_ISREG(st.st_mode)) {
-				statret = 0;
-				if (link_dest) {
-					if (!match_level) {
-						fallback_match = i;
-						match_level = 1;
-					} else if (match_level == 2
-					    && !unchanged_attrs(file, &st))
-						continue;
-					if (!unchanged_file(fnamecmpbuf, file, &st))
-						continue;
-					fallback_match = i;
-					match_level = 2;
-					if (!unchanged_attrs(file, &st))
-						continue;
-				}
-				match_level = 3;
-				break;
-			}
-		} while (basis_dir[++i] != NULL);
-		if (statret == 0) {
-			if (match_level < 3) {
-				i = fallback_match;
-				pathjoin(fnamecmpbuf, sizeof fnamecmpbuf,
-					 basis_dir[i], fname);
-			}
-#ifdef HAVE_LINK
-			if (link_dest && match_level == 3 && !dry_run) {
-				if (do_link(fnamecmpbuf, fname) < 0) {
-					if (verbose) {
-						rsyserr(FINFO, errno,
-							"link %s => %s",
-							full_fname(fnamecmpbuf),
-							safe_fname(fname));
-					}
-					fnamecmp = fnamecmpbuf;
-					fnamecmp_type = i;
-				}
-			} else
-#endif
-			{
-				fnamecmp = fnamecmpbuf;
-				fnamecmp_type = i;
-			}
-		}
-	}
-
 	if (statret == 0 && !S_ISREG(st.st_mode)) {
 		if (delete_item(fname, st.st_mode, DEL_TERSE) != 0)
 			return;
@@ -873,6 +853,88 @@ static void recv_generator(char *fname, struct file_list *flist,
 		stat_errno = ENOENT;
 	}
 
+	if (statret != 0 && basis_dir[0] != NULL) {
+		int best_match = -1;
+		int match_level = 0;
+		int i = 0;
+		do {
+			pathjoin(fnamecmpbuf, sizeof fnamecmpbuf,
+				 basis_dir[i], fname);
+			if (link_stat(fnamecmpbuf, &st, 0) < 0
+			    || !S_ISREG(st.st_mode))
+				continue;
+			switch (match_level) {
+			case 0:
+				best_match = i;
+				match_level = 1;
+				/* FALL THROUGH */
+			case 1:
+				if (!unchanged_file(fnamecmpbuf, file, &st))
+					continue;
+				best_match = i;
+				match_level = 2;
+				if (copy_dest)
+					break;
+				/* FALL THROUGH */
+			case 2:
+				if (!unchanged_attrs(file, &st))
+					continue;
+				best_match = i;
+				match_level = 3;
+				break;
+			}
+			break;
+		} while (basis_dir[++i] != NULL);
+		if (match_level) {
+			statret = 0;
+			if (i != best_match) {
+				i = best_match;
+				pathjoin(fnamecmpbuf, sizeof fnamecmpbuf,
+					 basis_dir[i], fname);
+				if (link_stat(fnamecmpbuf, &st, 0) < 0) {
+					match_level = 0;
+					statret = -1;
+					stat_errno = errno;
+				}
+			}
+#ifdef HAVE_LINK
+			if (link_dest && match_level == 3) {
+				if (hard_link_one(file, ndx, fname, -1, &st,
+						  fnamecmpbuf, 1,
+						  itemizing && verbose > 1,
+						  code) == 0)
+					return;
+				if (verbose) {
+					rsyserr(FINFO, errno, "link %s => %s",
+						full_fname(fnamecmpbuf),
+						safe_fname(fname));
+				}
+				match_level = 2;
+			}
+#endif
+			if (match_level == 2) {
+				/* Copy the file locally. */
+				if (copy_file(fnamecmpbuf, fname, file->mode) < 0) {
+					if (verbose) {
+						rsyserr(FINFO, errno,
+							"copy_file %s => %s",
+							full_fname(fnamecmpbuf),
+							safe_fname(fname));
+					}
+					match_level = 0;
+					statret = -1;
+				} else
+					set_perms(fname, file, NULL, 0);
+			} else if (compare_dest || match_level == 1) {
+				fnamecmp = fnamecmpbuf;
+				fnamecmp_type = i;
+			}
+		}
+	}
+
+	real_ret = statret;
+	real_st = st;
+
 	if (partial_dir && (partialptr = partial_dir_fname(fname)) != NULL
 	    && link_stat(partialptr, &partial_st, 0) == 0
 	    && S_ISREG(partial_st.st_mode)) {
@@ -890,9 +952,7 @@ static void recv_generator(char *fname, struct file_list *flist,
 				rprintf(FINFO, "fuzzy basis selected for %s: %s\n",
 					safe_fname(fname), safe_fname(fnamecmpbuf));
 			}
-			st.st_mode = fuzzy_file->mode;
 			st.st_size = fuzzy_file->length;
-			st.st_mtime = fuzzy_file->modtime;
 			statret = 0;
 			fnamecmp = fnamecmpbuf;
 			fnamecmp_type = FNAMECMP_FUZZY;
@@ -900,7 +960,7 @@ static void recv_generator(char *fname, struct file_list *flist,
 	}
 
 	if (statret != 0) {
-		if (preserve_hard_links && hard_link_check(file, HL_SKIP))
+		if (preserve_hard_links && hard_link_check(file, ndx, HL_SKIP))
 			return;
 		if (stat_errno == ENOENT)
 			goto notify_others;
@@ -912,32 +972,24 @@ static void recv_generator(char *fname, struct file_list *flist,
 		return;
 	}
 
-	if (opt_ignore_existing && fnamecmp_type == FNAMECMP_FNAME) {
-		if (verbose > 1)
-			rprintf(FINFO, "%s exists\n", safe_fname(fname));
-		return;
-	}
-
-	if (update_only && fnamecmp_type == FNAMECMP_FNAME
-	    && cmp_modtime(st.st_mtime, file->modtime) > 0) {
-		if (verbose > 1)
-			rprintf(FINFO, "%s is newer\n", safe_fname(fname));
-		return;
-	}
-
 	if (!compare_dest && fnamecmp_type <= FNAMECMP_BASIS_DIR_HIGH)
 		;
 	else if (fnamecmp_type == FNAMECMP_FUZZY)
 		;
 	else if (unchanged_file(fnamecmp, file, &st)) {
-		if (itemizing) {
-			itemize(file, statret, &st,
-				fnamecmp_type == FNAMECMP_FNAME
-					       ? 0 : SID_NO_DEST_AND_NO_UPDATE,
-				f_out, ndx);
-		}
-		if (fnamecmp_type == FNAMECMP_FNAME)
+		if (fnamecmp_type == FNAMECMP_FNAME) {
+			if (itemizing) {
+				itemize(file, ndx, real_ret, &real_st,
+					0, 0, NULL);
+			}
 			set_perms(fname, file, &st, maybe_PERMS_REPORT);
+			if (preserve_hard_links && file->link_u.links)
+				hard_link_cluster(file, ndx, itemizing, code);
+			return;
+		}
+		/* Only --compare-dest gets here. */
+		itemize(file, ndx, real_ret, &real_st,
+			ITEM_NO_DEST_AND_NO_UPDATE, 0, NULL);
 		return;
 	}
 
@@ -949,13 +1001,8 @@ prepare_to_open:
 		statret = 0;
 	}
 
-	if (dry_run || read_batch)
+	if (dry_run || read_batch || whole_file)
 		goto notify_others;
-	if (whole_file > 0) {
-		if (statret == 0)
-			statret = 1;
-		goto notify_others;
-	}
 
 	if (fuzzy_basis) {
 		int j = flist_find(fuzzy_dirlist, file);
@@ -971,13 +1018,13 @@ prepare_to_open:
 			full_fname(fnamecmp));
 	    pretend_missing:
 		/* pretend the file didn't exist */
-		if (preserve_hard_links && hard_link_check(file, HL_SKIP))
+		if (preserve_hard_links && hard_link_check(file, ndx, HL_SKIP))
 			return;
-		statret = -1;
+		statret = real_ret = -1;
 		goto notify_others;
 	}
 
-	if (inplace && make_backups) {
+	if (inplace && make_backups && fnamecmp_type == FNAMECMP_FNAME) {
 		if (!(backupptr = get_backup_name(fname))) {
 			close(fd);
 			return;
@@ -1015,59 +1062,49 @@ prepare_to_open:
 notify_others:
 	write_int(f_out, ndx);
 	if (itemizing) {
-		int iflags = SID_UPDATING;
+		int iflags = ITEM_TRANSFER;
 		if (always_checksum)
-			iflags |= SID_REPORT_CHECKSUM;
+			iflags |= ITEM_REPORT_CHECKSUM;
 		if (fnamecmp_type != FNAMECMP_FNAME)
-			iflags |= SID_USING_ALT_BASIS;
-		itemize(file, statret, &st, iflags, f_out, -1);
-	}
-	if (f_out_name >= 0) {
-		write_byte(f_out_name, fnamecmp_type);
-		if (fnamecmp_type == FNAMECMP_FUZZY) {
-			uchar lenbuf[3], *lb = lenbuf;
-			int len = strlen(fuzzy_file->basename);
-			if (len > 0x7F) {
-#if MAXPATHLEN > 0x7FFF
-				*lb++ = len / 0x10000 + 0x80;
-				*lb++ = len / 0x100;
-#else
-				*lb++ = len / 0x100 + 0x80;
-#endif
-			}
-			*lb = len;
-			write_buf(f_out_name, (char*)lenbuf, lb - lenbuf + 1);
-			write_buf(f_out_name, fuzzy_file->basename, len);
-		}
+			iflags |= ITEM_BASIS_TYPE_FOLLOWS;
+		if (fnamecmp_type == FNAMECMP_FUZZY)
+			iflags |= ITEM_XNAME_FOLLOWS;
+		itemize(file, -1, real_ret, &real_st, iflags, fnamecmp_type,
+			fuzzy_file ? fuzzy_file->basename : NULL);
 	}
 
-	if (dry_run || read_batch)
+	if (dry_run) {
+		if (preserve_hard_links && file->link_u.links)
+			hard_link_cluster(file, ndx, itemizing, code);
+		return;
+	}
+	if (read_batch)
 		return;
 
-	if (statret == 0) {
-		generate_and_send_sums(fd, st.st_size, f_out, f_copy);
-
-		if (f_copy >= 0) {
-			close(f_copy);
-			set_perms(backupptr, back_file, NULL, 0);
-			if (verbose > 1) {
-				rprintf(FINFO, "backed up %s to %s\n",
-					safe_fname(fname), safe_fname(backupptr));
-			}
-			free(back_file);
-		}
-
-		close(fd);
-	} else
+	if (statret != 0 || whole_file) {
 		write_sum_head(f_out, NULL);
+		return;
+	}
+
+	generate_and_send_sums(fd, st.st_size, f_out, f_copy);
+
+	if (f_copy >= 0) {
+		close(f_copy);
+		set_perms(backupptr, back_file, NULL, 0);
+		if (verbose > 1) {
+			rprintf(FINFO, "backed up %s to %s\n",
+				safe_fname(fname), safe_fname(backupptr));
+		}
+		free(back_file);
+	}
+
+	close(fd);
 }
 
 
-void generate_files(int f_out, struct file_list *flist, char *local_name,
-		    int f_out_name)
+void generate_files(int f_out, struct file_list *flist, char *local_name)
 {
-	int i;
-	int phase = 0;
+	int i, lull_mod;
 	char fbuf[MAXPATHLEN];
 	int itemizing, maybe_PERMS_REPORT;
 	enum logcode code;
@@ -1075,8 +1112,10 @@ void generate_files(int f_out, struct file_list *flist, char *local_name,
 	int need_retouch_dir_perms = 0;
 	int save_only_existing = only_existing;
 	int save_opt_ignore_existing = opt_ignore_existing;
-	int allowed_lull = read_batch ? 0 : (io_timeout + 1) / 2;
-	int lull_mod = allowed_lull * 5;
+	int save_do_progress = do_progress;
+
+	allowed_lull = read_batch ? 0 : (io_timeout + 1) / 2;
+	lull_mod = allowed_lull * 5;
 
 	if (protocol_version >= 29) {
 		itemizing = 1;
@@ -1102,11 +1141,14 @@ void generate_files(int f_out, struct file_list *flist, char *local_name,
 	}
 
 	if (delete_before && !local_name && flist->count > 0)
-		do_delete_pass(flist, allowed_lull);
+		do_delete_pass(flist);
+	do_progress = 0;
 
+	if (whole_file < 0)
+		whole_file = 0;
 	if (verbose >= 2) {
 		rprintf(FINFO, "delta-transmission %s\n",
-			whole_file > 0
+			whole_file
 			? "disabled for local transfer or --whole-file"
 			: "enabled");
 	}
@@ -1116,31 +1158,37 @@ void generate_files(int f_out, struct file_list *flist, char *local_name,
 
 	for (i = 0; i < flist->count; i++) {
 		struct file_struct *file = flist->files[i];
-		struct file_struct copy;
 
 		if (!file->basename)
 			continue;
 
+		recv_generator(local_name ? local_name : f_name_to(file, fbuf),
+			       file, i, itemizing, maybe_PERMS_REPORT, code,
+			       f_out);
+
 		/* We need to ensure that any dirs we create have writeable
 		 * permissions during the time we are putting files within
 		 * them.  This is then fixed after the transfer is done. */
 		if (!am_root && S_ISDIR(file->mode) && !(file->mode & S_IWUSR)) {
-			copy = *file;
-			copy.mode |= S_IWUSR; /* user write */
-			file = &copy;
+			int mode = file->mode | S_IWUSR; /* user write */
+			char *fname = local_name ? local_name : fbuf;
+			if (do_chmod(fname, mode & CHMOD_BITS) < 0) {
+				rsyserr(FERROR, errno,
+					"failed to modify permissions on %s",
+					full_fname(fname));
+			}
 			need_retouch_dir_perms = 1;
 		}
 
-		recv_generator(local_name ? local_name : f_name_to(file, fbuf),
-			       flist, file, i, itemizing, maybe_PERMS_REPORT,
-			       code, allowed_lull, f_out, f_out_name);
+		if (preserve_hard_links)
+			check_for_finished_hlinks(itemizing, code);
 
 		if (allowed_lull && !(i % lull_mod))
-			maybe_send_keepalive(allowed_lull, flist->count);
+			maybe_send_keepalive();
 	}
-	recv_generator(NULL, NULL, NULL, 0, 0, 0, code, 0, -1, -1);
+	recv_generator(NULL, NULL, 0, 0, 0, code, -1);
 	if (delete_during)
-		delete_in_dir(NULL, NULL, NULL, 0);
+		delete_in_dir(NULL, NULL, NULL);
 
 	phase++;
 	csum_length = SUM_LENGTH;
@@ -1160,11 +1208,11 @@ void generate_files(int f_out, struct file_list *flist, char *local_name,
 
 	/* files can cycle through the system more than once
 	 * to catch initial checksum errors */
-	while ((i = get_redo_num()) != -1) {
+	while ((i = get_redo_num(itemizing, code)) != -1) {
 		struct file_struct *file = flist->files[i];
 		recv_generator(local_name ? local_name : f_name_to(file, fbuf),
-			       flist, file, i, itemizing, maybe_PERMS_REPORT,
-			       code, allowed_lull, f_out, f_out_name);
+			       file, i, itemizing, maybe_PERMS_REPORT, code,
+			       f_out);
 	}
 
 	phase++;
@@ -1175,15 +1223,26 @@ void generate_files(int f_out, struct file_list *flist, char *local_name,
 		rprintf(FINFO,"generate_files phase=%d\n",phase);
 
 	write_int(f_out, -1);
+	/* Reduce round-trip lag-time for a useless delay-updates phase. */
+	if (protocol_version >= 29 && !delay_updates)
+		write_int(f_out, -1);
 
-	/* Read post-redo-phase MSG_DONE and any prior messages. */
-	get_redo_num();
+	/* Read MSG_DONE for the redo phase (and any prior messages). */
+	get_redo_num(itemizing, code);
 
-	if (preserve_hard_links)
-		do_hard_links(allowed_lull, flist->count);
+	if (protocol_version >= 29) {
+		phase++;
+		if (verbose > 2)
+			rprintf(FINFO, "generate_files phase=%d\n", phase);
+		if (delay_updates)
+			write_int(f_out, -1);
+		/* Read MSG_DONE for delay-updates phase & prior messages. */
+		get_redo_num(itemizing, code);
+	}
 
+	do_progress = save_do_progress;
 	if (delete_after && !local_name && flist->count > 0)
-		do_delete_pass(flist, allowed_lull);
+		do_delete_pass(flist);
 
 	if ((need_retouch_dir_perms || need_retouch_dir_times)
 	    && !list_only && !local_name && !dry_run) {
@@ -1197,15 +1256,20 @@ void generate_files(int f_out, struct file_list *flist, char *local_name,
 				continue;
 			if (!need_retouch_dir_times && file->mode & S_IWUSR)
 				continue;
-			recv_generator(local_name ? local_name : f_name(file),
-				       flist, file, i, itemizing,
-				       maybe_PERMS_REPORT, code, allowed_lull,
-				       -1, -1);
+			recv_generator(f_name(file), file, i, itemizing,
+				       maybe_PERMS_REPORT, code, -1);
 			if (allowed_lull && !(j++ % lull_mod))
-				maybe_send_keepalive(allowed_lull, flist->count);
+				maybe_send_keepalive();
 		}
 	}
-	recv_generator(NULL, NULL, NULL, 0, 0, 0, code, 0, -1, -1);
+	recv_generator(NULL, NULL, 0, 0, 0, code, -1);
+
+	if (max_delete > 0 && deletion_count > max_delete) {
+		rprintf(FINFO,
+			"Deletions stopped due to --max-delete limit (%d skipped)\n",
+			deletion_count - max_delete);
+		io_error |= IOERR_DEL_LIMIT;
+	}
 
 	if (verbose > 2)
 		rprintf(FINFO,"generate_files finished\n");

hlink.c

@@ -23,12 +23,18 @@
 extern int dry_run;
 extern int verbose;
 extern int make_backups;
+extern struct file_list *the_file_list;
 
 #ifdef SUPPORT_HARD_LINKS
-static int hlink_compare(struct file_struct **file1, struct file_struct **file2)
+
+#define FPTR(i) (the_file_list->files[i])
+#define LINKED(p1,p2) (FPTR(p1)->F_DEV == FPTR(p2)->F_DEV \
+		    && FPTR(p1)->F_INODE == FPTR(p2)->F_INODE)
+
+static int hlink_compare(int *int1, int *int2)
 {
-	struct file_struct *f1 = *file1;
-	struct file_struct *f2 = *file2;
+	struct file_struct *f1 = FPTR(*int1);
+	struct file_struct *f2 = FPTR(*int2);
 
 	if (f1->F_DEV != f2->F_DEV)
 		return (int) (f1->F_DEV > f2->F_DEV ? 1 : -1);
@@ -39,21 +45,17 @@ static int hlink_compare(struct file_struct **file1, struct file_struct **file2)
 	return f_name_cmp(f1, f2);
 }
 
-static struct file_struct **hlink_list;
+static int *hlink_list;
 static int hlink_count;
 
-#define LINKED(p1,p2) ((p1)->F_DEV == (p2)->F_DEV \
-		    && (p1)->F_INODE == (p2)->F_INODE)
-
 /* Analyze the data in the hlink_list[], remove items that aren't multiply
  * linked, and replace the dev+inode data with the hlindex+next linked list. */
-static void link_idev_data(struct file_list *flist)
+static void link_idev_data(void)
 {
-	struct file_struct *head;
-	int from, to, start;
+	int head, from, to, start;
 
 	alloc_pool_t hlink_pool;
-	alloc_pool_t idev_pool = flist->hlink_pool;
+	alloc_pool_t idev_pool = the_file_list->hlink_pool;
 
 	hlink_pool = pool_create(128 * 1024, sizeof (struct hlink),
 	    out_of_memory, POOL_INTERN);
@@ -63,26 +65,27 @@ static void link_idev_data(struct file_list *flist)
 		head = hlink_list[start];
 		while (from < hlink_count-1
 		    && LINKED(hlink_list[from], hlink_list[from+1])) {
-			pool_free(idev_pool, 0, hlink_list[from]->link_u.idev);
-			hlink_list[from]->link_u.links = pool_talloc(hlink_pool,
+			pool_free(idev_pool, 0, FPTR(hlink_list[from])->link_u.idev);
+			FPTR(hlink_list[from])->link_u.links = pool_talloc(hlink_pool,
 			    struct hlink, 1, "hlink_list");
 
-			hlink_list[from]->F_HLINDEX = to;
-			hlink_list[from]->F_NEXT = hlink_list[from+1];
+			FPTR(hlink_list[from])->F_HLINDEX = to;
+			FPTR(hlink_list[from])->F_NEXT = hlink_list[from+1];
 			from++;
 		}
 		if (from > start) {
-			pool_free(idev_pool, 0, hlink_list[from]->link_u.idev);
-			hlink_list[from]->link_u.links = pool_talloc(hlink_pool,
+			pool_free(idev_pool, 0, FPTR(hlink_list[from])->link_u.idev);
+			FPTR(hlink_list[from])->link_u.links = pool_talloc(hlink_pool,
 			    struct hlink, 1, "hlink_list");
 
-			hlink_list[from]->F_HLINDEX = to;
-			hlink_list[from]->F_NEXT = head;
-			hlink_list[from]->flags |= FLAG_HLINK_EOL;
+			FPTR(head)->flags |= FLAG_HLINK_TOL;
+			FPTR(hlink_list[from])->F_HLINDEX = to;
+			FPTR(hlink_list[from])->F_NEXT = head;
+			FPTR(hlink_list[from])->flags |= FLAG_HLINK_EOL;
 			hlink_list[to++] = head;
 		} else {
-			pool_free(idev_pool, 0, head->link_u.idev);
-			head->link_u.idev = NULL;
+			pool_free(idev_pool, 0, FPTR(head)->link_u.idev);
+			FPTR(head)->link_u.idev = NULL;
 		}
 	}
 
@@ -93,33 +96,33 @@ static void link_idev_data(struct file_list *flist)
 		hlink_pool = NULL;
 	} else {
 		hlink_count = to;
-		if (!(hlink_list = realloc_array(hlink_list,
-		    struct file_struct *, hlink_count)))
+		hlink_list = realloc_array(hlink_list, int, hlink_count);
+		if (!hlink_list)
 			out_of_memory("init_hard_links");
 	}
-	flist->hlink_pool = hlink_pool;
+	the_file_list->hlink_pool = hlink_pool;
 	pool_destroy(idev_pool);
 }
 #endif
 
-void init_hard_links(struct file_list *flist)
+void init_hard_links(void)
 {
 #ifdef SUPPORT_HARD_LINKS
 	int i;
 
-	if (flist->count < 2)
+	if (the_file_list->count < 2)
 		return;
 
 	if (hlink_list)
 		free(hlink_list);
 
-	if (!(hlink_list = new_array(struct file_struct *, flist->count)))
+	if (!(hlink_list = new_array(int, the_file_list->count)))
 		out_of_memory("init_hard_links");
 
 	hlink_count = 0;
-	for (i = 0; i < flist->count; i++) {
-		if (flist->files[i]->link_u.idev)
-			hlink_list[hlink_count++] = flist->files[i];
+	for (i = 0; i < the_file_list->count; i++) {
+		if (FPTR(i)->link_u.idev)
+			hlink_list[hlink_count++] = i;
 	}
 
 	qsort(hlink_list, hlink_count,
@@ -129,19 +132,19 @@ void init_hard_links(struct file_list *flist)
 		free(hlink_list);
 		hlink_list = NULL;
 	} else
-		link_idev_data(flist);
+		link_idev_data();
 #endif
 }
 
-int hard_link_check(struct file_struct *file, int skip)
+int hard_link_check(struct file_struct *file, int ndx, int skip)
 {
 #ifdef SUPPORT_HARD_LINKS
 	if (!hlink_list || !file->link_u.links)
 		return 0;
 	if (skip && !(file->flags & FLAG_HLINK_EOL))
 		hlink_list[file->F_HLINDEX] = file->F_NEXT;
-	if (hlink_list[file->F_HLINDEX] != file) {
-		if (verbose > 1) {
+	if (hlink_list[file->F_HLINDEX] != ndx) {
+		if (verbose > 2) {
 			rprintf(FINFO, "\"%s\" is a hard link\n",
 				safe_fname(f_name(file)));
 		}
@@ -152,63 +155,79 @@ int hard_link_check(struct file_struct *file, int skip)
 }
 
 #ifdef SUPPORT_HARD_LINKS
-static void hard_link_one(char *hlink1, char *hlink2)
+int hard_link_one(struct file_struct *file, int ndx, char *fname,
+		  int statret, STRUCT_STAT *st, char *toname, int terse,
+		  int itemizing, enum logcode code)
 {
-	if (do_link(hlink1, hlink2)) {
+	if (do_link(toname, fname)) {
 		if (verbose) {
-			rsyserr(FINFO, errno, "link %s => %s failed",
-				full_fname(hlink2), safe_fname(hlink1));
+			rsyserr(FERROR, errno, "link %s => %s failed",
+				full_fname(fname), safe_fname(toname));
 		}
+		return -1;
 	}
-	else if (verbose)
-		rprintf(FINFO, "%s => %s\n", safe_fname(hlink2), safe_fname(hlink1));
+
+	if (itemizing) {
+		itemize(file, ndx, statret, st,
+			ITEM_LOCAL_CHANGE | ITEM_XNAME_FOLLOWS, 0,
+			terse ? "" : toname);
+	}
+	if (code && verbose && !terse) {
+		rprintf(code, "%s => %s\n",
+			safe_fname(fname), safe_fname(toname));
+	}
+	return 0;
 }
 #endif
 
 
-
-/**
- * Create any hard links in the global hlink_list.  They were put
- * there by running init_hard_links on the filelist.
- **/
-void do_hard_links(int allowed_lull, int flist_count)
+void hard_link_cluster(struct file_struct *file, int master, int itemizing,
+		       enum logcode code)
 {
 #ifdef SUPPORT_HARD_LINKS
-	struct file_struct *file, *first;
 	char hlink1[MAXPATHLEN];
 	char *hlink2;
 	STRUCT_STAT st1, st2;
-	int i;
+	int statret, ndx = master;
 
-	if (!hlink_list)
+	if (link_stat(f_name_to(file, hlink1), &st1, 0) < 0)
 		return;
-
-	for (i = 0; i < hlink_count; i++) {
-		first = file = hlink_list[i];
-		if (link_stat(f_name_to(first, hlink1), &st1, 0) < 0)
-			continue;
-		while ((file = file->F_NEXT) != first) {
-			hlink2 = f_name(file);
-			if (link_stat(hlink2, &st2, 0) == 0) {
-				if (st2.st_dev == st1.st_dev
-				    && st2.st_ino == st1.st_ino)
-					continue;
-				if (make_backups) {
-					if (!make_backup(hlink2))
-						continue;
-				} else if (robust_unlink(hlink2)) {
-					if (verbose > 0) {
-						rsyserr(FINFO, errno,
-							"unlink %s failed",
-							full_fname(hlink2));
-					}
-					continue;
-				}
-			}
-			hard_link_one(hlink1, hlink2);
+	if (!(file->flags & FLAG_HLINK_TOL)) {
+		while (!(file->flags & FLAG_HLINK_EOL)) {
+			ndx = file->F_NEXT;
+			file = FPTR(ndx);
 		}
-		if (allowed_lull)
-			maybe_send_keepalive(allowed_lull, flist_count);
 	}
+	do {
+		ndx = file->F_NEXT;
+		file = FPTR(ndx);
+		if (ndx == master)
+			continue;
+		hlink2 = f_name(file);
+		if ((statret = link_stat(hlink2, &st2, 0)) == 0) {
+			if (st2.st_dev == st1.st_dev
+			    && st2.st_ino == st1.st_ino) {
+				if (itemizing) {
+					itemize(file, ndx, statret, &st2,
+						ITEM_LOCAL_CHANGE | ITEM_XNAME_FOLLOWS,
+						0, "");
+				}
+				continue;
+			}
+			if (make_backups) {
+				if (!make_backup(hlink2))
+					continue;
+			} else if (robust_unlink(hlink2)) {
+				if (verbose > 0) {
+					rsyserr(FINFO, errno,
+						"unlink %s failed",
+						full_fname(hlink2));
+				}
+				continue;
+			}
+		}
+		hard_link_one(file, ndx, hlink2, statret,
+			      &st2, hlink1, 0, itemizing, code);
+	} while (!(file->flags & FLAG_HLINK_EOL));
 #endif
 }

io.c

@@ -43,6 +43,7 @@ extern int bwlimit;
 extern size_t bwlimit_writemax;
 extern int verbose;
 extern int io_timeout;
+extern int allowed_lull;
 extern int am_server;
 extern int am_daemon;
 extern int am_sender;
@@ -51,8 +52,11 @@ extern int eol_nulls;
 extern int csum_length;
 extern int checksum_seed;
 extern int protocol_version;
+extern int remove_sent_files;
+extern int preserve_hard_links;
 extern char *filesfrom_host;
 extern struct stats stats;
+extern struct file_list *the_file_list;
 
 const char phase_unknown[] = "unknown";
 int select_timeout = SELECT_TIMEOUT;
@@ -81,11 +85,11 @@ int kluge_around_eof = 0;
 
 int msg_fd_in = -1;
 int msg_fd_out = -1;
+int sock_f_in = -1;
+int sock_f_out = -1;
 
 static int io_multiplexing_out;
 static int io_multiplexing_in;
-static int sock_f_in = -1;
-static int sock_f_out = -1;
 static time_t last_io;
 static int no_flush;
 
@@ -98,16 +102,20 @@ static char io_filesfrom_buf[2048];
 static char *io_filesfrom_bp;
 static char io_filesfrom_lastchar;
 static int io_filesfrom_buflen;
+static size_t contiguous_write_len = 0;
 
 static void read_loop(int fd, char *buf, size_t len);
 
-struct redo_list {
-	struct redo_list *next;
-	int num;
+struct flist_ndx_item {
+	struct flist_ndx_item *next;
+	int ndx;
 };
 
-static struct redo_list *redo_list_head;
-static struct redo_list *redo_list_tail;
+struct flist_ndx_list {
+	struct flist_ndx_item *head, *tail;
+};
+
+static struct flist_ndx_list redo_list, hlink_list;
 
 struct msg_list {
 	struct msg_list *next;
@@ -118,19 +126,37 @@ struct msg_list {
 static struct msg_list *msg_list_head;
 static struct msg_list *msg_list_tail;
 
-static void redo_list_add(int num)
+static void flist_ndx_push(struct flist_ndx_list *lp, int ndx)
 {
-	struct redo_list *rl;
+	struct flist_ndx_item *item;
 
-	if (!(rl = new(struct redo_list)))
-		exit_cleanup(RERR_MALLOC);
-	rl->next = NULL;
-	rl->num = num;
-	if (redo_list_tail)
-		redo_list_tail->next = rl;
+	if (!(item = new(struct flist_ndx_item)))
+		out_of_memory("flist_ndx_push");
+	item->next = NULL;
+	item->ndx = ndx;
+	if (lp->tail)
+		lp->tail->next = item;
 	else
-		redo_list_head = rl;
-	redo_list_tail = rl;
+		lp->head = item;
+	lp->tail = item;
+}
+
+static int flist_ndx_pop(struct flist_ndx_list *lp)
+{
+	struct flist_ndx_item *next;
+	int ndx;
+
+	if (!lp->head)
+		return -1;
+
+	ndx = lp->head->ndx;
+	next = lp->head->next;
+	free(lp->head);
+	lp->head = next;
+	if (!next)
+		lp->tail = NULL;
+
+	return ndx;
 }
 
 static void check_timeout(void)
@@ -187,10 +213,10 @@ static void msg_list_add(int code, char *buf, int len)
 	struct msg_list *ml;
 
 	if (!(ml = new(struct msg_list)))
-		exit_cleanup(RERR_MALLOC);
+		out_of_memory("msg_list_add");
 	ml->next = NULL;
 	if (!(ml->buf = new_array(char, len+4)))
-		exit_cleanup(RERR_MALLOC);
+		out_of_memory("msg_list_add");
 	SIVAL(ml->buf, 0, ((code+MPLEX_BASE)<<24) | len);
 	memcpy(ml->buf+4, buf, len);
 	ml->len = len+4;
@@ -223,7 +249,7 @@ static void read_msg_fd(void)
 	int tag, len;
 
 	/* Temporarily disable msg_fd_in.  This is needed to avoid looping back
-	 * to this routine from read_timeout() and writefd_unbuffered(). */
+	 * to this routine from writefd_unbuffered(). */
 	msg_fd_in = -1;
 
 	read_loop(fd, buf, 4);
@@ -238,7 +264,7 @@ static void read_msg_fd(void)
 			rprintf(FERROR, "invalid message %d:%d\n", tag, len);
 			exit_cleanup(RERR_STREAMIO);
 		}
-		redo_list_add(-1);
+		flist_ndx_push(&redo_list, -1);
 		break;
 	case MSG_REDO:
 		if (len != 4 || !am_generator) {
@@ -246,7 +272,7 @@ static void read_msg_fd(void)
 			exit_cleanup(RERR_STREAMIO);
 		}
 		read_loop(fd, buf, 4);
-		redo_list_add(IVAL(buf,0));
+		flist_ndx_push(&redo_list, IVAL(buf,0));
 		break;
 	case MSG_DELETED:
 		if (len >= (int)sizeof buf || !am_generator) {
@@ -262,7 +288,10 @@ static void read_msg_fd(void)
 			exit_cleanup(RERR_STREAMIO);
 		}
 		read_loop(fd, buf, len);
-		io_multiplex_write(MSG_SUCCESS, buf, len);
+		if (remove_sent_files)
+			io_multiplex_write(MSG_SUCCESS, buf, len);
+		if (preserve_hard_links)
+			flist_ndx_push(&hlink_list, IVAL(buf,0));
 		break;
 	case MSG_INFO:
 	case MSG_ERROR:
@@ -324,22 +353,22 @@ int msg_list_push(int flush_it_all)
 	return 1;
 }
 
-int get_redo_num(void)
+int get_redo_num(int itemizing, enum logcode code)
 {
-	struct redo_list *next;
-	int num;
-
-	while (!redo_list_head)
+	while (1) {
+		if (hlink_list.head)
+			check_for_finished_hlinks(itemizing, code);
+		if (redo_list.head)
+			break;
 		read_msg_fd();
+	}
 
-	num = redo_list_head->num;
-	next = redo_list_head->next;
-	free(redo_list_head);
-	redo_list_head = next;
-	if (!next)
-		redo_list_tail = NULL;
+	return flist_ndx_pop(&redo_list);
+}
 
-	return num;
+int get_hlink_num(void)
+{
+	return flist_ndx_pop(&hlink_list);
 }
 
 /**
@@ -419,11 +448,7 @@ static int read_timeout(int fd, char *buf, size_t len)
 		FD_ZERO(&r_fds);
 		FD_ZERO(&w_fds);
 		FD_SET(fd, &r_fds);
-		if (msg_fd_in >= 0) {
-			FD_SET(msg_fd_in, &r_fds);
-			if (msg_fd_in > maxfd)
-				maxfd = msg_fd_in;
-		} else if (msg_list_head) {
+		if (msg_list_head) {
 			FD_SET(msg_fd_out, &w_fds);
 			if (msg_fd_out > maxfd)
 				maxfd = msg_fd_out;
@@ -460,9 +485,7 @@ static int read_timeout(int fd, char *buf, size_t len)
 			continue;
 		}
 
-		if (msg_fd_in >= 0 && FD_ISSET(msg_fd_in, &r_fds))
-			read_msg_fd();
-		else if (msg_list_head && FD_ISSET(msg_fd_out, &w_fds))
+		if (msg_list_head && FD_ISSET(msg_fd_out, &w_fds))
 			msg_list_push(NORMAL_FLUSH);
 
 		if (io_filesfrom_f_out >= 0) {
@@ -642,13 +665,13 @@ void io_end_buffering(void)
 }
 
 
-void maybe_send_keepalive(int allowed_lull, int ndx)
+void maybe_send_keepalive(void)
 {
 	if (time(NULL) - last_io >= allowed_lull) {
 		if (!iobuf_out || !iobuf_out_cnt) {
 			if (protocol_version < 29)
 				return; /* there's nothing we can do */
-			write_int(sock_f_out, ndx);
+			write_int(sock_f_out, the_file_list->count);
 			write_shortint(sock_f_out, ITEM_IS_NEW);
 		}
 		if (iobuf_out)
@@ -737,8 +760,8 @@ static int readfd_unbuffered(int fd, char *buf, size_t len)
 			break;
 		case MSG_SUCCESS:
 			if (remaining != 4) {
-				rprintf(FERROR, "invalid multi-message %d:%ld\n",
-					tag, (long)remaining);
+				rprintf(FERROR, "invalid multi-message %d:%ld [%s]\n",
+					tag, (long)remaining, who_am_i());
 				exit_cleanup(RERR_STREAMIO);
 			}
 			read_loop(fd, line, remaining);
@@ -749,8 +772,8 @@ static int readfd_unbuffered(int fd, char *buf, size_t len)
 		case MSG_ERROR:
 			if (remaining >= sizeof line) {
 				rprintf(FERROR,
-					"[%s] multiplexing overflow %d:%ld\n\n",
-					who_am_i(), tag, (long)remaining);
+					"multiplexing overflow %d:%ld [%s]\n",
+					tag, (long)remaining, who_am_i());
 				exit_cleanup(RERR_STREAMIO);
 			}
 			read_loop(fd, line, remaining);
@@ -758,8 +781,8 @@ static int readfd_unbuffered(int fd, char *buf, size_t len)
 			remaining = 0;
 			break;
 		default:
-			rprintf(FERROR, "[%s] unexpected tag %d\n",
-				who_am_i(), tag);
+			rprintf(FERROR, "unexpected tag %d [%s]\n",
+				tag, who_am_i());
 			exit_cleanup(RERR_STREAMIO);
 		}
 	}
@@ -845,7 +868,7 @@ void read_buf(int f,char *buf,size_t len)
 void read_sbuf(int f,char *buf,size_t len)
 {
 	readfd(f, buf, len);
-	buf[len] = 0;
+	buf[len] = '\0';
 }
 
 uchar read_byte(int f)
@@ -855,6 +878,25 @@ uchar read_byte(int f)
 	return c;
 }
 
+int read_vstring(int f, char *buf, int bufsize)
+{
+	int len = read_byte(f);
+
+	if (len & 0x80)
+		len = (len & ~0x80) * 0x100 + read_byte(f);
+
+	if (len >= bufsize) {
+		rprintf(FERROR, "over-long vstring received (%d > %d)\n",
+			len, bufsize - 1);
+		return -1;
+	}
+
+	if (len)
+		readfd(f, buf, len);
+	buf[len] = '\0';
+	return len;
+}
+
 /* Populate a sum_struct with values from the socket.  This is
  * called by both the sender and the receiver. */
 void read_sum_head(int f, struct sum_struct *sum)
@@ -862,20 +904,20 @@ void read_sum_head(int f, struct sum_struct *sum)
 	sum->count = read_int(f);
 	sum->blength = read_int(f);
 	if (sum->blength < 0 || sum->blength > MAX_BLOCK_SIZE) {
-		rprintf(FERROR, "[%s] Invalid block length %ld\n",
-			who_am_i(), (long)sum->blength);
+		rprintf(FERROR, "Invalid block length %ld [%s]\n",
+			(long)sum->blength, who_am_i());
 		exit_cleanup(RERR_PROTOCOL);
 	}
 	sum->s2length = protocol_version < 27 ? csum_length : (int)read_int(f);
 	if (sum->s2length < 0 || sum->s2length > MD4_SUM_LENGTH) {
-		rprintf(FERROR, "[%s] Invalid checksum length %d\n",
-			who_am_i(), sum->s2length);
+		rprintf(FERROR, "Invalid checksum length %d [%s]\n",
+			sum->s2length, who_am_i());
 		exit_cleanup(RERR_PROTOCOL);
 	}
 	sum->remainder = read_int(f);
 	if (sum->remainder < 0 || sum->remainder > sum->blength) {
-		rprintf(FERROR, "[%s] Invalid remainder length %ld\n",
-			who_am_i(), (long)sum->remainder);
+		rprintf(FERROR, "Invalid remainder length %ld [%s]\n",
+			(long)sum->remainder, who_am_i());
 		exit_cleanup(RERR_PROTOCOL);
 	}
 }
@@ -955,8 +997,8 @@ static void sleep_for_bwlimit(int bytes_written)
 
 
 /* Write len bytes to the file descriptor fd, looping as necessary to get
- * the job done and also (in the generator) reading any data on msg_fd_in
- * (to avoid deadlock).
+ * the job done and also (in certain circumstnces) reading any data on
+ * msg_fd_in to avoid deadlock.
  *
  * This function underlies the multiplexing system.  The body of the
  * application never calls this function directly. */
@@ -974,7 +1016,7 @@ static void writefd_unbuffered(int fd,char *buf,size_t len)
 		FD_SET(fd,&w_fds);
 		maxfd = fd;
 
-		if (msg_fd_in >= 0) {
+		if (msg_fd_in >= 0 && len-total >= contiguous_write_len) {
 			FD_ZERO(&r_fds);
 			FD_SET(msg_fd_in,&r_fds);
 			if (msg_fd_in > maxfd)
@@ -1003,14 +1045,8 @@ static void writefd_unbuffered(int fd,char *buf,size_t len)
 		if (msg_fd_in >= 0 && FD_ISSET(msg_fd_in, &r_fds))
 			read_msg_fd();
 
-		if (!FD_ISSET(fd, &w_fds)) {
-			if (fd != sock_f_out && iobuf_out_cnt) {
-				no_flush--;
-				io_flush(NORMAL_FLUSH);
-				no_flush++;
-			}
+		if (!FD_ISSET(fd, &w_fds))
 			continue;
-		}
 
 		n = len - total;
 		if (bwlimit && n > bwlimit_writemax)
@@ -1068,6 +1104,13 @@ static void mplex_write(enum msgcode code, char *buf, size_t len)
 
 	SIVAL(buffer, 0, ((MPLEX_BASE + (int)code)<<24) + len);
 
+	/* When the generator reads messages from the msg_fd_in pipe, it can
+	 * cause output to occur down the socket.  Setting contiguous_write_len
+	 * prevents the reading of msg_fd_in once we actually start to write
+	 * this sequence of data (though we might read it before the start). */
+	if (am_generator && msg_fd_in >= 0)
+		contiguous_write_len = len + 4;
+
 	if (n > sizeof buffer - 4)
 		n = sizeof buffer - 4;
 
@@ -1079,6 +1122,9 @@ static void mplex_write(enum msgcode code, char *buf, size_t len)
 
 	if (len)
 		writefd_unbuffered(sock_f_out, buf, len);
+
+	if (am_generator && msg_fd_in >= 0)
+		contiguous_write_len = 0;
 }
 
 
@@ -1187,19 +1233,37 @@ void write_buf(int f,char *buf,size_t len)
 	writefd(f,buf,len);
 }
 
-
 /** Write a string to the connection */
 void write_sbuf(int f, char *buf)
 {
 	writefd(f, buf, strlen(buf));
 }
 
-
 void write_byte(int f, uchar c)
 {
 	writefd(f, (char *)&c, 1);
 }
 
+void write_vstring(int f, char *str, int len)
+{
+	uchar lenbuf[3], *lb = lenbuf;
+
+	if (len > 0x7F) {
+		if (len > 0x7FFF) {
+			rprintf(FERROR,
+				"attempting to send over-long vstring (%d > %d)\n",
+				len, 0x7FFF);
+			exit_cleanup(RERR_PROTOCOL);
+		}
+		*lb++ = len / 0x100 + 0x80;
+	}
+	*lb = len;
+
+	writefd(f, (char*)lenbuf, lb - lenbuf + 1);
+	if (len)
+		writefd(f, str, len);
+}
+
 
 /**
  * Read a line of up to @p maxlen characters into @p buf (not counting

log.c

@@ -32,6 +32,7 @@ extern int dry_run;
 extern int am_daemon;
 extern int am_server;
 extern int am_sender;
+extern int local_server;
 extern int quiet;
 extern int module_id;
 extern int msg_fd_out;
@@ -58,6 +59,7 @@ struct {
 	{ RERR_FILESELECT , "errors selecting input/output files, dirs" },
 	{ RERR_UNSUPPORTED, "requested action not supported" },
 	{ RERR_STARTCLIENT, "error starting client-server protocol" },
+	{ RERR_LOG_FAILURE, "daemon unable to append to log-file" },
 	{ RERR_SOCKETIO   , "error in socket IO" },
 	{ RERR_FILEIO     , "error in file IO" },
 	{ RERR_STREAMIO   , "error in rsync protocol data stream" },
@@ -71,8 +73,9 @@ struct {
 	{ RERR_TIMEOUT    , "timeout in data send/receive" },
 	{ RERR_CMD_FAILED , "remote shell failed" },
 	{ RERR_CMD_KILLED , "remote shell killed" },
-	{ RERR_CMD_RUN,     "remote command could not be run" },
-	{ RERR_CMD_NOTFOUND, "remote command not found" },
+	{ RERR_CMD_RUN    , "remote command could not be run" },
+	{ RERR_CMD_NOTFOUND,"remote command not found" },
+	{ RERR_DEL_LIMIT  , "the --max-delete limit stopped deletions" },
 	{ 0, NULL }
 };
 
@@ -155,7 +158,7 @@ void log_open(void)
 		if (!logfile) {
 			am_daemon = 0; /* avoid trying to log again */
 			rsyserr(FERROR, errno, "fopen() of log-file failed");
-			exit_cleanup(RERR_FILESELECT);
+			exit_cleanup(RERR_LOG_FAILURE);
 		}
 	}
 }
@@ -237,7 +240,7 @@ void rwrite(enum logcode code, char *buf, int len)
 	if (buf[len-1] == '\r' || buf[len-1] == '\n')
 		fflush(f);
 }
-		
+
 
 /* This is the rsync debugging function. Call it with FINFO, FERROR or
  * FLOG. */
@@ -317,7 +320,7 @@ void rsyserr(enum logcode code, int errcode, const char *format, ...)
 void rflush(enum logcode code)
 {
 	FILE *f = NULL;
-	
+
 	if (am_daemon) {
 		return;
 	}
@@ -345,48 +348,74 @@ void rflush(enum logcode code)
 
 /* a generic logging routine for send/recv, with parameter
  * substitiution */
-static void log_formatted(enum logcode code,
-			  char *format, char *op, struct file_struct *file,
-			  struct stats *initial_stats, int iflags)
+static void log_formatted(enum logcode code, char *format, char *op,
+			  struct file_struct *file, struct stats *initial_stats,
+			  int iflags, char *hlink)
 {
-	char buf[MAXPATHLEN+1024];
-	char buf2[MAXPATHLEN];
-	char *p, *n;
+	char buf[MAXPATHLEN+1024], buf2[MAXPATHLEN], fmt[32];
+	char *p, *s, *n;
 	size_t len, total;
 	int64 b;
 
+	*fmt = '%';
+
 	/* We expand % codes one by one in place in buf.  We don't
-	 * copy in the terminating nul of the inserted strings, but
-	 * rather keep going until we reach the nul of the format. */
+	 * copy in the terminating null of the inserted strings, but
+	 * rather keep going until we reach the null of the format. */
 	total = strlcpy(buf, format, sizeof buf);
-	
-	for (p = buf; (p = strchr(p, '%')) != NULL && p[1]; ) {
+	if (total > MAXPATHLEN) {
+		rprintf(FERROR, "log-format string is WAY too long!\n");
+		exit_cleanup(RERR_MESSAGEIO);
+	}
+	buf[total++] = '\n';
+	buf[total] = '\0';
+
+	for (p = buf; (p = strchr(p, '%')) != NULL; ) {
+		s = p++;
+		n = fmt + 1;
+		if (*p == '-')
+			*n++ = *p++;
+		while (isdigit(*(uchar*)p) && n - fmt < (int)(sizeof fmt) - 8)
+			*n++ = *p++;
+		if (!*p)
+			break;
+		*n = '\0';
 		n = NULL;
 
-		switch (p[1]) {
+		switch (*p) {
 		case 'h': if (am_daemon) n = client_name(0); break;
 		case 'a': if (am_daemon) n = client_addr(0); break;
 		case 'l':
-			snprintf(buf2, sizeof buf2, "%.0f",
+			strlcat(fmt, ".0f", sizeof fmt);
+			snprintf(buf2, sizeof buf2, fmt,
 				 (double)file->length);
 			n = buf2;
 			break;
 		case 'p':
-			snprintf(buf2, sizeof buf2, "%d",
-				 (int)getpid());
+			strlcat(fmt, "ld", sizeof fmt);
+			snprintf(buf2, sizeof buf2, fmt,
+				 (long)getpid());
 			n = buf2;
 			break;
 		case 'o': n = op; break;
 		case 'f':
-			pathjoin(buf2, sizeof buf2,
-			    am_sender && file->dir.root ? file->dir.root : "",
-			    safe_fname(f_name(file)));
-			clean_fname(buf2, 0);
-			n = buf2;
-			if (*n == '/') n++;
+			n = safe_fname(f_name(file));
+			if (am_sender && file->dir.root) {
+				pathjoin(buf2, sizeof buf2,
+					 file->dir.root, n);
+				/* The buffer from safe_fname() has more
+				 * room than MAXPATHLEN, so this is safe. */
+				if (fmt[1])
+					strcpy(n, buf2);
+				else
+					n = buf2;
+			}
+			clean_fname(n, 0);
+			if (*n == '/')
+				n++;
 			break;
 		case 'n':
-			n = (char*)safe_fname(f_name(file));
+			n = safe_fname(f_name(file));
 			if (S_ISDIR(file->mode)) {
 				/* The buffer from safe_fname() has more
 				 * room than MAXPATHLEN, so this is safe. */
@@ -394,12 +423,21 @@ static void log_formatted(enum logcode code,
 			}
 			break;
 		case 'L':
-			if (S_ISLNK(file->mode) && file->u.link) {
-				snprintf(buf2, sizeof buf2, " -> %s",
-					 safe_fname(file->u.link));
-				n = buf2;
-			} else
+			if (hlink && *hlink) {
+				n = safe_fname(hlink);
+				strcpy(buf2, " => ");
+			} else if (S_ISLNK(file->mode) && file->u.link) {
+				n = safe_fname(file->u.link);
+				strcpy(buf2, " -> ");
+			} else {
 				n = "";
+				if (!fmt[1])
+					break;
+				strcpy(buf2, "    ");
+			}
+			strlcat(fmt, "s", sizeof fmt);
+			snprintf(buf2 + 4, sizeof buf2 - 4, fmt, n);
+			n = buf2;
 			break;
 		case 'm': n = lp_name(module_id); break;
 		case 't': n = timestring(time(NULL)); break;
@@ -413,7 +451,8 @@ static void log_formatted(enum logcode code,
 				b = stats.total_read -
 					initial_stats->total_read;
 			}
-			snprintf(buf2, sizeof buf2, "%.0f", (double)b);
+			strlcat(fmt, ".0f", sizeof fmt);
+			snprintf(buf2, sizeof buf2, fmt, (double)b);
 			n = buf2;
 			break;
 		case 'c':
@@ -424,17 +463,20 @@ static void log_formatted(enum logcode code,
 				b = stats.total_read -
 					initial_stats->total_read;
 			}
-			snprintf(buf2, sizeof buf2, "%.0f", (double)b);
+			strlcat(fmt, ".0f", sizeof fmt);
+			snprintf(buf2, sizeof buf2, fmt, (double)b);
 			n = buf2;
 			break;
 		case 'i':
 			if (iflags & ITEM_DELETED) {
-				n = "deleting";
+				n = "*deleting";
 				break;
 			}
-			n = buf2;
-			n[0] = !(iflags & ITEM_UPDATING) ? '.'
-			     : *op == 's' ? '>' : '<';
+			n = buf2 + MAXPATHLEN - 32;
+			n[0] = iflags & ITEM_LOCAL_CHANGE
+			      ? iflags & ITEM_XNAME_FOLLOWS ? 'h' : 'c'
+			     : !(iflags & ITEM_TRANSFER) ? '.'
+			     : !local_server && *op == 's' ? '<' : '>';
 			n[1] = S_ISDIR(file->mode) ? 'd'
 			     : IS_DEVICE(file->mode) ? 'D'
 			     : S_ISLNK(file->mode) ? 'L' : 'f';
@@ -446,14 +488,15 @@ static void log_formatted(enum logcode code,
 			n[5] = !(iflags & ITEM_REPORT_PERMS) ? '.' : 'p';
 			n[6] = !(iflags & ITEM_REPORT_OWNER) ? '.' : 'o';
 			n[7] = !(iflags & ITEM_REPORT_GROUP) ? '.' : 'g';
-			n[8] = '\0';
+			n[8] = !(iflags & ITEM_REPORT_XATTRS) ? '.' : 'a';
+			n[9] = '\0';
 
 			if (iflags & (ITEM_IS_NEW|ITEM_MISSING_DATA)) {
 				char ch = iflags & ITEM_IS_NEW ? '+' : '?';
 				int i;
 				for (i = 2; n[i]; i++)
 					n[i] = ch;
-			} else if (!(iflags & ITEM_UPDATING)) {
+			} else if (!(iflags & (ITEM_TRANSFER|ITEM_LOCAL_CHANGE))) {
 				int i;
 				for (i = 2; n[i]; i++) {
 					if (n[i] != '.')
@@ -462,23 +505,25 @@ static void log_formatted(enum logcode code,
 				if (!n[i]) {
 					for (i = 2; n[i]; i++)
 						n[i] = ' ';
-					n[0] = '=';
 				}
 			}
 			break;
 		}
 
-		/* n is the string to be inserted in place of this %
-		 * code; len is its length not including the trailing
-		 * NUL */
-		if (!n) {
-			p += 2;
+		/* "n" is the string to be inserted in place of this % code. */
+		if (!n)
 			continue;
+		if (n != buf2 && fmt[1]) {
+			strlcat(fmt, "s", sizeof fmt);
+			snprintf(buf2, sizeof buf2, fmt, n);
+			n = buf2;
 		}
-
 		len = strlen(n);
 
-		if (len + total - 2 >= sizeof buf) {
+		/* Subtract the length of the escape from the string's size. */
+		total -= p - s + 1;
+
+		if (len + total >= (size_t)sizeof buf) {
 			rprintf(FERROR,
 				"buffer overflow expanding %%%c -- exiting\n",
 				p[0]);
@@ -486,45 +531,69 @@ static void log_formatted(enum logcode code,
 		}
 
 		/* Shuffle the rest of the string along to make space for n */
-		if (len != 2)
-			memmove(p + len, p + 2, total - (p + 2 - buf) + 1);
-		total += len - 2;
+		if (len != (size_t)(p - s + 1))
+			memmove(s + len, p + 1, total - (s - buf) + 1);
+		total += len;
 
-		/* Insert the contents of string "n", but NOT its nul. */
+		/* Insert the contents of string "n", but NOT its null. */
 		if (len)
-			memcpy(p, n, len);
+			memcpy(s, n, len);
 
 		/* Skip over inserted string; continue looking */
-		p += len;
+		p = s + len;
 	}
 
-	rprintf(code, "%s\n", buf);
+	rwrite(code, buf, total);
 }
 
-/* log the outgoing transfer of a file */
-void log_send(struct file_struct *file, struct stats *initial_stats, int iflags)
+/* Return 1 if the format escape is in the log-format string (e.g. look for
+ * the 'b' in the "%9b" format escape). */
+int log_format_has(const char *format, char esc)
 {
-	if (lp_transfer_logging(module_id)) {
-		log_formatted(FLOG, lp_log_format(module_id), "send",
-			      file, initial_stats, iflags);
-	} else if (log_format && !am_server) {
-		log_formatted(FINFO, log_format, "send",
-			      file, initial_stats, iflags);
+	const char *p;
+
+	if (!format)
+		return 0;
+
+	for (p = format; (p = strchr(p, '%')) != NULL; ) {
+		if (*++p == '-')
+			p++;
+		while (isdigit(*(uchar*)p))
+			p++;
+		if (!*p)
+			break;
+		if (*p == esc)
+			return 1;
 	}
+	return 0;
 }
 
-/* log the incoming transfer of a file */
-void log_recv(struct file_struct *file, struct stats *initial_stats, int iflags)
+/* log the transfer of a file */
+void log_item(struct file_struct *file, struct stats *initial_stats,
+	      int iflags, char *hlink)
 {
+	char *s_or_r = am_sender ? "send" : "recv";
+
 	if (lp_transfer_logging(module_id)) {
-		log_formatted(FLOG, lp_log_format(module_id), "recv",
-			      file, initial_stats, iflags);
+		log_formatted(FLOG, lp_log_format(module_id), s_or_r,
+			      file, initial_stats, iflags, hlink);
 	} else if (log_format && !am_server) {
-		log_formatted(FINFO, log_format, "recv",
-			      file, initial_stats, iflags);
+		log_formatted(FINFO, log_format, s_or_r,
+			      file, initial_stats, iflags, hlink);
 	}
 }
 
+void maybe_log_item(struct file_struct *file, int iflags, int itemizing,
+		    char *buf)
+{
+	int see_item = itemizing && (iflags || verbose > 1);
+	if (am_server) {
+		if (am_daemon && !dry_run && see_item)
+			log_item(file, &stats, iflags, buf);
+	} else if (see_item || iflags & ITEM_LOCAL_CHANGE || *buf
+	    || (S_ISDIR(file->mode) && iflags & SIGNIFICANT_ITEM_FLAGS))
+		log_item(file, &stats, iflags, buf);
+}
 
 void log_delete(char *fname, int mode)
 {
@@ -542,15 +611,16 @@ void log_delete(char *fname, int mode)
 			len++; /* directories include trailing null */
 		send_msg(MSG_DELETED, fname, len);
 	} else {
-		fmt = log_format_has_o_or_i ? log_format : "%i %n";
-		log_formatted(FCLIENT, fmt, "del.", &file, &stats, ITEM_DELETED);
+		fmt = log_format_has_o_or_i ? log_format : "deleting %n";
+		log_formatted(FCLIENT, fmt, "del.", &file, &stats,
+			      ITEM_DELETED, NULL);
 	}
 
 	if (!am_daemon || dry_run || !lp_transfer_logging(module_id))
 		return;
 
-	fmt = daemon_log_format_has_o_or_i ? lp_log_format(module_id) : "%i %n";
-	log_formatted(FLOG, fmt, "del.", &file, &stats, ITEM_DELETED);
+	fmt = daemon_log_format_has_o_or_i ? lp_log_format(module_id) : "deleting %n";
+	log_formatted(FLOG, fmt, "del.", &file, &stats, ITEM_DELETED, NULL);
 }
 
 

main.c

@@ -21,8 +21,6 @@
 
 #include "rsync.h"
 
-time_t starttime = 0;
-
 extern int verbose;
 extern int dry_run;
 extern int list_only;
@@ -66,6 +64,7 @@ extern char *shell_cmd;
 extern char *batch_name;
 
 int local_server = 0;
+struct file_list *the_file_list;
 
 /* There's probably never more than at most 2 outstanding child processes,
  * but set it higher, just in case. */
@@ -76,6 +75,9 @@ struct pid_status {
 	int   status;
 } pid_stat_table[MAXCHILDPROCS];
 
+static time_t starttime, endtime;
+static int64 total_read, total_written;
+
 static void show_malloc_stats(void);
 
 /****************************************************************************
@@ -120,12 +122,13 @@ void wait_process(pid_t pid, int *status)
  * the report.  All processes might also generate a set of debug stats, if
  * the verbose level is high enough (this is the only thing that the
  * generator process and the server receiver ever do here). */
-static void report(int f)
+static void handle_stats(int f)
 {
+	endtime = time(NULL);
+
 	/* Cache two stats because the read/write code can change it. */
-	int64 total_read = stats.total_read;
-	int64 total_written = stats.total_written;
-	time_t t = time(NULL);
+	total_read = stats.total_read;
+	total_written = stats.total_written;
 
 	if (do_stats && verbose > 1) {
 		/* These come out from every process */
@@ -206,11 +209,17 @@ static void report(int f)
 			(double)total_read);
 	}
 
+	fflush(stdout);
+	fflush(stderr);
+}
+
+static void output_summary(void)
+{
 	if (verbose || do_stats) {
 		rprintf(FINFO,
 			"\nsent %.0f bytes  received %.0f bytes  %.2f bytes/sec\n",
 			(double)total_written, (double)total_read,
-			(total_written + total_read)/(0.5 + (t - starttime)));
+			(total_written + total_read)/(0.5 + (endtime - starttime)));
 		rprintf(FINFO, "total size is %.0f  speedup is %.2f\n",
 			(double)stats.total_size,
 			(double)stats.total_size / (total_written+total_read));
@@ -429,23 +438,24 @@ static char *get_local_name(struct file_list *flist,char *name)
 
 
 /* This is only called by the sender. */
-static void read_final_goodbye(int f_in, int f_out, int flist_count)
+static void read_final_goodbye(int f_in, int f_out)
 {
 	int i;
 
 	if (protocol_version < 29)
 		i = read_int(f_in);
 	else {
-		while ((i = read_int(f_in)) == flist_count
+		while ((i = read_int(f_in)) == the_file_list->count
 		    && read_shortint(f_in) == ITEM_IS_NEW) {
 			/* Forward the keep-alive (no-op) to the receiver. */
-			write_int(f_out, flist_count);
+			write_int(f_out, the_file_list->count);
 			write_shortint(f_out, ITEM_IS_NEW);
 		}
 	}
 
 	if (i != -1) {
-		rprintf(FERROR, "Invalid packet from generator at end of run.\n");
+		rprintf(FERROR, "Invalid packet at end of run (%d) [%s]\n",
+			i, who_am_i());
 		exit_cleanup(RERR_PROTOCOL);
 	}
 }
@@ -500,15 +510,16 @@ static void do_server_sender(int f_in, int f_out, int argc,char *argv[])
 	if (!flist || flist->count == 0) {
 		exit_cleanup(0);
 	}
+	the_file_list = flist;
 
 	io_start_buffering_in();
 	io_start_buffering_out();
 
 	send_files(flist,f_out,f_in);
 	io_flush(FULL_FLUSH);
-	report(f_out);
+	handle_stats(f_out);
 	if (protocol_version >= 24)
-		read_final_goodbye(f_in, f_out, flist->count);
+		read_final_goodbye(f_in, f_out);
 	io_flush(FULL_FLUSH);
 	exit_cleanup(0);
 }
@@ -518,19 +529,16 @@ static int do_recv(int f_in,int f_out,struct file_list *flist,char *local_name)
 {
 	int pid;
 	int status = 0;
-	int error_pipe[2], name_pipe[2];
-	BOOL need_name_pipe = (basis_dir[0] || partial_dir || fuzzy_basis
-			    || (inplace && make_backups)) && !dry_run;
+	int error_pipe[2];
 
 	/* The receiving side mustn't obey this, or an existing symlink that
 	 * points to an identical file won't be replaced by the referent. */
 	copy_links = 0;
 
 	if (preserve_hard_links)
-		init_hard_links(flist);
+		init_hard_links();
 
-	if (fd_pair(error_pipe) < 0
-	    || (need_name_pipe && fd_pair(name_pipe) < 0)) {
+	if (fd_pair(error_pipe) < 0) {
 		rsyserr(FERROR, errno, "pipe failed in do_recv");
 		exit_cleanup(RERR_IPC);
 	}
@@ -544,11 +552,6 @@ static int do_recv(int f_in,int f_out,struct file_list *flist,char *local_name)
 
 	if (pid == 0) {
 		close(error_pipe[0]);
-		if (need_name_pipe) {
-			close(name_pipe[1]);
-			set_blocking(name_pipe[0]);
-		} else
-			name_pipe[0] = -1;
 		if (f_in != f_out)
 			close(f_out);
 
@@ -558,9 +561,9 @@ static int do_recv(int f_in,int f_out,struct file_list *flist,char *local_name)
 		/* set place to send errors */
 		set_msg_fd_out(error_pipe[1]);
 
-		recv_files(f_in, flist, local_name, name_pipe[0]);
+		recv_files(f_in, flist, local_name);
 		io_flush(FULL_FLUSH);
-		report(f_in);
+		handle_stats(f_in);
 
 		send_msg(MSG_DONE, "", 0);
 		io_flush(FULL_FLUSH);
@@ -574,7 +577,8 @@ static int do_recv(int f_in,int f_out,struct file_list *flist,char *local_name)
 			while (read_int(f_in) == flist->count
 			    && read_shortint(f_in) == ITEM_IS_NEW) {}
 
-			rprintf(FERROR, "Invalid packet from server at end of run.\n");
+			rprintf(FERROR, "Invalid packet at end of run [%s]\n",
+				who_am_i());
 			exit_cleanup(RERR_PROTOCOL);
 		}
 
@@ -591,11 +595,6 @@ static int do_recv(int f_in,int f_out,struct file_list *flist,char *local_name)
 		stop_write_batch();
 
 	close(error_pipe[1]);
-	if (need_name_pipe) {
-		close(name_pipe[0]);
-		set_nonblocking(name_pipe[1]);
-	} else
-		name_pipe[1] = -1;
 	if (f_in != f_out)
 		close(f_in);
 
@@ -603,9 +602,9 @@ static int do_recv(int f_in,int f_out,struct file_list *flist,char *local_name)
 
 	set_msg_fd_in(error_pipe[0]);
 
-	generate_files(f_out, flist, local_name, name_pipe[1]);
+	generate_files(f_out, flist, local_name);
 
-	report(-1);
+	handle_stats(-1);
 	io_flush(FULL_FLUSH);
 	if (protocol_version >= 24) {
 		/* send a final goodbye message */
@@ -676,6 +675,7 @@ static void do_server_recv(int f_in, int f_out, int argc,char *argv[])
 		rprintf(FERROR,"server_recv: recv_file_list error\n");
 		exit_cleanup(RERR_FILESELECT);
 	}
+	the_file_list = flist;
 
 	if (argc > 0) {
 		if (strcmp(dir,".")) {
@@ -766,24 +766,25 @@ int client_run(int f_in, int f_out, pid_t pid, int argc, char *argv[])
 
 		if (write_batch)
 			start_write_batch(f_out);
-		if (!read_batch) /* don't write to pipe */
-			flist = send_file_list(f_out,argc,argv);
+		flist = send_file_list(f_out, argc, argv);
 		set_msg_fd_in(-1);
 		if (verbose > 3)
 			rprintf(FINFO,"file list sent\n");
+		the_file_list = flist;
 
 		io_flush(NORMAL_FLUSH);
 		send_files(flist,f_out,f_in);
 		io_flush(FULL_FLUSH);
+		handle_stats(-1);
 		if (protocol_version >= 24)
-			read_final_goodbye(f_in, f_out, flist->count);
+			read_final_goodbye(f_in, f_out);
 		if (pid != -1) {
 			if (verbose > 3)
 				rprintf(FINFO,"client_run waiting on %d\n", (int) pid);
 			io_flush(FULL_FLUSH);
 			wait_process(pid, &status);
 		}
-		report(-1);
+		output_summary();
 		io_flush(FULL_FLUSH);
 		exit_cleanup(status);
 	}
@@ -810,6 +811,7 @@ int client_run(int f_in, int f_out, pid_t pid, int argc, char *argv[])
 			"the --recursive option?\n");
 		exit_cleanup(0);
 	}
+	the_file_list = flist;
 
 	local_name = get_local_name(flist,argv[0]);
 
@@ -984,8 +986,11 @@ static RETSIGTYPE sigusr1_handler(UNUSED(int val))
 
 static RETSIGTYPE sigusr2_handler(UNUSED(int val))
 {
+	if (!am_server)
+		output_summary();
 	close_all();
-	if (log_got_error) _exit(RERR_PARTIAL);
+	if (log_got_error)
+		_exit(RERR_PARTIAL);
 	_exit(0);
 }
 

match.c

@@ -133,12 +133,8 @@ static void matched(int f, struct sum_struct *s, struct map_struct *buf,
 	else
 		last_match = offset;
 
-	if (buf && do_progress) {
+	if (buf && do_progress)
 		show_progress(last_match, buf->file_size);
-
-		if (i == -1)
-			end_progress(buf->file_size);
-	}
 }
 
 

options.c

@@ -143,12 +143,13 @@ char *backup_dir = NULL;
 char backup_dir_buf[MAXPATHLEN];
 int rsync_port = 0;
 int compare_dest = 0;
+int copy_dest = 0;
 int link_dest = 0;
 int basis_dir_cnt = 0;
+char *dest_option = NULL;
 
 int verbose = 0;
 int quiet = 0;
-int itemize_changes = 0;
 int log_before_transfer = 0;
 int log_format_has_i = 0;
 int log_format_has_o_or_i = 0;
@@ -161,9 +162,9 @@ char *batch_name = NULL;
 static int daemon_opt;   /* sets am_daemon after option error-reporting */
 static int F_option_cnt = 0;
 static int modify_window_set;
+static int itemize_changes = 0;
 static int refused_delete, refused_archive_part;
 static int refused_partial, refused_progress, refused_delete_before;
-static char *dest_option = NULL;
 static char *max_size_arg;
 static char partialdir_for_delayupdate[] = ".~tmp~";
 
@@ -292,7 +293,7 @@ void usage(enum logcode F)
   rprintf(F," -x, --one-file-system       don't cross filesystem boundaries\n");
   rprintf(F," -B, --block-size=SIZE       force a fixed checksum block-size\n");
   rprintf(F," -e, --rsh=COMMAND           specify the remote shell to use\n");
-  rprintf(F,"     --rsync-path=PATH       specify path to rsync on the remote machine\n");
+  rprintf(F,"     --rsync-path=PROGRAM    specify the rsync to run on the remote machine\n");
   rprintf(F,"     --existing              only update files that already exist on receiver\n");
   rprintf(F,"     --ignore-existing       ignore files that already exist on receiving side\n");
   rprintf(F,"     --remove-sent-files     sent files/symlinks are removed from sending side\n");
@@ -317,6 +318,7 @@ void usage(enum logcode F)
   rprintf(F," -T, --temp-dir=DIR          create temporary files in directory DIR\n");
   rprintf(F," -y, --fuzzy                 find similar file for basis if no dest file\n");
   rprintf(F,"     --compare-dest=DIR      also compare destination files relative to DIR\n");
+  rprintf(F,"     --copy-dest=DIR         ... and include copies of unchanged files\n");
   rprintf(F,"     --link-dest=DIR         hardlink to files in DIR when unchanged\n");
   rprintf(F," -z, --compress              compress file data during the transfer\n");
   rprintf(F," -C, --cvs-exclude           auto-ignore files the same way CVS does\n");
@@ -343,6 +345,7 @@ void usage(enum logcode F)
   rprintf(F,"     --bwlimit=KBPS          limit I/O bandwidth; KBytes per second\n");
   rprintf(F,"     --write-batch=FILE      write a batched update to FILE\n");
   rprintf(F,"     --read-batch=FILE       read a batched update from FILE\n");
+  rprintf(F,"     --protocol=NUM          force an older protocol version to be used\n");
 #ifdef INET6
   rprintf(F," -4, --ipv4                  prefer IPv4\n");
   rprintf(F," -6, --ipv6                  prefer IPv6\n");
@@ -355,7 +358,7 @@ void usage(enum logcode F)
 }
 
 enum {OPT_VERSION = 1000, OPT_DAEMON, OPT_SENDER, OPT_EXCLUDE, OPT_EXCLUDE_FROM,
-      OPT_FILTER, OPT_COMPARE_DEST, OPT_LINK_DEST,
+      OPT_FILTER, OPT_COMPARE_DEST, OPT_COPY_DEST, OPT_LINK_DEST,
       OPT_INCLUDE, OPT_INCLUDE_FROM, OPT_MODIFY_WINDOW,
       OPT_READ_BATCH, OPT_WRITE_BATCH, OPT_TIMEOUT, OPT_MAX_SIZE,
       OPT_REFUSED_BASE = 9000};
@@ -413,7 +416,7 @@ static struct poptOption long_options[] = {
   {"archive",         'a', POPT_ARG_NONE,   &archive_mode, 0, 0, 0 },
   {"server",           0,  POPT_ARG_NONE,   &am_server, 0, 0, 0 },
   {"sender",           0,  POPT_ARG_NONE,   0, OPT_SENDER, 0, 0 },
-  {"recursive",       'r', POPT_ARG_VAL,    &recurse, -1, 0, 0 },
+  {"recursive",       'r', POPT_ARG_NONE,   &recurse, 0, 0, 0 },
   {"list-only",        0,  POPT_ARG_VAL,    &list_only, 2, 0, 0 },
   {"relative",        'R', POPT_ARG_VAL,    &relative_paths, 1, 0, 0 },
   {"no-relative",      0,  POPT_ARG_VAL,    &relative_paths, 0, 0, 0 },
@@ -424,6 +427,7 @@ static struct poptOption long_options[] = {
   {"timeout",          0,  POPT_ARG_INT,    &io_timeout, OPT_TIMEOUT, 0, 0 },
   {"temp-dir",        'T', POPT_ARG_STRING, &tmpdir, 0, 0, 0 },
   {"compare-dest",     0,  POPT_ARG_STRING, 0, OPT_COMPARE_DEST, 0, 0 },
+  {"copy-dest",        0,  POPT_ARG_STRING, 0, OPT_COPY_DEST, 0, 0 },
   {"link-dest",        0,  POPT_ARG_STRING, 0, OPT_LINK_DEST, 0, 0 },
   {"fuzzy",           'y', POPT_ARG_NONE,   &fuzzy_basis, 0, 0, 0 },
   /* TODO: Should this take an optional int giving the compression level? */
@@ -838,6 +842,11 @@ int parse_arguments(int *argc, const char ***argv, int frommain)
 			return 0;
 #endif
 
+		case OPT_COPY_DEST:
+			copy_dest = 1;
+			dest_option = "--copy-dest";
+			goto set_dest_dir;
+
 		case OPT_COMPARE_DEST:
 			compare_dest = 1;
 			dest_option = "--compare-dest";
@@ -893,12 +902,6 @@ int parse_arguments(int *argc, const char ***argv, int frommain)
 		return 0;
 	}
 	if (write_batch || read_batch) {
-		if (dry_run) {
-			snprintf(err_buf, sizeof err_buf,
-				"--%s-batch cannot be used with --dry_run (-n)\n",
-				write_batch ? "write" : "read");
-			return 0;
-		}
 		if (am_server) {
 			rprintf(FINFO,
 				"ignoring --%s-batch option sent to server\n",
@@ -908,7 +911,8 @@ int parse_arguments(int *argc, const char ***argv, int frommain)
 			 * batch args to server. */
 			read_batch = write_batch = 0;
 			batch_name = NULL;
-		}
+		} else if (dry_run)
+			write_batch = 0;
 	}
 	if (read_batch && files_from) {
 		snprintf(err_buf, sizeof err_buf,
@@ -928,9 +932,9 @@ int parse_arguments(int *argc, const char ***argv, int frommain)
 		return 0;
 	}
 
-	if (compare_dest + link_dest > 1) {
+	if (compare_dest + copy_dest + link_dest > 1) {
 		snprintf(err_buf, sizeof err_buf,
-			"You may not mix --compare-dest and --link-dest.\n");
+			"You may not mix --compare-dest, --copy-dest, and --link-dest.\n");
 		return 0;
 	}
 
@@ -940,7 +944,7 @@ int parse_arguments(int *argc, const char ***argv, int frommain)
 			return 0;
 		}
 		if (!files_from)
-			recurse = -1; /* infinite recursion */
+			recurse = 1;
 #ifdef SUPPORT_LINKS
 		preserve_links = 1;
 #endif
@@ -1074,10 +1078,10 @@ int parse_arguments(int *argc, const char ***argv, int frommain)
 		omit_dir_times = 1;
 
 	if (log_format) {
-		if (strstr(log_format, "%i") != NULL)
+		if (log_format_has(log_format, 'i'))
 			log_format_has_i = 1;
-		if (strstr(log_format, "%b") == NULL
-		 && strstr(log_format, "%c") == NULL)
+		if (!log_format_has(log_format, 'b')
+		 && !log_format_has(log_format, 'c'))
 			log_before_transfer = !am_server;
 	} else if (itemize_changes) {
 		log_format = "%i %n%L";
@@ -1093,8 +1097,7 @@ int parse_arguments(int *argc, const char ***argv, int frommain)
 		log_format = "%n%L";
 		log_before_transfer = !am_server;
 	}
-	if (log_format_has_i
-	    || (log_format && strstr(log_format, "%o") != NULL))
+	if (log_format_has_i || log_format_has(log_format, 'o'))
 		log_format_has_o_or_i = 1;
 
 	if (daemon_bwlimit && (!bwlimit || bwlimit > daemon_bwlimit))
@@ -1273,7 +1276,7 @@ void server_options(char **args,int *argc)
 		argstr[x++] = 'O';
 	if (preserve_perms)
 		argstr[x++] = 'p';
-	if (recurse < 0)
+	if (recurse)
 		argstr[x++] = 'r';
 	if (always_checksum)
 		argstr[x++] = 'c';
@@ -1293,7 +1296,7 @@ void server_options(char **args,int *argc)
 	/* This is a complete hack - blame Rusty.  FIXME!
 	 * This hack is only needed for older rsync versions that
 	 * don't understand the --list-only option. */
-	if (list_only == 1 && recurse >= 0)
+	if (list_only == 1 && !recurse)
 		argstr[x++] = 'r';
 
 	argstr[x] = 0;

packaging/lsb/rsync.spec

@@ -1,24 +1,46 @@
-Summary: Program for efficient remote updates of files.
+Summary: A program for synchronizing files over a network.
 Name: rsync
-Version: 2.6.4pre2
+Version: 2.6.4
 Release: 1
-Copyright: GPL
-Group: Applications/Networking
-Source:	ftp://samba.anu.edu.au/pub/rsync/rsync-%{version}.tar.gz
-URL: http://samba.anu.edu.au/rsync/
-Packager: Andrew Tridgell <tridge@samba.anu.edu.au>
-BuildRoot: /tmp/rsync
+Group: Applications/Internet
+Source:	ftp://rsync.samba.org/pub/rsync/rsync-%{version}.tar.gz
+URL: http://rsync.samba.org/
+
+Prefix: %{_prefix}
+BuildRoot: /var/tmp/%{name}-root
+License: GPL
 
 %description
-rsync is a replacement for rcp that has many more features.
+Rsync uses a reliable algorithm to bring remote and host files into
+sync very quickly. Rsync is fast because it just sends the differences
+in the files over the network instead of sending the complete
+files. Rsync is often used as a very powerful mirroring process or
+just as a more capable replacement for the rcp command. A technical
+report which describes the rsync algorithm is included in this
+package.
 
-rsync uses the "rsync algorithm" which provides a very fast method for
-bringing remote files into sync. It does this by sending just the
-differences in the files across the link, without requiring that both
-sets of files are present at one of the ends of the link beforehand.
+%prep
+%setup -q
 
-A technical report describing the rsync algorithm is included with
-this package. 
+%build
+%configure
+
+make
+
+%install
+rm -rf $RPM_BUILD_ROOT
+
+%makeinstall
+
+%clean
+rm -rf $RPM_BUILD_ROOT
+
+%files
+%defattr(-,root,root)
+%doc COPYING README tech_report.tex
+%{_prefix}/bin/rsync
+%{_mandir}/man1/rsync.1*
+%{_mandir}/man5/rsyncd.conf.5*
 
 %changelog
 * Thu Jan 30 2003 Horst von Brand <vonbrand@inf.utfsm.cl>
@@ -64,30 +86,3 @@ to '%build', removed '%prefix'.
 
 rsync-1.6.2-1 packaged.  (This entry by jam to credit Michael for the
 previous package(s).)
-
-%prep
-%setup
-
-%build
-./configure --prefix=/usr --mandir=%{_mandir}
-make CFLAGS="$RPM_OPT_FLAGS"
-strip rsync
-
-%install
-mkdir -p $RPM_BUILD_ROOT/usr/bin
-mkdir -p $RPM_BUILD_ROOT/%{_mandir}/man{1,5}
-install -m755 rsync $RPM_BUILD_ROOT/usr/bin
-install -m644 rsync.1 $RPM_BUILD_ROOT/%{_mandir}/man1
-install -m644 rsyncd.conf.5 $RPM_BUILD_ROOT/%{_mandir}/man5
-
-%clean
-rm -rf $RPM_BUILD_ROOT
-
-%files
-%attr(-,root,root) /usr/bin/rsync
-%attr(-,root,root) %{_mandir}/man1/rsync.1*
-%attr(-,root,root) %{_mandir}/man5/rsyncd.conf.5*
-%attr(-,root,root) %doc tech_report.tex
-%attr(-,root,root) %doc README
-%attr(-,root,root) %doc COPYING
-%attr(-,root,root) %doc doc/README-SGML doc/rsync.sgml

progress.c

@@ -67,17 +67,18 @@ static void rprint_progress(OFF_T ofs, OFF_T size, struct timeval *now,
 
 	if (is_last) {
 		/* Compute stats based on the starting info. */
-		diff = msdiff(&ph_start.time, now);
-		if (!diff)
+		if (!ph_start.time.tv_sec
+		    || !(diff = msdiff(&ph_start.time, now)))
 			diff = 1;
 		rate = (double) (ofs - ph_start.ofs) * 1000.0 / diff / 1024.0;
 		/* Switch to total time taken for our last update. */
 		remain = (double) diff / 1000.0;
 	} else {
 		/* Compute stats based on recent progress. */
-		diff = msdiff(&ph_list[oldest_hpos].time, now);
-		rate = diff ? (double) (ofs - ph_list[oldest_hpos].ofs) * 1000.0
-		    / diff / 1024.0 : 0;
+		if (!(diff = msdiff(&ph_list[oldest_hpos].time, now)))
+			diff = 1;
+		rate = (double) (ofs - ph_list[oldest_hpos].ofs) * 1000.0
+		     / diff / 1024.0;
 		remain = rate ? (double) (size - ofs) / rate / 1000.0 : 0.0;
 	}
 

receiver.c

@@ -53,7 +53,7 @@ extern char *log_format;
 extern char *tmpdir;
 extern char *partial_dir;
 extern char *basis_dir[];
-
+extern struct file_list *the_file_list;
 extern struct filter_list_struct server_filter_list;
 
 #define SLOT_SIZE	(16*1024)	/* Desired size in bytes */
@@ -62,6 +62,7 @@ extern struct filter_list_struct server_filter_list;
 
 static uint32 **delayed_bits = NULL;
 static int delayed_slot_cnt = 0;
+static int phase = 0;
 
 static void init_delayed_bits(int max_ndx)
 {
@@ -266,7 +267,7 @@ static int receive_data(int f_in, char *fname_r, int fd_r, OFF_T size_r,
 				continue;
 			}
 		}
-		if (fd != -1 && write_file(fd, map, len) != (int)len)
+		if (fd != -1 && map && write_file(fd, map, len) != (int)len)
 			goto report_write_error;
 		offset += len;
 	}
@@ -303,64 +304,86 @@ static int receive_data(int f_in, char *fname_r, int fd_r, OFF_T size_r,
 }
 
 
-static void read_gen_name(int fd, char *dirname, char *buf)
-{
-	int dlen;
-	int len = read_byte(fd);
-
-	if (len & 0x80) {
-#if MAXPATHLEN > 32767
-		uchar lenbuf[2];
-		read_buf(fd, (char *)lenbuf, 2);
-		len = (len & ~0x80) * 0x10000 + lenbuf[0] * 0x100 + lenbuf[1];
-#else
-		len = (len & ~0x80) * 0x100 + read_byte(fd);
-#endif
-	}
-
-	if (dirname) {
-		dlen = strlcpy(buf, dirname, MAXPATHLEN);
-		buf[dlen++] = '/';
-	} else
-		dlen = 0;
-
-	if (dlen + len >= MAXPATHLEN) {
-		rprintf(FERROR, "bogus data on generator name pipe\n");
-		exit_cleanup(RERR_PROTOCOL);
-	}
-
-	read_sbuf(fd, buf + dlen, len);
-}
-
-
 static void discard_receive_data(int f_in, OFF_T length)
 {
 	receive_data(f_in, NULL, -1, 0, NULL, -1, length);
 }
 
+static void handle_delayed_updates(struct file_list *flist, char *local_name)
+{
+	char *fname, *partialptr, numbuf[4];
+	int i;
+
+	for (i = -1; (i = next_delayed_bit(i)) >= 0; ) {
+		struct file_struct *file = flist->files[i];
+		fname = local_name ? local_name : f_name(file);
+		if ((partialptr = partial_dir_fname(fname)) != NULL) {
+			if (make_backups && !make_backup(fname))
+				continue;
+			if (verbose > 2) {
+				rprintf(FINFO, "renaming %s to %s\n",
+					safe_fname(partialptr),
+					safe_fname(fname));
+			}
+			if (do_rename(partialptr, fname) < 0) {
+				rsyserr(FERROR, errno,
+					"rename failed for %s (from %s)",
+					full_fname(fname),
+					safe_fname(partialptr));
+			} else {
+				if (remove_sent_files
+				    || (preserve_hard_links
+				     && file->link_u.links)) {
+					SIVAL(numbuf, 0, i);
+					send_msg(MSG_SUCCESS,numbuf,4);
+				}
+				handle_partial_dir(partialptr,
+						   PDIR_DELETE);
+			}
+		}
+	}
+}
+
+static int get_next_gen_i(int batch_gen_fd, int next_gen_i, int desired_i)
+{
+	while (next_gen_i < desired_i) {
+		if (next_gen_i >= 0) {
+			rprintf(FINFO,
+				"(No batched update for%s \"%s\")\n",
+				phase ? " resend of" : "",
+				safe_fname(f_name(the_file_list->files[next_gen_i])));
+		}
+		next_gen_i = read_int(batch_gen_fd);
+		if (next_gen_i == -1)
+			next_gen_i = the_file_list->count;
+	}
+	return next_gen_i;
+}
+
 
 /**
  * main routine for receiver process.
  *
  * Receiver process runs on the same host as the generator process. */
-int recv_files(int f_in, struct file_list *flist, char *local_name,
-	       int f_in_name)
+int recv_files(int f_in, struct file_list *flist, char *local_name)
 {
 	int next_gen_i = -1;
 	int fd1,fd2;
 	STRUCT_STAT st;
-	int iflags;
+	int iflags, xlen;
 	char *fname, fbuf[MAXPATHLEN];
-	char template[MAXPATHLEN];
+	char xname[MAXPATHLEN];
 	char fnametmp[MAXPATHLEN];
 	char *fnamecmp, *partialptr, numbuf[4];
 	char fnamecmpbuf[MAXPATHLEN];
+	uchar fnamecmp_type;
 	struct file_struct *file;
 	struct stats initial_stats;
 	int save_make_backups = make_backups;
 	int itemizing = am_daemon ? daemon_log_format_has_i
 		      : !am_server && log_format_has_i;
-	int i, recv_ok, phase = 0;
+	int max_phase = protocol_version >= 29 ? 2 : 1;
+	int i, recv_ok;
 
 	if (verbose > 2)
 		rprintf(FINFO,"recv_files(%d) starting\n",flist->count);
@@ -379,38 +402,27 @@ int recv_files(int f_in, struct file_list *flist, char *local_name,
 		i = read_int(f_in);
 		if (i == -1) {
 			if (read_batch) {
-				if (next_gen_i != flist->count) {
-					do {
-						if (f_in_name >= 0
-						    && next_gen_i >= 0)
-							read_byte(f_in_name);
-					} while (read_int(batch_gen_fd) != -1);
-				}
+				get_next_gen_i(batch_gen_fd, next_gen_i,
+					       flist->count);
 				next_gen_i = -1;
 			}
-
-			if (phase)
+			if (++phase > max_phase)
 				break;
-
-			phase = 1;
 			csum_length = SUM_LENGTH;
 			if (verbose > 2)
 				rprintf(FINFO, "recv_files phase=%d\n", phase);
+			if (phase == 2 && delay_updates)
+				handle_delayed_updates(flist, local_name);
 			send_msg(MSG_DONE, "", 0);
 			if (keep_partial && !partial_dir)
 				make_backups = 0; /* prevents double backup */
 			continue;
 		}
 
-		if (i < 0 || i >= flist->count) {
-			/* Handle the new keep-alive (no-op) packet. */
-			if (i == flist->count && protocol_version >= 29
-	 		    && read_shortint(f_in) == ITEM_IS_NEW)
-				continue;
-			rprintf(FERROR,"Invalid file index %d in recv_files (count=%d)\n",
-				i, flist->count);
-			exit_cleanup(RERR_PROTOCOL);
-		}
+		iflags = read_item_attrs(f_in, -1, i, &fnamecmp_type,
+					 xname, &xlen);
+		if (iflags == ITEM_IS_NEW) /* no-op packet */
+			continue;
 
 		file = flist->files[i];
 		fname = local_name ? local_name : f_name_to(file, fbuf);
@@ -418,25 +430,14 @@ int recv_files(int f_in, struct file_list *flist, char *local_name,
 		if (verbose > 2)
 			rprintf(FINFO, "recv_files(%s)\n", safe_fname(fname));
 
-		if (protocol_version >= 29) {
-			iflags = read_shortint(f_in);
-			if (!(iflags & ITEM_UPDATING) || !S_ISREG(file->mode)) {
-				int see_item = itemizing && (iflags || verbose > 1);
-				if (am_server) {
-					if (am_daemon && !dry_run && see_item)
-						log_recv(file, &stats, iflags);
-				} else if (see_item || iflags & ITEM_UPDATING
-				    || (S_ISDIR(file->mode)
-				     && iflags & ITEM_REPORT_TIME))
-					log_recv(file, &stats, iflags);
-				continue;
-			}
-		} else
-			iflags = ITEM_UPDATING | ITEM_MISSING_DATA;
-
-		if (!S_ISREG(file->mode)) {
-			rprintf(FERROR, "[%s] got index of non-regular file: %d\n",
-				who_am_i(), i);
+		if (!(iflags & ITEM_TRANSFER)) {
+			maybe_log_item(file, iflags, itemizing, xname);
+			continue;
+		}
+		if (phase == 2) {
+			rprintf(FERROR,
+				"got transfer request in phase 2 [%s]\n",
+				who_am_i());
 			exit_cleanup(RERR_PROTOCOL);
 		}
 
@@ -453,20 +454,16 @@ int recv_files(int f_in, struct file_list *flist, char *local_name,
 
 		if (dry_run) { /* log the transfer */
 			if (!am_server && log_format)
-				log_recv(file, &stats, iflags);
+				log_item(file, &stats, iflags, NULL);
+			if (read_batch)
+				discard_receive_data(f_in, file->length);
 			continue;
 		}
 
 		if (read_batch) {
-			while (i > next_gen_i) {
-				if (f_in_name >= 0 && next_gen_i >= 0)
-					read_byte(f_in_name);
-				next_gen_i = read_int(batch_gen_fd);
-				if (next_gen_i == -1)
-					next_gen_i = flist->count;
-			}
+			next_gen_i = get_next_gen_i(batch_gen_fd, next_gen_i, i);
 			if (i < next_gen_i) {
-				rprintf(FINFO, "skipping update for \"%s\"\n",
+				rprintf(FINFO, "(Skipping batched update for \"%s\")\n",
 					safe_fname(fname));
 				discard_receive_data(f_in, file->length);
 				continue;
@@ -476,42 +473,72 @@ int recv_files(int f_in, struct file_list *flist, char *local_name,
 
 		partialptr = partial_dir ? partial_dir_fname(fname) : fname;
 
-		if (f_in_name >= 0) {
-			uchar j;
-			switch (j = read_byte(f_in_name)) {
+		if (protocol_version >= 29) {
+			switch (fnamecmp_type) {
 			case FNAMECMP_FNAME:
 				fnamecmp = fname;
 				break;
 			case FNAMECMP_PARTIAL_DIR:
-				fnamecmp = partialptr ? partialptr : fname;
+				fnamecmp = partialptr;
 				break;
 			case FNAMECMP_BACKUP:
 				fnamecmp = get_backup_name(fname);
 				break;
 			case FNAMECMP_FUZZY:
-				read_gen_name(f_in_name, file->dirname, fnamecmpbuf);
-				fnamecmp = fnamecmpbuf;
+				if (file->dirname) {
+					pathjoin(fnamecmpbuf, MAXPATHLEN,
+						 file->dirname, xname);
+					fnamecmp = fnamecmpbuf;
+				} else
+					fnamecmp = xname;
 				break;
 			default:
-				if (j >= basis_dir_cnt) {
+				if (fnamecmp_type >= basis_dir_cnt) {
 					rprintf(FERROR,
 						"invalid basis_dir index: %d.\n",
-						j);
+						fnamecmp_type);
 					exit_cleanup(RERR_PROTOCOL);
 				}
 				pathjoin(fnamecmpbuf, sizeof fnamecmpbuf,
-					 basis_dir[j], fname);
+					 basis_dir[fnamecmp_type], fname);
 				fnamecmp = fnamecmpbuf;
 				break;
 			}
-		} else
-			fnamecmp = fname;
+			if (!fnamecmp || (server_filter_list.head
+			  && check_filter(&server_filter_list, fname, 0) < 0))
+				fnamecmp = fname;
+		} else {
+			/* Reminder: --inplace && --partial-dir are never
+			 * enabled at the same time. */
+			if (inplace && make_backups) {
+				if (!(fnamecmp = get_backup_name(fname)))
+					fnamecmp = fname;
+			} else if (partial_dir && partialptr)
+				fnamecmp = partialptr;
+			else
+				fnamecmp = fname;
+		}
 
 		initial_stats = stats;
 
 		/* open the file */
 		fd1 = do_open(fnamecmp, O_RDONLY, 0);
 
+		if (fd1 == -1 && protocol_version < 29) {
+			if (fnamecmp != fname) {
+				fnamecmp = fname;
+				fd1 = do_open(fnamecmp, O_RDONLY, 0);
+			}
+
+			if (fd1 == -1 && basis_dir[0]) {
+				/* pre-29 allowed only one alternate basis */
+				pathjoin(fnamecmpbuf, sizeof fnamecmpbuf,
+					 basis_dir[0], fname);
+				fnamecmp = fnamecmpbuf;
+				fd1 = do_open(fnamecmp, O_RDONLY, 0);
+			}
+		}
+
 		if (fd1 != -1 && do_fstat(fd1,&st) != 0) {
 			rsyserr(FERROR, errno, "fstat %s failed",
 				full_fname(fnamecmp));
@@ -564,8 +591,6 @@ int recv_files(int f_in, struct file_list *flist, char *local_name,
 				continue;
 			}
 
-			strlcpy(template, fnametmp, sizeof template);
-
 			/* we initially set the perms without the
 			 * setuid/setgid bits to ensure that there is no race
 			 * condition. They are then correctly updated after
@@ -579,7 +604,8 @@ int recv_files(int f_in, struct file_list *flist, char *local_name,
 			 * transferred, but that may not be the case with -R */
 			if (fd2 == -1 && relative_paths && errno == ENOENT
 			    && create_directory_path(fnametmp, orig_umask) == 0) {
-				strlcpy(fnametmp, template, sizeof fnametmp);
+				/* Get back to name with XXXXXX in it. */
+				get_tmpname(fnametmp, fname);
 				fd2 = do_mkstemp(fnametmp, file->mode & INITACCESSPERMS);
 			}
 			if (fd2 == -1) {
@@ -597,7 +623,7 @@ int recv_files(int f_in, struct file_list *flist, char *local_name,
 
 		/* log the transfer */
 		if (log_before_transfer)
-			log_recv(file, &initial_stats, iflags);
+			log_item(file, &initial_stats, iflags, NULL);
 		else if (!am_server && verbose && do_progress)
 			rprintf(FINFO, "%s\n", safe_fname(fname));
 
@@ -606,7 +632,7 @@ int recv_files(int f_in, struct file_list *flist, char *local_name,
 				       fname, fd2, file->length);
 
 		if (!log_before_transfer)
-			log_recv(file, &initial_stats, iflags);
+			log_item(file, &initial_stats, iflags, NULL);
 
 		if (fd1 != -1)
 			close(fd1);
@@ -637,14 +663,14 @@ int recv_files(int f_in, struct file_list *flist, char *local_name,
 
 		cleanup_disable();
 
-		if (recv_ok) {
-			if (remove_sent_files && recv_ok > 0) {
+		if (recv_ok > 0) {
+			if (remove_sent_files
+			    || (preserve_hard_links && file->link_u.links)) {
 				SIVAL(numbuf, 0, i);
 				send_msg(MSG_SUCCESS, numbuf, 4);
 			}
-		} else {
-			int msgtype = csum_length == SUM_LENGTH || read_batch ?
-				FERROR : FINFO;
+		} else if (!recv_ok) {
+			int msgtype = phase || read_batch ? FERROR : FINFO;
 			if (msgtype == FERROR || verbose) {
 				char *errstr, *redostr, *keptstr;
 				if (!(keep_partial && partialptr) && !inplace)
@@ -665,7 +691,7 @@ int recv_files(int f_in, struct file_list *flist, char *local_name,
 					errstr, safe_fname(fname),
 					keptstr, redostr);
 			}
-			if (csum_length != SUM_LENGTH) {
+			if (!phase) {
 				SIVAL(numbuf, 0, i);
 				send_msg(MSG_REDO, numbuf, 4);
 			}
@@ -673,35 +699,8 @@ int recv_files(int f_in, struct file_list *flist, char *local_name,
 	}
 	make_backups = save_make_backups;
 
-	if (delay_updates) {
-		for (i = -1; (i = next_delayed_bit(i)) >= 0; ) {
-			struct file_struct *file = flist->files[i];
-			fname = local_name ? local_name : f_name(file);
-			partialptr = partial_dir_fname(fname);
-			if (partialptr) {
-				if (make_backups && !make_backup(fname))
-					continue;
-				if (verbose > 2) {
-					rprintf(FINFO, "renaming %s to %s\n",
-						safe_fname(partialptr),
-						safe_fname(fname));
-				}
-				if (do_rename(partialptr, fname) < 0) {
-					rsyserr(FERROR, errno,
-						"rename failed for %s (from %s)",
-						full_fname(fname),
-						safe_fname(partialptr));
-				} else {
-					if (remove_sent_files) {
-						SIVAL(numbuf, 0, i);
-						send_msg(MSG_SUCCESS,numbuf,4);
-					}
-					handle_partial_dir(partialptr,
-							   PDIR_DELETE);
-				}
-			}
-		}
-	}
+	if (phase == 2 && delay_updates) /* for protocol_version < 29 */
+		handle_delayed_updates(flist, local_name);
 
 	if (verbose > 2)
 		rprintf(FINFO,"recv_files finished\n");

rsync.c

@@ -35,7 +35,6 @@ extern int am_starting_up;
 extern int preserve_uid;
 extern int preserve_gid;
 extern int inplace;
-extern int recurse;
 extern int keep_dirlinks;
 extern int make_backups;
 extern struct stats stats;

rsync.h

@@ -58,11 +58,12 @@
 /* These flags are used in the live flist data. */
 
 #define FLAG_TOP_DIR (1<<0)
-#define FLAG_HLINK_EOL (1<<1)	/* generator only */
-#define FLAG_MOUNT_POINT (1<<2)	/* sender only */
-#define FLAG_NO_FUZZY (1<<2)	/* generator only */
+#define FLAG_HLINK_EOL (1<<1)	/* receiver/generator */
+#define FLAG_MOUNT_POINT (1<<2)	/* sender */
+#define FLAG_NO_FUZZY (1<<2)	/* generator */
 #define FLAG_DEL_HERE (1<<3)	/* receiver/generator */
-#define FLAG_SENT (1<<3)	/* sender only */
+#define FLAG_SENT (1<<3)	/* sender */
+#define FLAG_HLINK_TOL (1<<4)	/* receiver/generator */
 
 /* update this if you make incompatible changes */
 #define PROTOCOL_VERSION 29
@@ -99,6 +100,7 @@
 
 #define IOERR_GENERAL	(1<<0) /* For backward compatibility, this must == 1 */
 #define IOERR_VANISHED	(1<<1)
+#define IOERR_DEL_LIMIT (1<<2)
 
 #define MAX_ARGS 1000
 
@@ -137,18 +139,27 @@
 #define DEL_TERSE		(1<<3)
 
 /* For use by the itemize_changes code */
-#define ITEM_UPDATING (1<<0)
+#define ITEM_DUMMY_BIT (1<<0)
 #define ITEM_REPORT_CHECKSUM (1<<1)
 #define ITEM_REPORT_SIZE (1<<2)
 #define ITEM_REPORT_TIME (1<<3)
 #define ITEM_REPORT_PERMS (1<<4)
 #define ITEM_REPORT_OWNER (1<<5)
 #define ITEM_REPORT_GROUP (1<<6)
-#define ITEM_IS_NEW (1<<7)
-#define ITEM_USING_ALT_BASIS (1<<8)
+#define ITEM_REPORT_XATTRS (1<<7)
+#define ITEM_BASIS_TYPE_FOLLOWS (1<<11)
+#define ITEM_XNAME_FOLLOWS (1<<12)
+#define ITEM_IS_NEW (1<<13)
+#define ITEM_LOCAL_CHANGE (1<<14)
+#define ITEM_TRANSFER (1<<15)
+/* These are outside the range of the transmitted flags. */
+#define ITEM_NO_DEST_AND_NO_UPDATE (1<<16) /* used by itemize() */
+#define ITEM_MISSING_DATA (1<<16)	   /* used by log_formatted() */
+#define ITEM_DELETED (1<<17)		   /* used by log_formatted() */
+
+#define SIGNIFICANT_ITEM_FLAGS (~(\
+	ITEM_BASIS_TYPE_FOLLOWS | ITEM_XNAME_FOLLOWS | ITEM_LOCAL_CHANGE))
 
-#define ITEM_MISSING_DATA (1<<16) /* these are outside the transmitted flags */
-#define ITEM_DELETED (1<<17)
 
 /* Log-message categories.  FLOG and FCLIENT are only used on the daemon
  * side for custom logging -- they don't get sent over the socket. */
@@ -471,7 +482,7 @@ struct idev {
 #define HL_SKIP		1
 
 struct hlink {
-	struct file_struct *next;
+	int next;
 	int hlindex;
 };
 

rsync.yo

@@ -1,5 +1,5 @@
 mailto(rsync-bugs@samba.org)
-manpage(rsync)(1)(28 Feb 2005)()()
+manpage(rsync)(1)(30 Mar 2005)()()
 manpagename(rsync)(faster, flexible replacement for rcp)
 manpagesynopsis()
 
@@ -297,7 +297,6 @@ Here is a short summary of the options available in rsync. Please refer
 to the detailed description below for a complete description.  verb(
  -v, --verbose               increase verbosity
  -q, --quiet                 suppress non-error messages
- -c, --checksum              always checksum
  -c, --checksum              skip based on checksum, not mod-time & size
  -a, --archive               archive mode; same as -rlptgoD (no -H)
  -r, --recursive             recurse into directories
@@ -329,7 +328,7 @@ to the detailed description below for a complete description.  verb(
  -x, --one-file-system       don't cross filesystem boundaries
  -B, --block-size=SIZE       force a fixed checksum block-size
  -e, --rsh=COMMAND           specify the remote shell to use
-     --rsync-path=PATH       specify path to rsync on the remote machine
+     --rsync-path=PROGRAM    specify the rsync to run on remote machine
      --existing              only update files that already exist
      --ignore-existing       ignore files that already exist on receiver
      --remove-sent-files     sent files/symlinks are removed from sender
@@ -354,6 +353,7 @@ to the detailed description below for a complete description.  verb(
  -T, --temp-dir=DIR          create temporary files in directory DIR
  -y, --fuzzy                 find similar file for basis if no dest file
      --compare-dest=DIR      also compare received files relative to DIR
+     --copy-dest=DIR         ... and include copies of unchanged files
      --link-dest=DIR         hardlink to files in DIR when unchanged
  -z, --compress              compress file data during the transfer
  -C, --cvs-exclude           auto-ignore files in the same way CVS does
@@ -380,6 +380,7 @@ to the detailed description below for a complete description.  verb(
      --bwlimit=KBPS          limit I/O bandwidth; KBytes per second
      --write-batch=FILE      write a batched update to FILE
      --read-batch=FILE       read a batched update from FILE
+     --protocol=NUM          force an older protocol version to be used
      --checksum-seed=NUM     set block/file checksum seed (advanced)
  -4, --ipv4                  prefer IPv4
  -6, --ipv6                  prefer IPv6
@@ -750,8 +751,8 @@ is only relevant without bf(--delete) because deletions are now done depth-first
 Requires the bf(--recursive) option (which is implied by bf(-a)) to have any effect.
 
 dit(bf(--max-delete=NUM)) This tells rsync not to delete more than NUM
-files or directories. This is useful when mirroring very large trees
-to prevent disasters.
+files or directories (NUM must be non-zero).
+This is useful when mirroring very large trees to prevent disasters.
 
 dit(bf(--max-size=SIZE)) This tells rsync to avoid transferring any
 file that is larger than the specified SIZE. The SIZE value can be
@@ -787,10 +788,18 @@ environment variable, which accepts the same range of values as bf(-e).
 
 See also the bf(--blocking-io) option which is affected by this option.
 
-dit(bf(--rsync-path=PATH)) Use this to specify the path to the copy of
-rsync on the remote machine. Useful when it's not in your path. Note
-that this is the full path to the binary, not just the directory that
-the binary is in.
+dit(bf(--rsync-path=PROGRAM)) Use this to specify what program is to be run
+on the remote machine to start-up rsync.  Often used when rsync is not in
+the default remote-shell's path (e.g. --rsync-path=/usr/local/bin/rsync).
+Note that PROGRAM is run with the help of a shell, so it can be any
+program, script, or command sequence you'd care to run, so long as it does
+not corrupt the standard-in & standard-out that rsync is using to
+communicate.
+
+One tricky example is to set a different default directory on the remote
+machine for use with the bf(--relative) option.  For instance:
+
+quote(tt(    rsync -avR --rsync-path="cd /a/b && rsync" hst:c/d /e/))
 
 dit(bf(-C, --cvs-exclude)) This is a useful shorthand for excluding a
 broad range of files that you often don't want to transfer between
@@ -814,7 +823,7 @@ See the bf(cvs(1)) manual for more information.
 
 If you're combining bf(-C) with your own bf(--filter) rules, you should
 note that these CVS excludes are appended at the end of your own rules,
-regardless of where the -C was placed on the command-line.  This makes them
+regardless of where the bf(-C) was placed on the command-line.  This makes them
 a lower priority than any rules you specified explicitly.  If you want to
 control where these CVS excludes get inserted into your filter rules, you
 should omit the bf(-C) as a command-line option and use a combination of
@@ -945,14 +954,32 @@ directory.  This is useful for creating a sparse backup of just files that
 have changed from an earlier backup.
 
 Beginning in version 2.6.4, multiple bf(--compare-dest) directories may be
-provided and rsync will search the list in the order specified until it
-finds an existing file.  That first discovery is used as the basis file,
-and also determines if the transfer needs to happen.
+provided, which will cause rsync to search the list in the order specified
+for an exact match.
+If a match is found that differs only in attributes, a local copy is made
+and the attributes updated.
+If a match is not found, a basis file from one of the em(DIR)s will be
+selected to try to speed up the transfer.
 
 If em(DIR) is a relative path, it is relative to the destination directory.
-See also bf(--link-dest).
+See also bf(--copy-dest) and bf(--link-dest).
 
-dit(bf(--link-dest=DIR)) This option behaves like bf(--compare-dest), but
+dit(bf(--copy-dest=DIR)) This option behaves like bf(--compare-dest), but
+rsync will also copy unchanged files found in em(DIR) to the destination
+directory using a local copy.
+This is useful for doing transfers to a new destination while leaving
+existing files intact, and then doing a flash-cutover when all files have
+been successfully transferred.
+
+Multiple bf(--copy-dest) directories may be provided, which will cause
+rsync to search the list in the order specified for an unchanged file.
+If a match is not found, a basis file from one of the em(DIR)s will be
+selected to try to speed up the transfer.
+
+If em(DIR) is a relative path, it is relative to the destination directory.
+See also bf(--compare-dest) and bf(--link-dest).
+
+dit(bf(--link-dest=DIR)) This option behaves like bf(--copy-dest), but
 unchanged files are hard linked from em(DIR) to the destination directory.
 The files must be identical in all preserved attributes (e.g. permissions,
 possibly ownership) in order for the files to be linked together.
@@ -960,13 +987,16 @@ An example:
 
 quote(tt(  rsync -av --link-dest=$PWD/prior_dir host:src_dir/ new_dir/))
 
-Beginning with version 2.6.4, if more than one bf(--link-dest) option is
-specified, rsync will try to find an exact match to link with (searching
-the list in the order specified), and if not found, a basis file from one
-of the em(DIR)s will be selected to try to speed up the transfer.
+Beginning in version 2.6.4, multiple bf(--link-dest) directories may be
+provided, which will cause rsync to search the list in the order specified
+for an exact match.
+If a match is found that differs only in attributes, a local copy is made
+and the attributes updated.
+If a match is not found, a basis file from one of the em(DIR)s will be
+selected to try to speed up the transfer.
 
 If em(DIR) is a relative path, it is relative to the destination directory.
-See also bf(--compare-dest).
+See also bf(--compare-dest) and bf(--copy-dest).
 
 Note that rsync versions prior to 2.6.1 had a bug that could prevent
 bf(--link-dest) from working properly for a non-root user when bf(-o) was specified
@@ -1021,25 +1051,35 @@ dit(bf(-i, --itemize-changes)) Requests a simple itemized list of the
 changes that are being made to each file, including attribute changes.
 This is exactly the same as specifying bf(--log-format='%i %n%L').
 
-The "%i" escape has a cryptic output that is 8 letters long.  The general
-format is as follows:
+The "%i" escape has a cryptic output that is 9 letters long.  The general
+format is like the string bf(UXcstpoga)), where bf(U) is replaced by the
+kind of update being done, bf(X) is replaced by the file-type, and the
+other letters represent attributes that may be output if they are being
+modified.
 
-quote(tt(  =Xcstpog))
+The update types that replace the bf(U) are as follows:
 
-The bf(=) is output as either a bf(<) (receive) or a bf(>) (send) if the
-item is being transferred, a bf(.) if only the attributes are being
-updated, or a bf(=) if the items are identical.  Note that when a symlink
-or a device gets its value changed, that is considered to be a transfer (as
-opposed to a change in permissions, ownership, etc.).
+quote(itemize(
+  it() A bf(<) means that a file is being transferred to the remote host
+  (sent).
+  it() A bf(>) means that a file is being transferred to the local host
+  (received).
+  it() A bf(c) means that a local change/creation is occuring for the item
+  (such as the creation of a directory or the changing of a symlink, etc.).
+  it() A bf(h) means that the item is a hard-link to another item (requires
+  bf(--hard-links)).
+  it() A bf(.) means that the item is not being updated (though it might
+  have attributes that are being modified).
+))
 
-The bf(X) will be replaced by one of the following: an "f" for a file, a
-"d" for a dir, an "L" for a symlink, or a "D" for a device.
+The file-types that replace the bf(X) are: bf(f) for a file, a bf(d) for a
+directory, an bf(L) for a symlink, and a bf(D) for a device.
 
-The rest of the letters in the string above are the actual letters that
+The other letters in the string above are the actual letters that
 will be output if the associated attribute for the item is being updated or
 a "." for no change.  Three exceptions to this are: (1) a newly created
-item replaces each letter with a "+", (2) an identical item replaces each
-letter with a space, and (3) an unknown attribute replaces each letter with
+item replaces each letter with a "+", (2) an identical item replaces the
+dots with spaces, and (3) an unknown attribute replaces each letter with
 a "?" (this happens when talking to an older rsync).
 
 The attribute that is associated with each letter is as follows:
@@ -1060,10 +1100,12 @@ quote(itemize(
   server's value (requires bf(--owner) and root privileges).
   it() A bf(g) means the group is different and is being updated to the
   server's value (requires bf(--group) and the authority to set the group).
+  it() The bf(a) is reserved for a future enhanced version that supports
+  extended file attributes, such as ACLs.
 ))
 
 One other output is possible:  when deleting files, the "%i" will output
-the string "deleting" for each item that is being removed (assuming that
+the string "*deleting" for each item that is being removed (assuming that
 you are talking to a recent enough rsync that it logs deletions instead of
 outputting them as a verbose message).
 
@@ -1234,6 +1276,14 @@ file previously generated by bf(--write-batch).
 If em(FILE) is "-" the batch data will be read from standard input.
 See the "BATCH MODE" section for details.
 
+dit(bf(--protocol=NUM)) Force an older protocol version to be used.  This
+is useful for creating a batch file that is compatible with an older
+version of rsync.  For instance, if rsync 2.6.4 is being used with the
+bf(--write-batch) option, but rsync 2.6.3 is what will be used to run the
+bf(--read-batch) option, you should use "--protocol=28" (when creating the
+batch file) to force the older protocol version to be used in the batch
+file (assuming you can't upgrade the rsync on the reading system to 2.6.4).
+
 dit(bf(-4, --ipv4) or bf(-6, --ipv6)) Tells rsync to prefer IPv4/IPv6
 when creating sockets.  This only affects sockets that rsync has direct
 control over, such as the outgoing socket when directly contacting an
@@ -1813,7 +1863,7 @@ Caveats:
 The read-batch option expects the destination tree that it is updating
 to be identical to the destination tree that was used to create the
 batch update fileset.  When a difference between the destination trees
-is encountered the update might be discarded with no error (if the file
+is encountered the update might be discarded with a warning (if the file
 appears to be up-to-date already) or the file-update may be attempted
 and then, if the file fails to verify, the update discarded with an
 error.  This means that it should be safe to re-run a read-batch operation
@@ -1828,10 +1878,10 @@ destination tree.
 The rsync version used on all destinations must be at least as new as the
 one used to generate the batch file.  Rsync will die with an error if the
 protocol version in the batch file is too new for the batch-reading rsync
-to handle.
-
-The bf(--dry-run) (bf(-n)) option does not work in batch mode and yields a runtime
-error.
+to handle.  See also the bf(--protocol) option for a way to have the
+creating rsync generate a batch file that an older rsync can understand.
+(Note that batch files changed format in version 2.6.3, so mixing versions
+older than that with newer versions will not work.)
 
 When reading a batch file, rsync will force the value of certain options
 to match the data in the batch file if you didn't set them to the same
@@ -1915,6 +1965,7 @@ was made to manipulate 64-bit files on a platform that cannot support
 them; or an option was specified that is supported by the client and
 not by the server.
 dit(bf(5)) Error starting client-server protocol
+dit(bf(6)) Daemon unable to append to log-file
 dit(bf(10)) Error in socket I/O
 dit(bf(11)) Error in file I/O
 dit(bf(12)) Error in rsync protocol data stream
@@ -1925,6 +1976,7 @@ dit(bf(21)) Some error returned by waitpid()
 dit(bf(22)) Error allocating core memory buffers
 dit(bf(23)) Partial transfer due to error
 dit(bf(24)) Partial transfer due to vanished source files
+dit(bf(25)) The --max-delete limit stopped deletions
 dit(bf(30)) Timeout in data send/receive
 enddit()
 

rsyncd.conf.yo

@@ -1,5 +1,5 @@
 mailto(rsync-bugs@samba.org)
-manpage(rsyncd.conf)(5)(28 Feb 2005)()()
+manpage(rsyncd.conf)(5)(30 Mar 2005)()()
 manpagename(rsyncd.conf)(configuration file for rsync server)
 manpagesynopsis()
 
@@ -156,9 +156,14 @@ Note that you are free to setup user/group information in the chroot area
 differently from your normal system.  For example, you could abbreviate
 the list of users and groups.  Also, you can protect this information from
 being downloaded/uploaded by adding an exclude rule to the rsync.conf file
-(e.g. "exclude = /etc/").  Note that having the exclusion affect uploads
+(e.g. "exclude = /etc/**").  Note that having the exclusion affect uploads
 is a relatively new feature in rsync, so make sure your server is running
-at least 2.6.3 to effect this.
+at least 2.6.3 to effect this.  Also note that it is safest to exclude a
+directory and all its contents combining the rule "/some/dir/" with the
+rule "/some/dir/**" just to be sure that rsync will not allow deeper
+access to some of the excluded files inside the directory (rsync tries to
+do this automatically, but you might as well specify both to be extra
+sure).
 
 dit(bf(port)) You can override the default port the daemon will listen on
 by specifying this value (defaults to 873).  This is ignored if the daemon
@@ -368,7 +373,9 @@ If you want to customize the log lines, see the "log format" option.
 dit(bf(log format)) The "log format" option allows you to specify the
 format used for logging file transfers when transfer logging is enabled.
 The format is a text string containing embedded single-character escape
-sequences prefixed with a percent (%) character.
+sequences prefixed with a percent (%) character.  An optional numeric
+field width may also be specified between the percent and the escape
+letter (e.g. "%-50n %8l %07p").
 
 The default log format is "%o %h [%a] %m (%u) %f %l", and a "%t [%p] "
 is always prefixed when using the "log file" option.
@@ -384,9 +391,11 @@ quote(itemize(
   it() %l for the length of the file in bytes
   it() %p for the process ID of this rsync session
   it() %o for the operation, which is "send", "recv", or "del."
+  (the latter includes the trailing period)
   it() %f for the filename (long form on sender; no trailing "/")
   it() %n for the filename (short form; trailing "/" on dir)
-  it() %L either the string " -> SYMLINK" or "" if not a symlink
+  it() %L either the string " -> SYMLINK", or " => HARDLINK" or an
+  empty string (where bf(SYMLINK) or bf(HARDLINK) is a filename)
   it() %P for the module path
   it() %m for the module name
   it() %t for the current date time
@@ -401,7 +410,7 @@ For a list of what the characters mean that are output by "%i", see the
 bf(--itemize-changes) option in the rsync manpage.
 
 Note that some of the logged output changes when talking with older
-rsync versions.  For instance, deleted files were only logged as verbose
+rsync versions.  For instance, deleted files were only output as verbose
 messages prior to rsync 2.6.4.
 
 dit(bf(timeout)) The "timeout" option allows you to override the

sender.c

@@ -35,6 +35,7 @@ extern int make_backups;
 extern int do_progress;
 extern int inplace;
 extern struct stats stats;
+extern struct file_list *the_file_list;
 extern char *log_format;
 
 
@@ -99,18 +100,16 @@ static struct sum_struct *receive_sums(int f)
 	return s;
 }
 
-static struct file_list *the_flist;
-
-void successful_send(int i)
+void successful_send(int ndx)
 {
 	char fname[MAXPATHLEN];
 	struct file_struct *file;
 	unsigned int offset;
 
-	if (!the_flist || i < 0 || i >= the_flist->count)
+	if (ndx < 0 || ndx >= the_file_list->count)
 		return;
 
-	file = the_flist->files[i];
+	file = the_file_list->files[ndx];
 	/* The generator might tell us about symlinks we didn't send. */
 	if (!(file->flags & FLAG_SENT) && !S_ISLNK(file->mode))
 		return;
@@ -126,6 +125,73 @@ void successful_send(int i)
 	}
 }
 
+static void write_ndx_and_attrs(int f_out, int ndx, int iflags,
+				uchar fnamecmp_type, char *buf, int len)
+{
+	write_int(f_out, ndx);
+	if (protocol_version < 29)
+		return;
+	write_shortint(f_out, iflags);
+	if (iflags & ITEM_BASIS_TYPE_FOLLOWS)
+		write_byte(f_out, fnamecmp_type);
+	if (iflags & ITEM_XNAME_FOLLOWS)
+		write_vstring(f_out, buf, len);
+}
+
+/* This is also used by receive.c with f_out = -1. */
+int read_item_attrs(int f_in, int f_out, int ndx, uchar *type_ptr,
+		    char *buf, int *len_ptr)
+{
+	int len;
+	uchar fnamecmp_type = FNAMECMP_FNAME;
+	int iflags = protocol_version >= 29 ? read_shortint(f_in)
+		   : ITEM_TRANSFER | ITEM_MISSING_DATA;
+
+	/* Handle the new keep-alive (no-op) packet. */
+	if (ndx == the_file_list->count && iflags == ITEM_IS_NEW)
+		;
+	else if (ndx < 0 || ndx >= the_file_list->count) {
+		rprintf(FERROR, "Invalid file index: %d (count=%d) [%s]\n",
+			ndx, the_file_list->count, who_am_i());
+		exit_cleanup(RERR_PROTOCOL);
+	} else if (iflags == ITEM_IS_NEW) {
+		rprintf(FERROR, "Invalid itemized flag word: %x [%s]\n",
+			iflags, who_am_i());
+		exit_cleanup(RERR_PROTOCOL);
+	}
+
+	if (iflags & ITEM_BASIS_TYPE_FOLLOWS)
+		fnamecmp_type = read_byte(f_in);
+	*type_ptr = fnamecmp_type;
+
+	if (iflags & ITEM_XNAME_FOLLOWS) {
+		if ((len = read_vstring(f_in, buf, MAXPATHLEN)) < 0)
+			exit_cleanup(RERR_PROTOCOL);
+	} else {
+		*buf = '\0';
+		len = -1;
+	}
+	*len_ptr = len;
+
+	/* Temporary handling of 2.6.4pre3 */
+	if (iflags & ITEM_DUMMY_BIT && iflags & (ITEM_LOCAL_CHANGE|ITEM_TRANSFER))
+		iflags &= ~ITEM_DUMMY_BIT;
+
+	if (iflags & ITEM_TRANSFER) {
+		if (!S_ISREG(the_file_list->files[ndx]->mode)) {
+			rprintf(FERROR,
+				"received request to transfer non-regular file: %d [%s]\n",
+				ndx, who_am_i());
+			exit_cleanup(RERR_PROTOCOL);
+		}
+	} else if (f_out >= 0) {
+		write_ndx_and_attrs(f_out, ndx, iflags,
+				    fnamecmp_type, buf, len);
+	}
+
+	return iflags;
+}
+
 void send_files(struct file_list *flist, int f_out, int f_in)
 {
 	int fd = -1;
@@ -133,9 +199,11 @@ void send_files(struct file_list *flist, int f_out, int f_in)
 	struct map_struct *mbuf = NULL;
 	STRUCT_STAT st;
 	char *fname2, fname[MAXPATHLEN];
-	int iflags;
+	char xname[MAXPATHLEN];
+	uchar fnamecmp_type;
+	int iflags, xlen;
 	struct file_struct *file;
-	int phase = 0;
+	int phase = 0, max_phase = protocol_version >= 29 ? 2 : 1;
 	struct stats initial_stats;
 	int save_make_backups = make_backups;
 	int itemizing = am_daemon ? daemon_log_format_has_i
@@ -145,39 +213,27 @@ void send_files(struct file_list *flist, int f_out, int f_in)
 	if (verbose > 2)
 		rprintf(FINFO, "send_files starting\n");
 
-	the_flist = flist;
-
 	while (1) {
 		unsigned int offset;
 
 		i = read_int(f_in);
 		if (i == -1) {
-			if (phase == 0) {
-				phase++;
-				csum_length = SUM_LENGTH;
-				write_int(f_out, -1);
-				if (verbose > 2)
-					rprintf(FINFO, "send_files phase=%d\n", phase);
-				/* For inplace: redo phase turns off the backup
-				 * flag so that we do a regular inplace send. */
-				make_backups = 0;
-				continue;
-			}
-			break;
+			if (++phase > max_phase)
+				break;
+			csum_length = SUM_LENGTH;
+			if (verbose > 2)
+				rprintf(FINFO, "send_files phase=%d\n", phase);
+			write_int(f_out, -1);
+			/* For inplace: redo phase turns off the backup
+			 * flag so that we do a regular inplace send. */
+			make_backups = 0;
+			continue;
 		}
 
-		if (i < 0 || i >= flist->count) {
-			/* Handle the new keep-alive (no-op) packet. */
-			if (i == flist->count && protocol_version >= 29
-	 		    && read_shortint(f_in) == ITEM_IS_NEW) {
-				write_int(f_out, i);
-				write_shortint(f_out, ITEM_IS_NEW);
-				continue;
-			}
-			rprintf(FERROR, "Invalid file index %d (count=%d)\n",
-				i, flist->count);
-			exit_cleanup(RERR_PROTOCOL);
-		}
+		iflags = read_item_attrs(f_in, f_out, i, &fnamecmp_type,
+					 xname, &xlen);
+		if (iflags == ITEM_IS_NEW) /* no-op packet */
+			continue;
 
 		file = flist->files[i];
 		if (file->dir.root) {
@@ -192,45 +248,29 @@ void send_files(struct file_list *flist, int f_out, int f_in)
 		if (verbose > 2)
 			rprintf(FINFO, "send_files(%d, %s)\n", i, fname);
 
-		if (protocol_version >= 29) {
-			iflags = read_shortint(f_in);
-			if (!(iflags & ITEM_UPDATING) || !S_ISREG(file->mode)) {
-				int see_item = itemizing && (iflags || verbose > 1);
-				write_int(f_out, i);
-				write_shortint(f_out, iflags);
-				if (am_server) {
-					if (am_daemon && !dry_run && see_item)
-						log_send(file, &stats, iflags);
-				} else if (see_item || iflags & ITEM_UPDATING
-				    || (S_ISDIR(file->mode)
-				     && iflags & ITEM_REPORT_TIME))
-					log_send(file, &stats, iflags);
-				continue;
-			}
-		} else
-			iflags = ITEM_UPDATING | ITEM_MISSING_DATA;
-
-		if (inplace && protocol_version >= 29) {
-			updating_basis_file = !(iflags & ITEM_USING_ALT_BASIS);
-		} else
-			updating_basis_file = inplace && !make_backups;
-
-		if (!S_ISREG(file->mode)) {
-			rprintf(FERROR, "[%s] got index of non-regular file: %d\n",
-				who_am_i(), i);
+		if (!(iflags & ITEM_TRANSFER)) {
+			maybe_log_item(file, iflags, itemizing, xname);
+			continue;
+		}
+		if (phase == 2) {
+			rprintf(FERROR,
+				"got transfer request in phase 2 [%s]\n",
+				who_am_i());
 			exit_cleanup(RERR_PROTOCOL);
 		}
 
+		updating_basis_file = inplace && (protocol_version >= 29
+			? fnamecmp_type == FNAMECMP_FNAME : !make_backups);
+
 		stats.current_file_index = i;
 		stats.num_transferred_files++;
 		stats.total_transferred_size += file->length;
 
 		if (dry_run) { /* log the transfer */
 			if (!am_server && log_format)
-				log_send(file, &stats, iflags);
-			write_int(f_out, i);
-			if (protocol_version >= 29)
-				write_shortint(f_out, iflags);
+				log_item(file, &stats, iflags, NULL);
+			write_ndx_and_attrs(f_out, i, iflags, fnamecmp_type,
+					    xname, xlen);
 			continue;
 		}
 
@@ -281,9 +321,8 @@ void send_files(struct file_list *flist, int f_out, int f_in)
 				safe_fname(fname), (double)st.st_size);
 		}
 
-		write_int(f_out, i);
-		if (protocol_version >= 29)
-			write_shortint(f_out, iflags);
+		write_ndx_and_attrs(f_out, i, iflags, fnamecmp_type,
+				    xname, xlen);
 		write_sum_head(f_out, s);
 
 		if (verbose > 2) {
@@ -292,15 +331,18 @@ void send_files(struct file_list *flist, int f_out, int f_in)
 		}
 
 		if (log_before_transfer)
-			log_send(file, &initial_stats, iflags);
+			log_item(file, &initial_stats, iflags, NULL);
 		else if (!am_server && verbose && do_progress)
 			rprintf(FINFO, "%s\n", safe_fname(fname2));
 
 		set_compression(fname);
 
 		match_sums(f_out, s, mbuf, st.st_size);
+		if (do_progress)
+			end_progress(st.st_size);
+
 		if (!log_before_transfer)
-			log_send(file, &initial_stats, iflags);
+			log_item(file, &initial_stats, iflags, NULL);
 
 		if (mbuf) {
 			j = unmap_file(mbuf);

testsuite/chown.test

@@ -15,6 +15,16 @@
 
 set -x
 
+case `id -u` in
+'') ;; # If "id" failed, try to continue...
+0)  ;;
+*)  if [ -f /usr/bin/fakeroot ]; then
+	echo "Let's try re-running the script under fakeroot..."
+	exec /usr/bin/fakeroot /bin/sh "$0"
+    fi
+    ;;
+esac
+
 # Build some hardlinks
 
 mkdir "$fromdir"

testsuite/compare-dest.test

@@ -31,9 +31,9 @@ $RSYNC -av --exclude=/text --exclude=etc-ltr-list "$fromdir/" "$chkdir/"
 checkit "$RSYNC -avv --no-whole-file \
     --compare-dest=\"$alt1dir\" --compare-dest=\"$alt2dir\" \
     \"$fromdir/\" \"$todir/\"" "$chkdir" "$todir"
-#checkit "$RSYNC -avv --no-whole-file \
-#    --copy-dest=\"$alt1dir\" --copy-dest=\"$alt2dir\" \
-#    \"$fromdir/\" \"$todir/\"" "$fromdir" "$todir"
+checkit "$RSYNC -avv --no-whole-file \
+    --copy-dest=\"$alt1dir\" --copy-dest=\"$alt2dir\" \
+    \"$fromdir/\" \"$todir/\"" "$fromdir" "$todir"
 
 # The script would have aborted on error, so getting here means we've won.
 exit 0

testsuite/devices.test

@@ -13,8 +13,13 @@
 
 case `id -u` in
 '') ;; # If "id" failed, try to continue...
-0) ;;
-*) test_skipped "Rsync won't copy devices unless we're root" ;;
+0)  ;;
+*)  if [ -f /usr/bin/fakeroot ]; then
+	echo "Let's try re-running the script under fakeroot..."
+	exec /usr/bin/fakeroot /bin/sh "$0"
+    fi
+    test_skipped "Rsync won't copy devices unless we're root"
+    ;;
 esac
 
 # TODO: Need to test whether hardlinks are possible on this OS/filesystem

testsuite/fuzzy.test

@@ -0,0 +1,25 @@
+#! /bin/sh
+
+# Copyright (C) 2005 by Wayne Davison <wayned@samba.org>
+
+# This program is distributable under the terms of the GNU GPL see
+# COPYING).
+
+# Test rsync handling of the --fuzzy option.
+
+. "$suitedir/rsync.fns"
+
+set -x
+
+mkdir "$fromdir"
+mkdir "$todir"
+
+cp -p "$srcdir"/rsync.c "$fromdir"/rsync.c
+cp -p "$fromdir"/rsync.c "$todir"/rsync2.c
+
+# Let's do it!
+checkit "$RSYNC -avvi --no-whole-file --fuzzy --delete-after \
+    \"$fromdir/\" \"$todir/\"" "$fromdir" "$todir"
+
+# The script would have aborted on error, so getting here means we've won.
+exit 0

testsuite/hardlinks.test

@@ -31,7 +31,7 @@ ln "$name1" "$name2" || fail "Can't create hardlink"
 ln "$name2" "$name3" || fail "Can't create hardlink"
 cp "$name2" "$name4" || fail "Can't copy file"
 
-checkit "$RSYNC -aHvv \"$fromdir/\" \"$todir/\"" "$fromdir" "$todir"
+checkit "$RSYNC -aHivv \"$fromdir/\" \"$todir/\"" "$fromdir" "$todir"
 
 # The script would have aborted on error, so getting here means we've won.
 exit 0

testsuite/longdir.test

@@ -17,7 +17,11 @@ longdir="$fromdir/$longname/$longname/$longname"
 makepath "$longdir" || test_skipped "unable to create long directory"
 touch "$longdir/1" || test_skipped "unable to create files in long directory"
 date > "$longdir/1"
-ls -la / > "$longdir/2"
+if [ -r /etc ]; then
+    ls -la /etc >"$longdir/2"
+else
+    ls -la / >"$longdir/2"
+fi
 checkit "$RSYNC --delete -avH \"$fromdir/\" \"$todir\"" "$fromdir/" "$todir"
 
 # The script would have aborted on error, so getting here means we've won.

util.c

@@ -245,7 +245,8 @@ static int safe_read(int desc, char *ptr, size_t len)
 
 /** Copy a file.
  *
- * This is used in conjunction with the --temp-dir and --backup options */
+ * This is used in conjunction with the --temp-dir, --backup, and
+ * --copy-dest options. */
 int copy_file(char *source, char *dest, mode_t mode)
 {
 	int ifd;
@@ -881,7 +882,7 @@ int pop_dir(char *dir)
  * empty line.  This function can return only MAX_SAFE_NAMES values at a
  * time!  The returned value can be longer than MAXPATHLEN (because we
  * may be trying to output an error about a too-long filename)! */
-const char *safe_fname(const char *fname)
+char *safe_fname(const char *fname)
 {
 #define MAX_SAFE_NAMES 4
 	static char fbuf[MAX_SAFE_NAMES][MAXPATHLEN*2];
@@ -1250,11 +1251,11 @@ const char *find_filename_suffix(const char *fn, int fn_len, int *len_ptr)
 			break;
 		s_len = fn_len - (s - fn);
 		fn_len = s - fn;
-		if (s_len == 3) {
+		if (s_len == 4) {
 			if (strcmp(s+1, "bak") == 0
 			 || strcmp(s+1, "old") == 0)
 				continue;
-		} else if (s_len == 4) {
+		} else if (s_len == 5) {
 			if (strcmp(s+1, "orig") == 0)
 				continue;
 		} else if (s_len > 2 && had_tilde