docs: add section on performance

docs/content/docs.md

@@ -778,6 +778,135 @@ modification time of the source object.
 Hashes are not included in system metadata as there is a well defined
 way of reading those already.
 
+## Performance {#performance}
+
+Rclone's defaults are fairly conservative. They are chosen to work on
+small machines, slow links and with cloud providers which rate limit
+heavily. If you have plenty of bandwidth, memory and a fast storage
+system then rclone can usually be made to run a great deal faster by
+adjusting a few flags.
+
+The right values depend on the size of your files, your network and
+the storage systems at each end, so treat the numbers below as
+starting points. Change one thing at a time and measure the result.
+
+### Measuring
+
+Use `rclone copy` with [-P/--progress](#p-progress) to watch the
+transfer rate while you are experimenting. For unattended runs log the
+stats instead, e.g. `--stats 1m --stats-one-line --stats-log-level
+NOTICE`.
+
+A transfer is always limited by something: the network bandwidth, the
+rate at which the storage system will accept requests, disk IO, CPU or
+memory. Tuning is working out which limit you are hitting and raising
+it until you hit the next one. Watch the network, disk, CPU and memory
+use of the machine rclone is running on as well as rclone's stats.
+
+### Lots of files
+
+The time taken to transfer small files is dominated by the round trip
+time of the requests rather than by the bandwidth, so the way to make
+them go faster is to do more of them at once.
+
+[--transfers](#transfers-int) sets the number of files transferred in
+parallel. The default is 4. Double this until the transfer rate stops
+increasing or the storage system starts returning errors. Values of 32
+to 64 are common and values in the hundreds work well with the large
+object storage systems (e.g. s3, azureblob, googlecloudstorage, b2)
+when transferring lots of small files.
+
+[--checkers](#checkers-int) sets the number of directory listings and
+file comparisons done in parallel. The default is 8. Raising this
+speeds up syncs where most of the files are already present on the
+destination and stops the checking being the bottleneck when
+`--transfers` is large. 16 to 64 is a reasonable range.
+
+Not all storage systems like lots of parallel requests. If you see rate
+limiting errors (e.g. HTTP 429 or 503 errors) or lots of low level
+retries in the log then you have gone too far, so reduce `--transfers`
+and `--checkers` or use [--tpslimit](#tpslimit-float) to cap the
+request rate. Local disks, especially spinning ones, and network file
+systems may get slower rather than faster with too much parallelism.
+
+### Large files
+
+A single stream often won't fill a fast link, so rclone can transfer a
+large file as several chunks in parallel. These are called multi-thread
+transfers and are used for files bigger than
+[--multi-thread-cutoff](#multi-thread-cutoff) on backends which
+support them.
+
+The number of chunks transferred in parallel is set by
+[--multi-thread-streams](#multi-thread-streams-int) (default 4) and
+this works for all backends so increase this first to make large files
+transfer faster.
+
+The size of the chunks is set by the backend, usually with a
+`--backend-chunk-size` flag, e.g. `--s3-chunk-size`. Backends without
+a natural chunk size (e.g. local, smb) use
+[--multi-thread-chunk-size](#multi-thread-chunk-size-sizesuffix).
+Larger chunks mean fewer requests which increases throughput on big
+files. See the docs for the backend you are using for details.
+
+Chunked uploads are usually buffered in memory, so in the worst case
+rclone will use
+
+```text
+--transfers * upload concurrency * chunk size
+```
+
+of memory for buffers. For example `--transfers 256` with an upload
+concurrency of 8 and a chunk size of 16 MiB could use up to 32 GiB.
+When you raise `--transfers` to speed up small files you may therefore
+need to lower the upload concurrency, or set
+[--max-buffer-memory](#max-buffer-memory) which limits the total
+memory used for buffers and makes transfers wait when the limit is
+reached.
+
+[--use-mmap](#use-mmap) makes rclone return buffer memory to the OS as
+soon as it has finished with it which is helpful when using lots of
+transfers or large chunks.
+
+### A mixture of large and small files
+
+By default rclone transfers files in roughly the order it finds them.
+With a mixture of file sizes this means that at times all the
+transfers are large files, which fill the bandwidth but make few
+requests, and at other times all the transfers are small files, which
+make lots of requests but use little of the bandwidth.
+
+Using [--order-by](#order-by-string) `size,mixed,50` makes half of the
+transfers take the smallest files in the backlog and half of them take
+the largest. This keeps both large and small files moving at once so
+the bandwidth and the request rate are both kept full for the whole of
+the run.
+
+This works best if rclone has lots of files to choose from so use it
+with a big [--max-backlog](#max-backlog-int), e.g. `--max-backlog
+2000000`. Each item in the backlog uses in the order of 1 KiB of memory.
+
+### Doing less work
+
+Fewer requests to the storage system means a faster transfer.
+
+- If the source and destination are on the same remote then rclone
+  will use [server-side copy](#server-side-copy) where it is supported
+  so the data doesn't need to be downloaded and uploaded again.
+- [--fast-list](#fast-list) lists bucket-based remotes with fewer
+  transactions at the cost of using more memory. This may be faster or
+  slower than the default so it is worth testing.
+- [--no-traverse](#no-traverse) stops rclone listing the destination
+  which helps when copying a small number of files into a large
+  destination.
+- [--size-only](#size-only) or [--checksum](#c-checksum) avoid reading
+  the modification time. On some object storage systems (e.g. s3,
+  swift) this needs an extra request for each file. Alternatively use
+  [--use-server-modtime](#use-server-modtime) with `--update`.
+- [--no-check-dest](#no-check-dest) skips checking the destination
+  entirely. Only use this if you know that none of the files are
+  there already.
+
 ## Options
 
 Rclone has a number of options to control its behaviour. These are