diff --git a/docs/content/docs.md b/docs/content/docs.md index a5f93d008..4398c9c5b 100644 --- a/docs/content/docs.md +++ b/docs/content/docs.md @@ -778,6 +778,135 @@ modification time of the source object. Hashes are not included in system metadata as there is a well defined way of reading those already. +## Performance {#performance} + +Rclone's defaults are fairly conservative. They are chosen to work on +small machines, slow links and with cloud providers which rate limit +heavily. If you have plenty of bandwidth, memory and a fast storage +system then rclone can usually be made to run a great deal faster by +adjusting a few flags. + +The right values depend on the size of your files, your network and +the storage systems at each end, so treat the numbers below as +starting points. Change one thing at a time and measure the result. + +### Measuring + +Use `rclone copy` with [-P/--progress](#p-progress) to watch the +transfer rate while you are experimenting. For unattended runs log the +stats instead, e.g. `--stats 1m --stats-one-line --stats-log-level +NOTICE`. + +A transfer is always limited by something: the network bandwidth, the +rate at which the storage system will accept requests, disk IO, CPU or +memory. Tuning is working out which limit you are hitting and raising +it until you hit the next one. Watch the network, disk, CPU and memory +use of the machine rclone is running on as well as rclone's stats. + +### Lots of files + +The time taken to transfer small files is dominated by the round trip +time of the requests rather than by the bandwidth, so the way to make +them go faster is to do more of them at once. + +[--transfers](#transfers-int) sets the number of files transferred in +parallel. The default is 4. Double this until the transfer rate stops +increasing or the storage system starts returning errors. Values of 32 +to 64 are common and values in the hundreds work well with the large +object storage systems (e.g. s3, azureblob, googlecloudstorage, b2) +when transferring lots of small files. + +[--checkers](#checkers-int) sets the number of directory listings and +file comparisons done in parallel. The default is 8. Raising this +speeds up syncs where most of the files are already present on the +destination and stops the checking being the bottleneck when +`--transfers` is large. 16 to 64 is a reasonable range. + +Not all storage systems like lots of parallel requests. If you see rate +limiting errors (e.g. HTTP 429 or 503 errors) or lots of low level +retries in the log then you have gone too far, so reduce `--transfers` +and `--checkers` or use [--tpslimit](#tpslimit-float) to cap the +request rate. Local disks, especially spinning ones, and network file +systems may get slower rather than faster with too much parallelism. + +### Large files + +A single stream often won't fill a fast link, so rclone can transfer a +large file as several chunks in parallel. These are called multi-thread +transfers and are used for files bigger than +[--multi-thread-cutoff](#multi-thread-cutoff) on backends which +support them. + +The number of chunks transferred in parallel is set by +[--multi-thread-streams](#multi-thread-streams-int) (default 4) and +this works for all backends so increase this first to make large files +transfer faster. + +The size of the chunks is set by the backend, usually with a +`--backend-chunk-size` flag, e.g. `--s3-chunk-size`. Backends without +a natural chunk size (e.g. local, smb) use +[--multi-thread-chunk-size](#multi-thread-chunk-size-sizesuffix). +Larger chunks mean fewer requests which increases throughput on big +files. See the docs for the backend you are using for details. + +Chunked uploads are usually buffered in memory, so in the worst case +rclone will use + +```text +--transfers * upload concurrency * chunk size +``` + +of memory for buffers. For example `--transfers 256` with an upload +concurrency of 8 and a chunk size of 16 MiB could use up to 32 GiB. +When you raise `--transfers` to speed up small files you may therefore +need to lower the upload concurrency, or set +[--max-buffer-memory](#max-buffer-memory) which limits the total +memory used for buffers and makes transfers wait when the limit is +reached. + +[--use-mmap](#use-mmap) makes rclone return buffer memory to the OS as +soon as it has finished with it which is helpful when using lots of +transfers or large chunks. + +### A mixture of large and small files + +By default rclone transfers files in roughly the order it finds them. +With a mixture of file sizes this means that at times all the +transfers are large files, which fill the bandwidth but make few +requests, and at other times all the transfers are small files, which +make lots of requests but use little of the bandwidth. + +Using [--order-by](#order-by-string) `size,mixed,50` makes half of the +transfers take the smallest files in the backlog and half of them take +the largest. This keeps both large and small files moving at once so +the bandwidth and the request rate are both kept full for the whole of +the run. + +This works best if rclone has lots of files to choose from so use it +with a big [--max-backlog](#max-backlog-int), e.g. `--max-backlog +2000000`. Each item in the backlog uses in the order of 1 KiB of memory. + +### Doing less work + +Fewer requests to the storage system means a faster transfer. + +- If the source and destination are on the same remote then rclone + will use [server-side copy](#server-side-copy) where it is supported + so the data doesn't need to be downloaded and uploaded again. +- [--fast-list](#fast-list) lists bucket-based remotes with fewer + transactions at the cost of using more memory. This may be faster or + slower than the default so it is worth testing. +- [--no-traverse](#no-traverse) stops rclone listing the destination + which helps when copying a small number of files into a large + destination. +- [--size-only](#size-only) or [--checksum](#c-checksum) avoid reading + the modification time. On some object storage systems (e.g. s3, + swift) this needs an extra request for each file. Alternatively use + [--use-server-modtime](#use-server-modtime) with `--update`. +- [--no-check-dest](#no-check-dest) skips checking the destination + entirely. Only use this if you know that none of the files are + there already. + ## Options Rclone has a number of options to control its behaviour. These are