mirror of
https://github.com/rclone/rclone.git
synced 2026-09-14 07:14:20 -04:00
Nick Craig-Wood
f5bdceab49
The multipart upload allocated a fresh chunk-sized buffer (64 MiB by default) plus a 1 MiB scratch buffer per large file, copying every byte twice, and never returned them to rclone's memory pool. Buffer each part with multipart.NewRW instead so the memory is reused across uploads and part of rclone's central memory management. The pooled buffer is seekable, so a part can now be re-sent. uploadPart previously had no retry at all and any transient error failed the whole upload. It is now wrapped in the pacer with the backend's usual shouldRetry rules, seeking to the start before each attempt. The body is wrapped in readers.NoCloser so the http transport can't close the pooled buffer between attempts, and Content-Length is set explicitly since net/http can't infer it from a pool.RW. The FsPutRetry integration test covers the retry of a failed upload request and checks the buffers are returned to the pool.