mirror of
https://github.com/rclone/rclone.git
synced 2026-09-13 06:35:59 -04:00
Nick Craig-Wood
fb8783732d
Each part of a multipart upload allocated a fresh part-sized buffer (the size is chosen by Box, typically 8-32 MiB) with up to --transfers parts in flight, so large uploads churned allocations and GC. Buffer parts with multipart.NewRW instead so the memory comes from rclone's global pool, is reused across parts and files, and is part of rclone's central memory management. The pool.RW is seekable so the retry closure seeks back to the start before each attempt instead of rebuilding a bytes.Reader, and the per-part SHA1 digest is computed by reading the buffer and seeking back. The body goes through lib/rest which already stops the transport from closing it; the uploading goroutine owns and closes the buffer. The whole-file SHA1 used for the commit is unchanged. The FsPutRetry integration test covers the retry of a failed upload request and checks the buffers are returned to the pool.