mirror of
https://github.com/rclone/rclone.git
synced 2026-09-14 15:26:30 -04:00
Nick Craig-Wood
b2d0642825
Streamed multipart uploads had two problems when the client uploaded parts concurrently: Parts arriving ahead of the next part needed by the backend stream were buffered in memory without limit, acknowledging each part as soon as it was received. A client uploading faster than the backend could drain would therefore balloon the server's memory to the size of the upload. Buffering is now bounded a new --multipart-streaming-buffer-limit flag (default 256Mi, 0 for unlimited): a part that would take the buffer over the limit is not read until the stream drains, applying backpressure to the client instead of using unbounded memory. A part uploaded again with the same number - typically a client retrying after its request timed out - left a stale copy in the reorder buffer which made CompleteMultipartUpload fail with InvalidPart, aborting the whole upload. Re-uploaded parts are now handled properly: a copy still in the buffer is replaced, an identical copy of an already streamed part is accepted as a no-op, and only replacing an already streamed part with different content (which the in-order stream cannot honour) is rejected.