Every multipart part was buffered in memory in full before being written
to the backend, even the part the stream needed next. There is no limit
on part size, and several concurrent requests for the current part were
each admitted, so a client could make the server hold as much memory as
it sent, whatever --multipart-streaming-buffer-limit was set to.
The part the stream needs next is now written straight through to the
backend as it arrives, so only parts which arrive ahead of their turn
are buffered, and those always within the limit - a part bigger than
the limit waits until it is the next part rather than being admitted
when the buffer is empty. Further requests for the part being streamed
wait, holding no memory, and are then checked against it.
Since what reaches the backend can't be taken back, a part whose body
fails part way records how far it got, and the client's retry of the
part, which must have the same content, carries on from there. Such an
upload can't be completed without the rest of the part.
The body is read to its end, so that an error it only reports there -
as the Content-MD5 check does - fails the part rather than being
missed, and a body longer than declared is rejected. A part rejected
once it has been streamed leaves data which can be neither completed
nor replaced, so the whole upload is failed and cleaned up and the
client starts again.
Nick Craig-Wood
7773bb6487