Zoltan Kochan
369a9d6c93
Four decode paths grew to whatever the archive or the server produced, and nothing upstream of them bounded it. The lockfile's integrity does not: a gzip bomb is a legitimately published package whose hash matches. The registry's `dist.unpackedSize` does not either — it is the publisher's claim, and on a frozen install there is none at all, since a lockfile records no unpacked size. - The whole-archive gzip decode now stops at `MAX_UNTRUSTED_PREALLOC_BYTES`, the same ceiling `should_stream_extract` routes on, instead of relying on zune-inflate's 1 GiB default that every concurrent extraction may claim. Reaching it is not a refusal: the archive is re-run through the streaming extractor and decoded in full. `read_local_tarball_metadata` gets the same fallback, so a large `file:` tarball still resolves. - A response body that keeps arriving is handed to the streaming extractor once it reaches the size at which it would be extracted as a stream anyway, so a chunked response — which advertises no length — and a body that outruns the length it advertised both stop growing the buffer. The buffer is also pre-sized only as far as it can fill, so an advertised `Content-Length` no longer decides the size of an allocation it will never occupy. A body that is not a gzip stream is drained and dropped rather than accumulated, after being hashed, so a pinned integrity still produces the verdict that outranks the decode failure. - The queue joining the download loop to the streaming extractor is bounded. An unbounded channel between an async producer and a blocking consumer holds whatever a server sends faster than the extractor consumes it, which would hand back the very buffer this path exists to avoid; it was only ever bounded by the buffered alternative holding the whole body too. A bounded tokio channel with a blocking receive on the extractor's thread applies backpressure to the download instead. - A zip entry is read only as far as the size its central directory records, and one above the entry buffer ceiling streams straight into the store. Nothing in a zip bounds an entry's decompressed size on its own — unlike a tar entry, whose header size genuinely delimits it. No archive is refused for being large, so nothing that installed before stops installing. Routing now consults the gzip trailer's unpacked size, which the frozen-install path had no equivalent of, so a large archive goes to the streaming extractor on the first pass rather than being decoded twice to discover how large it was. It is read only behind a well-formed deflate header, so a corrupt body keeps reporting the decode error rather than being routed by four bytes of whatever it ends with. One error code moves: a non-gzip response body larger than 16 MiB reported `ERR_PNPM_TARBALL_IO_ERROR`, having been routed to the streaming extractor by its compressed size and failed in the tar reader, and now reports `ERR_PNPM_TARBALL_DECODE_GZIP` — what the same body under 16 MiB already reported. Both are retryable, so retry behavior is unchanged. The TypeScript CLI has the same exposure on all three of the issue's paths — it buffers the whole body, `gunzipSync`es the whole archive, and inflates each zip entry through adm-zip, with no ceiling anywhere. Bounding pacquet's decode changes no observable behavior, so there is nothing to mirror for parity; giving the TypeScript stack the same property means replacing `parseTarball`'s index-scan-over-one-buffer design, and is tracked separately. Closes https://github.com/pnpm/pnpm/issues/13656
520 B
520 B
pacquet
| pacquet |
|---|
| patch |
pnpm now decodes package archives using a bounded amount of memory, whatever an archive or a registry claims about its size. A gzipped tarball that inflates past what a whole-archive decode may hold is extracted as a stream instead, a response body that keeps arriving is extracted while it downloads rather than accumulated in full, and a zip entry is read only as far as the size its archive records. No archive is refused for being large — everything that installed before still installs.