Listing a prefix with no delimiter walked the entire subtree below it into
memory and only then sliced out the requested page, so every page of a
listing cost a full traversal of the tree.
Walk the tree lazily instead, stopping as soon as the page is full and
skipping the subtrees an earlier page already returned. A page now costs a
number of directory reads proportional to the keys it returns rather than to
the size of the subtree, and ETags are computed only for the objects that
are actually returned.
Entries are emitted in the order their keys have in a flat keyspace, with a
directory sorting as if it carried its trailing slash, so that "a.txt" comes
before "a/b" as it does in a real S3 bucket. Without this a resumed listing
would silently skip keys across a page boundary.
Nick Craig-Wood
04697cc02a