Jarek Kowalski 1566b74263 blob: support for custom blob store sharding (#1299) ...

* blob: support for custom blob store sharding

This is experimental.

The .shards file can reside in the root of any blob storage that uses
sharding (filesystem/sftp/webdav) and can specify rules for sharding.

{
  "default": [3,2,1],
  "overrides": [
    { "prefix": "p", "shards": [2,2] },
    { "prefix": "x", "shards": [1,1,1] }
  ],
  "maxNonShardedLength": 2
}

With this in place we'll be later able to do resharding of the
repository to optimize get/put/list performance for both repositories
and caches.

* cli: command line tools to manipulate shards in a directory

2021-09-19 18:50:38 -07:00

481 B

Raw Blame History

View Raw