Ali Dowair 7ca8b85a57 feat(providers): expand PutBlob API to allow for idempotent puts (#1654) ...

* Add a new PutBlob option and blob error type

When `DoNotRecreate` is set as true, the blob put operation should
only succeed if no blob with the given blob ID already exists.
Othwerwise, `ErrBlobAlreadyExists` is returned.

* Validate default storage providers' support

By default, storage providers should not support idempotent creates.
This commit adds error handling to exit early if `DoNotRecreate` is
set to true. The commit also verifies this behavior in the provider
validation test.

* Implement support for new option in GCS storage

* Push PutBlob option handling down to Impl

When PutBlob options were introduced, error handling logic for them
was implemented for the Sharded storage interface. However, the
behavior of different providers that implement Sharded can be
different, so it's better to push the options down to be processed in
the provider implementations.

* Introduce new error type for unsupported put opts

To unify error handling code and make it more maintainable, introduce
a new error type `blob.ErrUnsupportedPutBlobOption`, which is to be
returned whenever a storage provider implementation is given put
options it does not support.

2022-01-27 08:49:06 -08:00

..

app.go

chore(general): updated linter to 1.44.0 (#1681)

2022-01-25 21:21:13 -08:00

auto_upgrade.go

Maintenance improvements for epoch-based index structures (#1225)

2021-08-02 21:08:54 -07:00

cli_progress.go

minor spelling and grammar fixes (#1318)

2021-09-28 15:19:02 -07:00

command_acl_add.go

upgraded linter to 1.41.0 (#1144)

2021-06-16 19:44:55 -07:00

command_acl_delete.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

command_acl_enable.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

command_acl_list.go

cli: plumbed through 'textOutput' which controls stdout/stderr writers (#1053)

2021-05-06 20:26:35 -07:00

command_acl.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

command_benchmark_compression.go

cli: benchmark compressor memory usage (#1416)

2021-10-18 21:39:00 -07:00

command_benchmark_crypto.go

huge: replaced the use of allocated byte slices with populating gather.WriteBuffer in the repository (#1244)

2021-08-20 08:45:10 -07:00

command_benchmark_splitters.go

minor spelling and grammar fixes (#1318)

2021-09-28 15:19:02 -07:00

command_benchmark_test.go

refactor: move from io/ioutil to io and os package (#1360)

2021-10-06 08:39:10 -07:00

command_benchmark.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

command_blob_delete.go

Refactored most of the CLI tests to run in-process as opposed to using sub-processes (#1059)

2021-05-11 22:26:28 -07:00

command_blob_gc.go

Refactored most of the CLI tests to run in-process as opposed to using sub-processes (#1059)

2021-05-11 22:26:28 -07:00

command_blob_list.go

Maintenance improvements for epoch-based index structures (#1225)

2021-08-02 21:08:54 -07:00

command_blob_shards_modify_test.go

blob: changed default shards from {3,3} to {1,3} (#1513)

2021-11-16 06:02:04 -08:00

command_blob_shards_modify.go

chore(general): updated linter to 1.44.0 (#1681)

2022-01-25 21:21:13 -08:00

command_blob_shards.go

blob: support for custom blob store sharding (#1299)

2021-09-19 18:50:38 -07:00

command_blob_show_test.go

Consolidated format version flags (#1284)

2021-09-08 18:44:03 -07:00

command_blob_show.go

huge: replaced the use of allocated byte slices with populating gather.WriteBuffer in the repository (#1244)

2021-08-20 08:45:10 -07:00

command_blob_stats.go

upgraded linter to 1.41.0 (#1144)

2021-06-16 19:44:55 -07:00

command_blob.go

blob: support for custom blob store sharding (#1299)

2021-09-19 18:50:38 -07:00

command_cache_clear.go

upgraded linter to 1.41.0 (#1144)

2021-06-16 19:44:55 -07:00

command_cache_info.go

cache: prevent metadata cache thrashing if working set exceeds max defined size (#1557)

2021-12-03 15:35:01 -08:00

command_cache_set_test.go

Consolidated format version flags (#1284)

2021-09-08 18:44:03 -07:00

command_cache_set.go

cache: prevent metadata cache thrashing if working set exceeds max defined size (#1557)

2021-12-03 15:35:01 -08:00

command_cache_sync_test.go

Consolidated format version flags (#1284)

2021-09-08 18:44:03 -07:00

command_cache_sync.go

Upgraded linter to 1.40.1 (#1072)

2021-05-15 12:12:34 -07:00

command_cache.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

command_content_delete.go

Refactored most of the CLI tests to run in-process as opposed to using sub-processes (#1059)

2021-05-11 22:26:28 -07:00

command_content_list.go

Support for content-level compression (#1076)

2021-05-22 05:35:27 -07:00

command_content_range_flags.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

command_content_rewrite.go

Upgraded linter to 1.40.1 (#1072)

2021-05-15 12:12:34 -07:00

command_content_show.go

Refactored most of the CLI tests to run in-process as opposed to using sub-processes (#1059)

2021-05-11 22:26:28 -07:00

command_content_stats.go

upgraded linter to 1.41.0 (#1144)

2021-06-16 19:44:55 -07:00

command_content_verify_test.go

refactor: move from io/ioutil to io and os package (#1360)

2021-10-06 08:39:10 -07:00

command_content_verify.go

Fix endurance test (#1254)

2021-08-27 04:22:18 -07:00

command_content.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

command_diff.go

Upgraded linter to 1.40.1 (#1072)

2021-05-15 12:12:34 -07:00

command_index_epoch_list.go

Implemented epoch-based index manager (#1174)

2021-07-06 21:38:08 -07:00

command_index_epoch.go

Implemented epoch-based index manager (#1174)

2021-07-06 21:38:08 -07:00

command_index_inspect_test.go

Consolidated format version flags (#1284)

2021-09-08 18:44:03 -07:00

command_index_inspect.go

cli: added 'index inspect --parallel' flag (#1404)

2021-10-17 18:35:21 -07:00

command_index_list.go

Implemented epoch-based index manager (#1174)

2021-07-06 21:38:08 -07:00

command_index_optimize.go

Upgraded linter to 1.40.1 (#1072)

2021-05-15 12:12:34 -07:00

command_index_recover.go

cli: optimized 'kopia index recover' by leveraging partial read and parallelism (#1094)

2021-05-23 22:18:31 -07:00

command_index.go

Implemented epoch-based index manager (#1174)

2021-07-06 21:38:08 -07:00

command_logs_cleanup.go

Added content manager internal logging (#1116)

2021-06-05 08:48:43 -07:00

command_logs_list.go

Added content manager internal logging (#1116)

2021-06-05 08:48:43 -07:00

command_logs_session.go

clock: discard monotonic clock component in clock.Now() (#1437)

2021-10-22 15:35:09 -07:00

command_logs_show.go

minor spelling and grammar fixes (#1318)

2021-09-28 15:19:02 -07:00

command_logs_test.go

maintenance: send logs to content log as well (#1496)

2021-11-06 23:08:00 -07:00

command_logs.go

Added content manager internal logging (#1116)

2021-06-05 08:48:43 -07:00

command_ls.go

chore(general): updated linter to 1.44.0 (#1681)

2022-01-25 21:21:13 -08:00

command_maintenance_info_test.go

cli: include parameters in maintenance info JSON output (#981)

2021-09-10 17:51:55 -07:00

command_maintenance_info.go

clock: discard monotonic clock component in clock.Now() (#1437)

2021-10-22 15:35:09 -07:00

command_maintenance_run.go

Maintenance improvements for epoch-based index structures (#1225)

2021-08-02 21:08:54 -07:00

command_maintenance_set.go

Added content manager internal logging (#1116)

2021-06-05 08:48:43 -07:00

command_maintenance.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

command_manifest_delete.go

Refactored most of the CLI tests to run in-process as opposed to using sub-processes (#1059)

2021-05-11 22:26:28 -07:00

command_manifest_ls.go

cli: plumbed through 'textOutput' which controls stdout/stderr writers (#1053)

2021-05-06 20:26:35 -07:00

command_manifest_show.go

Refactored most of the CLI tests to run in-process as opposed to using sub-processes (#1059)

2021-05-11 22:26:28 -07:00

command_manifest.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

command_mount.go

Upgraded linter to 1.40.1 (#1072)

2021-05-15 12:12:34 -07:00

command_policy_edit.go

chore(general): updated linter to 1.44.0 (#1681)

2022-01-25 21:21:13 -08:00

command_policy_ls.go

cli: plumbed through 'textOutput' which controls stdout/stderr writers (#1053)

2021-05-06 20:26:35 -07:00

command_policy_remove.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

command_policy_set_actions.go

Upgraded linter to 1.43.0 (#1505)

2021-11-11 17:03:11 -08:00

command_policy_set_compression.go

Added support for logging policies (#1472)

2021-11-06 10:06:05 -07:00

command_policy_set_error_handling.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

command_policy_set_files.go

Added support for logging policies (#1472)

2021-11-06 10:06:05 -07:00

command_policy_set_logging_test.go

Added support for logging policies (#1472)

2021-11-06 10:06:05 -07:00

command_policy_set_logging.go

Added support for logging policies (#1472)

2021-11-06 10:06:05 -07:00

command_policy_set_retention.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

command_policy_set_scheduling.go

Added support for logging policies (#1472)

2021-11-06 10:06:05 -07:00

command_policy_set_test.go

Added support for logging policies (#1472)

2021-11-06 10:06:05 -07:00

command_policy_set.go

chore(general): updated linter to 1.44.0 (#1681)

2022-01-25 21:21:13 -08:00

command_policy_show.go

policy: revamped policy merge mechanism (#1538)

2021-11-27 18:14:45 -08:00

command_policy.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

command_repository_change_password_test.go

Consolidated format version flags (#1284)

2021-09-08 18:44:03 -07:00

command_repository_change_password.go

Repository password change support (#1197)

2021-07-17 07:58:02 -07:00

command_repository_connect_from_config.go

blob: changed default shards from {3,3} to {1,3} (#1513)

2021-11-16 06:02:04 -08:00

command_repository_connect_server.go

repo: refactored password persistence (#1065)

2021-05-11 21:53:36 -07:00

command_repository_connect.go

build(lint): enabled gochecknoglobals and tagged existing globals (#1664)

2022-01-15 12:54:56 -08:00

command_repository_create.go

feat(repository): persistence for blob-retention configuration (#1596)

2022-01-22 08:37:00 -08:00

command_repository_disconnect.go

repo: refactored password persistence (#1065)

2021-05-11 21:53:36 -07:00

command_repository_repair.go

build(lint): enabled gochecknoglobals and tagged existing globals (#1664)

2022-01-15 12:54:56 -08:00

command_repository_set_client.go

repo: limit the duration of kopia.repository caching to 15 minutes (#1196)

2021-07-15 11:32:55 -07:00

command_repository_set_parameters_test.go

content: fixed repo upgrade version (#1286)

2021-09-10 22:51:51 -07:00

command_repository_set_parameters.go

build(lint): enabled gochecknoglobals and tagged existing globals (#1664)

2022-01-15 12:54:56 -08:00

command_repository_status.go

fixed directory read race condition (#1489)

2021-11-05 10:18:03 -07:00

command_repository_sync.go

build(lint): enabled gochecknoglobals and tagged existing globals (#1664)

2022-01-15 12:54:56 -08:00

command_repository_validate_provider.go

feat(providers): expand PutBlob API to allow for idempotent puts (#1654)

2022-01-27 08:49:06 -08:00

command_repository.go

Upgraded linter to 1.43.0 (#1505)

2021-11-11 17:03:11 -08:00

command_restore.go

chore(general): updated linter to 1.44.0 (#1681)

2022-01-25 21:21:13 -08:00

command_server_cancel.go

server: support for server control APIs and tooling (#1644)

2022-01-03 18:48:38 -08:00

command_server_control_test.go

server: support for server control APIs and tooling (#1644)

2022-01-03 18:48:38 -08:00

command_server_flush.go

server: support for server control APIs and tooling (#1644)

2022-01-03 18:48:38 -08:00

command_server_pause.go

server: support for server control APIs and tooling (#1644)

2022-01-03 18:48:38 -08:00

command_server_refresh.go

server: support for server control APIs and tooling (#1644)

2022-01-03 18:48:38 -08:00

command_server_resume.go

server: support for server control APIs and tooling (#1644)

2022-01-03 18:48:38 -08:00

command_server_shutdown.go

server: support for server control APIs and tooling (#1644)

2022-01-03 18:48:38 -08:00

command_server_snapshot.go

server: support for server control APIs and tooling (#1644)

2022-01-03 18:48:38 -08:00

command_server_source_manager_action.go

server: support for server control APIs and tooling (#1644)

2022-01-03 18:48:38 -08:00

command_server_start.go

chore(general): updated linter to 1.44.0 (#1681)

2022-01-25 21:21:13 -08:00

command_server_status.go

server: support for server control APIs and tooling (#1644)

2022-01-03 18:48:38 -08:00

command_server_tls.go

chore(general): updated linter to 1.44.0 (#1681)

2022-01-25 21:21:13 -08:00

command_server.go

server: support for server control APIs and tooling (#1644)

2022-01-03 18:48:38 -08:00

command_session_list.go

cli: plumbed through 'textOutput' which controls stdout/stderr writers (#1053)

2021-05-06 20:26:35 -07:00

command_session.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

command_show.go

minor spelling and grammar fixes (#1318)

2021-09-28 15:19:02 -07:00

command_snapshot_copy_move_history.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

command_snapshot_create.go

chore(general): updated linter to 1.44.0 (#1681)

2022-01-25 21:21:13 -08:00

command_snapshot_delete.go

cli: added 'snapshot delete --all-snapshots-for-source' (#1397)

2021-10-17 09:46:44 -07:00

command_snapshot_estimate_test.go

Added support for logging policies (#1472)

2021-11-06 10:06:05 -07:00

command_snapshot_estimate.go

cli: test + misc improvements (#1083)

2021-05-17 21:47:11 -07:00

command_snapshot_expire.go

Upgraded linter to 1.40.1 (#1072)

2021-05-15 12:12:34 -07:00

command_snapshot_gc.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

command_snapshot_list.go

chore(general): updated linter to 1.44.0 (#1681)

2022-01-25 21:21:13 -08:00

command_snapshot_migrate.go

cli: when migrating snapshot honor destination policies (#1433)

2021-10-21 22:12:06 -07:00

command_snapshot_pin_test.go

cli: added 'snapshot pin' command (#1528)

2021-11-20 20:53:25 -08:00

command_snapshot_pin.go

cli: added 'snapshot pin' command (#1528)

2021-11-20 20:53:25 -08:00

command_snapshot_restore.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

command_snapshot_verify.go

chore(general): updated linter to 1.44.0 (#1681)

2022-01-25 21:21:13 -08:00

command_snapshot.go

cli: added 'snapshot pin' command (#1528)

2021-11-20 20:53:25 -08:00

command_user_add_set.go

cli: plumbed through 'textOutput' which controls stdout/stderr writers (#1053)

2021-05-06 20:26:35 -07:00

command_user_delete.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

command_user_info.go

cli: plumbed through 'textOutput' which controls stdout/stderr writers (#1053)

2021-05-06 20:26:35 -07:00

command_user_list.go

cli: plumbed through 'textOutput' which controls stdout/stderr writers (#1053)

2021-05-06 20:26:35 -07:00

command_user.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

config.go

introduced structural debug logging and optional JSON output (#1475)

2021-11-03 21:57:37 -07:00

inproc.go

Upgraded linter to 1.43.0 (#1505)

2021-11-11 17:03:11 -08:00

json_output.go

cli: plumbed through 'textOutput' which controls stdout/stderr writers (#1053)

2021-05-06 20:26:35 -07:00

memory_tracking.go

huge: replaced the use of allocated byte slices with populating gather.WriteBuffer in the repository (#1244)

2021-08-20 08:45:10 -07:00

password_darwin.go

repo: refactored password persistence (#1065)

2021-05-11 21:53:36 -07:00

password_linux.go

cli: linux: read KOPIA_USE_KEYRING env (#1337)

2021-10-01 22:56:50 -07:00

password_other.go

linter: upgrade to 1.42.1 (#1292)

2021-09-14 19:11:39 -07:00

password_windows.go

repo: refactored password persistence (#1065)

2021-05-11 21:53:36 -07:00

password.go

blob: changed default shards from {3,3} to {1,3} (#1513)

2021-11-16 06:02:04 -08:00

profile.go

huge: replaced the use of allocated byte slices with populating gather.WriteBuffer in the repository (#1244)

2021-08-20 08:45:10 -07:00

show_utils.go

chore(general): updated linter to 1.44.0 (#1681)

2022-01-25 21:21:13 -08:00

sighup_unix.go

linter: upgrade to 1.42.1 (#1292)

2021-09-14 19:11:39 -07:00

sighup_windows.go

Added repository user authorization support + server flag refactoring + refresh (#890)

2021-03-18 23:03:27 -07:00

storage_azure.go

throtting: implemented a Throttler based on token bucket and configur… (#1512)

2021-11-16 07:39:26 -08:00

storage_b2.go

throtting: implemented a Throttler based on token bucket and configur… (#1512)

2021-11-16 07:39:26 -08:00

storage_filesystem.go

throtting: implemented a Throttler based on token bucket and configur… (#1512)

2021-11-16 07:39:26 -08:00

storage_gcs.go

throtting: implemented a Throttler based on token bucket and configur… (#1512)

2021-11-16 07:39:26 -08:00

storage_providers.go

build(lint): enabled gochecknoglobals and tagged existing globals (#1664)

2022-01-15 12:54:56 -08:00

storage_rclone.go

throtting: implemented a Throttler based on token bucket and configur… (#1512)

2021-11-16 07:39:26 -08:00

storage_s3.go

throtting: implemented a Throttler based on token bucket and configur… (#1512)

2021-11-16 07:39:26 -08:00

storage_sftp_test.go

blob: changed default shards from {3,3} to {1,3} (#1513)

2021-11-16 06:02:04 -08:00

storage_sftp.go

throtting: implemented a Throttler based on token bucket and configur… (#1512)

2021-11-16 07:39:26 -08:00

storage_webdav.go

throtting: implemented a Throttler based on token bucket and configur… (#1512)

2021-11-16 07:39:26 -08:00

suite_test.go

Consolidated format version flags (#1284)

2021-09-08 18:44:03 -07:00

update_check.go

Upgraded linter to 1.43.0 (#1505)

2021-11-11 17:03:11 -08:00