Ali Dowair aafe56cd6f feat(snapshots): support restoring sparse files (#1823) ...

* feat(snapshots): support restoring sparse files

This commit implements basic support for restoring sparse files from
a snapshot. When specifying "--mode=sparse" in a snapshot restore
command, Kopia will make a best effort to make sure the underlying
filesystem allocates the minimum amount of blocks needed to persist
restored files. In other words, enabling this feature will "force"
all restored files to be sparse-blocks of zero bytes in the source
file should not be allocated.

* Address review comments

- Separate sparse option into its own bool flag
- Implement sparsefile packagewith copySparse method
- Truncate once before writing sparse file
- Check error from Truncate
- Add unit test for copySparse
- Invoke GetBlockSize once per file copy
- Remove support for Windows and explain why
- Add unit test for stat package

Co-authored-by: Dave Smith-Uchida <dave@kasten.io>

2022-03-22 19:09:50 -07:00

..

app.go

chore(repository): switched from opencensus to directly exporting prometheus metrics (#1831)

2022-03-17 23:39:36 -07:00

auto_upgrade.go

Maintenance improvements for epoch-based index structures (#1225)

2021-08-02 21:08:54 -07:00

cli_progress.go

chore(ci): added checklocks static analyzer (#1838)

2022-03-19 22:42:59 -07:00

command_acl_add.go

upgraded linter to 1.41.0 (#1144)

2021-06-16 19:44:55 -07:00

command_acl_delete.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

command_acl_enable.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

command_acl_list.go

cli: plumbed through 'textOutput' which controls stdout/stderr writers (#1053)

2021-05-06 20:26:35 -07:00

command_acl.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

command_benchmark_compression.go

cli: benchmark compressor memory usage (#1416)

2021-10-18 21:39:00 -07:00

command_benchmark_crypto.go

huge: replaced the use of allocated byte slices with populating gather.WriteBuffer in the repository (#1244)

2021-08-20 08:45:10 -07:00

command_benchmark_splitters.go

minor spelling and grammar fixes (#1318)

2021-09-28 15:19:02 -07:00

command_benchmark_test.go

refactor: move from io/ioutil to io and os package (#1360)

2021-10-06 08:39:10 -07:00

command_benchmark.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

command_blob_delete.go

Refactored most of the CLI tests to run in-process as opposed to using sub-processes (#1059)

2021-05-11 22:26:28 -07:00

command_blob_gc.go

Refactored most of the CLI tests to run in-process as opposed to using sub-processes (#1059)

2021-05-11 22:26:28 -07:00

command_blob_list.go

Maintenance improvements for epoch-based index structures (#1225)

2021-08-02 21:08:54 -07:00

command_blob_shards_modify_test.go

blob: changed default shards from {3,3} to {1,3} (#1513)

2021-11-16 06:02:04 -08:00

command_blob_shards_modify.go

chore(general): updated linter to 1.44.0 (#1681)

2022-01-25 21:21:13 -08:00

command_blob_shards.go

blob: support for custom blob store sharding (#1299)

2021-09-19 18:50:38 -07:00

command_blob_show_test.go

Consolidated format version flags (#1284)

2021-09-08 18:44:03 -07:00

command_blob_show.go

huge: replaced the use of allocated byte slices with populating gather.WriteBuffer in the repository (#1244)

2021-08-20 08:45:10 -07:00

command_blob_stats.go

upgraded linter to 1.41.0 (#1144)

2021-06-16 19:44:55 -07:00

command_blob.go

blob: support for custom blob store sharding (#1299)

2021-09-19 18:50:38 -07:00

command_cache_clear.go

upgraded linter to 1.41.0 (#1144)

2021-06-16 19:44:55 -07:00

command_cache_info.go

cache: prevent metadata cache thrashing if working set exceeds max defined size (#1557)

2021-12-03 15:35:01 -08:00

command_cache_prefetch.go

feat(repository): added 'hint' to Prefetch methods. (#1825)

2022-03-12 23:16:39 -08:00

command_cache_set_test.go

Consolidated format version flags (#1284)

2021-09-08 18:44:03 -07:00

command_cache_set.go

cache: prevent metadata cache thrashing if working set exceeds max defined size (#1557)

2021-12-03 15:35:01 -08:00

command_cache_sync_test.go

Consolidated format version flags (#1284)

2021-09-08 18:44:03 -07:00

command_cache_sync.go

chore(repository): streamlined internal content cache API (#1828)

2022-03-15 23:57:04 -07:00

command_cache.go

feat(repository): added PrefetchObjects() API (#1779)

2022-03-06 14:30:58 -08:00

command_content_delete.go

Refactored most of the CLI tests to run in-process as opposed to using sub-processes (#1059)

2021-05-11 22:26:28 -07:00

command_content_list.go

Support for content-level compression (#1076)

2021-05-22 05:35:27 -07:00

command_content_range_flags.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

command_content_rewrite.go

Upgraded linter to 1.40.1 (#1072)

2021-05-15 12:12:34 -07:00

command_content_show.go

Refactored most of the CLI tests to run in-process as opposed to using sub-processes (#1059)

2021-05-11 22:26:28 -07:00

command_content_stats.go

chore(ci): added checklocks static analyzer (#1838)

2022-03-19 22:42:59 -07:00

command_content_verify_test.go

refactor: move from io/ioutil to io and os package (#1360)

2021-10-06 08:39:10 -07:00

command_content_verify.go

Fix endurance test (#1254)

2021-08-27 04:22:18 -07:00

command_content.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

command_diff.go

Upgraded linter to 1.40.1 (#1072)

2021-05-15 12:12:34 -07:00

command_index_epoch_list.go

Implemented epoch-based index manager (#1174)

2021-07-06 21:38:08 -07:00

command_index_epoch.go

Implemented epoch-based index manager (#1174)

2021-07-06 21:38:08 -07:00

command_index_inspect_test.go

Consolidated format version flags (#1284)

2021-09-08 18:44:03 -07:00

command_index_inspect.go

cli: added 'index inspect --parallel' flag (#1404)

2021-10-17 18:35:21 -07:00

command_index_list.go

Implemented epoch-based index manager (#1174)

2021-07-06 21:38:08 -07:00

command_index_optimize.go

Upgraded linter to 1.40.1 (#1072)

2021-05-15 12:12:34 -07:00

command_index_recover.go

cli: optimized 'kopia index recover' by leveraging partial read and parallelism (#1094)

2021-05-23 22:18:31 -07:00

command_index.go

Implemented epoch-based index manager (#1174)

2021-07-06 21:38:08 -07:00

command_logs_cleanup.go

Added content manager internal logging (#1116)

2021-06-05 08:48:43 -07:00

command_logs_list.go

Added content manager internal logging (#1116)

2021-06-05 08:48:43 -07:00

command_logs_session.go

clock: discard monotonic clock component in clock.Now() (#1437)

2021-10-22 15:35:09 -07:00

command_logs_show.go

minor spelling and grammar fixes (#1318)

2021-09-28 15:19:02 -07:00

command_logs_test.go

maintenance: send logs to content log as well (#1496)

2021-11-06 23:08:00 -07:00

command_logs.go

Added content manager internal logging (#1116)

2021-06-05 08:48:43 -07:00

command_ls.go

chore(general): updated linter to 1.44.0 (#1681)

2022-01-25 21:21:13 -08:00

command_maintenance_info_test.go

cli: include parameters in maintenance info JSON output (#981)

2021-09-10 17:51:55 -07:00

command_maintenance_info.go

clock: discard monotonic clock component in clock.Now() (#1437)

2021-10-22 15:35:09 -07:00

command_maintenance_run.go

Maintenance improvements for epoch-based index structures (#1225)

2021-08-02 21:08:54 -07:00

command_maintenance_set.go

Added content manager internal logging (#1116)

2021-06-05 08:48:43 -07:00

command_maintenance.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

command_manifest_delete.go

Refactored most of the CLI tests to run in-process as opposed to using sub-processes (#1059)

2021-05-11 22:26:28 -07:00

command_manifest_ls.go

cli: plumbed through 'textOutput' which controls stdout/stderr writers (#1053)

2021-05-06 20:26:35 -07:00

command_manifest_show.go

Refactored most of the CLI tests to run in-process as opposed to using sub-processes (#1059)

2021-05-11 22:26:28 -07:00

command_manifest.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

command_mount.go

Upgraded linter to 1.40.1 (#1072)

2021-05-15 12:12:34 -07:00

command_policy_edit.go

fix(cli): fixed and unified help text for policy commands (#1829)

2022-03-16 00:46:57 -07:00

command_policy_ls.go

cli: plumbed through 'textOutput' which controls stdout/stderr writers (#1053)

2021-05-06 20:26:35 -07:00

command_policy_remove.go

fix(cli): fixed and unified help text for policy commands (#1829)

2022-03-16 00:46:57 -07:00

command_policy_set_actions.go

Upgraded linter to 1.43.0 (#1505)

2021-11-11 17:03:11 -08:00

command_policy_set_compression.go

Added support for logging policies (#1472)

2021-11-06 10:06:05 -07:00

command_policy_set_error_handling.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

command_policy_set_files.go

Added support for logging policies (#1472)

2021-11-06 10:06:05 -07:00

command_policy_set_logging_test.go

feat(cli): improved formatting of 'policy show' outputs (#1767)

2022-02-22 22:21:48 -08:00

command_policy_set_logging.go

Added support for logging policies (#1472)

2021-11-06 10:06:05 -07:00

command_policy_set_retention.go

fix(snapshots): treat empty retention policy as retaining ALL, not NONE (#1733)

2022-02-07 11:40:27 -08:00

command_policy_set_scheduling.go

Added support for logging policies (#1472)

2021-11-06 10:06:05 -07:00

command_policy_set_test.go

Added support for logging policies (#1472)

2021-11-06 10:06:05 -07:00

command_policy_set.go

fix(cli): fixed and unified help text for policy commands (#1829)

2022-03-16 00:46:57 -07:00

command_policy_show.go

chore(ci): upgraded linter to 1.45 (#1836)

2022-03-18 22:24:42 -07:00

command_policy.go

fix(cli): fixed and unified help text for policy commands (#1829)

2022-03-16 00:46:57 -07:00

command_repo_status_test.go

feat(cli): add --json output to 'repo status' command (#1834)

2022-03-17 22:22:24 -07:00

command_repository_change_password_test.go

Consolidated format version flags (#1284)

2021-09-08 18:44:03 -07:00

command_repository_change_password.go

Repository password change support (#1197)

2021-07-17 07:58:02 -07:00

command_repository_connect_from_config.go

refactor(cli): An in-memory storage mock setup for CLI tests (#1697)

2022-02-01 10:29:13 -08:00

command_repository_connect_server.go

repo: refactored password persistence (#1065)

2021-05-11 21:53:36 -07:00

command_repository_connect.go

refactor(cli): An in-memory storage mock setup for CLI tests (#1697)

2022-02-01 10:29:13 -08:00

command_repository_create.go

feat(cli): improved formatting of 'policy show' outputs (#1767)

2022-02-22 22:21:48 -08:00

command_repository_disconnect.go

repo: refactored password persistence (#1065)

2021-05-11 21:53:36 -07:00

command_repository_repair.go

refactor(cli): An in-memory storage mock setup for CLI tests (#1697)

2022-02-01 10:29:13 -08:00

command_repository_set_client.go

repo: limit the duration of kopia.repository caching to 15 minutes (#1196)

2021-07-15 11:32:55 -07:00

command_repository_set_parameters_test.go

feat(cli): allow changing retention parameters from CLI (#1680)

2022-02-02 19:04:22 -08:00

command_repository_set_parameters.go

feat(cli): allow changing retention parameters from CLI (#1680)

2022-02-02 19:04:22 -08:00

command_repository_status.go

feat(cli): add --json output to 'repo status' command (#1834)

2022-03-17 22:22:24 -07:00

command_repository_sync.go

refactor(cli): An in-memory storage mock setup for CLI tests (#1697)

2022-02-01 10:29:13 -08:00

command_repository_validate_provider.go

fix(providers): fixed DoNotRecreate and tests for gcs (#1688)

2022-01-29 09:12:07 -08:00

command_repository.go

Upgraded linter to 1.43.0 (#1505)

2021-11-11 17:03:11 -08:00

command_restore.go

feat(snapshots): support restoring sparse files (#1823)

2022-03-22 19:09:50 -07:00

command_server_cancel.go

server: support for server control APIs and tooling (#1644)

2022-01-03 18:48:38 -08:00

command_server_control_test.go

fix(ci): fixed flaky TestServerControl (#1840)

2022-03-20 20:19:21 -07:00

command_server_flush.go

server: support for server control APIs and tooling (#1644)

2022-01-03 18:48:38 -08:00

command_server_pause.go

server: support for server control APIs and tooling (#1644)

2022-01-03 18:48:38 -08:00

command_server_refresh.go

server: support for server control APIs and tooling (#1644)

2022-01-03 18:48:38 -08:00

command_server_resume.go

server: support for server control APIs and tooling (#1644)

2022-01-03 18:48:38 -08:00

command_server_shutdown.go

server: support for server control APIs and tooling (#1644)

2022-01-03 18:48:38 -08:00

command_server_snapshot.go

server: support for server control APIs and tooling (#1644)

2022-01-03 18:48:38 -08:00

command_server_source_manager_action.go

server: support for server control APIs and tooling (#1644)

2022-01-03 18:48:38 -08:00

command_server_start.go

chore(repository): switched from opencensus to directly exporting prometheus metrics (#1831)

2022-03-17 23:39:36 -07:00

command_server_status.go

server: support for server control APIs and tooling (#1644)

2022-01-03 18:48:38 -08:00

command_server_tls.go

chore(general): updated linter to 1.44.0 (#1681)

2022-01-25 21:21:13 -08:00

command_server.go

server: support for server control APIs and tooling (#1644)

2022-01-03 18:48:38 -08:00

command_session_list.go

cli: plumbed through 'textOutput' which controls stdout/stderr writers (#1053)

2021-05-06 20:26:35 -07:00

command_session.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

command_show.go

minor spelling and grammar fixes (#1318)

2021-09-28 15:19:02 -07:00

command_snapshot_copy_move_history.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

command_snapshot_create.go

chore(general): updated linter to 1.44.0 (#1681)

2022-01-25 21:21:13 -08:00

command_snapshot_delete.go

cli: added 'snapshot delete --all-snapshots-for-source' (#1397)

2021-10-17 09:46:44 -07:00

command_snapshot_estimate_test.go

Added support for logging policies (#1472)

2021-11-06 10:06:05 -07:00

command_snapshot_estimate.go

cli: test + misc improvements (#1083)

2021-05-17 21:47:11 -07:00

command_snapshot_expire.go

Upgraded linter to 1.40.1 (#1072)

2021-05-15 12:12:34 -07:00

command_snapshot_gc.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

command_snapshot_list.go

chore(ci): added checklocks static analyzer (#1838)

2022-03-19 22:42:59 -07:00

command_snapshot_migrate.go

cli: when migrating snapshot honor destination policies (#1433)

2021-10-21 22:12:06 -07:00

command_snapshot_pin_test.go

cli: added 'snapshot pin' command (#1528)

2021-11-20 20:53:25 -08:00

command_snapshot_pin.go

cli: added 'snapshot pin' command (#1528)

2021-11-20 20:53:25 -08:00

command_snapshot_restore.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

command_snapshot_storage_stats_test.go

feat(cli): added snapshot list flags: --storage-stats and --reverse (#1766)

2022-02-21 20:24:13 -08:00

command_snapshot_verify.go

chore(ci): added checklocks static analyzer (#1838)

2022-03-19 22:42:59 -07:00

command_snapshot.go

cli: added 'snapshot pin' command (#1528)

2021-11-20 20:53:25 -08:00

command_user_add_set.go

cli: plumbed through 'textOutput' which controls stdout/stderr writers (#1053)

2021-05-06 20:26:35 -07:00

command_user_delete.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

command_user_info.go

cli: plumbed through 'textOutput' which controls stdout/stderr writers (#1053)

2021-05-06 20:26:35 -07:00

command_user_list.go

cli: plumbed through 'textOutput' which controls stdout/stderr writers (#1053)

2021-05-06 20:26:35 -07:00

command_user.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

config.go

introduced structural debug logging and optional JSON output (#1475)

2021-11-03 21:57:37 -07:00

inproc.go

Upgraded linter to 1.43.0 (#1505)

2021-11-11 17:03:11 -08:00

json_output.go

cli: plumbed through 'textOutput' which controls stdout/stderr writers (#1053)

2021-05-06 20:26:35 -07:00

memory_tracking.go

huge: replaced the use of allocated byte slices with populating gather.WriteBuffer in the repository (#1244)

2021-08-20 08:45:10 -07:00

password_darwin.go

repo: refactored password persistence (#1065)

2021-05-11 21:53:36 -07:00

password_linux.go

cli: linux: read KOPIA_USE_KEYRING env (#1337)

2021-10-01 22:56:50 -07:00

password_other.go

linter: upgrade to 1.42.1 (#1292)

2021-09-14 19:11:39 -07:00

password_windows.go

repo: refactored password persistence (#1065)

2021-05-11 21:53:36 -07:00

password.go

blob: changed default shards from {3,3} to {1,3} (#1513)

2021-11-16 06:02:04 -08:00

profile.go

huge: replaced the use of allocated byte slices with populating gather.WriteBuffer in the repository (#1244)

2021-08-20 08:45:10 -07:00

show_utils.go

feat(cli): improved formatting of 'policy show' outputs (#1767)

2022-02-22 22:21:48 -08:00

sighup_unix.go

linter: upgrade to 1.42.1 (#1292)

2021-09-14 19:11:39 -07:00

sighup_windows.go

Added repository user authorization support + server flag refactoring + refresh (#890)

2021-03-18 23:03:27 -07:00

storage_azure.go

refactor(cli): An in-memory storage mock setup for CLI tests (#1697)

2022-02-01 10:29:13 -08:00

storage_b2.go

refactor(cli): An in-memory storage mock setup for CLI tests (#1697)

2022-02-01 10:29:13 -08:00

storage_filesystem.go

refactor(cli): An in-memory storage mock setup for CLI tests (#1697)

2022-02-01 10:29:13 -08:00

storage_gcs.go

refactor(cli): An in-memory storage mock setup for CLI tests (#1697)

2022-02-01 10:29:13 -08:00

storage_gdrive.go

feat(providers): add Google Drive support (#1731)

2022-02-16 22:34:48 -08:00

storage_providers.go

refactor(cli): An in-memory storage mock setup for CLI tests (#1697)

2022-02-01 10:29:13 -08:00

storage_rclone.go

refactor(cli): An in-memory storage mock setup for CLI tests (#1697)

2022-02-01 10:29:13 -08:00

storage_s3.go

refactor(cli): An in-memory storage mock setup for CLI tests (#1697)

2022-02-01 10:29:13 -08:00

storage_sftp_test.go

blob: changed default shards from {3,3} to {1,3} (#1513)

2021-11-16 06:02:04 -08:00

storage_sftp.go

refactor(cli): An in-memory storage mock setup for CLI tests (#1697)

2022-02-01 10:29:13 -08:00

storage_webdav.go

refactor(cli): An in-memory storage mock setup for CLI tests (#1697)

2022-02-01 10:29:13 -08:00

suite_test.go

Consolidated format version flags (#1284)

2021-09-08 18:44:03 -07:00

update_check.go

Upgraded linter to 1.43.0 (#1505)

2021-11-11 17:03:11 -08:00