Jarek Kowalski f49bcdd883 feat(cli): implementation for 'kopia snapshot fix' (#1930) ...

* feat(cli): implementation for 'kopia snapshot fix'

This allows modifications and fixes to the snapshots after they have
been taken.

Supported are:

* `kopia snapshot fix remove-invalid-files [--verify-files-percent=X]`

Removes all directory entries where the underlying files cannot be
read based on index analysis (this does not read the files, only index
structures so is reasonably quick).

`--verify-files-percent=100` can be used to trigger full read for
all files.

* `kopia snapshot fix remove-files --object-id=<object-id>`

Removes the object with a given ID from the entire snapshot tree.
Useful when you accidentally snapshot a sensitive file.

* `kopia snapshot fix remove-files --filename=<wildcard>`

Removes the files with a given name from the entire snapshot tree.
Useful when you accidentally snapshot a sensitive file.

By default all snapshots are analyzed and rewritten. To limit the scope
use:

--source=user@host:/path
--manifest-id=manifestID

By default the rewrite operation writes new directory entries but
does not replace the manifests. To do that pass `--commit`.

Related #1906
Fixes #799

reorganized CLI per PR suggestion

* additional logging for diff command

* added Clone() method to snapshot manifst and directory entry

* added a comprehensive test, moved DirRewriter  to separate file

* pr feedback

* more pr feedback

* improved logging output

* disable test in -race configuration since it's way to slow

* pr feedback

2022-05-25 01:17:55 +00:00

..

app.go

fix(repository): fixed slow goroutine leak from indexBlobCache, added tests (#1950)

2022-05-16 01:21:30 +00:00

auto_upgrade.go

Maintenance improvements for epoch-based index structures (#1225)

2021-08-02 21:08:54 -07:00

cli_progress.go

chore(ci): added checklocks static analyzer (#1838)

2022-03-19 22:42:59 -07:00

command_acl_add.go

upgraded linter to 1.41.0 (#1144)

2021-06-16 19:44:55 -07:00

command_acl_delete.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

command_acl_enable.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

command_acl_list.go

cli: plumbed through 'textOutput' which controls stdout/stderr writers (#1053)

2021-05-06 20:26:35 -07:00

command_acl.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

command_benchmark_compression.go

feat(cli): improved kopia benchmark commands (#1849)

2022-03-26 14:28:08 +00:00

command_benchmark_crypto.go

feat(cli): improved kopia benchmark commands (#1849)

2022-03-26 14:28:08 +00:00

command_benchmark_encryption.go

feat(cli): improved kopia benchmark commands (#1849)

2022-03-26 14:28:08 +00:00

command_benchmark_hashing.go

feat(cli): improved kopia benchmark commands (#1849)

2022-03-26 14:28:08 +00:00

command_benchmark_splitters.go

feat(cli): improved kopia benchmark commands (#1849)

2022-03-26 14:28:08 +00:00

command_benchmark_test.go

feat(cli): improved kopia benchmark commands (#1849)

2022-03-26 14:28:08 +00:00

command_benchmark.go

feat(cli): improved kopia benchmark commands (#1849)

2022-03-26 14:28:08 +00:00

command_blob_delete.go

Refactored most of the CLI tests to run in-process as opposed to using sub-processes (#1059)

2021-05-11 22:26:28 -07:00

command_blob_gc.go

Refactored most of the CLI tests to run in-process as opposed to using sub-processes (#1059)

2021-05-11 22:26:28 -07:00

command_blob_list.go

Maintenance improvements for epoch-based index structures (#1225)

2021-08-02 21:08:54 -07:00

command_blob_shards_modify_test.go

blob: changed default shards from {3,3} to {1,3} (#1513)

2021-11-16 06:02:04 -08:00

command_blob_shards_modify.go

chore(general): updated linter to 1.44.0 (#1681)

2022-01-25 21:21:13 -08:00

command_blob_shards.go

blob: support for custom blob store sharding (#1299)

2021-09-19 18:50:38 -07:00

command_blob_show_test.go

Consolidated format version flags (#1284)

2021-09-08 18:44:03 -07:00

command_blob_show.go

huge: replaced the use of allocated byte slices with populating gather.WriteBuffer in the repository (#1244)

2021-08-20 08:45:10 -07:00

command_blob_stats.go

upgraded linter to 1.41.0 (#1144)

2021-06-16 19:44:55 -07:00

command_blob.go

blob: support for custom blob store sharding (#1299)

2021-09-19 18:50:38 -07:00

command_cache_clear.go

upgraded linter to 1.41.0 (#1144)

2021-06-16 19:44:55 -07:00

command_cache_info.go

cache: prevent metadata cache thrashing if working set exceeds max defined size (#1557)

2021-12-03 15:35:01 -08:00

command_cache_prefetch.go

feat(repository): added 'hint' to Prefetch methods. (#1825)

2022-03-12 23:16:39 -08:00

command_cache_set_test.go

Consolidated format version flags (#1284)

2021-09-08 18:44:03 -07:00

command_cache_set.go

cache: prevent metadata cache thrashing if working set exceeds max defined size (#1557)

2021-12-03 15:35:01 -08:00

command_cache_sync_test.go

Consolidated format version flags (#1284)

2021-09-08 18:44:03 -07:00

command_cache_sync.go

chore(repository): streamlined internal content cache API (#1828)

2022-03-15 23:57:04 -07:00

command_cache.go

feat(repository): added PrefetchObjects() API (#1779)

2022-03-06 14:30:58 -08:00

command_content_delete.go

feat(cli): REVERT added 'content delete --forget' flag (#1932) (#1940)

2022-05-10 03:45:25 +00:00

command_content_list.go

Support for content-level compression (#1076)

2021-05-22 05:35:27 -07:00

command_content_range_flags.go

chore(repository): extracted content index building and parsing into repo/content/index (#1881)

2022-04-05 18:04:50 -07:00

command_content_rewrite.go

Upgraded linter to 1.40.1 (#1072)

2021-05-15 12:12:34 -07:00

command_content_show.go

Refactored most of the CLI tests to run in-process as opposed to using sub-processes (#1059)

2021-05-11 22:26:28 -07:00

command_content_stats.go

chore(ci): added checklocks static analyzer (#1838)

2022-03-19 22:42:59 -07:00

command_content_verify_test.go

refactor: move from io/ioutil to io and os package (#1360)

2021-10-06 08:39:10 -07:00

command_content_verify.go

refactor(snapshots): extracted snapshotfs.Verifier component (#1921)

2022-05-02 04:03:28 +00:00

command_content.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

command_diff.go

Upgraded linter to 1.40.1 (#1072)

2021-05-15 12:12:34 -07:00

command_index_epoch_list.go

Implemented epoch-based index manager (#1174)

2021-07-06 21:38:08 -07:00

command_index_epoch.go

Implemented epoch-based index manager (#1174)

2021-07-06 21:38:08 -07:00

command_index_inspect_test.go

Consolidated format version flags (#1284)

2021-09-08 18:44:03 -07:00

command_index_inspect.go

cli: added 'index inspect --parallel' flag (#1404)

2021-10-17 18:35:21 -07:00

command_index_list.go

Implemented epoch-based index manager (#1174)

2021-07-06 21:38:08 -07:00

command_index_optimize.go

Upgraded linter to 1.40.1 (#1072)

2021-05-15 12:12:34 -07:00

command_index_recover.go

cli: optimized 'kopia index recover' by leveraging partial read and parallelism (#1094)

2021-05-23 22:18:31 -07:00

command_index.go

Implemented epoch-based index manager (#1174)

2021-07-06 21:38:08 -07:00

command_logs_cleanup.go

Added content manager internal logging (#1116)

2021-06-05 08:48:43 -07:00

command_logs_list.go

Added content manager internal logging (#1116)

2021-06-05 08:48:43 -07:00

command_logs_session.go

chore(general): minor cleanups (#1959)

2022-05-23 15:16:25 -07:00

command_logs_show.go

chore(general): minor cleanups (#1959)

2022-05-23 15:16:25 -07:00

command_logs_test.go

maintenance: send logs to content log as well (#1496)

2021-11-06 23:08:00 -07:00

command_logs.go

Added content manager internal logging (#1116)

2021-06-05 08:48:43 -07:00

command_ls.go

chore(general): updated linter to 1.44.0 (#1681)

2022-01-25 21:21:13 -08:00

command_maintenance_info_test.go

cli: include parameters in maintenance info JSON output (#981)

2021-09-10 17:51:55 -07:00

command_maintenance_info.go

clock: discard monotonic clock component in clock.Now() (#1437)

2021-10-22 15:35:09 -07:00

command_maintenance_run.go

Maintenance improvements for epoch-based index structures (#1225)

2021-08-02 21:08:54 -07:00

command_maintenance_set.go

Added content manager internal logging (#1116)

2021-06-05 08:48:43 -07:00

command_maintenance.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

command_manifest_delete.go

Refactored most of the CLI tests to run in-process as opposed to using sub-processes (#1059)

2021-05-11 22:26:28 -07:00

command_manifest_ls.go

cli: plumbed through 'textOutput' which controls stdout/stderr writers (#1053)

2021-05-06 20:26:35 -07:00

command_manifest_show.go

Refactored most of the CLI tests to run in-process as opposed to using sub-processes (#1059)

2021-05-11 22:26:28 -07:00

command_manifest.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

command_mount.go

Upgraded linter to 1.40.1 (#1072)

2021-05-15 12:12:34 -07:00

command_policy_edit.go

fix(cli): fixed and unified help text for policy commands (#1829)

2022-03-16 00:46:57 -07:00

command_policy_ls.go

cli: plumbed through 'textOutput' which controls stdout/stderr writers (#1053)

2021-05-06 20:26:35 -07:00

command_policy_remove.go

fix(cli): fixed and unified help text for policy commands (#1829)

2022-03-16 00:46:57 -07:00

command_policy_set_actions.go

Upgraded linter to 1.43.0 (#1505)

2021-11-11 17:03:11 -08:00

command_policy_set_compression.go

Added support for logging policies (#1472)

2021-11-06 10:06:05 -07:00

command_policy_set_error_handling.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

command_policy_set_files.go

Added support for logging policies (#1472)

2021-11-06 10:06:05 -07:00

command_policy_set_logging_test.go

feat(cli): improved formatting of 'policy show' outputs (#1767)

2022-02-22 22:21:48 -08:00

command_policy_set_logging.go

Added support for logging policies (#1472)

2021-11-06 10:06:05 -07:00

command_policy_set_retention.go

fix(snapshots): treat empty retention policy as retaining ALL, not NONE (#1733)

2022-02-07 11:40:27 -08:00

command_policy_set_scheduling.go

Added support for logging policies (#1472)

2021-11-06 10:06:05 -07:00

command_policy_set_test.go

Added support for logging policies (#1472)

2021-11-06 10:06:05 -07:00

command_policy_set_upload_test.go

feat(snapshots): support for controlling upload parallelism via policies (#1850)

2022-03-26 19:20:49 +00:00

command_policy_set_upload.go

feat(snapshots): support for controlling upload parallelism via policies (#1850)

2022-03-26 19:20:49 +00:00

command_policy_set.go

feat(snapshots): support for controlling upload parallelism via policies (#1850)

2022-03-26 19:20:49 +00:00

command_policy_show.go

feat(snapshots): support for controlling upload parallelism via policies (#1850)

2022-03-26 19:20:49 +00:00

command_policy.go

fix(cli): fixed and unified help text for policy commands (#1829)

2022-03-16 00:46:57 -07:00

command_repo_status_test.go

feat(cli): add --json output to 'repo status' command (#1834)

2022-03-17 22:22:24 -07:00

command_repo_throttle_test.go

feat(cli): added CLI for controlling throttler (#1956)

2022-05-18 01:27:06 -07:00

command_repository_change_password_test.go

Consolidated format version flags (#1284)

2021-09-08 18:44:03 -07:00

command_repository_change_password.go

Repository password change support (#1197)

2021-07-17 07:58:02 -07:00

command_repository_connect_from_config.go

refactor(cli): An in-memory storage mock setup for CLI tests (#1697)

2022-02-01 10:29:13 -08:00

command_repository_connect_server.go

repo: refactored password persistence (#1065)

2021-05-11 21:53:36 -07:00

command_repository_connect.go

refactor(cli): An in-memory storage mock setup for CLI tests (#1697)

2022-02-01 10:29:13 -08:00

command_repository_create.go

feat(cli): improved formatting of 'policy show' outputs (#1767)

2022-02-22 22:21:48 -08:00

command_repository_disconnect.go

repo: refactored password persistence (#1065)

2021-05-11 21:53:36 -07:00

command_repository_repair.go

refactor(cli): An in-memory storage mock setup for CLI tests (#1697)

2022-02-01 10:29:13 -08:00

command_repository_set_client.go

repo: limit the duration of kopia.repository caching to 15 minutes (#1196)

2021-07-15 11:32:55 -07:00

command_repository_set_parameters_test.go

feat(cli): allow changing retention parameters from CLI (#1680)

2022-02-02 19:04:22 -08:00

command_repository_set_parameters.go

feat(cli): allow changing retention parameters from CLI (#1680)

2022-02-02 19:04:22 -08:00

command_repository_status.go

feat(cli): show storage capacity in repo status (#1867)

2022-04-01 01:03:51 +00:00

command_repository_sync.go

chore(infra): added more leak checks to tests (#1953)

2022-05-16 06:37:57 +00:00

command_repository_throttle_get.go

feat(cli): added CLI for controlling throttler (#1956)

2022-05-18 01:27:06 -07:00

command_repository_throttle_set.go

feat(cli): added CLI for controlling throttler (#1956)

2022-05-18 01:27:06 -07:00

command_repository_throttle.go

feat(cli): added CLI for controlling throttler (#1956)

2022-05-18 01:27:06 -07:00

command_repository_validate_provider.go

fix(providers): fixed DoNotRecreate and tests for gcs (#1688)

2022-01-29 09:12:07 -08:00

command_repository.go

feat(cli): added CLI for controlling throttler (#1956)

2022-05-18 01:27:06 -07:00

command_restore.go

feat(snapshots): support restoring sparse files (#1823)

2022-03-22 19:09:50 -07:00

command_server_cancel.go

server: support for server control APIs and tooling (#1644)

2022-01-03 18:48:38 -08:00

command_server_control_test.go

feat(cli): added CLI for controlling throttler (#1956)

2022-05-18 01:27:06 -07:00

command_server_flush.go

server: support for server control APIs and tooling (#1644)

2022-01-03 18:48:38 -08:00

command_server_pause.go

server: support for server control APIs and tooling (#1644)

2022-01-03 18:48:38 -08:00

command_server_refresh.go

server: support for server control APIs and tooling (#1644)

2022-01-03 18:48:38 -08:00

command_server_resume.go

server: support for server control APIs and tooling (#1644)

2022-01-03 18:48:38 -08:00

command_server_shutdown.go

server: support for server control APIs and tooling (#1644)

2022-01-03 18:48:38 -08:00

command_server_snapshot.go

server: support for server control APIs and tooling (#1644)

2022-01-03 18:48:38 -08:00

command_server_source_manager_action.go

server: support for server control APIs and tooling (#1644)

2022-01-03 18:48:38 -08:00

command_server_start.go

chore(repository): switched from opencensus to directly exporting prometheus metrics (#1831)

2022-03-17 23:39:36 -07:00

command_server_status.go

server: support for server control APIs and tooling (#1644)

2022-01-03 18:48:38 -08:00

command_server_throttle_get.go

feat(cli): added CLI for controlling throttler (#1956)

2022-05-18 01:27:06 -07:00

command_server_throttle_set.go

feat(cli): added CLI for controlling throttler (#1956)

2022-05-18 01:27:06 -07:00

command_server_throttle.go

feat(cli): added CLI for controlling throttler (#1956)

2022-05-18 01:27:06 -07:00

command_server_tls.go

chore(general): updated linter to 1.44.0 (#1681)

2022-01-25 21:21:13 -08:00

command_server.go

feat(cli): added CLI for controlling throttler (#1956)

2022-05-18 01:27:06 -07:00

command_session_list.go

cli: plumbed through 'textOutput' which controls stdout/stderr writers (#1053)

2021-05-06 20:26:35 -07:00

command_session.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

command_show.go

minor spelling and grammar fixes (#1318)

2021-09-28 15:19:02 -07:00

command_snapshot_copy_move_history.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

command_snapshot_create.go

chore(general): updated linter to 1.44.0 (#1681)

2022-01-25 21:21:13 -08:00

command_snapshot_delete.go

cli: added 'snapshot delete --all-snapshots-for-source' (#1397)

2021-10-17 09:46:44 -07:00

command_snapshot_estimate_test.go

Added support for logging policies (#1472)

2021-11-06 10:06:05 -07:00

command_snapshot_estimate.go

cli: test + misc improvements (#1083)

2021-05-17 21:47:11 -07:00

command_snapshot_expire.go

Upgraded linter to 1.40.1 (#1072)

2021-05-15 12:12:34 -07:00

command_snapshot_fix_invalid_files.go

feat(cli): implementation for 'kopia snapshot fix' (#1930)

2022-05-25 01:17:55 +00:00

command_snapshot_fix_remove_files.go

feat(cli): implementation for 'kopia snapshot fix' (#1930)

2022-05-25 01:17:55 +00:00

command_snapshot_fix_test.go

feat(cli): implementation for 'kopia snapshot fix' (#1930)

2022-05-25 01:17:55 +00:00

command_snapshot_fix.go

feat(cli): implementation for 'kopia snapshot fix' (#1930)

2022-05-25 01:17:55 +00:00

command_snapshot_gc.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

command_snapshot_list.go

chore(ci): added checklocks static analyzer (#1838)

2022-03-19 22:42:59 -07:00

command_snapshot_migrate.go

chore(infra): added more leak checks to tests (#1953)

2022-05-16 06:37:57 +00:00

command_snapshot_pin_test.go

cli: added 'snapshot pin' command (#1528)

2021-11-20 20:53:25 -08:00

command_snapshot_pin.go

cli: added 'snapshot pin' command (#1528)

2021-11-20 20:53:25 -08:00

command_snapshot_restore.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

command_snapshot_storage_stats_test.go

feat(cli): added snapshot list flags: --storage-stats and --reverse (#1766)

2022-02-21 20:24:13 -08:00

command_snapshot_verify.go

refactor(snapshots): extracted snapshotfs.Verifier component (#1921)

2022-05-02 04:03:28 +00:00

command_snapshot.go

feat(cli): implementation for 'kopia snapshot fix' (#1930)

2022-05-25 01:17:55 +00:00

command_user_add_set.go

cli: plumbed through 'textOutput' which controls stdout/stderr writers (#1053)

2021-05-06 20:26:35 -07:00

command_user_delete.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

command_user_info.go

cli: plumbed through 'textOutput' which controls stdout/stderr writers (#1053)

2021-05-06 20:26:35 -07:00

command_user_list.go

cli: plumbed through 'textOutput' which controls stdout/stderr writers (#1053)

2021-05-06 20:26:35 -07:00

command_user.go

cli: major refactoring (#1046)

2021-05-03 10:28:00 -07:00

config.go

introduced structural debug logging and optional JSON output (#1475)

2021-11-03 21:57:37 -07:00

inproc.go

Upgraded linter to 1.43.0 (#1505)

2021-11-11 17:03:11 -08:00

json_output.go

cli: plumbed through 'textOutput' which controls stdout/stderr writers (#1053)

2021-05-06 20:26:35 -07:00

memory_tracking.go

huge: replaced the use of allocated byte slices with populating gather.WriteBuffer in the repository (#1244)

2021-08-20 08:45:10 -07:00

password_darwin.go

repo: refactored password persistence (#1065)

2021-05-11 21:53:36 -07:00

password_linux.go

cli: linux: read KOPIA_USE_KEYRING env (#1337)

2021-10-01 22:56:50 -07:00

password_other.go

linter: upgrade to 1.42.1 (#1292)

2021-09-14 19:11:39 -07:00

password_windows.go

repo: refactored password persistence (#1065)

2021-05-11 21:53:36 -07:00

password.go

blob: changed default shards from {3,3} to {1,3} (#1513)

2021-11-16 06:02:04 -08:00

profile.go

huge: replaced the use of allocated byte slices with populating gather.WriteBuffer in the repository (#1244)

2021-08-20 08:45:10 -07:00

show_utils.go

feat(cli): improved formatting of 'policy show' outputs (#1767)

2022-02-22 22:21:48 -08:00

sighup_unix.go

linter: upgrade to 1.42.1 (#1292)

2021-09-14 19:11:39 -07:00

sighup_windows.go

Added repository user authorization support + server flag refactoring + refresh (#890)

2021-03-18 23:03:27 -07:00

storage_azure.go

refactor(cli): An in-memory storage mock setup for CLI tests (#1697)

2022-02-01 10:29:13 -08:00

storage_b2.go

refactor(cli): An in-memory storage mock setup for CLI tests (#1697)

2022-02-01 10:29:13 -08:00

storage_filesystem.go

refactor(cli): An in-memory storage mock setup for CLI tests (#1697)

2022-02-01 10:29:13 -08:00

storage_gcs.go

refactor(cli): An in-memory storage mock setup for CLI tests (#1697)

2022-02-01 10:29:13 -08:00

storage_gdrive.go

feat(providers): add Google Drive support (#1731)

2022-02-16 22:34:48 -08:00

storage_providers.go

chore(general): minor cleanups (#1959)

2022-05-23 15:16:25 -07:00

storage_rclone.go

refactor(cli): An in-memory storage mock setup for CLI tests (#1697)

2022-02-01 10:29:13 -08:00

storage_s3.go

refactor(cli): An in-memory storage mock setup for CLI tests (#1697)

2022-02-01 10:29:13 -08:00

storage_sftp_test.go

blob: changed default shards from {3,3} to {1,3} (#1513)

2021-11-16 06:02:04 -08:00

storage_sftp.go

refactor(cli): An in-memory storage mock setup for CLI tests (#1697)

2022-02-01 10:29:13 -08:00

storage_webdav.go

refactor(cli): An in-memory storage mock setup for CLI tests (#1697)

2022-02-01 10:29:13 -08:00

suite_test.go

fix(repository): fixed slow goroutine leak from indexBlobCache, added tests (#1950)

2022-05-16 01:21:30 +00:00

throttle_get.go

feat(cli): added CLI for controlling throttler (#1956)

2022-05-18 01:27:06 -07:00

throttle_set.go

feat(cli): added CLI for controlling throttler (#1956)

2022-05-18 01:27:06 -07:00

update_check.go

Upgraded linter to 1.43.0 (#1505)

2021-11-11 17:03:11 -08:00