- check all mismatched fields in a content info
- change return type to `error` for `CheckIndexInfo`
- log errors as soon as they are detected, instead of accumulating them in memory
- refactor `drainAllClients`
Also:
- unexport `cli.checkIndexInfo`
- reduce `err` scope
- separate matching vs non-matching tests
- Track return error with errgroup.
- Check for ctx error in scan loop.
- Drain reader streams on error to prevent deadlock.
- Check stderr scanning error after starting async stderr draining.
- Disconnect repo before test context is cancelled.
Defer instead of t.Cleanup in test. Needed to preserve
the cleanup order relative to repo disconnect.
Avoid calling `RunAndProcessStderr` in a background goroutine in tests.
Refactor TestServerControl and TestServerControlUDS.
Simplifies tests by removing unnecessary async processing.
Improves test cleanup.
- Required for #5626
`--parallel` has had no effect on `snapshot verify` since #1921, which
merged the tree-walk and file-verification parallelism into the single
`VerifierOptions.Parallelism` field and wired it to `--file-parallelism`.
`verifyCommandParallel` has had no reader since then: the flag parses,
advertises a default of 8, and is silently discarded.
The Consistency documentation page still recommends `--parallel=10` in
four places, so users following it believe they are tuning something.
Follow the existing convention in this file for `--all-sources`: hide the
flag, label it as having no effect, and log a deprecation message when it
is passed. Dropping `Default("8")` lets the zero value distinguish an
explicit `--parallel` from an absent one, and stops the help text from
advertising a default that does nothing. The flag still parses, so
existing scripts and cron jobs keep working.
Restoring a separate tree-walk parallelism knob would be the alternative
fix, but that changes the public `snapshotfs.VerifierOptions` surface.
The --parallel flag was declared with kingpin .IntVar on 12 commands,
which accepted negative values and led to a panic at runtime (e.g.
`snapshot migrate --parallel=-1`).
Declare the --parallel family as uint so kingpin rejects negative input
at parse time.
The uint->int conversion the callee APIs need is done through a small
helper .
Add CLI test that asserts the rejection message (a bare failure check passed
even without the fix since the commands also fail with no repository connected).
Parse via ParseFloat like kingpin's built-in int flag so the only behavior
change is rejecting negatives.
- use struct field initialization idiom
- clarify error message
- directly use s.rootctx in server request processing
- refactor: use require in e2e ACL test
- inline constant definition
- modernize with `strings.Cut`
- modernize with slices.Backward
- simplify timeFormat initialization
- refactor testing: check `Scanner.Err()` in testenv
- remove spurious parenthesis
- trueStr and falseStr consts
- use inheritPolicyString
- use const in cache info
- "bytes" const in ui task counters
- use common consts in gettool
Fixes for new linter warnings:
- fix: add //nolint annotations where it makes sense
- fix: remove no-longer necessary linter annotations
- fix linter: add or remove new lines
- fix linter: use Fprintf instead of WriteString
- fix: check for overflow in conversion to uint8 / byte
- remove no longer needed `repositoryAccessMode` struct
- pulls up auto-maintenance run check to the call path
where it is actually used, after write session flushes to
keep existing behavior
- nit: put nil check inside function
* fix(cli): only run automatic maintenance after data-modifying operations
Automatic maintenance previously ran after any successful repository-writer
command, including control-only operations such as `policy set`, `acl ...` and
`user ...`. Make repositoryWriterAction skip maintenance by default and add
repositoryWriterActionWithMaintenance, used only by data-modifying commands
(snapshot create/delete/expire/migrate/fix-*/copy-move-history/pin,
manifest delete).
Fixes#3174
* fix(cli): treat snapshot pin and manifest delete as control-only
Per review, move both to repositoryWriterAction so they no longer trigger
opportunistic automatic maintenance. pin orphans no content and manifest
delete is a low-level operation not meant for direct use; any content it
orphans is reclaimed at the next scheduled maintenance.
---------
Co-authored-by: max <max@example.com>
Add fadvise hints to backup read path to reduce OS cache pollution
Issue FADV_SEQUENTIAL at open and FADV_DONTNEED at close for every file read during backup. This tells the kernel to read-ahead aggressively and then reclaim pages once the file is fully consumed, reducing cache pressure from large backups without adding measurable overhead.
Uses SyscallConn().Control() instead of os.File.Fd() to avoid switching file descriptors to blocking mode, which would remove them from Go's epoll poller and stall the upload pipeline.
Incremental FADV_DONTNEED during reads was tested and removed — it adds ~15% overhead on large-file workloads by fighting the kernel's own LRU. Restore write-path cache eviction was also tested and not included.
Authored-by: Rajat Gupta rajat.gupta@veeam.com
* feat(cli): restrict insecure unauthenticated server to loopback binds
Reject starting the server with --insecure and --without-password when
--address would bind outside loopback (including empty host / all
interfaces). Validate the actual listener after Listen so systemd socket
activation cannot bypass the check.
Add hidden --allow-extremely-dangerous-unauthenticated-server-on-the-network
to opt into the previous behavior for isolated environments.
Extract validation into internal/insecureserverbind with table-driven tests.
Add end-to-end smoke tests for rejection and escape hatch.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Update internal/insecureserverbind/insecureserverbind.go
Co-authored-by: Julio López <1953782+julio-lopez@users.noreply.github.com>
* pr feedback
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Julio López <1953782+julio-lopez@users.noreply.github.com>
- make benchmarking params uint
- prevent error in compression benchmarking
- lint on Windows and address linter warnings
- upgrade golang.org/x/exp
- upgrade github.com/cncf/xds/go
- upgrade github.com/dustinkirkland/golang-petname
- direct users to forum
- add warning about _recovery recipes_
Minor robustness improvements
- add safety check when converting file descriptor (`os.File.Fd()`) to `int`
- check that file descriptor returned by Open (on Linux) is non-negative before converting to `uintptr`. This addresses a linter warning for a most-likely-non-existent-edge-case of converting a negative file descriptor.
- check that parsed content IDs (hashes) do not exceed the maximum id hash length.
- add nolint annotation for safe conversion
Cleanup nits:
- get error handling policy upfront and improve readability in uploader
- update error message
- update field documentation and update flag description
- remove unused function
- const `isWindows` and remove redundant condition check
- add `getEnvVarBool` helper
- refactor common helper for mockfs.AddError* functions, and
add mockfs.AddErrorEntry<Type> wrappers for clarity.
- removed list of skipped tests from gotestsum summary
When running without a tty, disable progress by default. The progress
includes binary characters such as a spinner that do not display
properly without a tty.
Refactor `--profile-*` flags:
- Multiple profile types can be enabled at once, before only
a single type profiling could be done during a process execution.
- The new `--profiles-store-on-exit` enables all available profile
types, except for CPU profiling which needs to be explicitly enabled.
- Profiling parameters can now be set via new flags. This allows setting
the profile parameters for the pprof endpoint, as well as when saving
profiles to files on exit.
- Group profiling flags with other observability flags
- Adds a `--diagnostics-output-directory` flag that unifies and
supersedes the `--profile-dir` and `--metrics-directory` flags
Enhancements and behavior changes:
- Profile flags now have effect for all kopia commands, including
`server start`. Before these flags did not have any effect
in a few commands.
- Multiple profile types can be enabled at once, before only
a single type profiling could be done during a process execution.
- The new `--profiles-store-on-exit` enables all available profile
types, except for CPU profiling which needs to be explicitly enabled.
- Profiling parameters can now be set via new flags. This allows setting
the profile parameters for the pprof endpoint, as well as when saving
profiles to files on exit.
The following flags have been removed:
- `--profile-dir`: superseded by the `--diagnostics-output-directory` flag
- `--profile-blocking`: the `--profile-store-on-exit` flag enables blocking
profiling. Use `--profile-blocking-rate=0` to explicitly disable it.
- `--profile-memory`: the `--profile-store-on-exit` flag enables memory
profiling. Use `--profile-memory-rate=0` to explicitly disable it.
- `--profile-mutex`: the `--profile-store-on-exit` flag enables mutex
profiling. Use `--profile-mutex-fraction=0` to explicitly disable it.
Add CLI test for profile flags.
- upgrade to golangci-lint 2.6.1
- updates for gosec
- updates for govet
- updates for perfsprint
- updates modernize
Leaves out modernize:omitempty due to conflicts with tests
In kopia, "blob" is a generic term to refer to either
an object in an object storage provider, or a file
in a file system storage provider. There are various
types of blobs in a kopia repository.
In kopia, the term "pack" is used to refer to specific types
of blobs, namely 'p' & 'q' pack blobs, that store
"content" data, as opposed to say, "index" blobs.
This change attempts to use the term "pack" consistently
in the functions and types used for pack deletion.
Note that the corresponding task names, shown below, remain
unchanged since these names are used in the persistent
maintenance run metadata, and that is used to make decisions
about the safety of the execution of those tasks.
```
TaskDeleteOrphanedBlobsQuick = "quick-delete-blobs"
TaskDeleteOrphanedBlobsFull = "full-delete-blobs"
```
Julio López