`--parallel` has had no effect on `snapshot verify` since #1921, which
merged the tree-walk and file-verification parallelism into the single
`VerifierOptions.Parallelism` field and wired it to `--file-parallelism`.
`verifyCommandParallel` has had no reader since then: the flag parses,
advertises a default of 8, and is silently discarded.
The Consistency documentation page still recommends `--parallel=10` in
four places, so users following it believe they are tuning something.
Follow the existing convention in this file for `--all-sources`: hide the
flag, label it as having no effect, and log a deprecation message when it
is passed. Dropping `Default("8")` lets the zero value distinguish an
explicit `--parallel` from an absent one, and stops the help text from
advertising a default that does nothing. The flag still parses, so
existing scripts and cron jobs keep working.
Restoring a separate tree-walk parallelism knob would be the alternative
fix, but that changes the public `snapshotfs.VerifierOptions` surface.
The --parallel flag was declared with kingpin .IntVar on 12 commands,
which accepted negative values and led to a panic at runtime (e.g.
`snapshot migrate --parallel=-1`).
Declare the --parallel family as uint so kingpin rejects negative input
at parse time.
The uint->int conversion the callee APIs need is done through a small
helper .
Add CLI test that asserts the rejection message (a bare failure check passed
even without the fix since the commands also fail with no repository connected).
Parse via ParseFloat like kingpin's built-in int flag so the only behavior
change is rejecting negatives.
- use struct field initialization idiom
- clarify error message
- directly use s.rootctx in server request processing
- refactor: use require in e2e ACL test
- inline constant definition
- modernize with `strings.Cut`
- modernize with slices.Backward
- simplify timeFormat initialization
- refactor testing: check `Scanner.Err()` in testenv
- remove spurious parenthesis
- trueStr and falseStr consts
- use inheritPolicyString
- use const in cache info
- "bytes" const in ui task counters
- use common consts in gettool
Fixes for new linter warnings:
- fix: add //nolint annotations where it makes sense
- fix: remove no-longer necessary linter annotations
- fix linter: add or remove new lines
- fix linter: use Fprintf instead of WriteString
- fix: check for overflow in conversion to uint8 / byte
- remove no longer needed `repositoryAccessMode` struct
- pulls up auto-maintenance run check to the call path
where it is actually used, after write session flushes to
keep existing behavior
- nit: put nil check inside function
* fix(cli): only run automatic maintenance after data-modifying operations
Automatic maintenance previously ran after any successful repository-writer
command, including control-only operations such as `policy set`, `acl ...` and
`user ...`. Make repositoryWriterAction skip maintenance by default and add
repositoryWriterActionWithMaintenance, used only by data-modifying commands
(snapshot create/delete/expire/migrate/fix-*/copy-move-history/pin,
manifest delete).
Fixes#3174
* fix(cli): treat snapshot pin and manifest delete as control-only
Per review, move both to repositoryWriterAction so they no longer trigger
opportunistic automatic maintenance. pin orphans no content and manifest
delete is a low-level operation not meant for direct use; any content it
orphans is reclaimed at the next scheduled maintenance.
---------
Co-authored-by: max <max@example.com>
Add fadvise hints to backup read path to reduce OS cache pollution
Issue FADV_SEQUENTIAL at open and FADV_DONTNEED at close for every file read during backup. This tells the kernel to read-ahead aggressively and then reclaim pages once the file is fully consumed, reducing cache pressure from large backups without adding measurable overhead.
Uses SyscallConn().Control() instead of os.File.Fd() to avoid switching file descriptors to blocking mode, which would remove them from Go's epoll poller and stall the upload pipeline.
Incremental FADV_DONTNEED during reads was tested and removed — it adds ~15% overhead on large-file workloads by fighting the kernel's own LRU. Restore write-path cache eviction was also tested and not included.
Authored-by: Rajat Gupta rajat.gupta@veeam.com
* feat(cli): restrict insecure unauthenticated server to loopback binds
Reject starting the server with --insecure and --without-password when
--address would bind outside loopback (including empty host / all
interfaces). Validate the actual listener after Listen so systemd socket
activation cannot bypass the check.
Add hidden --allow-extremely-dangerous-unauthenticated-server-on-the-network
to opt into the previous behavior for isolated environments.
Extract validation into internal/insecureserverbind with table-driven tests.
Add end-to-end smoke tests for rejection and escape hatch.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Update internal/insecureserverbind/insecureserverbind.go
Co-authored-by: Julio López <1953782+julio-lopez@users.noreply.github.com>
* pr feedback
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Julio López <1953782+julio-lopez@users.noreply.github.com>
- make benchmarking params uint
- prevent error in compression benchmarking
- lint on Windows and address linter warnings
- upgrade golang.org/x/exp
- upgrade github.com/cncf/xds/go
- upgrade github.com/dustinkirkland/golang-petname
- direct users to forum
- add warning about _recovery recipes_
Minor robustness improvements
- add safety check when converting file descriptor (`os.File.Fd()`) to `int`
- check that file descriptor returned by Open (on Linux) is non-negative before converting to `uintptr`. This addresses a linter warning for a most-likely-non-existent-edge-case of converting a negative file descriptor.
- check that parsed content IDs (hashes) do not exceed the maximum id hash length.
- add nolint annotation for safe conversion
Cleanup nits:
- get error handling policy upfront and improve readability in uploader
- update error message
- update field documentation and update flag description
- remove unused function
- const `isWindows` and remove redundant condition check
- add `getEnvVarBool` helper
- refactor common helper for mockfs.AddError* functions, and
add mockfs.AddErrorEntry<Type> wrappers for clarity.
- removed list of skipped tests from gotestsum summary
When running without a tty, disable progress by default. The progress
includes binary characters such as a spinner that do not display
properly without a tty.
Refactor `--profile-*` flags:
- Multiple profile types can be enabled at once, before only
a single type profiling could be done during a process execution.
- The new `--profiles-store-on-exit` enables all available profile
types, except for CPU profiling which needs to be explicitly enabled.
- Profiling parameters can now be set via new flags. This allows setting
the profile parameters for the pprof endpoint, as well as when saving
profiles to files on exit.
- Group profiling flags with other observability flags
- Adds a `--diagnostics-output-directory` flag that unifies and
supersedes the `--profile-dir` and `--metrics-directory` flags
Enhancements and behavior changes:
- Profile flags now have effect for all kopia commands, including
`server start`. Before these flags did not have any effect
in a few commands.
- Multiple profile types can be enabled at once, before only
a single type profiling could be done during a process execution.
- The new `--profiles-store-on-exit` enables all available profile
types, except for CPU profiling which needs to be explicitly enabled.
- Profiling parameters can now be set via new flags. This allows setting
the profile parameters for the pprof endpoint, as well as when saving
profiles to files on exit.
The following flags have been removed:
- `--profile-dir`: superseded by the `--diagnostics-output-directory` flag
- `--profile-blocking`: the `--profile-store-on-exit` flag enables blocking
profiling. Use `--profile-blocking-rate=0` to explicitly disable it.
- `--profile-memory`: the `--profile-store-on-exit` flag enables memory
profiling. Use `--profile-memory-rate=0` to explicitly disable it.
- `--profile-mutex`: the `--profile-store-on-exit` flag enables mutex
profiling. Use `--profile-mutex-fraction=0` to explicitly disable it.
Add CLI test for profile flags.
- upgrade to golangci-lint 2.6.1
- updates for gosec
- updates for govet
- updates for perfsprint
- updates modernize
Leaves out modernize:omitempty due to conflicts with tests
In kopia, "blob" is a generic term to refer to either
an object in an object storage provider, or a file
in a file system storage provider. There are various
types of blobs in a kopia repository.
In kopia, the term "pack" is used to refer to specific types
of blobs, namely 'p' & 'q' pack blobs, that store
"content" data, as opposed to say, "index" blobs.
This change attempts to use the term "pack" consistently
in the functions and types used for pack deletion.
Note that the corresponding task names, shown below, remain
unchanged since these names are used in the persistent
maintenance run metadata, and that is used to make decisions
about the safety of the execution of those tasks.
```
TaskDeleteOrphanedBlobsQuick = "quick-delete-blobs"
TaskDeleteOrphanedBlobsFull = "full-delete-blobs"
```
Maintenance is critical for healthy of the repository.
On the other hand, Maintenance is complex, because
it runs multiple sub tasks each may generate different
results according to the maintenance policy.
The results may include deleting/combining/adding
data/metadata to the repository.
It is worthy to add more observability for these
tasks for below reasons:
It is helpful for troubleshooting. Any data change
to the repository is critical, the observability info
helps to understand what happened during the
maintenance and why that happened.
It is helpful for users to understand/predict the
repo's behavior. The repo data may be stored
in a public cloud for which costs are sensitive
to scale/duration of data stored. On the other
hand, repository has its own policy to manage
the data, so the data is not deleted until it is
safe enough according to the policy.
The observability info helps users to
understand how much data is in-use,
how much data is out of use and
when it is deleted
* fix(cli): improve progress output control in repository sync and documentation
- Update progress flag description from "progress bar" to "progress output" for clarity
- Document progress control features in Logging, Synchronization, and Command-Line reference
- Support --no-progress flag for cleaner automation and scripting usage
- nit: rename function to repositoryAction.
It always calls the action with a repository
- move allocator stats functionality to observability
- rename observability functions to start/stop. They
start and stop more than just the metrics services.
- rename field to c.enablePProfEndpoint for clarity.
- add observability run function to make it explicit
where start and stop are called.
Ensure auto-maintenance errors are propagated.
This enables sending notifications for failed "auto-maintenances".
Preserve action callback error when closing the repository fails.
This is a breaking change to users who might be using Kopia as a library.
### Log Format
```json
{"t":"<timestamp-rfc-3389-microseconds>", "span:T1":"V1", "span:T2":"V2", "n":"<source>", "m":"<message>", /*parameters*/}
```
Where each record is associated with one or more spans that describe its scope:
* `"span:client": "<hash-of-username@hostname>"`
* `"span:repo": "<random>"` - random identifier of a repository connection (from `repo.Open`)
* `"span:maintenance": "<random>"` - random identifier of a maintenance session
* `"span:upload": "<hash-of-username@host:/path>"` - uniquely identifies upload session of a given directory
* `"span:checkpoint": "<random>"` - encapsulates each checkpoint operation during Upload
* `"span:server-session": "<random>"` -single client connection to the server
* `"span:flush": "<random>"` - encapsulates each Flush session
* `"span:maintenance": "<random>"` - encapsulates each maintenance operation
* `"span:loadIndex" : "<random>"` - encapsulates index loading operation
* `"span:emr" : "<random>"` - encapsulates epoch manager refresh
* `"span:writePack": "<pack-blob-ID>"` - encapsulates pack blob preparation and writing
(plus additional minor spans for various phases of the maintenance).
Notable points:
- Used internal zero allocation JSON writer for reduced memory usage.
- renamed `--disable-internal-log` to `--disable-repository-log` (controls saving blobs to repository)
- added `--disable-content-log` (controls writing of `content-log` files)
- all storage operations are also logged in a structural way and associated with the corresponding spans.
- all content IDs are logged in a truncated format (since first N bytes that are usually enough to be unique) to improve compressibility of logs (blob IDs are frequently repeated but content IDs usually appear just once).
This format should make it possible to recreate the journey of any single content throughout pack blobs, indexes and compaction events.
* Remove unused return value from ListIndexBlobInfos
* Unexport index.Builder.buildStable
* Remove unnecessary OneUseBuilder.BuildStable
* Remove unnecessary `BuilderCreator` interface,
use a function type instead.
* Cleanup comment
Vetea