Julio Lopez
b0f99ccb7f
refactor(general): remove dry run option for blob retention ext ( #4962 )
2025-11-06 11:12:46 -08:00
lyndon-li
3b7f193c29
feature(general): maintenance stats for rewrite contents phase ( #4961 )
2025-11-06 10:22:58 -08:00
Julio Lopez
1961ed4dc3
refactor(general): extendBlobRetentionTime ( #4960 )
...
Refactor extendeBlobRetentionTime
- unexport extendBlobRetentionTime
- use `atomic.Uint32` type instead of `uint32` with atomic functions
- rename `cnt` to `extendedCount` for clarity
- simplify logging
- return blob.ID from GetLockingStoragePrefixes
- simplify extendBlobRetentionTime with errgroup
- remove linter annotation
- remove ineffective dryRun check
2025-11-05 23:28:17 -08:00
lyndon-li
3446837bed
feat(general): maintenance stats for clean up logs phase ( #4959 )
2025-11-05 23:23:47 -08:00
Julio Lopez
c7ea2c6e09
refactor(general): consistent use "pack" where possible ( #4952 )
...
In kopia, "blob" is a generic term to refer to either
an object in an object storage provider, or a file
in a file system storage provider. There are various
types of blobs in a kopia repository.
In kopia, the term "pack" is used to refer to specific types
of blobs, namely 'p' & 'q' pack blobs, that store
"content" data, as opposed to say, "index" blobs.
This change attempts to use the term "pack" consistently
in the functions and types used for pack deletion.
Note that the corresponding task names, shown below, remain
unchanged since these names are used in the persistent
maintenance run metadata, and that is used to make decisions
about the safety of the execution of those tasks.
```
TaskDeleteOrphanedBlobsQuick = "quick-delete-blobs"
TaskDeleteOrphanedBlobsFull = "full-delete-blobs"
```
2025-11-05 22:03:21 -08:00
Julio Lopez
16b88cb305
refactor(general): fix field names in ExtendBlobRetentionStats ( #4958 )
2025-11-05 16:36:47 -08:00
lyndon-li
af38dfc69d
feat(general) maintenance stats for blob retention extension ( #4956 )
2025-11-05 16:16:44 -08:00
Julio Lopez
f400ec6037
deps(build): upgrade Go to 1.24.10 ( #4957 )
2025-11-05 15:18:31 -08:00
lyndon-li
db48fad95e
feat(general): maintenance stats for delete unreferenced packs phase ( #4955 )
2025-11-04 22:51:17 -08:00
Kopia Builder [bot]
f1ad165294
feat(ui): upgraded htmlui to the latest version ( #4954 )
2025-11-04 20:49:38 -08:00
lyndon-li
83bd4d45da
feat(general): maintenance stats for drop deleted contents and compact indexes ( #4948 )
2025-11-04 11:40:20 -08:00
dependabot[bot]
4ab8ecc341
build(deps): bump github.com/prometheus/common ( #4951 )
...
Bumps the telemetry-dependencies group with 1 update: [github.com/prometheus/common](https://github.com/prometheus/common ).
Updates `github.com/prometheus/common` from 0.67.1 to 0.67.2
- [Release notes](https://github.com/prometheus/common/releases )
- [Changelog](https://github.com/prometheus/common/blob/main/CHANGELOG.md )
- [Commits](https://github.com/prometheus/common/compare/v0.67.1...v0.67.2 )
---
updated-dependencies:
- dependency-name: github.com/prometheus/common
dependency-version: 0.67.2
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: telemetry-dependencies
...
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-11-04 02:58:31 +00:00
dependabot[bot]
99738c8f4f
build(deps): bump the common-golang-dependencies group with 2 updates ( #4950 )
...
Bumps the common-golang-dependencies group with 2 updates: [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go ) and [google.golang.org/api](https://github.com/googleapis/google-api-go-client ).
Updates `cloud.google.com/go/storage` from 1.57.0 to 1.57.1
- [Release notes](https://github.com/googleapis/google-cloud-go/releases )
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md )
- [Commits](https://github.com/googleapis/google-cloud-go/compare/spanner/v1.57.0...storage/v1.57.1 )
Updates `google.golang.org/api` from 0.253.0 to 0.254.0
- [Release notes](https://github.com/googleapis/google-api-go-client/releases )
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md )
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.253.0...v0.254.0 )
---
updated-dependencies:
- dependency-name: cloud.google.com/go/storage
dependency-version: 1.57.1
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: common-golang-dependencies
- dependency-name: google.golang.org/api
dependency-version: 0.254.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: common-golang-dependencies
...
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-11-03 15:26:49 -08:00
Julio Lopez
952058a744
refactor(general): fix field names in CompactSingleEpochStats ( #4947 )
...
- Followup to #4941
2025-11-02 21:01:10 -08:00
lyndon-li
29ee7b18ea
compact single epoch stats ( #4941 )
...
Add stats to maintenance phase: `CompactSingleEpoch`
2025-11-02 20:38:23 -08:00
dependabot[bot]
b130e77e3b
build(deps): bump the github-actions group with 5 updates ( #4945 )
...
Bumps the github-actions group with 5 updates:
| Package | From | To |
| --- | --- | --- |
| [actions/upload-artifact](https://github.com/actions/upload-artifact ) | `4.6.2` | `5.0.0` |
| [actions/dependency-review-action](https://github.com/actions/dependency-review-action ) | `4.8.0` | `4.8.1` |
| [actions/download-artifact](https://github.com/actions/download-artifact ) | `5.0.0` | `6.0.0` |
| [github/codeql-action](https://github.com/github/codeql-action ) | `3.30.5` | `4.31.2` |
| [actions/stale](https://github.com/actions/stale ) | `10.0.0` | `10.1.0` |
Updates `actions/upload-artifact` from 4.6.2 to 5.0.0
- [Release notes](https://github.com/actions/upload-artifact/releases )
- [Commits](ea165f8d65...330a01c490https://github.com/actions/dependency-review-action/releases )
- [Commits](56339e523c...40c09b7dc9https://github.com/actions/download-artifact/releases )
- [Commits](634f93cb29...018cc2cf5bhttps://github.com/github/codeql-action/releases )
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md )
- [Commits](3599b3baa1...0499de31b9https://github.com/actions/stale/releases )
- [Changelog](https://github.com/actions/stale/blob/main/CHANGELOG.md )
- [Commits](3a9db7e6a4...5f858e3efbsupport@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-11-02 20:15:18 -08:00
Julio Lopez
069018adca
refactor(general): use explicit return values ( #4940 )
...
Add return values in long helper function. This Improves clarity
and avoids 'naked' return and removes `//nolint:nakedret` annotation.
Explicit return values for GetShardedPathAndFilePath
2025-10-30 23:14:58 -07:00
lyndon-li
ed7ef85a2d
feat(general): add stats to maintenance run - AdvanceEpoch ( #4937 )
2025-10-30 23:11:15 -07:00
Julio Lopez
934529038d
refactor(general): diskCommitedContentIndexCache ( #4939 )
...
No functional changes.
- simplifies function name to mmapFile to better reflect what it does
- rename parameter to `filename` for clarity
- expands comment for clarity
2025-10-30 22:37:29 -07:00
Julio Lopez
763e88d177
refactor(general): buildinfo helpers ( #4936 )
...
- add explicit return values for `getBuildInfoAndVersion`
- rename package alias to `stdlog` for clarity
2025-10-29 22:46:22 -07:00
Julio Lopez
eacee60d19
chore(ci): update copilot instructions for reviews ( #4935 )
2025-10-29 22:19:57 -07:00
Julio Lopez
020092d973
refactor(general): improve randRead helper ( #4934 )
...
also, removes unnecessary naked return
2025-10-29 22:07:14 -07:00
Julio Lopez
a51c71004c
chore(ci): add copilot instructions ( #4933 )
...
General and Go-specific instructions for Copilot
2025-10-29 22:04:03 -07:00
Julio Lopez
4150cf933b
refactor(general): index.maxContentLengths ( #4932 )
...
- leverage `max` builtin
- explicit return values (to satisfy linters)
2025-10-29 17:08:22 -07:00
Julio Lopez
9cdba57783
chore(ci): add linter config for gofumpt ( #4929 )
2025-10-29 12:33:30 -07:00
Julio Lopez
356930b1a9
refactor(general): simplify function and generics signatures ( #4930 )
2025-10-29 12:32:51 -07:00
Julio Lopez
5ca49e8eb4
refactor(general): lastSettledEpochNumber() helper ( #4927 )
2025-10-28 21:17:45 -07:00
Julio Lopez
2b2efeb031
refactor(general): fix return value order to match return names ( #4926 )
2025-10-28 18:28:01 -07:00
Julio Lopez
736ee451b9
refactor(general): rename GenerateRangeCheckpointStats fields ( #4925 )
...
- Rename GenerateRangeCheckpointStats fields for consistency
- Add marshaling test for GenerateRangeCheckpointStats
2025-10-28 17:32:39 -07:00
lyndon-li
930793c754
feat(general) maintenance stats for generate range checkpoint ( #4924 )
2025-10-28 16:48:28 -07:00
dependabot[bot]
47a43c7ce6
build(deps): bump google.golang.org/api ( #4923 )
...
Bumps the common-golang-dependencies group with 1 update: [google.golang.org/api](https://github.com/googleapis/google-api-go-client ).
Updates `google.golang.org/api` from 0.252.0 to 0.253.0
- [Release notes](https://github.com/googleapis/google-api-go-client/releases )
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md )
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.252.0...v0.253.0 )
---
updated-dependencies:
- dependency-name: google.golang.org/api
dependency-version: 0.253.0
dependency-type: direct:production
update-type: version-update:semver-minor
dependency-group: common-golang-dependencies
...
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-10-28 00:15:54 +00:00
dependabot[bot]
20aae8b600
build(deps): bump github.com/hanwen/go-fuse/v2 from 2.8.0 to 2.9.0 ( #4922 )
...
Bumps [github.com/hanwen/go-fuse/v2](https://github.com/hanwen/go-fuse ) from 2.8.0 to 2.9.0.
- [Commits](https://github.com/hanwen/go-fuse/compare/v2.8.0...v2.9.0 )
---
updated-dependencies:
- dependency-name: github.com/hanwen/go-fuse/v2
dependency-version: 2.9.0
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-10-27 23:46:47 +00:00
dependabot[bot]
6743bfb8c7
build(deps): bump github.com/pkg/sftp from 1.13.9 to 1.13.10 ( #4921 )
...
Bumps [github.com/pkg/sftp](https://github.com/pkg/sftp ) from 1.13.9 to 1.13.10.
- [Release notes](https://github.com/pkg/sftp/releases )
- [Commits](https://github.com/pkg/sftp/compare/v1.13.9...v1.13.10 )
---
updated-dependencies:
- dependency-name: github.com/pkg/sftp
dependency-version: 1.13.10
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com >
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-10-27 23:39:12 +00:00
Julio Lopez
c3dc18f7e5
fix typo ( #4920 )
2025-10-27 16:12:20 -07:00
lyndon-li
727fea0801
feat(general): add maintenance phase stats - CleanupSupersededIndexes ( #4916 )
2025-10-27 12:49:38 -07:00
Julio Lopez
25b1492d65
fix(general): field and variable names ( #4911 )
...
Fixes field names in CleanupMarkersStats persistent struct.
Also renames variables for consistency.
- Followup fix for: #4900 add stats to maintenance run - CleanupMarkers
- Ref: #4848
2025-10-24 16:09:36 -07:00
lyndon-li
e3e4e09bc8
feat(general): add stats to maintenance run - CleanupMarkers ( #4900 )
2025-10-24 13:49:40 -07:00
Julio Lopez
22f0919644
refactor(general): simplify construction of error messages ( #4913 )
2025-10-24 13:45:42 -07:00
Julio Lopez
d8519ab9fa
refactor(test): run cases as sub-tests ( #4912 )
2025-10-24 12:40:45 -07:00
Julio Lopez
5bc467e2ca
refactor(general): omitzero JSON tag ( #4907 )
...
* remove omitempty from object id fields
* remove omitempty from storage stats fields
* use omitzero for other structs
* remove `omitempty` from `UpgradeLockIntent` fields
The `omitempty` JSON tag is ineffective for fields of
type `Time` and `Duration`.
Also, this struct is only used during the "upgrade"
protocol, and it is OK to explicitly serialize 0 values,
as it was done before.
2025-10-23 22:59:52 -07:00
Julio Lopez
bab11c181d
nit: remove unused constant ( #4906 )
2025-10-23 22:19:56 -07:00
Julio Lopez
4cbd7026a6
refactor(general): leverage maps.Clone ( #4905 )
2025-10-23 19:45:29 -07:00
Julio Lopez
c5cd3c5651
refactor(general): modernize bloop ( #4904 )
2025-10-23 17:49:20 -07:00
Julio Lopez
995e7fd893
refactor(general): modernize ( #4903 )
...
Applies the modernize changes for the following categories:
- mapsloop
- stringsseq
- stringscutprefix
- sortslice
2025-10-23 17:11:38 -07:00
Kopia Builder [bot]
6472bf386d
feat(ui): upgraded htmlui to the latest version ( #4902 )
2025-10-23 17:49:47 +00:00
Kopia Builder [bot]
4683d37874
feat(ui): upgraded htmlui to the latest version ( #4899 )
2025-10-23 05:29:14 +00:00
Julio Lopez
af62386ecd
chore(general): modernize min/max ( #4898 )
2025-10-22 22:22:54 -07:00
Samuel Maynard
11765d696c
feat(general): reduce number of open file descriptors for indexes on Unix ( #4866 )
...
Close file after mmap on Unix to reduce open file descriptors.
On Unix-like platforms, close the file descriptor immediately after a successful
mmap.Map of .sndx index cache files. This keeps the mapping valid until Unmap
(per POSIX semantics) and significantly reduces steady-state FD usage when many
indexes are open, helping avoid EMFILE ("too many open files").
- Split mmapOpenWithRetry into platform-specific implementations:
- committed_content_index_disk_cache_unix.go (!windows):
- Map RDONLY, close FD immediately.
- Return closer that only unmaps.
- committed_content_index_disk_cache_windows.go (windows):
- Keep FD open until Unmap.
- Return closer that unmaps and closes FD.
- Remove old mmapOpenWithRetry and mmap import from
repo/content/committed_content_index_disk_cache.go.
- Add Linux-only unit test verifying FD count does not grow proportionally:
repo/content/committed_content_index_fd_linux_test.go
- Creates N small indexes, opens them all, checks /proc/self/fd delta stays low.
Notes:
- Behavior unchanged on Windows due to OS semantics.
- Mapping failures close the FD to avoid leaks.
- Unlink semantics remain correct; mappings stay valid until Unmap.
* remove windows only retry logic under unix
2025-10-22 21:16:35 -07:00
Julio Lopez
b6da922e3a
fix(general): address linter warnings ( #4897 )
2025-10-22 19:01:10 -07:00
lyndon-li
657db9706b
feat(general): interfaces and structures for maintenance stats ( #4890 )
...
Maintenance is critical for healthy of the repository.
On the other hand, Maintenance is complex, because
it runs multiple sub tasks each may generate different
results according to the maintenance policy.
The results may include deleting/combining/adding
data/metadata to the repository.
It is worthy to add more observability for these
tasks for below reasons:
It is helpful for troubleshooting. Any data change
to the repository is critical, the observability info
helps to understand what happened during the
maintenance and why that happened.
It is helpful for users to understand/predict the
repo's behavior. The repo data may be stored
in a public cloud for which costs are sensitive
to scale/duration of data stored. On the other
hand, repository has its own policy to manage
the data, so the data is not deleted until it is
safe enough according to the policy.
The observability info helps users to
understand how much data is in-use,
how much data is out of use and
when it is deleted
2025-10-22 18:14:10 -07:00
