mirror/kopia
1
0 Fork 0
mirror of https://github.com/kopia/kopia.git synced 2026-10-04 16:14:54 -04:00
Code Issues Packages Projects Releases Wiki Activity
3,903 Commits 11 Branches 90 Tags
b0f99ccb7f1a4a9ffadefa3668cdf6ec00568cd3
Commit Graph
3903 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Julio Lopez b0f99ccb7f refactor(general): remove dry run option for blob retention ext (#4962) 2025-11-06 11:12:46 -08:00
lyndon-li 3b7f193c29 feature(general): maintenance stats for rewrite contents phase (#4961) 2025-11-06 10:22:58 -08:00
Julio Lopez 1961ed4dc3 refactor(general): extendBlobRetentionTime (#4960) 
Refactor extendeBlobRetentionTime

- unexport extendBlobRetentionTime
- use `atomic.Uint32` type instead of `uint32` with atomic functions
- rename `cnt` to `extendedCount` for clarity
- simplify logging
- return blob.ID from GetLockingStoragePrefixes
- simplify extendBlobRetentionTime with errgroup
- remove linter annotation
- remove ineffective dryRun check
 2025-11-05 23:28:17 -08:00
lyndon-li 3446837bed feat(general): maintenance stats for clean up logs phase (#4959) 2025-11-05 23:23:47 -08:00
Julio Lopez c7ea2c6e09 refactor(general): consistent use "pack" where possible (#4952) 
In kopia, "blob" is a generic term to refer to either
an object in an object storage provider, or a file
in a file system storage provider. There are various
types of blobs in a kopia repository.

In kopia, the term "pack" is used to refer to specific types
of blobs, namely 'p' & 'q' pack blobs, that store
"content" data, as opposed to say, "index" blobs.

This change attempts to use the term "pack" consistently
in the functions and types used for pack deletion.

Note that the corresponding task names, shown below, remain
unchanged since these names are used in the persistent
maintenance run metadata, and that is used to make decisions
about the safety of the execution of those tasks.

```
	TaskDeleteOrphanedBlobsQuick     = "quick-delete-blobs"
	TaskDeleteOrphanedBlobsFull      = "full-delete-blobs"
```
 2025-11-05 22:03:21 -08:00
Julio Lopez 16b88cb305 refactor(general): fix field names in ExtendBlobRetentionStats (#4958) 2025-11-05 16:36:47 -08:00
lyndon-li af38dfc69d feat(general) maintenance stats for blob retention extension (#4956) 2025-11-05 16:16:44 -08:00
Julio Lopez f400ec6037 deps(build): upgrade Go to 1.24.10 (#4957) 2025-11-05 15:18:31 -08:00
lyndon-li db48fad95e feat(general): maintenance stats for delete unreferenced packs phase (#4955) 2025-11-04 22:51:17 -08:00
Kopia Builder [bot] f1ad165294 feat(ui): upgraded htmlui to the latest version (#4954) 2025-11-04 20:49:38 -08:00
lyndon-li 83bd4d45da feat(general): maintenance stats for drop deleted contents and compact indexes (#4948) 2025-11-04 11:40:20 -08:00
dependabot[bot] 4ab8ecc341 build(deps): bump github.com/prometheus/common (#4951) 
Bumps the telemetry-dependencies group with 1 update: [github.com/prometheus/common](https://github.com/prometheus/common).


Updates `github.com/prometheus/common` from 0.67.1 to 0.67.2
- [Release notes](https://github.com/prometheus/common/releases)
- [Changelog](https://github.com/prometheus/common/blob/main/CHANGELOG.md)
- [Commits](https://github.com/prometheus/common/compare/v0.67.1...v0.67.2)

---
updated-dependencies:
- dependency-name: github.com/prometheus/common
  dependency-version: 0.67.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: telemetry-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2025-11-04 02:58:31 +00:00
dependabot[bot] 99738c8f4f build(deps): bump the common-golang-dependencies group with 2 updates (#4950) 
Bumps the common-golang-dependencies group with 2 updates: [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) and [google.golang.org/api](https://github.com/googleapis/google-api-go-client).


Updates `cloud.google.com/go/storage` from 1.57.0 to 1.57.1
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/spanner/v1.57.0...storage/v1.57.1)

Updates `google.golang.org/api` from 0.253.0 to 0.254.0
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.253.0...v0.254.0)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/storage
  dependency-version: 1.57.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: common-golang-dependencies
- dependency-name: google.golang.org/api
  dependency-version: 0.254.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: common-golang-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2025-11-03 15:26:49 -08:00
Julio Lopez 952058a744 refactor(general): fix field names in CompactSingleEpochStats (#4947) 
- Followup to #4941
 2025-11-02 21:01:10 -08:00
lyndon-li 29ee7b18ea compact single epoch stats (#4941) 
Add stats to maintenance phase: `CompactSingleEpoch`
 2025-11-02 20:38:23 -08:00
dependabot[bot] b130e77e3b build(deps): bump the github-actions group with 5 updates (#4945) 
Bumps the github-actions group with 5 updates:

| Package | From | To |
| --- | --- | --- |
| [actions/upload-artifact](https://github.com/actions/upload-artifact) | `4.6.2` | `5.0.0` |
| [actions/dependency-review-action](https://github.com/actions/dependency-review-action) | `4.8.0` | `4.8.1` |
| [actions/download-artifact](https://github.com/actions/download-artifact) | `5.0.0` | `6.0.0` |
| [github/codeql-action](https://github.com/github/codeql-action) | `3.30.5` | `4.31.2` |
| [actions/stale](https://github.com/actions/stale) | `10.0.0` | `10.1.0` |


Updates `actions/upload-artifact` from 4.6.2 to 5.0.0
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/ea165f8d65b6e75b540449e92b4886f43607fa02...330a01c490aca151604b8cf639adc76d48f6c5d4)

Updates `actions/dependency-review-action` from 4.8.0 to 4.8.1
- [Release notes](https://github.com/actions/dependency-review-action/releases)
- [Commits](https://github.com/actions/dependency-review-action/compare/56339e523c0409420f6c2c9a2f4292bbb3c07dd3...40c09b7dc99638e5ddb0bfd91c1673effc064d8a)

Updates `actions/download-artifact` from 5.0.0 to 6.0.0
- [Release notes](https://github.com/actions/download-artifact/releases)
- [Commits](https://github.com/actions/download-artifact/compare/634f93cb2916e3fdff6788551b99b062d0335ce0...018cc2cf5baa6db3ef3c5f8a56943fffe632ef53)

Updates `github/codeql-action` from 3.30.5 to 4.31.2
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/3599b3baa15b485a2e49ef411a7a4bb2452e7f93...0499de31b99561a6d14a36a5f662c2a54f91beee)

Updates `actions/stale` from 10.0.0 to 10.1.0
- [Release notes](https://github.com/actions/stale/releases)
- [Changelog](https://github.com/actions/stale/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/stale/compare/3a9db7e6a41a89f618792c92c0e97cc736e1b13f...5f858e3efba33a5ca4407a664cc011ad407f2008)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-version: 5.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions
- dependency-name: actions/dependency-review-action
  dependency-version: 4.8.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: github-actions
- dependency-name: actions/download-artifact
  dependency-version: 6.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions
- dependency-name: github/codeql-action
  dependency-version: 4.31.2
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions
- dependency-name: actions/stale
  dependency-version: 10.1.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: github-actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2025-11-02 20:15:18 -08:00
Julio Lopez 069018adca refactor(general): use explicit return values (#4940) 
Add return values in long helper function. This Improves clarity
and avoids 'naked' return and removes `//nolint:nakedret` annotation.

Explicit return values for GetShardedPathAndFilePath
 2025-10-30 23:14:58 -07:00
lyndon-li ed7ef85a2d feat(general): add stats to maintenance run - AdvanceEpoch (#4937) 2025-10-30 23:11:15 -07:00
Julio Lopez 934529038d refactor(general): diskCommitedContentIndexCache (#4939) 
No functional changes.
- simplifies function name to mmapFile to better reflect what it does
- rename parameter to `filename` for clarity
- expands comment for clarity
 2025-10-30 22:37:29 -07:00
Julio Lopez 763e88d177 refactor(general): buildinfo helpers (#4936) 
- add explicit return values for `getBuildInfoAndVersion`
- rename package alias to `stdlog` for clarity
 2025-10-29 22:46:22 -07:00
Julio Lopez eacee60d19 chore(ci): update copilot instructions for reviews (#4935) 2025-10-29 22:19:57 -07:00
Julio Lopez 020092d973 refactor(general): improve randRead helper (#4934) 
also, removes unnecessary naked return
 2025-10-29 22:07:14 -07:00
Julio Lopez a51c71004c chore(ci): add copilot instructions (#4933) 
General and Go-specific instructions for Copilot
 2025-10-29 22:04:03 -07:00
Julio Lopez 4150cf933b refactor(general): index.maxContentLengths (#4932) 
- leverage `max` builtin
- explicit return values (to satisfy linters)
 2025-10-29 17:08:22 -07:00
Julio Lopez 9cdba57783 chore(ci): add linter config for gofumpt (#4929) 2025-10-29 12:33:30 -07:00
Julio Lopez 356930b1a9 refactor(general): simplify function and generics signatures (#4930) 2025-10-29 12:32:51 -07:00
Julio Lopez 5ca49e8eb4 refactor(general): lastSettledEpochNumber() helper (#4927) 2025-10-28 21:17:45 -07:00
Julio Lopez 2b2efeb031 refactor(general): fix return value order to match return names (#4926) 2025-10-28 18:28:01 -07:00
Julio Lopez 736ee451b9 refactor(general): rename GenerateRangeCheckpointStats fields (#4925) 
- Rename GenerateRangeCheckpointStats fields for consistency
- Add marshaling test for GenerateRangeCheckpointStats
 2025-10-28 17:32:39 -07:00
lyndon-li 930793c754 feat(general) maintenance stats for generate range checkpoint (#4924) 2025-10-28 16:48:28 -07:00
dependabot[bot] 47a43c7ce6 build(deps): bump google.golang.org/api (#4923) 
Bumps the common-golang-dependencies group with 1 update: [google.golang.org/api](https://github.com/googleapis/google-api-go-client).


Updates `google.golang.org/api` from 0.252.0 to 0.253.0
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.252.0...v0.253.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-version: 0.253.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: common-golang-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2025-10-28 00:15:54 +00:00
dependabot[bot] 20aae8b600 build(deps): bump github.com/hanwen/go-fuse/v2 from 2.8.0 to 2.9.0 (#4922) 
Bumps [github.com/hanwen/go-fuse/v2](https://github.com/hanwen/go-fuse) from 2.8.0 to 2.9.0.
- [Commits](https://github.com/hanwen/go-fuse/compare/v2.8.0...v2.9.0)

---
updated-dependencies:
- dependency-name: github.com/hanwen/go-fuse/v2
  dependency-version: 2.9.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2025-10-27 23:46:47 +00:00
dependabot[bot] 6743bfb8c7 build(deps): bump github.com/pkg/sftp from 1.13.9 to 1.13.10 (#4921) 
Bumps [github.com/pkg/sftp](https://github.com/pkg/sftp) from 1.13.9 to 1.13.10.
- [Release notes](https://github.com/pkg/sftp/releases)
- [Commits](https://github.com/pkg/sftp/compare/v1.13.9...v1.13.10)

---
updated-dependencies:
- dependency-name: github.com/pkg/sftp
  dependency-version: 1.13.10
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2025-10-27 23:39:12 +00:00
Julio Lopez c3dc18f7e5 fix typo (#4920) 2025-10-27 16:12:20 -07:00
lyndon-li 727fea0801 feat(general): add maintenance phase stats - CleanupSupersededIndexes (#4916) 2025-10-27 12:49:38 -07:00
Julio Lopez 25b1492d65 fix(general): field and variable names (#4911) 
Fixes field names in CleanupMarkersStats persistent struct.
Also renames variables for consistency.

- Followup fix for: #4900 add stats to maintenance run - CleanupMarkers 
- Ref: #4848
 2025-10-24 16:09:36 -07:00
lyndon-li e3e4e09bc8 feat(general): add stats to maintenance run - CleanupMarkers (#4900) 2025-10-24 13:49:40 -07:00
Julio Lopez 22f0919644 refactor(general): simplify construction of error messages (#4913) 2025-10-24 13:45:42 -07:00
Julio Lopez d8519ab9fa refactor(test): run cases as sub-tests (#4912) 2025-10-24 12:40:45 -07:00
Julio Lopez 5bc467e2ca refactor(general): omitzero JSON tag (#4907) 
* remove omitempty from object id fields
* remove omitempty from storage stats fields
* use omitzero for other structs

* remove `omitempty` from `UpgradeLockIntent` fields

The `omitempty` JSON tag is ineffective for fields of 
type `Time` and `Duration`.
Also, this struct is only used during the "upgrade"
protocol, and it is OK to explicitly serialize 0 values,
as it was done before.
 2025-10-23 22:59:52 -07:00
Julio Lopez bab11c181d nit: remove unused constant (#4906) 2025-10-23 22:19:56 -07:00
Julio Lopez 4cbd7026a6 refactor(general): leverage maps.Clone (#4905) 2025-10-23 19:45:29 -07:00
Julio Lopez c5cd3c5651 refactor(general): modernize bloop (#4904) 2025-10-23 17:49:20 -07:00
Julio Lopez 995e7fd893 refactor(general): modernize (#4903) 
Applies the modernize changes for the following categories:

- mapsloop
- stringsseq
- stringscutprefix
- sortslice
 2025-10-23 17:11:38 -07:00
Kopia Builder [bot] 6472bf386d feat(ui): upgraded htmlui to the latest version (#4902) 2025-10-23 17:49:47 +00:00
Kopia Builder [bot] 4683d37874 feat(ui): upgraded htmlui to the latest version (#4899) 2025-10-23 05:29:14 +00:00
Julio Lopez af62386ecd chore(general): modernize min/max (#4898) 2025-10-22 22:22:54 -07:00
Samuel Maynard 11765d696c feat(general): reduce number of open file descriptors for indexes on Unix (#4866) 
Close file after mmap on Unix to reduce open file descriptors.

On Unix-like platforms, close the file descriptor immediately after a successful
mmap.Map of .sndx index cache files. This keeps the mapping valid until Unmap
(per POSIX semantics) and significantly reduces steady-state FD usage when many
indexes are open, helping avoid EMFILE ("too many open files").

- Split mmapOpenWithRetry into platform-specific implementations:
  - committed_content_index_disk_cache_unix.go (!windows):
    - Map RDONLY, close FD immediately.
    - Return closer that only unmaps.
  - committed_content_index_disk_cache_windows.go (windows):
    - Keep FD open until Unmap.
    - Return closer that unmaps and closes FD.
- Remove old mmapOpenWithRetry and mmap import from
  repo/content/committed_content_index_disk_cache.go.
- Add Linux-only unit test verifying FD count does not grow proportionally:
  repo/content/committed_content_index_fd_linux_test.go
  - Creates N small indexes, opens them all, checks /proc/self/fd delta stays low.

Notes:
- Behavior unchanged on Windows due to OS semantics.
- Mapping failures close the FD to avoid leaks.
- Unlink semantics remain correct; mappings stay valid until Unmap.

* remove windows only retry logic under unix
 2025-10-22 21:16:35 -07:00
Julio Lopez b6da922e3a fix(general): address linter warnings (#4897) 2025-10-22 19:01:10 -07:00
lyndon-li 657db9706b feat(general): interfaces and structures for maintenance stats (#4890) 
Maintenance is critical for healthy of the repository.

On the other hand, Maintenance is complex, because
it runs multiple sub tasks each may generate different
results according to the maintenance policy.
The results may include deleting/combining/adding
data/metadata to the repository.

It is worthy to add more observability for these
tasks for below reasons:

It is helpful for troubleshooting. Any data change
to the repository is critical, the observability info
helps to understand what happened during the
maintenance and why that happened.

It is helpful for users to understand/predict the
repo's behavior. The repo data may be stored
in a public cloud for which costs are sensitive
to scale/duration of data stored. On the other
hand, repository has its own policy to manage
the data, so the data is not deleted until it is
safe enough according to the policy.

The observability info helps users to
understand how much data is in-use,
how much data is out of use and
when it is deleted
 2025-10-22 18:14:10 -07:00