Compare commits

...
34 Commits
Author SHA1 Message Date
OpenCloud Devops c8769532d4 🎉 Release 4.0.4 (#2338)
* 🎉 Release 4.0.4

* 🎉 Release 4.0.4

* 🎉 Release 4.0.4

* 🎉 Release 4.0.4

* 🎉 Release 4.0.4
2026-03-30 15:50:34 +02:00
Viktor Scharf 9995160624 [stable-4.0] bump version 4.0.4 (#2546) 2026-03-30 15:07:08 +02:00
Viktor Scharf b55462b185 [stable-4.0] chore: bump reva to v2.40.4 (#2545) 2026-03-30 12:23:10 +02:00
Ralf Haferkamp 9c4ac18e47 chore: bump github.com/nats-io/nats-server/v2
Fixes various CVEs (see https://github.com/nats-io/nats-server/releases/tag/v2.12.6
for details)
2026-03-30 12:22:23 +02:00
Ralf Haferkamp 8cf768bf80 chore: bump github.com/russellhaering/goxmldsig
CVE-2026-33487
2026-03-30 12:22:23 +02:00
Viktor Scharf 336acffb16 add final step for branch protection (#2501) (#2532)
* add final step for branch protection

* add step for ready release
2026-03-30 10:35:53 +02:00
Andre Duffeck 09d95eb415 Merge pull request #2527 from aduffeck/backport-bleve-batch-fix
Backport bleve batch fix
2026-03-26 08:52:39 +01:00
Viktor Scharf 8f7ad7f328 change php image 2026-03-26 07:42:11 +01:00
André Duffeck 228adbe177 Reduce default batch size to prevent memory issues with large documents 2026-03-25 18:32:57 +01:00
André Duffeck de1120210e Commit batches when the limit is reached while iterating over children 2026-03-25 18:32:54 +01:00
Florian Schade ffea35ebf5 Merge pull request #2407 from dragonchaser/backport-backchannel-logout
Backport backchannel logout
2026-03-02 13:19:36 +01:00
Florian Schade f8bed839ce fix: send the backchannel logout event only if a session exists 2026-03-02 10:25:55 +01:00
Christian Richter 8a667ff1c7 readd missing import
Signed-off-by: Christian Richter <c.richter@opencloud.eu>
2026-02-26 09:50:20 +01:00
Ralf Haferkamp 2a2d925ee4 fix(idp): Remove kpop dependency
The built package (https://download.kopano.io/community/kapp:/kpop-2.7.2.tgz)
seems to be no longer available and upstream lico already switched away
from it quite a while ago.

Fixes: #2364
2026-02-26 09:27:53 +01:00
Christian Richter 9cf02e5c7e fix: simplify subject.session key parsing
Signed-off-by: Christian Richter <c.richter@opencloud.eu>
2026-02-26 09:11:25 +01:00
Florian Schade 44fc25dbf6 refactor: make the logout mode private 2026-02-26 09:08:24 +01:00
Florian Schade 8d99cf3f8b fix: use base64 record keys to prevent separator clashes with subjects or sessionIds that contain a dot 2026-02-26 09:08:17 +01:00
Florian Schade a5f5009f9d chore: change naming 2026-02-26 09:08:11 +01:00
Florian Schade 6af2c44f7f test: add more backchannellogout tests 2026-02-26 09:08:04 +01:00
Florian Schade f7a86d681a chore: cleanup backchannel logout pr for review 2026-02-26 09:07:57 +01:00
Florian Schade a9660e3e8c enhancement: document idp side-effects 2026-02-26 09:07:50 +01:00
Florian Schade 66d220ff44 enhancement: finalize backchannel logout 2026-02-26 09:07:29 +01:00
2e36859816 refactor deletion
Co-authored-by: Jörn Dreyer <j.dreyer@opencloud.eu>
Co-authored-by: Michael Barz <m.barz@opencloud.eu>
Signed-off-by: Christian Richter <c.richter@opencloud.eu>
2026-02-26 08:53:21 +01:00
Christian Richter 06708d94a9 create mapping in cache for subject => sessionid
Signed-off-by: Christian Richter <c.richter@opencloud.eu>
2026-02-26 08:53:12 +01:00
Christian Richter 5aefcb9544 add mapping to backchannel logout for subject => sessionid
Signed-off-by: Christian Richter <c.richter@opencloud.eu>
2026-02-26 08:53:05 +01:00
Christian RichterandMichael Barz ca559bc3fb add additional validation to logout token
Signed-off-by: Christian Richter <c.richter@opencloud.eu>
Co-authored-by: Michael Barz <m.barz@opencloud.eu>
2026-02-26 08:52:59 +01:00
Sawjan Gurung 535a5d67e9 Merge pull request #2330 from opencloud-eu/port/ci-skip-prs
[full-ci][tests-only] ci: port ci skip prs
2026-02-13 09:57:56 +05:45
Sawjan Gurung 98b6c53270 ci: fix unwanted workflow skip in the cron pipelines (#2117)
* ci: run workflow if not found in the list

Signed-off-by: Saw-jan <saw.jan.grg3e@gmail.com>

* ci: suffix event in cache key to separate cron and push pipelines

Signed-off-by: Saw-jan <saw.jan.grg3e@gmail.com>

* ci: prefix test workflow with test

Signed-off-by: Saw-jan <saw.jan.grg3e@gmail.com>

* ci: create empty cache dir for unit tests

Signed-off-by: Saw-jan <saw.jan.grg3e@gmail.com>

---------

Signed-off-by: Saw-jan <saw.jan.grg3e@gmail.com>
2026-02-12 17:49:21 +05:45
Sawjan Gurung fe84d0dec4 [POC] ci: skip previously passed workflows on pipeline restart (#2099)
* ci: add pipeline info check scripts

Signed-off-by: Saw-jan <saw.jan.grg3e@gmail.com>

* ci: prefix test pipelines with test

Signed-off-by: Saw-jan <saw.jan.grg3e@gmail.com>

* ci: implement skip-on-pass for test workflows

Signed-off-by: Saw-jan <saw.jan.grg3e@gmail.com>

* ci: add cache purge workflow

Signed-off-by: Saw-jan <saw.jan.grg3e@gmail.com>

---------

Signed-off-by: Saw-jan <saw.jan.grg3e@gmail.com>
2026-02-12 17:49:07 +05:45
Sawjan Gurung 17606da390 ci: generate api, cli and e2e pipelines in a similar way (#2095)
* ci: refactor ci config

Signed-off-by: Saw-jan <saw.jan.grg3e@gmail.com>

* ci: refactor storage and configs

Signed-off-by: Saw-jan <saw.jan.grg3e@gmail.com>

* ci: fix format

Signed-off-by: Saw-jan <saw.jan.grg3e@gmail.com>

---------

Signed-off-by: Saw-jan <saw.jan.grg3e@gmail.com>
2026-02-12 17:46:59 +05:45
OpenCloud Devops 239ce6a794 🎉 Release 4.0.3 (#2281) 2026-02-05 17:11:52 +01:00
Andre Duffeck 4f36445048 Merge pull request #2275 from aduffeck/bump-reva-2.40.3
Bump reva 2.40.3
2026-02-05 16:14:26 +01:00
André Duffeck 312eacd46a Bump latest tag 2026-02-05 15:35:05 +01:00
André Duffeck 3c906e1dae Bump reva 2026-02-05 15:34:11 +01:00
292 changed files with 22632 additions and 65469 deletions

No files matched your search

+513 -336
View File
File diff suppressed because it is too large. Load diff
+25
View File
@@ -1,5 +1,30 @@
# Changelog
## [4.0.4](https://github.com/opencloud-eu/opencloud/releases/tag/v4.0.4) - 2026-03-30
### ❤️ Thanks to all contributors! ❤️
@ScharfViktor, @aduffeck, @dragonchaser, @rhafer, @saw-jan
### 🐛 Bug Fixes
- Backport bleve batch fix [[#2527](https://github.com/opencloud-eu/opencloud/pull/2527)]
- Backport backchannel logout [[#2407](https://github.com/opencloud-eu/opencloud/pull/2407)]
### 📦️ Dependencies
- [stable-4.0] Bump some go dependencies [[#2544](https://github.com/opencloud-eu/opencloud/pull/2544)]
## [4.0.3](https://github.com/opencloud-eu/opencloud/releases/tag/v4.0.3) - 2026-02-05
### ❤️ Thanks to all contributors! ❤️
@aduffeck
### 🐛 Bug Fixes
- Bump reva 2.40.3 [[#2275](https://github.com/opencloud-eu/opencloud/pull/2275)]
## [4.0.2](https://github.com/opencloud-eu/opencloud/releases/tag/v4.0.2) - 2026-02-05
### ❤️ Thanks to all contributors! ❤️
+2 -2
View File
@@ -1,8 +1,8 @@
{
"name": "opencloud-eu/opencloud",
"config" : {
"config": {
"platform": {
"php": "8.2"
"php": "8.4"
},
"vendor-dir": "./vendor-php",
"allow-plugins": {
+20 -19
View File
@@ -1,6 +1,6 @@
module github.com/opencloud-eu/opencloud
go 1.24.6
go 1.25.0
require (
dario.cat/mergo v1.0.2
@@ -54,8 +54,8 @@ require (
github.com/mitchellh/mapstructure v1.5.0
github.com/mna/pigeon v1.3.0
github.com/mohae/deepcopy v0.0.0-20170929034955-c48cc78d4826
github.com/nats-io/nats-server/v2 v2.12.2
github.com/nats-io/nats.go v1.47.0
github.com/nats-io/nats-server/v2 v2.12.6
github.com/nats-io/nats.go v1.49.0
github.com/oklog/run v1.2.0
github.com/olekukonko/tablewriter v1.1.1
github.com/onsi/ginkgo v1.16.5
@@ -64,7 +64,7 @@ require (
github.com/open-policy-agent/opa v1.10.1
github.com/opencloud-eu/icap-client v0.0.0-20250930132611-28a2afe62d89
github.com/opencloud-eu/libre-graph-api-go v1.0.8-0.20250724122329-41ba6b191e76
github.com/opencloud-eu/reva/v2 v2.40.2
github.com/opencloud-eu/reva/v2 v2.40.4
github.com/opensearch-project/opensearch-go/v4 v4.5.0
github.com/orcaman/concurrent-map v1.0.0
github.com/pkg/errors v0.9.1
@@ -101,14 +101,14 @@ require (
go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.38.0
go.opentelemetry.io/otel/sdk v1.38.0
go.opentelemetry.io/otel/trace v1.38.0
golang.org/x/crypto v0.45.0
golang.org/x/crypto v0.49.0
golang.org/x/exp v0.0.0-20250210185358-939b2ce775ac
golang.org/x/image v0.33.0
golang.org/x/net v0.47.0
golang.org/x/net v0.51.0
golang.org/x/oauth2 v0.33.0
golang.org/x/sync v0.18.0
golang.org/x/term v0.37.0
golang.org/x/text v0.31.0
golang.org/x/sync v0.20.0
golang.org/x/term v0.41.0
golang.org/x/text v0.35.0
google.golang.org/genproto/googleapis/api v0.0.0-20251022142026-3a174f9686a8
google.golang.org/grpc v1.77.0
google.golang.org/protobuf v1.36.10
@@ -134,7 +134,7 @@ require (
github.com/ajg/form v1.5.1 // indirect
github.com/alexedwards/argon2id v1.0.0 // indirect
github.com/amoghe/go-crypt v0.0.0-20220222110647-20eada5f5964 // indirect
github.com/antithesishq/antithesis-sdk-go v0.4.3-default-no-op // indirect
github.com/antithesishq/antithesis-sdk-go v0.6.0-default-no-op // indirect
github.com/armon/go-radix v1.0.0 // indirect
github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 // indirect
github.com/beorn7/perks v1.0.1 // indirect
@@ -240,7 +240,7 @@ require (
github.com/golang/snappy v0.0.4 // indirect
github.com/gomodule/redigo v1.9.3 // indirect
github.com/google/go-querystring v1.1.0 // indirect
github.com/google/go-tpm v0.9.6 // indirect
github.com/google/go-tpm v0.9.8 // indirect
github.com/google/pprof v0.0.0-20250403155104-27863c87afa6 // indirect
github.com/google/renameio/v2 v2.0.1 // indirect
github.com/gookit/goutil v0.7.1 // indirect
@@ -249,6 +249,7 @@ require (
github.com/grpc-ecosystem/go-grpc-middleware v1.4.0 // indirect
github.com/hashicorp/go-hclog v1.6.3 // indirect
github.com/hashicorp/go-plugin v1.7.0 // indirect
github.com/hashicorp/golang-lru/v2 v2.0.7 // indirect
github.com/hashicorp/yamux v0.1.2 // indirect
github.com/huandu/xstrings v1.5.0 // indirect
github.com/iancoleman/strcase v0.3.0 // indirect
@@ -259,7 +260,7 @@ require (
github.com/json-iterator/go v1.1.12 // indirect
github.com/juliangruber/go-intersect v1.1.0 // indirect
github.com/kevinburke/ssh_config v1.2.0 // indirect
github.com/klauspost/compress v1.18.1 // indirect
github.com/klauspost/compress v1.18.4 // indirect
github.com/klauspost/cpuid/v2 v2.2.11 // indirect
github.com/klauspost/crc32 v1.3.0 // indirect
github.com/kovidgoyal/go-parallel v1.1.1 // indirect
@@ -305,8 +306,8 @@ require (
github.com/morikuni/aec v1.0.0 // indirect
github.com/mschoch/smat v0.2.0 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/nats-io/jwt/v2 v2.8.0 // indirect
github.com/nats-io/nkeys v0.4.11 // indirect
github.com/nats-io/jwt/v2 v2.8.1 // indirect
github.com/nats-io/nkeys v0.4.15 // indirect
github.com/nats-io/nuid v1.0.1 // indirect
github.com/nxadm/tail v1.4.8 // indirect
github.com/olekukonko/cat v0.0.0-20250911104152-50322a0618f6 // indirect
@@ -332,7 +333,7 @@ require (
github.com/prometheus/statsd_exporter v0.22.8 // indirect
github.com/rcrowley/go-metrics v0.0.0-20250401214520-65e299d6c5c9 // indirect
github.com/rs/xid v1.6.0 // indirect
github.com/russellhaering/goxmldsig v1.5.0 // indirect
github.com/russellhaering/goxmldsig v1.6.0 // indirect
github.com/russross/blackfriday/v2 v2.1.0 // indirect
github.com/rwcarlsen/goexif v0.0.0-20190401172101-9e8deecbddbd // indirect
github.com/samber/lo v1.51.0 // indirect
@@ -384,10 +385,10 @@ require (
go.uber.org/zap v1.27.0 // indirect
go.yaml.in/yaml/v2 v2.4.3 // indirect
go.yaml.in/yaml/v3 v3.0.4 // indirect
golang.org/x/mod v0.29.0 // indirect
golang.org/x/sys v0.38.0 // indirect
golang.org/x/time v0.14.0 // indirect
golang.org/x/tools v0.38.0 // indirect
golang.org/x/mod v0.33.0 // indirect
golang.org/x/sys v0.42.0 // indirect
golang.org/x/time v0.15.0 // indirect
golang.org/x/tools v0.42.0 // indirect
google.golang.org/genproto v0.0.0-20250303144028-a0af3efb3deb // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20251022142026-3a174f9686a8 // indirect
gopkg.in/cenkalti/backoff.v1 v1.1.0 // indirect
+38 -36
View File
@@ -119,8 +119,8 @@ github.com/andreyvit/diff v0.0.0-20170406064948-c7f18ee00883 h1:bvNMNQO63//z+xNg
github.com/andreyvit/diff v0.0.0-20170406064948-c7f18ee00883/go.mod h1:rCTlJbsFo29Kk6CurOXKm700vrz8f0KW0JNfpkRJY/8=
github.com/anmitsu/go-shlex v0.0.0-20200514113438-38f4b401e2be h1:9AeTilPcZAjCFIImctFaOjnTIavg87rW78vTPkQqLI8=
github.com/anmitsu/go-shlex v0.0.0-20200514113438-38f4b401e2be/go.mod h1:ySMOLuWl6zY27l47sB3qLNK6tF2fkHG55UZxx8oIVo4=
github.com/antithesishq/antithesis-sdk-go v0.4.3-default-no-op h1:+OSa/t11TFhqfrX0EOSqQBDJ0YlpmK0rDSiB19dg9M0=
github.com/antithesishq/antithesis-sdk-go v0.4.3-default-no-op/go.mod h1:IUpT2DPAKh6i/YhSbt6Gl3v2yvUZjmKncl7U91fup7E=
github.com/antithesishq/antithesis-sdk-go v0.6.0-default-no-op h1:kpBdlEPbRvff0mDD1gk7o9BhI16b9p5yYAXRlidpqJE=
github.com/antithesishq/antithesis-sdk-go v0.6.0-default-no-op/go.mod h1:IUpT2DPAKh6i/YhSbt6Gl3v2yvUZjmKncl7U91fup7E=
github.com/apache/thrift v0.12.0/go.mod h1:cp2SuWMxlEZw2r+iP2GNCdIi4C1qmUzdZFSVb+bacwQ=
github.com/arbovm/levenshtein v0.0.0-20160628152529-48b4e1c0c4d0 h1:jfIu9sQUG6Ig+0+Ap1h4unLjW6YQJpKZVmUzxsD4E/Q=
github.com/arbovm/levenshtein v0.0.0-20160628152529-48b4e1c0c4d0/go.mod h1:t2tdKJDJF9BV14lnkjHmOQgcvEKgtqs5a1N3LNdJhGE=
@@ -572,8 +572,8 @@ github.com/google/go-querystring v1.1.0 h1:AnCroh3fv4ZBgVIf1Iwtovgjaw/GiKJo8M8yD
github.com/google/go-querystring v1.1.0/go.mod h1:Kcdr2DB4koayq7X8pmAG4sNG59So17icRSOU623lUBU=
github.com/google/go-tika v0.3.1 h1:l+jr10hDhZjcgxFRfcQChRLo1bPXQeLFluMyvDhXTTA=
github.com/google/go-tika v0.3.1/go.mod h1:DJh5N8qxXIl85QkqmXknd+PeeRkUOTbvwyYf7ieDz6c=
github.com/google/go-tpm v0.9.6 h1:Ku42PT4LmjDu1H5C5ISWLlpI1mj+Zq7sPGKoRw2XROA=
github.com/google/go-tpm v0.9.6/go.mod h1:h9jEsEECg7gtLis0upRBQU+GhYVH6jMjrFxI8u6bVUY=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
github.com/google/go-tpm v0.9.8/go.mod h1:h9jEsEECg7gtLis0upRBQU+GhYVH6jMjrFxI8u6bVUY=
github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
github.com/google/martian v2.1.0+incompatible/go.mod h1:9I4somxYTbIHy5NJKHRl3wXiIaQGbYVAs8BPL6v8lEs=
github.com/google/martian/v3 v3.0.0/go.mod h1:y5Zk1BBys9G+gd6Jrk0W3cC1+ELVxBWuIGO+w/tUAp0=
@@ -648,6 +648,8 @@ github.com/hashicorp/go-uuid v1.0.3/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/b
github.com/hashicorp/go.net v0.0.1/go.mod h1:hjKkEWcCURg++eb33jQU7oqQcI9XDCnUzHA0oac0k90=
github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ=
github.com/hashicorp/logutils v1.0.0/go.mod h1:QIAnNjmIWmVIIkWDTG1z5v++HQmx9WQRO+LraFDTW64=
github.com/hashicorp/mdns v1.0.0/go.mod h1:tL+uN++7HEJ6SQLQ2/p+z2pH24WQKWjBPkE0mNTz8vQ=
@@ -725,8 +727,8 @@ github.com/kevinburke/ssh_config v1.2.0/go.mod h1:CT57kijsi8u/K/BOFA39wgDQJ9CxiF
github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q=
github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
github.com/klauspost/compress v1.18.1 h1:bcSGx7UbpBqMChDtsF28Lw6v/G94LPrrbMbdC3JH2co=
github.com/klauspost/compress v1.18.1/go.mod h1:ZQFFVG+MdnR0P+l6wpXgIL4NTtwiKIdBnrBd8Nrxr+0=
github.com/klauspost/compress v1.18.4 h1:RPhnKRAQ4Fh8zU2FY/6ZFDwTVTxgJ/EMydqSTzE9a2c=
github.com/klauspost/compress v1.18.4/go.mod h1:R0h/fSBs8DE4ENlcrlib3PsXS61voFxhIs2DeRhCvJ4=
github.com/klauspost/cpuid/v2 v2.0.1/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
github.com/klauspost/cpuid/v2 v2.2.11 h1:0OwqZRYI2rFrjS4kvkDnqJkKHdHaRnCm68/DY4OxRzU=
github.com/klauspost/cpuid/v2 v2.2.11/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0=
@@ -908,14 +910,14 @@ github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8m
github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
github.com/namedotcom/go v0.0.0-20180403034216-08470befbe04/go.mod h1:5sN+Lt1CaY4wsPvgQH/jsuJi4XO2ssZbdsIizr4CVC8=
github.com/nats-io/jwt/v2 v2.8.0 h1:K7uzyz50+yGZDO5o772eRE7atlcSEENpL7P+b74JV1g=
github.com/nats-io/jwt/v2 v2.8.0/go.mod h1:me11pOkwObtcBNR8AiMrUbtVOUGkqYjMQZ6jnSdVUIA=
github.com/nats-io/nats-server/v2 v2.12.2 h1:4TEQd0Y4zvcW0IsVxjlXnRso1hBkQl3TS0BI+SxgPhE=
github.com/nats-io/nats-server/v2 v2.12.2/go.mod h1:j1AAttYeu7WnvD8HLJ+WWKNMSyxsqmZ160pNtCQRMyE=
github.com/nats-io/nats.go v1.47.0 h1:YQdADw6J/UfGUd2Oy6tn4Hq6YHxCaJrVKayxxFqYrgM=
github.com/nats-io/nats.go v1.47.0/go.mod h1:iRWIPokVIFbVijxuMQq4y9ttaBTMe0SFdlZfMDd+33g=
github.com/nats-io/nkeys v0.4.11 h1:q44qGV008kYd9W1b1nEBkNzvnWxtRSQ7A8BoqRrcfa0=
github.com/nats-io/nkeys v0.4.11/go.mod h1:szDimtgmfOi9n25JpfIdGw12tZFYXqhGxjhVxsatHVE=
github.com/nats-io/jwt/v2 v2.8.1 h1:V0xpGuD/N8Mi+fQNDynXohVvp7ZztevW5io8CUWlPmU=
github.com/nats-io/jwt/v2 v2.8.1/go.mod h1:nWnOEEiVMiKHQpnAy4eXlizVEtSfzacZ1Q43LIRavZg=
github.com/nats-io/nats-server/v2 v2.12.6 h1:Egbx9Vl7Ch8wTtpXPGqbehkZ+IncKqShUxvrt1+Enc8=
github.com/nats-io/nats-server/v2 v2.12.6/go.mod h1:4HPlrvtmSO3yd7KcElDNMx9kv5EBJBnJJzQPptXlheo=
github.com/nats-io/nats.go v1.49.0 h1:yh/WvY59gXqYpgl33ZI+XoVPKyut/IcEaqtsiuTJpoE=
github.com/nats-io/nats.go v1.49.0/go.mod h1:fDCn3mN5cY8HooHwE2ukiLb4p4G4ImmzvXyJt+tGwdw=
github.com/nats-io/nkeys v0.4.15 h1:JACV5jRVO9V856KOapQ7x+EY8Jo3qw1vJt/9Jpwzkk4=
github.com/nats-io/nkeys v0.4.15/go.mod h1:CpMchTXC9fxA5zrMo4KpySxNjiDVvr8ANOSZdiNfUrs=
github.com/nats-io/nuid v1.0.1 h1:5iA8DT8V7q8WK2EScv2padNa/rTESc1KdnPw4TC2paw=
github.com/nats-io/nuid v1.0.1/go.mod h1:19wcPz3Ph3q0Jbyiqsd0kePYG7A95tJPxeL+1OSON2c=
github.com/nbio/st v0.0.0-20140626010706-e9e8d9816f32/go.mod h1:9wM+0iRr9ahx58uYLpLIr5fm8diHn0JbqRycJi6w0Ms=
@@ -963,8 +965,8 @@ github.com/opencloud-eu/inotifywaitgo v0.0.0-20251111171128-a390bae3c5e9 h1:dIft
github.com/opencloud-eu/inotifywaitgo v0.0.0-20251111171128-a390bae3c5e9/go.mod h1:JWyDC6H+5oZRdUJUgKuaye+8Ph5hEs6HVzVoPKzWSGI=
github.com/opencloud-eu/libre-graph-api-go v1.0.8-0.20250724122329-41ba6b191e76 h1:vD/EdfDUrv4omSFjrinT8Mvf+8D7f9g4vgQ2oiDrVUI=
github.com/opencloud-eu/libre-graph-api-go v1.0.8-0.20250724122329-41ba6b191e76/go.mod h1:pzatilMEHZFT3qV7C/X3MqOa3NlRQuYhlRhZTL+hN6Q=
github.com/opencloud-eu/reva/v2 v2.40.2 h1:rsgOkwA6MwReuwu7RvhWh675PJEQhLWrTOew5OSpR/E=
github.com/opencloud-eu/reva/v2 v2.40.2/go.mod h1:DGH08n2mvtsQLkt8o15FV6m51FwSJJGhjR8Ty+iIJww=
github.com/opencloud-eu/reva/v2 v2.40.4 h1:KTeFLHSwRp2bnK9J1dtaAGwPCreEiijYIXSyQ1A5tbo=
github.com/opencloud-eu/reva/v2 v2.40.4/go.mod h1:NN88hQAbjCVq011D2zhr0IOZ9Xclz3HS+Wf961YHwbg=
github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8Oi/yOhh5U=
github.com/opencontainers/go-digest v1.0.0/go.mod h1:0JzlMkj0TRzQZfJkVvzbP0HBR3IKzErnv2BNG4W4MAM=
github.com/opencontainers/image-spec v1.1.1 h1:y0fUlFfIZhPF1W537XOLg0/fcx6zcHCJwooC2xJA040=
@@ -1084,8 +1086,8 @@ github.com/rs/xid v1.6.0 h1:fV591PaemRlL6JfRxGDEPl69wICngIQ3shQtzfy2gxU=
github.com/rs/xid v1.6.0/go.mod h1:7XoLgs4eV+QndskICGsho+ADou8ySMSjJKDIan90Nz0=
github.com/rs/zerolog v1.34.0 h1:k43nTLIwcTVQAncfCw4KZ2VY6ukYoZaBPNOE8txlOeY=
github.com/rs/zerolog v1.34.0/go.mod h1:bJsvje4Z08ROH4Nhs5iH600c3IkWhwp44iRc54W6wYQ=
github.com/russellhaering/goxmldsig v1.5.0 h1:AU2UkkYIUOTyZRbe08XMThaOCelArgvNfYapcmSjBNw=
github.com/russellhaering/goxmldsig v1.5.0/go.mod h1:x98CjQNFJcWfMxeOrMnMKg70lvDP6tE0nTaeUnjXDmk=
github.com/russellhaering/goxmldsig v1.6.0 h1:8fdWXEPh2k/NZNQBPFNoVfS3JmzS4ZprY/sAOpKQLks=
github.com/russellhaering/goxmldsig v1.6.0/go.mod h1:TrnaquDcYxWXfJrOjeMBTX4mLBeYAqaHEyUeWPxZlBM=
github.com/russross/blackfriday/v2 v2.0.1/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/russross/blackfriday/v2 v2.1.0 h1:JIOH55/0cWyOuilr9/qlrm0BSXldqnqwMsf35Ld67mk=
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
@@ -1359,8 +1361,8 @@ golang.org/x/crypto v0.14.0/go.mod h1:MVFd36DqK4CsrnJYDkBA3VC4m2GkXAM0PvzMCn4JQf
golang.org/x/crypto v0.19.0/go.mod h1:Iy9bg/ha4yyC70EfRS8jz+B6ybOBKMaSxLj6P6oBDfU=
golang.org/x/crypto v0.21.0/go.mod h1:0BP7YvVV9gBbVKyeTG0Gyn+gZm94bibOW5BjDEYAOMs=
golang.org/x/crypto v0.23.0/go.mod h1:CKFgDieR+mRhux2Lsu27y0fO304Db0wZe70UKqHu0v8=
golang.org/x/crypto v0.45.0 h1:jMBrvKuj23MTlT0bQEOBcAE0mjg8mK9RXFhRH6nyF3Q=
golang.org/x/crypto v0.45.0/go.mod h1:XTGrrkGJve7CYK7J8PEww4aY7gM3qMCElcJQ8n8JdX4=
golang.org/x/crypto v0.49.0 h1:+Ng2ULVvLHnJ/ZFEq4KdcDd/cfjrrjjNSXNzxg0Y4U4=
golang.org/x/crypto v0.49.0/go.mod h1:ErX4dUh2UM+CFYiXZRTcMpEcN8b/1gxEuv3nODoYtCA=
golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8=
@@ -1402,8 +1404,8 @@ golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/mod v0.12.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/mod v0.15.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
golang.org/x/mod v0.17.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
golang.org/x/mod v0.29.0 h1:HV8lRxZC4l2cr3Zq1LvtOsi/ThTgWnUk/y64QSs8GwA=
golang.org/x/mod v0.29.0/go.mod h1:NyhrlYXJ2H4eJiRy/WDBO6HMqZQ6q9nk4JzS3NuCK+w=
golang.org/x/mod v0.33.0 h1:tHFzIWbBifEmbwtGz65eaWyGiGZatSrT9prnU8DbVL8=
golang.org/x/mod v0.33.0/go.mod h1:swjeQEj+6r7fODbD2cqrnje9PnziFuw4bmLbBZFrQ5w=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
@@ -1457,8 +1459,8 @@ golang.org/x/net v0.15.0/go.mod h1:idbUs1IY1+zTqbi8yxTbhexhEEk5ur9LInksu6HrEpk=
golang.org/x/net v0.21.0/go.mod h1:bIjVDfnllIU7BJ2DNgfnXvpSvtn8VRwhlsaeUTyUS44=
golang.org/x/net v0.23.0/go.mod h1:JKghWKKOSdJwpW2GEx0Ja7fmaKnMsbu+MWVZTokSYmg=
golang.org/x/net v0.25.0/go.mod h1:JkAGAh7GEvH74S6FOH42FLoXpXbE/aqXSrIQjXgsiwM=
golang.org/x/net v0.47.0 h1:Mx+4dIFzqraBXUugkia1OOvlD6LemFo1ALMHjrXDOhY=
golang.org/x/net v0.47.0/go.mod h1:/jNxtkgq5yWUGYkaZGqo27cfGZ1c5Nen03aYrrKpVRU=
golang.org/x/net v0.51.0 h1:94R/GTO7mt3/4wIKpcR5gkGmRLOuE/2hNGeWq/GBIFo=
golang.org/x/net v0.51.0/go.mod h1:aamm+2QF5ogm02fjy5Bb7CQ0WMt1/WVM7FtyaTLlA9Y=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
@@ -1485,8 +1487,8 @@ golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.3.0/go.mod h1:FU7BRWz2tNW+3quACPkgCx/L+uEAv1htQ0V83Z9Rj+Y=
golang.org/x/sync v0.6.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sync v0.7.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sync v0.18.0 h1:kr88TuHDroi+UVf+0hZnirlk8o8T+4MrK6mr60WkH/I=
golang.org/x/sync v0.18.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
golang.org/x/sync v0.20.0 h1:e0PTpb7pjO8GAtTs2dQ6jYa5BWYlMuX047Dco/pItO4=
golang.org/x/sync v0.20.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/sys v0.0.0-20180622082034-63fc586f45fe/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20180823144017-11551d06cbcc/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
@@ -1570,8 +1572,8 @@ golang.org/x/sys v0.17.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.18.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.20.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.21.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.38.0 h1:3yZWxaJjBmCWXqhN1qh02AkOnCQ1poK6oF+a7xWL6Gc=
golang.org/x/sys v0.38.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
golang.org/x/sys v0.42.0 h1:omrd2nAlyT5ESRdCLYdm3+fMfNFE/+Rf4bDIQImRJeo=
golang.org/x/sys v0.42.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/telemetry v0.0.0-20240228155512-f48c80bd79b2/go.mod h1:TeRTkGYfJXctD9OcfyVLyj2J3IxLnKwHJR8f4D8a3YE=
golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
@@ -1583,8 +1585,8 @@ golang.org/x/term v0.13.0/go.mod h1:LTmsnFJwVN6bCy1rVCoS+qHT1HhALEFxKncY3WNNh4U=
golang.org/x/term v0.17.0/go.mod h1:lLRBjIVuehSbZlaOtGMbcMncT+aqLLLmKrsjNrUguwk=
golang.org/x/term v0.18.0/go.mod h1:ILwASektA3OnRv7amZ1xhE/KTR+u50pbXfZ03+6Nx58=
golang.org/x/term v0.20.0/go.mod h1:8UkIAJTvZgivsXaD6/pH6U9ecQzZ45awqEOzuCvwpFY=
golang.org/x/term v0.37.0 h1:8EGAD0qCmHYZg6J17DvsMy9/wJ7/D/4pV/wfnld5lTU=
golang.org/x/term v0.37.0/go.mod h1:5pB4lxRNYYVZuTLmy8oR2BH8dflOR+IbTYFD8fi3254=
golang.org/x/term v0.41.0 h1:QCgPso/Q3RTJx2Th4bDLqML4W6iJiaXFq2/ftQF13YU=
golang.org/x/term v0.41.0/go.mod h1:3pfBgksrReYfZ5lvYM0kSO0LIkAl4Yl2bXOkKP7Ec2A=
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
@@ -1599,16 +1601,16 @@ golang.org/x/text v0.13.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE=
golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
golang.org/x/text v0.15.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
golang.org/x/text v0.16.0/go.mod h1:GhwF1Be+LQoKShO3cGOHzqOgRrGaYc9AvblQOmPVHnI=
golang.org/x/text v0.31.0 h1:aC8ghyu4JhP8VojJ2lEHBnochRno1sgL6nEi9WGFGMM=
golang.org/x/text v0.31.0/go.mod h1:tKRAlv61yKIjGGHX/4tP1LTbc13YSec1pxVEWXzfoeM=
golang.org/x/text v0.35.0 h1:JOVx6vVDFokkpaq1AEptVzLTpDe9KGpj5tR4/X+ybL8=
golang.org/x/text v0.35.0/go.mod h1:khi/HExzZJ2pGnjenulevKNX1W67CUy0AsXcNubPGCA=
golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20200630173020-3af7569d3a1e/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20201208040808-7e3f01d25324/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20210220033141-f8bda1e9f3ba/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.14.0 h1:MRx4UaLrDotUKUdCIqzPC48t1Y9hANFKIRpNx+Te8PI=
golang.org/x/time v0.14.0/go.mod h1:eL/Oa2bBBK0TkX57Fyni+NgnyQQN4LitPmob2Hjnqw4=
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
golang.org/x/tools v0.0.0-20180221164845-07fd8470d635/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20180828015842-6cd1fcedba52/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
@@ -1663,8 +1665,8 @@ golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc
golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
golang.org/x/tools v0.13.0/go.mod h1:HvlwmtVNQAhOuCjW7xxvovg8wbNq7LwfXh/k7wXUl58=
golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d/go.mod h1:aiJjzUbINMkxbQROHiO6hDPo2LHcIPhhQsa9DLh0yGk=
golang.org/x/tools v0.38.0 h1:Hx2Xv8hISq8Lm16jvBZ2VQf+RLmbd7wVUsALibYI/IQ=
golang.org/x/tools v0.38.0/go.mod h1:yEsQ/d/YK8cjh0L6rZlY8tgtlKiBNTL14pGDJPJpYQs=
golang.org/x/tools v0.42.0 h1:uNgphsn75Tdz5Ji2q36v/nsFSfR/9BRFvqhGBaJGd5k=
golang.org/x/tools v0.42.0/go.mod h1:Ma6lCIwGZvHK6XtgbswSoWroEkhugApmsXyrUmBhfr0=
golang.org/x/tools/godoc v0.1.0-deprecated h1:o+aZ1BOj6Hsx/GBdJO/s815sqftjSnrZZwyYTHODvtk=
golang.org/x/tools/godoc v0.1.0-deprecated/go.mod h1:qM63CriJ961IHWmnWa9CjZnBndniPt4a3CK0PVB9bIg=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
+1 -1
View File
@@ -34,7 +34,7 @@ var (
// LatestTag is the latest released version plus the dev meta version.
// Will be overwritten by the release pipeline
// Needs a manual change for every tagged release
LatestTag = "4.0.1+dev"
LatestTag = "4.0.4+dev"
// Date indicates the build date.
// This has been removed, it looks like you can only replace static strings with recent go versions
+2 -8
View File
@@ -7,7 +7,7 @@
"analyze": "source-map-explorer 'build/static/js/*.js'",
"build": "node --openssl-legacy-provider scripts/build.js && rm -f build/service-worker.js",
"licenses": "NODE_PATH=./node_modules node ../scripts/js-license-ranger.js",
"licenses:check": "license-checker-rseidelsohn --summary --relativeLicensePath --onlyAllow 'Python-2.0;Apache*;Apache License, Version 2.0;Apache-2.0;Apache 2.0;Artistic-2.0;BSD;BSD-3-Clause;CC-BY-3.0;CC-BY-4.0;CC0-1.0;ISC;MIT;MPL-2.0;Public Domain;Unicode-TOU;Unlicense;WTFPL;ODC-By-1.0;BlueOak-1.0.0;OFL-1.1' --excludePackages 'identifier;kpop;unicoderegexp' --clarificationsFile license-checker-clarifications.json",
"licenses:check": "license-checker-rseidelsohn --summary --relativeLicensePath --onlyAllow 'Python-2.0;Apache*;Apache License, Version 2.0;Apache-2.0;Apache 2.0;Artistic-2.0;BSD;BSD-3-Clause;CC-BY-3.0;CC-BY-4.0;CC0-1.0;ISC;MIT;MPL-2.0;Public Domain;Unicode-TOU;Unlicense;WTFPL;ODC-By-1.0;BlueOak-1.0.0;OFL-1.1' --excludePackages 'identifier;unicoderegexp' --clarificationsFile license-checker-clarifications.json",
"licenses:csv": "license-checker-rseidelsohn --relativeLicensePath --csv --out ../../third-party-licenses/node/idp/third-party-licenses.csv",
"licenses:save": "license-checker-rseidelsohn --relativeLicensePath --out /dev/null --files ../../third-party-licenses/node/idp/third-party-licenses",
"lint": "eslint ./**/*.{tsx,ts,jsx,js}",
@@ -89,7 +89,6 @@
"i18next-browser-languagedetector": "^8.1.0",
"i18next-http-backend": "^3.0.2",
"i18next-resources-to-backend": "^1.2.1",
"kpop": "https://download.kopano.io/community/kapp:/kpop-2.7.2.tgz",
"query-string": "^9.2.0",
"react": "^17.0.2",
"react-app-polyfill": "^3.0.0",
@@ -154,10 +153,5 @@
"webpack-manifest-plugin": "5.0.0",
"workbox-webpack-plugin": "7.1.0"
},
"packageManager": "pnpm@9.15.4",
"pnpm": {
"overrides": {
"kpop>cldr": ""
}
}
"packageManager": "pnpm@9.15.4"
}
-151
View File
@@ -4,9 +4,6 @@ settings:
autoInstallPeers: true
excludeLinksFromLockfile: false
overrides:
kpop>cldr: ''
importers:
.:
@@ -65,9 +62,6 @@ importers:
i18next-resources-to-backend:
specifier: ^1.2.1
version: 1.2.1
kpop:
specifier: https://download.kopano.io/community/kapp:/kpop-2.7.2.tgz
version: https://download.kopano.io/community/kapp:/kpop-2.7.2.tgz(@gluejs/glue@0.3.0)(@material-ui/core@4.12.4(@types/react@17.0.80)(react-dom@17.0.2(react@17.0.2))(react@17.0.2))(@material-ui/icons@4.11.3(@material-ui/core@4.12.4(@types/react@17.0.80)(react-dom@17.0.2(react@17.0.2))(react@17.0.2))(@types/react@17.0.80)(react-dom@17.0.2(react@17.0.2))(react@17.0.2))(notistack@0.8.9(@material-ui/core@4.12.4(@types/react@17.0.80)(react-dom@17.0.2(react@17.0.2))(react@17.0.2))(react-dom@17.0.2(react@17.0.2))(react@17.0.2))(oidc-client@1.11.5)(react-dom@17.0.2(react@17.0.2))(react-intl@2.9.0(prop-types@15.8.1)(react@17.0.2))(react@17.0.2)
query-string:
specifier: ^9.2.0
version: 9.2.0
@@ -1516,9 +1510,6 @@ packages:
'@fontsource/roboto@5.2.5':
resolution: {integrity: sha512-70r2UZ0raqLn5W+sPeKhqlf8wGvUXFWlofaDlcbt/S3d06+17gXKr3VNqDODB0I1ASme3dGT5OJj9NABt7OTZQ==}
'@gluejs/glue@0.3.0':
resolution: {integrity: sha512-byvFoZCbZW+A3Pg8JUU+8FjoPuF5l1v7mDeLJQP/YSeEcEDiD/YdUKLBUapPrcuyxclrtS8+peX4cxkh6awwTw==}
'@gulpjs/to-absolute-glob@4.0.0':
resolution: {integrity: sha512-kjotm7XJrJ6v+7knhPaRgaT6q8F8K2jiafwYdNHLzmV0uGLuZY43FK6smNSHUPrhq5kX2slCUy+RGG/xGqmIKA==}
engines: {node: '>=10.13.0'}
@@ -2724,9 +2715,6 @@ packages:
core-js@3.40.0:
resolution: {integrity: sha512-7vsMc/Lty6AGnn7uFpYT56QesI5D2Y/UkgKounk87OP9Z2H9Z8kj6jzcSGAxFmUtDOS0ntK6lbQz+Nsa0Jj6mQ==}
core-js@3.43.0:
resolution: {integrity: sha512-N6wEbTTZSYOY2rYAn85CuvWWkCK6QweMn7/4Nr3w+gDBeBhk/x4EJeY6FPo4QzDoJZxVTv8U7CMvgWk6pOHHqA==}
core-util-is@1.0.3:
resolution: {integrity: sha512-ZQBvi1DcpJ4GDqanjucZ2Hj3wEO5pZDS89BWbkcrvdxksJorwUDDZamX9ldFkp9aw2lmBDLgkObEA4DWNJ9FYQ==}
@@ -2738,11 +2726,6 @@ packages:
resolution: {integrity: sha512-AdmX6xUzdNASswsFtmwSt7Vj8po9IuqXm0UXz7QKPuEUmPB4XyjGfaAr2PSuELMwkRMVH1EpIkX5bTZGRB3eCA==}
engines: {node: '>=10'}
crc32@0.2.2:
resolution: {integrity: sha512-PFZEGbDUeoNbL2GHIEpJRQGheXReDody/9axKTxhXtQqIL443wnNigtVZO9iuCIMPApKZRv7k2xr8euXHqNxQQ==}
engines: {node: '>= 0.4.0'}
hasBin: true
cross-fetch@4.0.0:
resolution: {integrity: sha512-e4a5N8lVvuLgAWgnCrLr2PP0YyDOTHa9H/Rj54dirp61qXnNq46m82bRhNqIA5VccJtWBvPTFRV3TtvHUKPB1g==}
@@ -2754,9 +2737,6 @@ packages:
resolution: {integrity: sha512-uV2QOWP2nWzsy2aMp8aRibhi9dlzF5Hgh5SHaB9OiTGEyDTiJJyx0uy51QXdyWbtAHNua4XJzUKca3OzKUd3vA==}
engines: {node: '>= 8'}
crypto-js@4.2.0:
resolution: {integrity: sha512-KALDyEYgpY+Rlob/iriUtjV6d5Eq+Y191A5g4UqLAi8CyGP9N1+FdVbkc1SxKc2r4YAYqG8JzO2KGL+AizD70Q==}
crypto-random-string@2.0.0:
resolution: {integrity: sha512-v1plID3y9r/lPhviJ1wrXpLeyUIGAZ2SHNYTEapm7/8A9nLPoyvVp3RK/EPFqn5kEznyWgYZNsRtYYIWbuG8KA==}
engines: {node: '>=8'}
@@ -3718,9 +3698,6 @@ packages:
resolution: {integrity: sha512-r0EI+HBMcXadMrugk0GCQ+6BQV39PiWAZVfq7oIckeGiN7sjRGyQxPdft3nQekFTCQbYxLBH+/axZMeH8UX6+w==}
engines: {node: ^14.17.0 || ^16.13.0 || >=18.0.0}
hsv-rgb@1.0.0:
resolution: {integrity: sha512-Azd6IP11LZm0cEczEnJw5B6zIgWdGlE4TSoM2eh+RPRbXSQCy/0JS2POEq0wOtbAZtxTJhEMGm3GUYGbnTIJGw==}
html-escaper@2.0.2:
resolution: {integrity: sha512-H2iMtd0I4Mt5eYiapRdIDjp+XzelXQ0tFE4JS7YFwFevXXMmOp9myNrUvCg0D6ws8iqkRPBfKHgbwig1SmlLfg==}
@@ -3850,23 +3827,6 @@ packages:
resolution: {integrity: sha512-4gd7VpWNQNB4UKKCFFVcp1AVv+FMOgs9NKzjHKusc8jTMhd5eL1NqQqOpE0KzMds804/yHlglp3uxgluOqAPLw==}
engines: {node: '>= 0.4'}
intl-format-cache@2.2.9:
resolution: {integrity: sha512-Zv/u8wRpekckv0cLkwpVdABYST4hZNTDaX7reFetrYTJwxExR2VyTqQm+l0WmL0Qo8Mjb9Tf33qnfj0T7pjxdQ==}
intl-messageformat-parser@1.4.0:
resolution: {integrity: sha512-/XkqFHKezO6UcF4Av2/Lzfrez18R0jyw7kRFhSeB/YRakdrgSc9QfFZUwNJI9swMwMoNPygK1ArC5wdFSjPw+A==}
deprecated: We've written a new parser that's 6x faster and is backwards compatible. Please use @formatjs/icu-messageformat-parser
intl-messageformat@2.2.0:
resolution: {integrity: sha512-I+tSvHnXqJYjDfNmY95tpFMj30yoakC6OXAo+wu/wTMy6tA/4Fd4mvV7Uzs4cqK/Ap29sHhwjcY+78a8eifcXw==}
intl-relativeformat@2.2.0:
resolution: {integrity: sha512-4bV/7kSKaPEmu6ArxXf9xjv1ny74Zkwuey8Pm01NH4zggPP7JHwg2STk8Y3JdspCKRDriwIyLRfEXnj2ZLr4Bw==}
deprecated: This package has been deprecated, please see migration guide at 'https://github.com/formatjs/formatjs/tree/master/packages/intl-relativeformat#migration-guide'
invariant@2.2.4:
resolution: {integrity: sha512-phJfQVBuaJM5raOpJjSfkiD6BpbCE4Ns//LaXl6wGYtUBY83nWS6Rf9tXm2e8VaK60JEjYldbPif/A2B1C2gNA==}
is-arguments@1.2.0:
resolution: {integrity: sha512-7bVbi0huj/wrIAOzb8U1aszg9kdi3KN/CyU19CTI7tAoZYEZoL9yCDXpbXN+uPsuWnP02cyug1gleqq+TU+YCA==}
engines: {node: '>= 0.4'}
@@ -4045,10 +4005,6 @@ packages:
isexe@2.0.0:
resolution: {integrity: sha512-RHxMLp9lnKHGHRng9QFhRCMbYAcVpn69smSGcq3f36xjgVVWThj4qqLbTLlq7Ssj8B+fIQ1EuCEGI2lKsyQeIw==}
iso-639-1@2.1.15:
resolution: {integrity: sha512-7c7mBznZu2ktfvyT582E2msM+Udc1EjOyhVRE/0ZsjD9LBtWSm23h3PtiRh2a35XoUsTQQjJXaJzuLjXsOdFDg==}
engines: {node: '>=6.0'}
istanbul-lib-coverage@3.2.2:
resolution: {integrity: sha512-O8dpsF+r0WV/8MNRKfnmrtCWhuKjxrq2w+jpzBL5UZKTi2LeVWnWOmWRxFlesJONmc+wLAGvKQZEOanko0LFTg==}
engines: {node: '>=8'}
@@ -4346,20 +4302,6 @@ packages:
resolution: {integrity: sha512-dhG34DXATL5hSxJbIexCft8FChFXtmskoZYnoPWjXQuebWYCNkVeV3KkGegCK9CP1oswI/vQibS2GY7Em/sJJA==}
engines: {node: '>= 8'}
kpop@https://download.kopano.io/community/kapp:/kpop-2.7.2.tgz:
resolution: {tarball: https://download.kopano.io/community/kapp:/kpop-2.7.2.tgz}
version: 2.7.1
engines: {node: '>=6.11.0'}
peerDependencies:
'@gluejs/glue': ^0.3.0
'@material-ui/core': ^4.11.0
'@material-ui/icons': ^4.9.1
notistack: ^0.8.8
oidc-client: ^1.11.0
react: ^16.8.0 || ^17.0.0
react-dom: ^16.8.0 || ^17.0.0
react-intl: ^2.6.0
language-subtag-registry@0.3.23:
resolution: {integrity: sha512-0K65Lea881pHotoGEa5gDlMxt3pctLi2RplBb7Ezh4rRdLEOtgi7n4EwK9lamnUCkKBqaeKRVebTq6BAxSkpXQ==}
@@ -4636,13 +4578,6 @@ packages:
resolution: {integrity: sha512-bdok/XvKII3nUpklnV6P2hxtMNrCboOjAcyBuQnWEhO665FwrSNRxU+AqpsyvO6LgGYPspN+lu5CLtw4jPRKNA==}
engines: {node: '>=0.10.0'}
notistack@0.8.9:
resolution: {integrity: sha512-nRHQVWUfgHnvnKrjRbRX9f+YAnbyh96yRyO5bEP/FCLVLuTZcJOwUr0GZ7Xr/8wK3+hXa9JYpXUkUhSxj1K8NQ==}
peerDependencies:
'@material-ui/core': ^3.2.0 || ^4.0.0
react: ^16.8.0
react-dom: ^16.8.0
now-and-later@3.0.0:
resolution: {integrity: sha512-pGO4pzSdaxhWTGkfSfHx3hVzJVslFPwBp2Myq9MYN/ChfJZF87ochMAXnvz6/58RJSf5ik2q9tXprBBrk2cpcg==}
engines: {node: '>= 10.13.0'}
@@ -4698,9 +4633,6 @@ packages:
resolution: {integrity: sha512-gXah6aZrcUxjWg2zR2MwouP2eHlCBzdV4pygudehaKXSGW4v2AsRQUK+lwwXhii6KFZcunEnmSUoYp5CXibxtA==}
engines: {node: '>= 0.4'}
oidc-client@1.11.5:
resolution: {integrity: sha512-LcKrKC8Av0m/KD/4EFmo9Sg8fSQ+WFJWBrmtWd+tZkNn3WT/sQG3REmPANE9tzzhbjW6VkTNy4xhAXCfPApAOg==}
once@1.4.0:
resolution: {integrity: sha512-lNaJgI+2Q5URQBkccEKHTQOPaXdUxnZZElQTZY0MFUAuaEqe1E+Nyvgdz/aIyNi6Z9MzO5dv1H8n58/GELp3+w==}
@@ -5364,12 +5296,6 @@ packages:
typescript:
optional: true
react-intl@2.9.0:
resolution: {integrity: sha512-27jnDlb/d2A7mSJwrbOBnUgD+rPep+abmoJE511Tf8BnoONIAUehy/U1zZCHGO17mnOwMWxqN4qC0nW11cD6rA==}
peerDependencies:
prop-types: ^15.5.4
react: ^0.14.9 || ^15.0.0 || ^16.0.0
react-is@16.13.1:
resolution: {integrity: sha512-24e6ynE2H+OKt4kqsOvNd8kBpV65zoxbA4BVsEOB3ARVWQki/DHzaUoC5KuON/BiccDaCCTZBuOcfZs70kR8bQ==}
@@ -5698,9 +5624,6 @@ packages:
seq@0.3.5:
resolution: {integrity: sha512-sisY2Ln1fj43KBkRtXkesnRHYNdswIkIibvNe/0UKm2GZxjMbqmccpiatoKr/k2qX5VKiLU8xm+tz/74LAho4g==}
serialize-javascript@4.0.0:
resolution: {integrity: sha512-GaNA54380uFefWghODBWEGisLZFj00nS5ACs6yHa9nLqlLpVLO8ChDGeKRjZnV4Nh4n0Qi7nhYZD/9fCPzEqkw==}
serialize-javascript@6.0.2:
resolution: {integrity: sha512-Saa1xPByTTq2gdeFZYLLo+RFE35NHZkAbqZeWNd3BpzppeVisAqpDjcp8dyf6uIvEqJRd46jemmyA4iFIeVk8g==}
@@ -7918,8 +7841,6 @@ snapshots:
'@fontsource/roboto@5.2.5': {}
'@gluejs/glue@0.3.0': {}
'@gulpjs/to-absolute-glob@4.0.0':
dependencies:
is-negated-glob: 1.0.0
@@ -9403,8 +9324,6 @@ snapshots:
core-js@3.40.0: {}
core-js@3.43.0: {}
core-util-is@1.0.3: {}
cosmiconfig@6.0.0:
@@ -9423,8 +9342,6 @@ snapshots:
path-type: 4.0.0
yaml: 1.10.2
crc32@0.2.2: {}
cross-fetch@4.0.0(encoding@0.1.13):
dependencies:
node-fetch: 2.7.0(encoding@0.1.13)
@@ -9443,8 +9360,6 @@ snapshots:
shebang-command: 2.0.0
which: 2.0.2
crypto-js@4.2.0: {}
crypto-random-string@2.0.0: {}
css-blank-pseudo@7.0.1(postcss@8.5.4):
@@ -10683,8 +10598,6 @@ snapshots:
dependencies:
lru-cache: 7.18.3
hsv-rgb@1.0.0: {}
html-escaper@2.0.2: {}
html-minifier-terser@6.1.0:
@@ -10833,22 +10746,6 @@ snapshots:
hasown: 2.0.2
side-channel: 1.1.0
intl-format-cache@2.2.9: {}
intl-messageformat-parser@1.4.0: {}
intl-messageformat@2.2.0:
dependencies:
intl-messageformat-parser: 1.4.0
intl-relativeformat@2.2.0:
dependencies:
intl-messageformat: 2.2.0
invariant@2.2.4:
dependencies:
loose-envify: 1.4.0
is-arguments@1.2.0:
dependencies:
call-bound: 1.0.4
@@ -11007,8 +10904,6 @@ snapshots:
isexe@2.0.0: {}
iso-639-1@2.1.15: {}
istanbul-lib-coverage@3.2.2: {}
istanbul-lib-instrument@6.0.3:
@@ -11539,20 +11434,6 @@ snapshots:
klona@2.0.6: {}
kpop@https://download.kopano.io/community/kapp:/kpop-2.7.2.tgz(@gluejs/glue@0.3.0)(@material-ui/core@4.12.4(@types/react@17.0.80)(react-dom@17.0.2(react@17.0.2))(react@17.0.2))(@material-ui/icons@4.11.3(@material-ui/core@4.12.4(@types/react@17.0.80)(react-dom@17.0.2(react@17.0.2))(react@17.0.2))(@types/react@17.0.80)(react-dom@17.0.2(react@17.0.2))(react@17.0.2))(notistack@0.8.9(@material-ui/core@4.12.4(@types/react@17.0.80)(react-dom@17.0.2(react@17.0.2))(react@17.0.2))(react-dom@17.0.2(react@17.0.2))(react@17.0.2))(oidc-client@1.11.5)(react-dom@17.0.2(react@17.0.2))(react-intl@2.9.0(prop-types@15.8.1)(react@17.0.2))(react@17.0.2):
dependencies:
'@gluejs/glue': 0.3.0
'@material-ui/core': 4.12.4(@types/react@17.0.80)(react-dom@17.0.2(react@17.0.2))(react@17.0.2)
'@material-ui/icons': 4.11.3(@material-ui/core@4.12.4(@types/react@17.0.80)(react-dom@17.0.2(react@17.0.2))(react@17.0.2))(@types/react@17.0.80)(react-dom@17.0.2(react@17.0.2))(react@17.0.2)
crc32: 0.2.2
hsv-rgb: 1.0.0
iso-639-1: 2.1.15
notistack: 0.8.9(@material-ui/core@4.12.4(@types/react@17.0.80)(react-dom@17.0.2(react@17.0.2))(react@17.0.2))(react-dom@17.0.2(react@17.0.2))(react@17.0.2)
oidc-client: 1.11.5
react: 17.0.2
react-dom: 17.0.2(react@17.0.2)
react-intl: 2.9.0(prop-types@15.8.1)(react@17.0.2)
language-subtag-registry@0.3.23: {}
language-tags@1.0.9:
@@ -11787,16 +11668,6 @@ snapshots:
normalize-range@0.1.2: {}
notistack@0.8.9(@material-ui/core@4.12.4(@types/react@17.0.80)(react-dom@17.0.2(react@17.0.2))(react@17.0.2))(react-dom@17.0.2(react@17.0.2))(react@17.0.2):
dependencies:
'@material-ui/core': 4.12.4(@types/react@17.0.80)(react-dom@17.0.2(react@17.0.2))(react@17.0.2)
classnames: 2.5.1
hoist-non-react-statics: 3.3.2
prop-types: 15.8.1
react: 17.0.2
react-dom: 17.0.2(react@17.0.2)
react-is: 16.13.1
now-and-later@3.0.0:
dependencies:
once: 1.4.0
@@ -11860,14 +11731,6 @@ snapshots:
define-properties: 1.2.1
es-object-atoms: 1.1.1
oidc-client@1.11.5:
dependencies:
acorn: 7.4.1
base64-js: 1.5.1
core-js: 3.43.0
crypto-js: 4.2.0
serialize-javascript: 4.0.0
once@1.4.0:
dependencies:
wrappy: 1.0.2
@@ -12599,16 +12462,6 @@ snapshots:
react-dom: 17.0.2(react@17.0.2)
typescript: 5.8.3
react-intl@2.9.0(prop-types@15.8.1)(react@17.0.2):
dependencies:
hoist-non-react-statics: 3.3.2
intl-format-cache: 2.2.9
intl-messageformat: 2.2.0
intl-relativeformat: 2.2.0
invariant: 2.2.4
prop-types: 15.8.1
react: 17.0.2
react-is@16.13.1: {}
react-is@17.0.2: {}
@@ -12975,10 +12828,6 @@ snapshots:
chainsaw: 0.0.9
hashish: 0.0.4
serialize-javascript@4.0.0:
dependencies:
randombytes: 2.1.0
serialize-javascript@6.0.2:
dependencies:
randombytes: 2.1.0
+2 -5
View File
@@ -2,10 +2,7 @@ import React, {ReactElement, Suspense, lazy, useState, useEffect} from 'react';
import PropTypes from 'prop-types';
import {MuiThemeProvider} from '@material-ui/core/styles';
import {defaultTheme} from 'kpop/es/theme';
import 'kpop/static/css/base.css';
import 'kpop/static/css/scrollbar.css';
import muiTheme from './theme';
import Spinner from './components/Spinner';
import * as version from './version';
@@ -52,7 +49,7 @@ const App = ({ bgImg }): ReactElement => {
className={`oc-login-bg ${bgImg ? 'oc-login-bg-image' : ''}`}
style={{backgroundImage: bgImg ? `url(${bgImg})` : undefined}}
>
<MuiThemeProvider theme={defaultTheme}>
<MuiThemeProvider theme={muiTheme}>
<Suspense fallback={<Spinner/>}>
<LazyMain/>
</Suspense>
+9 -1
View File
@@ -1,4 +1,3 @@
/* additional css on top of kpop */
@font-face {
font-family: OpenCloud;
src: url('./fonts/OpenCloud500-Regular.woff2') format('woff2');
@@ -17,16 +16,25 @@
html {
font-feature-settings: "cv11";
color: #20434f !important;
height: 100%;
}
body {
font-family: OpenCloud, sans-serif;
height: 100%;
margin: 0;
padding: 0;
}
strong {
font-weight: 600;
}
#root {
height: 100%;
display: flex;
}
.oc-font-weight-light {
font-weight: 300;
}
+5
View File
@@ -12,3 +12,8 @@ packages:
github.com/opencloud-eu/opencloud/services/proxy/pkg/userroles:
interfaces:
UserRoleAssigner: {}
go-micro.dev/v4/store:
config:
dir: pkg/staticroutes/internal/backchannellogout/mocks
interfaces:
Store: {}
+2
View File
@@ -11,6 +11,7 @@ import (
gateway "github.com/cs3org/go-cs3apis/cs3/gateway/v1beta1"
chimiddleware "github.com/go-chi/chi/v5/middleware"
"github.com/justinas/alice"
"github.com/opencloud-eu/opencloud/pkg/config/configlog"
"github.com/opencloud-eu/opencloud/pkg/generators"
"github.com/opencloud-eu/opencloud/pkg/log"
@@ -73,6 +74,7 @@ func Server(cfg *config.Config) *cli.Command {
microstore.Nodes(cfg.PreSignedURL.SigningKeys.Nodes...),
microstore.Database("proxy"),
microstore.Table("signing-keys"),
store.DisablePersistence(cfg.PreSignedURL.SigningKeys.DisablePersistence),
store.Authentication(cfg.PreSignedURL.SigningKeys.AuthUsername, cfg.PreSignedURL.SigningKeys.AuthPassword),
)
+25 -14
View File
@@ -8,14 +8,16 @@ import (
"strings"
"time"
"github.com/golang-jwt/jwt/v5"
"github.com/opencloud-eu/opencloud/pkg/log"
"github.com/opencloud-eu/opencloud/pkg/oidc"
"github.com/pkg/errors"
"github.com/vmihailenco/msgpack/v5"
store "go-micro.dev/v4/store"
"go-micro.dev/v4/store"
"golang.org/x/crypto/sha3"
"golang.org/x/oauth2"
"github.com/golang-jwt/jwt/v5"
"github.com/opencloud-eu/opencloud/pkg/log"
"github.com/opencloud-eu/opencloud/pkg/oidc"
"github.com/opencloud-eu/opencloud/services/proxy/pkg/staticroutes"
)
const (
@@ -114,16 +116,25 @@ func (m *OIDCAuthenticator) getClaims(token string, req *http.Request) (map[stri
m.Logger.Error().Err(err).Msg("failed to write to userinfo cache")
}
if sid := aClaims.SessionID; sid != "" {
// reuse user cache for session id lookup
err = m.userInfoCache.Write(&store.Record{
Key: sid,
Value: []byte(encodedHash),
Expiry: time.Until(expiration),
})
if err != nil {
m.Logger.Error().Err(err).Msg("failed to write session lookup cache")
}
// fail if creating the storage key fails,
// it means there is no subject and no session.
//
// ok: {key: ".sessionId"}
// ok: {key: "subject."}
// ok: {key: "subject.sessionId"}
// fail: {key: "."}
subjectSessionKey, err := staticroutes.NewRecordKey(aClaims.Subject, aClaims.SessionID)
if err != nil {
m.Logger.Error().Err(err).Msg("failed to build subject.session")
return
}
if err := m.userInfoCache.Write(&store.Record{
Key: subjectSessionKey,
Value: []byte(encodedHash),
Expiry: time.Until(expiration),
}); err != nil {
m.Logger.Error().Err(err).Msg("failed to write session lookup cache")
}
}
}()
@@ -6,17 +6,40 @@ import (
"net/http"
"github.com/go-chi/render"
"github.com/opencloud-eu/opencloud/pkg/oidc"
"github.com/opencloud-eu/reva/v2/pkg/events"
"github.com/opencloud-eu/reva/v2/pkg/utils"
"github.com/pkg/errors"
"github.com/vmihailenco/msgpack/v5"
microstore "go-micro.dev/v4/store"
bcl "github.com/opencloud-eu/opencloud/services/proxy/pkg/staticroutes/internal/backchannellogout"
"github.com/opencloud-eu/reva/v2/pkg/events"
"github.com/opencloud-eu/reva/v2/pkg/utils"
)
// handle backchannel logout requests as per https://openid.net/specs/openid-connect-backchannel-1_0.html#BCRequest
// NewRecordKey converts the subject and session to a base64 encoded key
var NewRecordKey = bcl.NewKey
// backchannelLogout handles backchannel logout requests from the identity provider and invalidates the related sessions in the cache
// spec: https://openid.net/specs/openid-connect-backchannel-1_0.html#BCRequest
//
// known side effects of backchannel logout in keycloak:
//
// - keyCloak "Sign out all active sessions" does not send a backchannel logout request,
// as the devs mention, this may lead to thousands of backchannel logout requests,
// therefore, they recommend a short token lifetime.
// https://github.com/keycloak/keycloak/issues/27342#issuecomment-2408461913
//
// - keyCloak user self-service portal, "Sign out all devices" may not send a backchannel
// logout request for each session, it's not mentionex explicitly,
// but maybe the reason for that is the same as for "Sign out all active sessions"
// to prevent a flood of backchannel logout requests.
//
// - if the keycloak setting "Backchannel logout session required" is disabled (or the token has no session id),
// we resolve the session by the subject which can lead to multiple session records (subject.*),
// we then send a logout event (sse) to each connected client and delete our stored cache record (subject.session & claim).
// all sessions besides the one that triggered the backchannel logout continue to exist in the identity provider,
// so the user will not be fully logged out until all sessions are logged out or expired.
// this leads to the situation that web renders the logout view even if the instance is not fully logged out yet.
func (s *StaticRouteHandler) backchannelLogout(w http.ResponseWriter, r *http.Request) {
// parse the application/x-www-form-urlencoded POST request
logger := s.Logger.SubloggerWithRequestID(r.Context())
if err := r.ParseForm(); err != nil {
logger.Warn().Err(err).Msg("ParseForm failed")
@@ -27,45 +50,86 @@ func (s *StaticRouteHandler) backchannelLogout(w http.ResponseWriter, r *http.Re
logoutToken, err := s.OidcClient.VerifyLogoutToken(r.Context(), r.PostFormValue("logout_token"))
if err != nil {
logger.Warn().Err(err).Msg("VerifyLogoutToken failed")
msg := "failed to verify logout token"
logger.Warn().Err(err).Msg(msg)
render.Status(r, http.StatusBadRequest)
render.JSON(w, r, jse{Error: "invalid_request", ErrorDescription: err.Error()})
render.JSON(w, r, jse{Error: "invalid_request", ErrorDescription: msg})
return
}
records, err := s.UserInfoCache.Read(logoutToken.SessionId)
if errors.Is(err, microstore.ErrNotFound) || len(records) == 0 {
lookupKey, err := bcl.NewKey(logoutToken.Subject, logoutToken.SessionId)
if err != nil {
msg := "failed to build key from logout token"
logger.Warn().Err(err).Msg(msg)
render.Status(r, http.StatusBadRequest)
render.JSON(w, r, jse{Error: "invalid_request", ErrorDescription: msg})
return
}
requestSubjectAndSession, err := bcl.NewSuSe(lookupKey)
if err != nil {
msg := "failed to build subjec.session from lookupKey"
logger.Error().Err(err).Msg(msg)
render.Status(r, http.StatusBadRequest)
render.JSON(w, r, jse{Error: "invalid_request", ErrorDescription: msg})
return
}
lookupRecords, err := bcl.GetLogoutRecords(requestSubjectAndSession, s.UserInfoCache)
if errors.Is(err, microstore.ErrNotFound) || len(lookupRecords) == 0 {
render.Status(r, http.StatusOK)
render.JSON(w, r, nil)
return
}
if err != nil {
logger.Error().Err(err).Msg("Error reading userinfo cache")
msg := "failed to read userinfo cache"
logger.Error().Err(err).Msg(msg)
render.Status(r, http.StatusBadRequest)
render.JSON(w, r, jse{Error: "invalid_request", ErrorDescription: err.Error()})
render.JSON(w, r, jse{Error: "invalid_request", ErrorDescription: msg})
return
}
for _, record := range records {
err := s.publishBackchannelLogoutEvent(r.Context(), record, logoutToken)
for _, record := range lookupRecords {
// the record key is in the format "subject.session" or ".session"
// the record value is the key of the record that contains the claim in its value
key, value := record.Key, string(record.Value)
subjectSession, err := bcl.NewSuSe(key)
if err != nil {
s.Logger.Warn().Err(err).Msg("could not publish backchannel logout event")
// never leak any key-related information
logger.Warn().Err(err).Msgf("failed to parse key: %s", key)
continue
}
err = s.UserInfoCache.Delete(string(record.Value))
session, err := subjectSession.Session()
if err != nil {
logger.Warn().Err(err).Msgf("failed to read session for: %s", key)
continue
}
if requestSubjectAndSession.Mode() == bcl.LogoutModeSession {
if err := s.publishBackchannelLogoutEvent(r.Context(), session, value); err != nil {
s.Logger.Warn().Err(err).Msgf("failed to publish backchannel logout event for: %s", key)
continue
}
}
err = s.UserInfoCache.Delete(value)
if err != nil && !errors.Is(err, microstore.ErrNotFound) {
// Spec requires us to return a 400 BadRequest when the session could not be destroyed
logger.Err(err).Msg("could not delete user info from cache")
// we have to return a 400 BadRequest when we fail to delete the session
// https://openid.net/specs/openid-connect-backchannel-1_0.html#rfc.section.2.8
msg := "failed to delete record"
s.Logger.Warn().Err(err).Msgf("%s for: %s", msg, key)
render.Status(r, http.StatusBadRequest)
render.JSON(w, r, jse{Error: "invalid_request", ErrorDescription: err.Error()})
render.JSON(w, r, jse{Error: "invalid_request", ErrorDescription: msg})
return
}
logger.Debug().Msg("Deleted userinfo from cache")
}
// we can ignore errors when cleaning up the lookup table
err = s.UserInfoCache.Delete(logoutToken.SessionId)
if err != nil {
logger.Debug().Err(err).Msg("Failed to cleanup sessionid lookup entry")
// we can ignore errors when deleting the lookup record
err = s.UserInfoCache.Delete(key)
if err != nil {
logger.Debug().Err(err).Msgf("failed to delete record for: %s", key)
}
}
render.Status(r, http.StatusOK)
@@ -73,41 +137,42 @@ func (s *StaticRouteHandler) backchannelLogout(w http.ResponseWriter, r *http.Re
}
// publishBackchannelLogoutEvent publishes a backchannel logout event when the callback revived from the identity provider
func (s StaticRouteHandler) publishBackchannelLogoutEvent(ctx context.Context, record *microstore.Record, logoutToken *oidc.LogoutToken) error {
func (s *StaticRouteHandler) publishBackchannelLogoutEvent(ctx context.Context, sessionId, claimKey string) error {
if s.EventsPublisher == nil {
return fmt.Errorf("the events publisher is not set")
return errors.New("events publisher not set")
}
urecords, err := s.UserInfoCache.Read(string(record.Value))
if err != nil {
return fmt.Errorf("reading userinfo cache: %w", err)
}
if len(urecords) == 0 {
return fmt.Errorf("userinfo not found")
claimRecords, err := s.UserInfoCache.Read(claimKey)
switch {
case err != nil:
return fmt.Errorf("failed to read userinfo cache: %w", err)
case len(claimRecords) == 0:
return fmt.Errorf("no claim found for key: %s", claimKey)
}
var claims map[string]interface{}
if err = msgpack.Unmarshal(urecords[0].Value, &claims); err != nil {
return fmt.Errorf("could not unmarshal userinfo: %w", err)
if err = msgpack.Unmarshal(claimRecords[0].Value, &claims); err != nil {
return fmt.Errorf("failed to unmarshal claims: %w", err)
}
oidcClaim, ok := claims[s.Config.UserOIDCClaim].(string)
if !ok {
return fmt.Errorf("could not get claim %w", err)
return fmt.Errorf("failed to get claim %w", err)
}
user, _, err := s.UserProvider.GetUserByClaims(ctx, s.Config.UserCS3Claim, oidcClaim)
if err != nil || user.GetId() == nil {
return fmt.Errorf("could not get user by claims: %w", err)
return fmt.Errorf("failed to get user by claims: %w", err)
}
e := events.BackchannelLogout{
Executant: user.GetId(),
SessionId: logoutToken.SessionId,
SessionId: sessionId,
Timestamp: utils.TSNow(),
}
if err := events.Publish(ctx, s.EventsPublisher, e); err != nil {
return fmt.Errorf("could not publish user created event %w", err)
return fmt.Errorf("failed to publish user logout event %w", err)
}
return nil
}
@@ -0,0 +1,186 @@
// package backchannellogout provides functions to classify and lookup
// backchannel logout records from the cache store.
package backchannellogout
import (
"encoding/base64"
"errors"
"strings"
microstore "go-micro.dev/v4/store"
)
// keyEncoding is the base64 encoding used for session and subject keys
var keyEncoding = base64.URLEncoding
// ErrInvalidKey indicates that the provided key does not conform to the expected format.
var ErrInvalidKey = errors.New("invalid key format")
// NewKey converts the subject and session to a base64 encoded key
func NewKey(subject, session string) (string, error) {
subjectSession := strings.Join([]string{
keyEncoding.EncodeToString([]byte(subject)),
keyEncoding.EncodeToString([]byte(session)),
}, ".")
if subjectSession == "." {
return "", ErrInvalidKey
}
return subjectSession, nil
}
// LogoutMode defines the mode of backchannel logout, either by session or by subject
type LogoutMode int
const (
// LogoutModeUndefined is used when the logout mode cannot be determined
LogoutModeUndefined LogoutMode = iota
// LogoutModeSubject is used when the logout mode is determined by the subject
LogoutModeSubject
// LogoutModeSession is used when the logout mode is determined by the session id
LogoutModeSession
)
// ErrDecoding is returned when decoding fails
var ErrDecoding = errors.New("failed to decode")
// SuSe 🦎 ;) is a struct that groups the subject and session together
// to prevent mix-ups for ('session, subject' || 'subject, session')
// return values.
type SuSe struct {
encodedSubject string
encodedSession string
}
// Subject decodes and returns the subject or an error
func (suse SuSe) Subject() (string, error) {
subject, err := keyEncoding.DecodeString(suse.encodedSubject)
if err != nil {
return "", errors.Join(errors.New("failed to decode subject"), ErrDecoding, err)
}
return string(subject), nil
}
// Session decodes and returns the session or an error
func (suse SuSe) Session() (string, error) {
subject, err := keyEncoding.DecodeString(suse.encodedSession)
if err != nil {
return "", errors.Join(errors.New("failed to decode session"), ErrDecoding, err)
}
return string(subject), nil
}
// Mode determines the backchannel logout mode based on the presence of subject and session
func (suse SuSe) Mode() LogoutMode {
switch {
case suse.encodedSession == "" && suse.encodedSubject != "":
return LogoutModeSubject
case suse.encodedSession != "":
return LogoutModeSession
default:
return LogoutModeUndefined
}
}
// ErrInvalidSubjectOrSession is returned when the provided key does not match the expected key format
var ErrInvalidSubjectOrSession = errors.New("invalid subject or session")
// NewSuSe parses the subject and session id from the given key and returns a SuSe struct
func NewSuSe(key string) (SuSe, error) {
suse := SuSe{}
keys := strings.Split(key, ".")
switch len(keys) {
case 1:
suse.encodedSession = keys[0]
case 2:
suse.encodedSubject = keys[0]
suse.encodedSession = keys[1]
default:
return suse, ErrInvalidSubjectOrSession
}
if suse.encodedSubject == "" && suse.encodedSession == "" {
return suse, ErrInvalidSubjectOrSession
}
if _, err := suse.Subject(); err != nil {
return suse, errors.Join(ErrInvalidSubjectOrSession, err)
}
if _, err := suse.Session(); err != nil {
return suse, errors.Join(ErrInvalidSubjectOrSession, err)
}
if mode := suse.Mode(); mode == LogoutModeUndefined {
return suse, ErrInvalidSubjectOrSession
}
return suse, nil
}
// ErrSuspiciousCacheResult is returned when the cache result is suspicious
var ErrSuspiciousCacheResult = errors.New("suspicious cache result")
// GetLogoutRecords retrieves the records from the user info cache based on the backchannel
// logout mode and the provided SuSe struct.
// it uses a seperator to prevent sufix and prefix exploration in the cache and checks
// if the retrieved records match the requested subject and or session id as well, to prevent false positives.
func GetLogoutRecords(suse SuSe, store microstore.Store) ([]*microstore.Record, error) {
var key string
var opts []microstore.ReadOption
switch {
case suse.Mode() == LogoutModeSubject && suse.encodedSubject != "":
// the dot at the end prevents prefix exploration in the cache,
// so only keys that start with 'subject.*' will be returned, but not 'sub*'.
key = suse.encodedSubject + "."
opts = append(opts, microstore.ReadPrefix())
case suse.Mode() == LogoutModeSession && suse.encodedSession != "":
// the dot at the beginning prevents sufix exploration in the cache,
// so only keys that end with '*.session' will be returned, but not '*sion'.
key = "." + suse.encodedSession
opts = append(opts, microstore.ReadSuffix())
default:
return nil, errors.Join(errors.New("cannot determine logout mode"), ErrSuspiciousCacheResult)
}
// the go micro memory store requires a limit to work, why???
records, err := store.Read(key, append(opts, microstore.ReadLimit(1000))...)
if err != nil {
return nil, err
}
if len(records) == 0 {
return nil, microstore.ErrNotFound
}
if suse.Mode() == LogoutModeSession && len(records) > 1 {
return nil, errors.Join(errors.New("multiple session records found"), ErrSuspiciousCacheResult)
}
// double-check if the found records match the requested subject and or session id as well,
// to prevent false positives.
for _, record := range records {
recordSuSe, err := NewSuSe(record.Key)
if err != nil {
// never leak any key-related information
return nil, errors.Join(errors.New("failed to parse key"), ErrSuspiciousCacheResult, err)
}
switch {
// in subject mode, the subject must match, but the session id can be different
case suse.Mode() == LogoutModeSubject && suse.encodedSubject == recordSuSe.encodedSubject:
continue
// in session mode, the session id must match, but the subject can be different
case suse.Mode() == LogoutModeSession && suse.encodedSession == recordSuSe.encodedSession:
continue
}
return nil, errors.Join(errors.New("key does not match the requested subject or session"), ErrSuspiciousCacheResult)
}
return records, nil
}
@@ -0,0 +1,306 @@
package backchannellogout
import (
"slices"
"strings"
"testing"
"github.com/stretchr/testify/mock"
"github.com/stretchr/testify/require"
"go-micro.dev/v4/store"
"github.com/opencloud-eu/opencloud/services/proxy/pkg/staticroutes/internal/backchannellogout/mocks"
)
func mustNewKey(t *testing.T, subject, session string) string {
key, err := NewKey(subject, session)
require.NoError(t, err)
return key
}
func mustNewSuSe(t *testing.T, subject, session string) SuSe {
suse, err := NewSuSe(mustNewKey(t, subject, session))
require.NoError(t, err)
return suse
}
func TestNewKey(t *testing.T) {
tests := []struct {
name string
subject string
session string
wantKey string
wantErr error
}{
{
name: "key variation: 'subject.session'",
subject: "subject",
session: "session",
wantKey: "c3ViamVjdA==.c2Vzc2lvbg==",
},
{
name: "key variation: 'subject.'",
subject: "subject",
wantKey: "c3ViamVjdA==.",
},
{
name: "key variation: '.session'",
session: "session",
wantKey: ".c2Vzc2lvbg==",
},
{
name: "key variation: '.'",
wantErr: ErrInvalidKey,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
key, err := NewKey(tt.subject, tt.session)
require.ErrorIs(t, err, tt.wantErr)
require.Equal(t, tt.wantKey, key)
})
}
}
func TestNewSuSe(t *testing.T) {
tests := []struct {
name string
key string
wantSubject string
wantSession string
wantMode LogoutMode
wantErr error
}{
{
name: "key variation: '.session'",
key: mustNewKey(t, "", "session"),
wantSession: "session",
wantMode: LogoutModeSession,
},
{
name: "key variation: 'session'",
key: mustNewKey(t, "", "session"),
wantSession: "session",
wantMode: LogoutModeSession,
},
{
name: "key variation: 'subject.'",
key: mustNewKey(t, "subject", ""),
wantSubject: "subject",
wantMode: LogoutModeSubject,
},
{
name: "key variation: 'subject.session'",
key: mustNewKey(t, "subject", "session"),
wantSubject: "subject",
wantSession: "session",
wantMode: LogoutModeSession,
},
{
name: "key variation: 'dot'",
key: ".",
wantErr: ErrInvalidSubjectOrSession,
},
{
name: "key variation: 'empty'",
key: "",
wantErr: ErrInvalidSubjectOrSession,
},
{
name: "key variation: string('subject.session')",
key: "subject.session",
wantErr: ErrInvalidSubjectOrSession,
wantMode: LogoutModeSession,
},
{
name: "key variation: string('subject.')",
key: "subject.",
wantErr: ErrInvalidSubjectOrSession,
wantMode: LogoutModeSubject,
},
{
name: "key variation: string('.session')",
key: ".session",
wantErr: ErrInvalidSubjectOrSession,
wantMode: LogoutModeSession,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
suSe, err := NewSuSe(tt.key)
require.ErrorIs(t, err, tt.wantErr)
mode := suSe.Mode()
require.Equal(t, tt.wantMode, mode)
subject, _ := suSe.Subject()
require.Equal(t, tt.wantSubject, subject)
session, _ := suSe.Session()
require.Equal(t, tt.wantSession, session)
})
}
}
func TestGetLogoutRecords(t *testing.T) {
sessionStore := store.NewMemoryStore()
recordClaimA := &store.Record{Key: "claim-a", Value: []byte("claim-a-data")}
recordClaimB := &store.Record{Key: "claim-b", Value: []byte("claim-b-data")}
recordClaimC := &store.Record{Key: "claim-c", Value: []byte("claim-c-data")}
recordClaimD := &store.Record{Key: "claim-d", Value: []byte("claim-d-data")}
recordSessionA := &store.Record{Key: mustNewKey(t, "", "session-a"), Value: []byte(recordClaimA.Key)}
recordSessionB := &store.Record{Key: mustNewKey(t, "", "session-b"), Value: []byte(recordClaimB.Key)}
recordSubjectASessionC := &store.Record{Key: mustNewKey(t, "subject-a", "session-c"), Value: []byte(recordSessionA.Key)}
recordSubjectASessionD := &store.Record{Key: mustNewKey(t, "subject-a", "session-d"), Value: []byte(recordSessionA.Key)}
for _, r := range []*store.Record{
recordClaimA,
recordClaimB,
recordClaimC,
recordClaimD,
recordSessionA,
recordSessionB,
recordSubjectASessionC,
recordSubjectASessionD,
} {
require.NoError(t, sessionStore.Write(r))
}
tests := []struct {
name string
suSe SuSe
store func(t *testing.T) store.Store
wantRecords []*store.Record
wantErrs []error
}{
{
name: "fails if multiple session records are found",
suSe: mustNewSuSe(t, "", "session-a"),
store: func(t *testing.T) store.Store {
s := mocks.NewStore(t)
s.EXPECT().Read(mock.Anything, mock.Anything).Return([]*store.Record{
recordSessionA,
recordSessionB,
}, nil)
return s
},
wantRecords: []*store.Record{},
wantErrs: []error{ErrSuspiciousCacheResult}},
{
name: "fails if the record key is not ok",
suSe: mustNewSuSe(t, "", "session-a"),
store: func(t *testing.T) store.Store {
s := mocks.NewStore(t)
s.EXPECT().Read(mock.Anything, mock.Anything).Return([]*store.Record{
{Key: "invalid.record.key"},
}, nil)
return s
},
wantRecords: []*store.Record{},
wantErrs: []error{ErrInvalidSubjectOrSession, ErrSuspiciousCacheResult},
},
{
name: "fails if the session does not match the retrieved record",
suSe: mustNewSuSe(t, "", "session-a"),
store: func(t *testing.T) store.Store {
s := mocks.NewStore(t)
s.EXPECT().Read(mock.Anything, mock.Anything).Return([]*store.Record{
recordSessionB,
}, nil)
return s
},
wantRecords: []*store.Record{},
wantErrs: []error{ErrSuspiciousCacheResult}},
{
name: "fails if the subject does not match the retrieved record",
suSe: mustNewSuSe(t, "subject-a", ""),
store: func(t *testing.T) store.Store {
s := mocks.NewStore(t)
s.EXPECT().Read(mock.Anything, mock.Anything).Return([]*store.Record{
recordSessionB,
}, nil)
return s
},
wantRecords: []*store.Record{},
wantErrs: []error{ErrSuspiciousCacheResult}},
// key variation tests
{
name: "key variation: 'session-a'",
suSe: mustNewSuSe(t, "", "session-a"),
store: func(*testing.T) store.Store {
return sessionStore
},
wantRecords: []*store.Record{recordSessionA},
},
{
name: "key variation: 'session-b'",
suSe: mustNewSuSe(t, "", "session-b"),
store: func(*testing.T) store.Store {
return sessionStore
},
wantRecords: []*store.Record{recordSessionB},
},
{
name: "key variation: 'session-c'",
suSe: mustNewSuSe(t, "", "session-c"),
store: func(*testing.T) store.Store {
return sessionStore
},
wantRecords: []*store.Record{recordSubjectASessionC},
},
{
name: "key variation: 'ession-c'",
suSe: mustNewSuSe(t, "", "ession-c"),
store: func(*testing.T) store.Store {
return sessionStore
},
wantRecords: []*store.Record{},
wantErrs: []error{store.ErrNotFound},
},
{
name: "key variation: 'subject-a'",
suSe: mustNewSuSe(t, "subject-a", ""),
store: func(*testing.T) store.Store {
return sessionStore
},
wantRecords: []*store.Record{recordSubjectASessionC, recordSubjectASessionD},
},
{
name: "key variation: 'subject-'",
suSe: mustNewSuSe(t, "subject-", ""),
store: func(*testing.T) store.Store {
return sessionStore
},
wantRecords: []*store.Record{},
wantErrs: []error{store.ErrNotFound},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
records, err := GetLogoutRecords(tt.suSe, tt.store(t))
for _, wantErr := range tt.wantErrs {
require.ErrorIs(t, err, wantErr)
}
require.Len(t, records, len(tt.wantRecords))
sortRecords := func(r []*store.Record) []*store.Record {
slices.SortFunc(r, func(a, b *store.Record) int {
return strings.Compare(a.Key, b.Key)
})
return r
}
records = sortRecords(records)
for i, wantRecords := range sortRecords(tt.wantRecords) {
require.True(t, len(records) >= i+1)
require.Equal(t, wantRecords.Key, records[i].Key)
require.Equal(t, wantRecords.Value, records[i].Value)
}
})
}
}
@@ -0,0 +1,509 @@
// Code generated by mockery; DO NOT EDIT.
// github.com/vektra/mockery
// template: testify
package mocks
import (
mock "github.com/stretchr/testify/mock"
"go-micro.dev/v4/store"
)
// NewStore creates a new instance of Store. It also registers a testing interface on the mock and a cleanup function to assert the mocks expectations.
// The first argument is typically a *testing.T value.
func NewStore(t interface {
mock.TestingT
Cleanup(func())
}) *Store {
mock := &Store{}
mock.Mock.Test(t)
t.Cleanup(func() { mock.AssertExpectations(t) })
return mock
}
// Store is an autogenerated mock type for the Store type
type Store struct {
mock.Mock
}
type Store_Expecter struct {
mock *mock.Mock
}
func (_m *Store) EXPECT() *Store_Expecter {
return &Store_Expecter{mock: &_m.Mock}
}
// Close provides a mock function for the type Store
func (_mock *Store) Close() error {
ret := _mock.Called()
if len(ret) == 0 {
panic("no return value specified for Close")
}
var r0 error
if returnFunc, ok := ret.Get(0).(func() error); ok {
r0 = returnFunc()
} else {
r0 = ret.Error(0)
}
return r0
}
// Store_Close_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'Close'
type Store_Close_Call struct {
*mock.Call
}
// Close is a helper method to define mock.On call
func (_e *Store_Expecter) Close() *Store_Close_Call {
return &Store_Close_Call{Call: _e.mock.On("Close")}
}
func (_c *Store_Close_Call) Run(run func()) *Store_Close_Call {
_c.Call.Run(func(args mock.Arguments) {
run()
})
return _c
}
func (_c *Store_Close_Call) Return(err error) *Store_Close_Call {
_c.Call.Return(err)
return _c
}
func (_c *Store_Close_Call) RunAndReturn(run func() error) *Store_Close_Call {
_c.Call.Return(run)
return _c
}
// Delete provides a mock function for the type Store
func (_mock *Store) Delete(key string, opts ...store.DeleteOption) error {
var tmpRet mock.Arguments
if len(opts) > 0 {
tmpRet = _mock.Called(key, opts)
} else {
tmpRet = _mock.Called(key)
}
ret := tmpRet
if len(ret) == 0 {
panic("no return value specified for Delete")
}
var r0 error
if returnFunc, ok := ret.Get(0).(func(string, ...store.DeleteOption) error); ok {
r0 = returnFunc(key, opts...)
} else {
r0 = ret.Error(0)
}
return r0
}
// Store_Delete_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'Delete'
type Store_Delete_Call struct {
*mock.Call
}
// Delete is a helper method to define mock.On call
// - key string
// - opts ...store.DeleteOption
func (_e *Store_Expecter) Delete(key interface{}, opts ...interface{}) *Store_Delete_Call {
return &Store_Delete_Call{Call: _e.mock.On("Delete",
append([]interface{}{key}, opts...)...)}
}
func (_c *Store_Delete_Call) Run(run func(key string, opts ...store.DeleteOption)) *Store_Delete_Call {
_c.Call.Run(func(args mock.Arguments) {
var arg0 string
if args[0] != nil {
arg0 = args[0].(string)
}
var arg1 []store.DeleteOption
var variadicArgs []store.DeleteOption
if len(args) > 1 {
variadicArgs = args[1].([]store.DeleteOption)
}
arg1 = variadicArgs
run(
arg0,
arg1...,
)
})
return _c
}
func (_c *Store_Delete_Call) Return(err error) *Store_Delete_Call {
_c.Call.Return(err)
return _c
}
func (_c *Store_Delete_Call) RunAndReturn(run func(key string, opts ...store.DeleteOption) error) *Store_Delete_Call {
_c.Call.Return(run)
return _c
}
// Init provides a mock function for the type Store
func (_mock *Store) Init(options ...store.Option) error {
var tmpRet mock.Arguments
if len(options) > 0 {
tmpRet = _mock.Called(options)
} else {
tmpRet = _mock.Called()
}
ret := tmpRet
if len(ret) == 0 {
panic("no return value specified for Init")
}
var r0 error
if returnFunc, ok := ret.Get(0).(func(...store.Option) error); ok {
r0 = returnFunc(options...)
} else {
r0 = ret.Error(0)
}
return r0
}
// Store_Init_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'Init'
type Store_Init_Call struct {
*mock.Call
}
// Init is a helper method to define mock.On call
// - options ...store.Option
func (_e *Store_Expecter) Init(options ...interface{}) *Store_Init_Call {
return &Store_Init_Call{Call: _e.mock.On("Init",
append([]interface{}{}, options...)...)}
}
func (_c *Store_Init_Call) Run(run func(options ...store.Option)) *Store_Init_Call {
_c.Call.Run(func(args mock.Arguments) {
var arg0 []store.Option
var variadicArgs []store.Option
if len(args) > 0 {
variadicArgs = args[0].([]store.Option)
}
arg0 = variadicArgs
run(
arg0...,
)
})
return _c
}
func (_c *Store_Init_Call) Return(err error) *Store_Init_Call {
_c.Call.Return(err)
return _c
}
func (_c *Store_Init_Call) RunAndReturn(run func(options ...store.Option) error) *Store_Init_Call {
_c.Call.Return(run)
return _c
}
// List provides a mock function for the type Store
func (_mock *Store) List(opts ...store.ListOption) ([]string, error) {
var tmpRet mock.Arguments
if len(opts) > 0 {
tmpRet = _mock.Called(opts)
} else {
tmpRet = _mock.Called()
}
ret := tmpRet
if len(ret) == 0 {
panic("no return value specified for List")
}
var r0 []string
var r1 error
if returnFunc, ok := ret.Get(0).(func(...store.ListOption) ([]string, error)); ok {
return returnFunc(opts...)
}
if returnFunc, ok := ret.Get(0).(func(...store.ListOption) []string); ok {
r0 = returnFunc(opts...)
} else {
if ret.Get(0) != nil {
r0 = ret.Get(0).([]string)
}
}
if returnFunc, ok := ret.Get(1).(func(...store.ListOption) error); ok {
r1 = returnFunc(opts...)
} else {
r1 = ret.Error(1)
}
return r0, r1
}
// Store_List_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'List'
type Store_List_Call struct {
*mock.Call
}
// List is a helper method to define mock.On call
// - opts ...store.ListOption
func (_e *Store_Expecter) List(opts ...interface{}) *Store_List_Call {
return &Store_List_Call{Call: _e.mock.On("List",
append([]interface{}{}, opts...)...)}
}
func (_c *Store_List_Call) Run(run func(opts ...store.ListOption)) *Store_List_Call {
_c.Call.Run(func(args mock.Arguments) {
var arg0 []store.ListOption
var variadicArgs []store.ListOption
if len(args) > 0 {
variadicArgs = args[0].([]store.ListOption)
}
arg0 = variadicArgs
run(
arg0...,
)
})
return _c
}
func (_c *Store_List_Call) Return(strings []string, err error) *Store_List_Call {
_c.Call.Return(strings, err)
return _c
}
func (_c *Store_List_Call) RunAndReturn(run func(opts ...store.ListOption) ([]string, error)) *Store_List_Call {
_c.Call.Return(run)
return _c
}
// Options provides a mock function for the type Store
func (_mock *Store) Options() store.Options {
ret := _mock.Called()
if len(ret) == 0 {
panic("no return value specified for Options")
}
var r0 store.Options
if returnFunc, ok := ret.Get(0).(func() store.Options); ok {
r0 = returnFunc()
} else {
r0 = ret.Get(0).(store.Options)
}
return r0
}
// Store_Options_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'Options'
type Store_Options_Call struct {
*mock.Call
}
// Options is a helper method to define mock.On call
func (_e *Store_Expecter) Options() *Store_Options_Call {
return &Store_Options_Call{Call: _e.mock.On("Options")}
}
func (_c *Store_Options_Call) Run(run func()) *Store_Options_Call {
_c.Call.Run(func(args mock.Arguments) {
run()
})
return _c
}
func (_c *Store_Options_Call) Return(options store.Options) *Store_Options_Call {
_c.Call.Return(options)
return _c
}
func (_c *Store_Options_Call) RunAndReturn(run func() store.Options) *Store_Options_Call {
_c.Call.Return(run)
return _c
}
// Read provides a mock function for the type Store
func (_mock *Store) Read(key string, opts ...store.ReadOption) ([]*store.Record, error) {
var tmpRet mock.Arguments
if len(opts) > 0 {
tmpRet = _mock.Called(key, opts)
} else {
tmpRet = _mock.Called(key)
}
ret := tmpRet
if len(ret) == 0 {
panic("no return value specified for Read")
}
var r0 []*store.Record
var r1 error
if returnFunc, ok := ret.Get(0).(func(string, ...store.ReadOption) ([]*store.Record, error)); ok {
return returnFunc(key, opts...)
}
if returnFunc, ok := ret.Get(0).(func(string, ...store.ReadOption) []*store.Record); ok {
r0 = returnFunc(key, opts...)
} else {
if ret.Get(0) != nil {
r0 = ret.Get(0).([]*store.Record)
}
}
if returnFunc, ok := ret.Get(1).(func(string, ...store.ReadOption) error); ok {
r1 = returnFunc(key, opts...)
} else {
r1 = ret.Error(1)
}
return r0, r1
}
// Store_Read_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'Read'
type Store_Read_Call struct {
*mock.Call
}
// Read is a helper method to define mock.On call
// - key string
// - opts ...store.ReadOption
func (_e *Store_Expecter) Read(key interface{}, opts ...interface{}) *Store_Read_Call {
return &Store_Read_Call{Call: _e.mock.On("Read",
append([]interface{}{key}, opts...)...)}
}
func (_c *Store_Read_Call) Run(run func(key string, opts ...store.ReadOption)) *Store_Read_Call {
_c.Call.Run(func(args mock.Arguments) {
var arg0 string
if args[0] != nil {
arg0 = args[0].(string)
}
var arg1 []store.ReadOption
var variadicArgs []store.ReadOption
if len(args) > 1 {
variadicArgs = args[1].([]store.ReadOption)
}
arg1 = variadicArgs
run(
arg0,
arg1...,
)
})
return _c
}
func (_c *Store_Read_Call) Return(records []*store.Record, err error) *Store_Read_Call {
_c.Call.Return(records, err)
return _c
}
func (_c *Store_Read_Call) RunAndReturn(run func(key string, opts ...store.ReadOption) ([]*store.Record, error)) *Store_Read_Call {
_c.Call.Return(run)
return _c
}
// String provides a mock function for the type Store
func (_mock *Store) String() string {
ret := _mock.Called()
if len(ret) == 0 {
panic("no return value specified for String")
}
var r0 string
if returnFunc, ok := ret.Get(0).(func() string); ok {
r0 = returnFunc()
} else {
r0 = ret.Get(0).(string)
}
return r0
}
// Store_String_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'String'
type Store_String_Call struct {
*mock.Call
}
// String is a helper method to define mock.On call
func (_e *Store_Expecter) String() *Store_String_Call {
return &Store_String_Call{Call: _e.mock.On("String")}
}
func (_c *Store_String_Call) Run(run func()) *Store_String_Call {
_c.Call.Run(func(args mock.Arguments) {
run()
})
return _c
}
func (_c *Store_String_Call) Return(s string) *Store_String_Call {
_c.Call.Return(s)
return _c
}
func (_c *Store_String_Call) RunAndReturn(run func() string) *Store_String_Call {
_c.Call.Return(run)
return _c
}
// Write provides a mock function for the type Store
func (_mock *Store) Write(r *store.Record, opts ...store.WriteOption) error {
var tmpRet mock.Arguments
if len(opts) > 0 {
tmpRet = _mock.Called(r, opts)
} else {
tmpRet = _mock.Called(r)
}
ret := tmpRet
if len(ret) == 0 {
panic("no return value specified for Write")
}
var r0 error
if returnFunc, ok := ret.Get(0).(func(*store.Record, ...store.WriteOption) error); ok {
r0 = returnFunc(r, opts...)
} else {
r0 = ret.Error(0)
}
return r0
}
// Store_Write_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'Write'
type Store_Write_Call struct {
*mock.Call
}
// Write is a helper method to define mock.On call
// - r *store.Record
// - opts ...store.WriteOption
func (_e *Store_Expecter) Write(r interface{}, opts ...interface{}) *Store_Write_Call {
return &Store_Write_Call{Call: _e.mock.On("Write",
append([]interface{}{r}, opts...)...)}
}
func (_c *Store_Write_Call) Run(run func(r *store.Record, opts ...store.WriteOption)) *Store_Write_Call {
_c.Call.Run(func(args mock.Arguments) {
var arg0 *store.Record
if args[0] != nil {
arg0 = args[0].(*store.Record)
}
var arg1 []store.WriteOption
var variadicArgs []store.WriteOption
if len(args) > 1 {
variadicArgs = args[1].([]store.WriteOption)
}
arg1 = variadicArgs
run(
arg0,
arg1...,
)
})
return _c
}
func (_c *Store_Write_Call) Return(err error) *Store_Write_Call {
_c.Call.Return(err)
return _c
}
func (_c *Store_Write_Call) RunAndReturn(run func(r *store.Record, opts ...store.WriteOption) error) *Store_Write_Call {
_c.Call.Return(run)
return _c
}
+20
View File
@@ -71,6 +71,11 @@ func (b *Batch) Move(id, parentID, location string) error {
if err := b.batch.Index(resource.ID, resource); err != nil {
return err
}
if b.batch.Size() >= b.size {
if err := b.Push(); err != nil {
return err
}
}
}
return nil
@@ -88,6 +93,11 @@ func (b *Batch) Delete(id string) error {
if err := b.batch.Index(resource.ID, resource); err != nil {
return err
}
if b.batch.Size() >= b.size {
if err := b.Push(); err != nil {
return err
}
}
}
return nil
@@ -105,6 +115,11 @@ func (b *Batch) Restore(id string) error {
if err := b.batch.Index(resource.ID, resource); err != nil {
return err
}
if b.batch.Size() >= b.size {
if err := b.Push(); err != nil {
return err
}
}
}
return nil
@@ -142,6 +157,11 @@ func (b *Batch) Purge(id string, onlyDeleted bool) error {
for _, resource := range affectResources {
b.batch.Delete(resource.ID)
if b.batch.Size() >= b.size {
if err := b.Push(); err != nil {
return err
}
}
}
return nil
@@ -64,7 +64,7 @@ func DefaultConfig() *config.Config {
AckWait: 1 * time.Minute,
},
ContentExtractionSizeLimit: 20 * 1024 * 1024, // Limit content extraction to <20MB files by default
BatchSize: 500,
BatchSize: 50,
}
}
@@ -0,0 +1,102 @@
const fs = require("fs");
const CI_REPO_NAME = process.env.CI_REPO_NAME;
const CI_COMMIT_SHA = process.env.CI_COMMIT_SHA;
const CI_WORKFLOW_NAME = process.env.CI_WORKFLOW_NAME;
const CI_PIPELINE_EVENT = process.env.CI_PIPELINE_EVENT;
const opencloudBuildWorkflow = "build-opencloud-for-testing";
const webCacheWorkflows = ["cache-web", "cache-web-pnpm", "cache-browsers"];
const INFO_URL = `https://s3.ci.opencloud.eu/public/${CI_REPO_NAME}/pipelines/${CI_COMMIT_SHA}-${CI_PIPELINE_EVENT}/pipeline_info.json`;
function getWorkflowNames(workflows) {
const allWorkflows = [];
for (const workflow of workflows) {
allWorkflows.push(workflow.name);
}
return allWorkflows;
}
function getFailedWorkflows(workflows) {
const failedWorkflows = [];
for (const workflow of workflows) {
if (workflow.state !== "success") {
failedWorkflows.push(workflow.name);
}
}
return failedWorkflows;
}
function hasFailingTestWorkflow(failedWorkflows) {
for (const workflowName of failedWorkflows) {
if (workflowName.startsWith("test-")) {
return true;
}
}
return false;
}
function hasFailingE2eTestWorkflow(failedWorkflows) {
for (const workflowName of failedWorkflows) {
if (workflowName.startsWith("test-e2e-")) {
return true;
}
}
return false;
}
async function main() {
const infoResponse = await fetch(INFO_URL);
if (infoResponse.status === 404) {
console.log("[INFO] No matching previous pipeline found. Continue...");
process.exit(0);
} else if (!infoResponse.ok) {
console.error(
"[ERROR] Failed to fetch previous pipeline info:" +
`\n URL: ${INFO_URL}\n Status: ${infoResponse.status}`
);
process.exit(1);
}
const info = await infoResponse.json();
console.log(info);
if (info.status === "success") {
console.log(
"[INFO] All workflows passed in previous pipeline. Full restart. Continue..."
);
process.exit(0);
}
const allWorkflows = getWorkflowNames(info.workflows);
const failedWorkflows = getFailedWorkflows(info.workflows);
// NOTE: implement for test pipelines only for now
// // run the build workflow if any test workflow has failed
// if (
// CI_WORKFLOW_NAME === opencloudBuildWorkflow &&
// hasFailingTestWorkflow(failedWorkflows)
// ) {
// process.exit(0);
// }
// // run the web cache workflows if any e2e test workflow has failed
// if (
// webCacheWorkflows.includes(CI_WORKFLOW_NAME) &&
// hasFailingE2eTestWorkflow(failedWorkflows)
// ) {
// process.exit(0);
// }
if (!allWorkflows.includes(CI_WORKFLOW_NAME)) {
process.exit(0);
}
if (!failedWorkflows.includes(CI_WORKFLOW_NAME)) {
console.log("[INFO] Workflow passed in previous pipeline. Skip...");
fs.appendFileSync(".woodpecker.env", "SKIP_WORKFLOW=true\n");
process.exit(0);
}
console.log("[INFO] Restarting previously failed workflow. Continue...");
}
main();
@@ -0,0 +1,29 @@
#!/bin/bash
set -e
CACHE_KEY="$PUBLIC_BUCKET/$CI_REPO_NAME/pipelines/$CI_COMMIT_SHA-$CI_PIPELINE_EVENT"
mc alias set s3 $MC_HOST $AWS_ACCESS_KEY_ID $AWS_SECRET_ACCESS_KEY
# check previous pipeline
URL="https://s3.ci.opencloud.eu/$CACHE_KEY/prev_pipeline"
status=$(curl -s -o prev_pipeline "$URL" -w '%{http_code}')
if [ "$status" == "200" ];
then
source prev_pipeline
REPO_ID=$(printf '%s' "$CI_PIPELINE_URL" | sed 's|.*/repos/\([0-9]*\)/.*|\1|')
p_status=$(curl -s -o pipeline_info.json "$CI_SYSTEM_URL/api/repos/$REPO_ID/pipelines/$PREV_PIPELINE_NUMBER" -w "%{http_code}")
if [ "$p_status" != "200" ];
then
echo -e "[ERROR] Failed to fetch previous pipeline info.\n URL: $CI_SYSTEM_URL/api/repos/$REPO_ID/pipelines/$PREV_PIPELINE_NUMBER\n Status: $p_status"
exit 1
fi
# update previous pipeline info
mc cp -a pipeline_info.json "s3/$CACHE_KEY/"
fi
# upload current pipeline number for the next pipeline
echo "PREV_PIPELINE_NUMBER=$CI_PIPELINE_NUMBER" > prev_pipeline
mc cp -a prev_pipeline "s3/$CACHE_KEY/"
+2 -2
View File
@@ -1,7 +1,7 @@
{
"config": {
"platform": {
"php": "8.2"
"php": "8.4"
},
"allow-plugins": {
"composer/package-versions-deprecated": true
@@ -20,7 +20,7 @@
"guzzlehttp/guzzle": "^7.7",
"phpunit/phpunit": "^9.6",
"laminas/laminas-ldap": "^2.15",
"ankitpokhrel/tus-php": "^2.3",
"ankitpokhrel/tus-php": "^2.4",
"swaggest/json-schema": "^0.12.42",
"splitbrain/php-archive": "^1.3"
}
+65 -2
View File
@@ -17,13 +17,18 @@
// [test properties]: https://antithesis.com/docs/using_antithesis/properties/
// [workload]: https://antithesis.com/docs/getting_started/first_test/
// [antithesis-go-generator]: https://antithesis.com/docs/using_antithesis/sdk/go/instrumentor/
// [triage report]: https://antithesis.com/docs/reports/triage/
// [triage report]: https://antithesis.com/docs/reports/
// [here]: https://antithesis.com/docs/using_antithesis/sdk/fallback/
// [Sometimes assertions]: https://antithesis.com/docs/best_practices/sometimes_assertions/
//
// [details]: https://antithesis.com/docs/reports/triage/#details
// [details]: https://antithesis.com/docs/reports/
package assert
import (
"encoding/json"
"fmt"
)
type assertInfo struct {
Location *locationInfo `json:"location"`
Details map[string]any `json:"details"`
@@ -36,6 +41,64 @@ type assertInfo struct {
Condition bool `json:"condition"`
}
// Create a custom json marshaler for assertInfo so that we can force Errors to be marshaled with their error details.
// Without this, custom errors are marshaled as an empty object because the default json marshaler doesn't include the error
// (because it's a method - not an exported struct field).
func (f assertInfo) MarshalJSON() ([]byte, error) {
type alias assertInfo // prevent infinite recursion
a := alias(f)
if a.Details != nil {
a.Details = normalizeMap(a.Details)
}
return json.Marshal(a)
}
type jsonError struct {
innerError error
}
func (e jsonError) MarshalJSON() ([]byte, error) {
// Marshal this as the debug output string instead of e.Error(). These should be equivalent, but Sprintf correctly
// handles nil values for us (which otherwise are annoying to defend against due to this - https://go.dev/doc/faq#nil_error)
return json.Marshal(fmt.Sprintf("%+v", e.innerError))
}
// Recursively replace any `error` with jsonError while doing a deep copy.
// Most of the logic is in the normalize method below. This method exists to localize the type assertions
// and provide a function that takes in/out a map instead of any.
func normalizeMap(v map[string]any) map[string]any {
return normalize(v).(map[string]any)
}
func normalize(input any) any {
// This switch will miss some cases (pointers, structs, non-any types), but should catch a very large proportion of real error interfaces
// in real details objects. We can augment this if we find other cases common enough to support.
switch inputTyped := input.(type) {
case error:
// Check if the underlying error implements json.Marshaler, so that if the error
// already knows who to marshal itself, we don't override that.
if _, ok := inputTyped.(json.Marshaler); ok {
return inputTyped
} else {
return jsonError{inputTyped}
}
case map[string]any:
out := make(map[string]any, len(inputTyped))
for k, v := range inputTyped {
out[k] = normalize(v)
}
return out
case []any:
out := make([]any, len(inputTyped))
for i := range inputTyped {
out[i] = normalize(inputTyped[i])
}
return out
default:
return input
}
}
type wrappedAssertInfo struct {
A *assertInfo `json:"antithesis_assert"`
}
+4 -146
View File
@@ -4,67 +4,11 @@ package internal
import (
"encoding/json"
"fmt"
"log"
"math/rand"
"os"
"unsafe"
)
// --------------------------------------------------------------------------------
// To build and run an executable with this package
//
// CC=clang CGO_ENABLED=1 go run ./main.go
// --------------------------------------------------------------------------------
// \/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/
//
// The commented lines below, and the `import "C"` line which must directly follow
// the commented lines are used by CGO. They are load-bearing, and should not be
// changed without first understanding how CGO uses them.
//
// \/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/
// #cgo LDFLAGS: -ldl
//
// #include <dlfcn.h>
// #include <stdbool.h>
// #include <stdint.h>
// #include <stdlib.h>
//
// typedef void (*go_fuzz_json_data_fn)(const char *data, size_t size);
// void
// go_fuzz_json_data(void *f, const char *data, size_t size) {
// ((go_fuzz_json_data_fn)f)(data, size);
// }
//
// typedef void (*go_fuzz_flush_fn)(void);
// void
// go_fuzz_flush(void *f) {
// ((go_fuzz_flush_fn)f)();
// }
//
// typedef uint64_t (*go_fuzz_get_random_fn)(void);
// uint64_t
// go_fuzz_get_random(void *f) {
// return ((go_fuzz_get_random_fn)f)();
// }
//
// typedef bool (*go_notify_coverage_fn)(size_t);
// int
// go_notify_coverage(void *f, size_t edges) {
// bool b = ((go_notify_coverage_fn)f)(edges);
// return b ? 1 : 0;
// }
//
// typedef uint64_t (*go_init_coverage_fn)(size_t num_edges, const char *symbols);
// uint64_t
// go_init_coverage(void *f, size_t num_edges, const char *symbols) {
// return ((go_init_coverage_fn)f)(num_edges, symbols);
// }
//
import "C"
func Json_data(v any) error {
if data, err := json.Marshal(v); err != nil {
return err
@@ -95,45 +39,10 @@ type libHandler interface {
const (
errorLogLinePrefix = "[* antithesis-sdk-go *]"
defaultNativeLibraryPath = "/usr/lib/libvoidstar.so"
)
var handler libHandler
type voidstarHandler struct {
fuzzJsonData unsafe.Pointer
fuzzFlush unsafe.Pointer
fuzzGetRandom unsafe.Pointer
initCoverage unsafe.Pointer
notifyCoverage unsafe.Pointer
}
func (h *voidstarHandler) output(message string) {
msg_len := len(message)
if msg_len == 0 {
return
}
cstrMessage := C.CString(message)
defer C.free(unsafe.Pointer(cstrMessage))
C.go_fuzz_json_data(h.fuzzJsonData, cstrMessage, C.ulong(msg_len))
C.go_fuzz_flush(h.fuzzFlush)
}
func (h *voidstarHandler) random() uint64 {
return uint64(C.go_fuzz_get_random(h.fuzzGetRandom))
}
func (h *voidstarHandler) init_coverage(num_edge uint64, symbols string) uint64 {
cstrSymbols := C.CString(symbols)
defer C.free(unsafe.Pointer(cstrSymbols))
return uint64(C.go_init_coverage(h.initCoverage, C.ulong(num_edge), cstrSymbols))
}
func (h *voidstarHandler) notify(edge uint64) bool {
ival := int(C.go_notify_coverage(h.notifyCoverage, C.ulong(edge)))
return ival == 1
}
type localHandler struct {
outputFile *os.File // can be nil
}
@@ -160,63 +69,12 @@ func (h *localHandler) init_coverage(num_edges uint64, symbols string) uint64 {
return 0
}
// If we have a file at `defaultNativeLibraryPath`, we load the shared library
// (and panic on any error encountered during load).
// Otherwise fallback to the local handler.
func init() {
if _, err := os.Stat(defaultNativeLibraryPath); err == nil {
if handler, err = openSharedLib(defaultNativeLibraryPath); err != nil {
panic(err)
}
return
handler = init_in_antithesis()
if handler == nil {
// Otherwise fallback to the local handler.
handler = openLocalHandler()
}
handler = openLocalHandler()
}
// Attempt to load libvoidstar and some symbols from `path`
func openSharedLib(path string) (*voidstarHandler, error) {
cstrPath := C.CString(path)
defer C.free(unsafe.Pointer(cstrPath))
dlError := func(message string) error {
return fmt.Errorf("%s: (%s)", message, C.GoString(C.dlerror()))
}
sharedLib := C.dlopen(cstrPath, C.int(C.RTLD_NOW))
if sharedLib == nil {
return nil, dlError("Can not load the Antithesis native library")
}
loadFunc := func(name string) (symbol unsafe.Pointer, err error) {
cstrName := C.CString(name)
defer C.free(unsafe.Pointer(cstrName))
if symbol = C.dlsym(sharedLib, cstrName); symbol == nil {
err = dlError(fmt.Sprintf("Can not access symbol %s", name))
}
return
}
fuzzJsonData, err := loadFunc("fuzz_json_data")
if err != nil {
return nil, err
}
fuzzFlush, err := loadFunc("fuzz_flush")
if err != nil {
return nil, err
}
fuzzGetRandom, err := loadFunc("fuzz_get_random")
if err != nil {
return nil, err
}
notifyCoverage, err := loadFunc("notify_coverage")
if err != nil {
return nil, err
}
initCoverage, err := loadFunc("init_coverage_module")
if err != nil {
return nil, err
}
return &voidstarHandler{fuzzJsonData, fuzzFlush, fuzzGetRandom, initCoverage, notifyCoverage}, nil
}
// If `localOutputEnvVar` is set to a non-empty path, attempt to open that path and truncate the file
+1 -1
View File
@@ -3,7 +3,7 @@ package internal
// --------------------------------------------------------------------------------
// Versions
// --------------------------------------------------------------------------------
const SDK_Version = "0.4.3"
const SDK_Version = "0.6.0"
const Protocol_Version = "1.1.0"
// --------------------------------------------------------------------------------
@@ -0,0 +1,160 @@
//go:build enable_antithesis_sdk && linux && amd64 && cgo
package internal
import (
"fmt"
"unsafe"
"os"
)
// --------------------------------------------------------------------------------
// To build and run an executable with this package
//
// CC=clang CGO_ENABLED=1 go run ./main.go
// --------------------------------------------------------------------------------
// \/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/
//
// The commented lines below, and the `import "C"` line which must directly follow
// the commented lines are used by CGO. They are load-bearing, and should not be
// changed without first understanding how CGO uses them.
//
// \/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/
// #cgo LDFLAGS: -ldl
//
// #include <dlfcn.h>
// #include <stdbool.h>
// #include <stdint.h>
// #include <stdlib.h>
//
// typedef void (*go_fuzz_json_data_fn)(const char *data, size_t size);
// void
// go_fuzz_json_data(void *f, const char *data, size_t size) {
// ((go_fuzz_json_data_fn)f)(data, size);
// }
//
// typedef void (*go_fuzz_flush_fn)(void);
// void
// go_fuzz_flush(void *f) {
// ((go_fuzz_flush_fn)f)();
// }
//
// typedef uint64_t (*go_fuzz_get_random_fn)(void);
// uint64_t
// go_fuzz_get_random(void *f) {
// return ((go_fuzz_get_random_fn)f)();
// }
//
// typedef bool (*go_notify_coverage_fn)(size_t);
// int
// go_notify_coverage(void *f, size_t edges) {
// bool b = ((go_notify_coverage_fn)f)(edges);
// return b ? 1 : 0;
// }
//
// typedef uint64_t (*go_init_coverage_fn)(size_t num_edges, const char *symbols);
// uint64_t
// go_init_coverage(void *f, size_t num_edges, const char *symbols) {
// return ((go_init_coverage_fn)f)(num_edges, symbols);
// }
//
import "C"
const (
defaultNativeLibraryPath = "/usr/lib/libvoidstar.so"
)
type voidstarHandler struct {
fuzzJsonData unsafe.Pointer
fuzzFlush unsafe.Pointer
fuzzGetRandom unsafe.Pointer
initCoverage unsafe.Pointer
notifyCoverage unsafe.Pointer
}
func (h *voidstarHandler) output(message string) {
msg_len := len(message)
if msg_len == 0 {
return
}
cstrMessage := C.CString(message)
defer C.free(unsafe.Pointer(cstrMessage))
C.go_fuzz_json_data(h.fuzzJsonData, cstrMessage, C.ulong(msg_len))
C.go_fuzz_flush(h.fuzzFlush)
}
func (h *voidstarHandler) random() uint64 {
return uint64(C.go_fuzz_get_random(h.fuzzGetRandom))
}
func (h *voidstarHandler) init_coverage(num_edge uint64, symbols string) uint64 {
cstrSymbols := C.CString(symbols)
defer C.free(unsafe.Pointer(cstrSymbols))
return uint64(C.go_init_coverage(h.initCoverage, C.ulong(num_edge), cstrSymbols))
}
func (h *voidstarHandler) notify(edge uint64) bool {
ival := int(C.go_notify_coverage(h.notifyCoverage, C.ulong(edge)))
return ival == 1
}
// Attempt to load libvoidstar and some symbols from `path`
func openSharedLib(path string) (*voidstarHandler, error) {
cstrPath := C.CString(path)
defer C.free(unsafe.Pointer(cstrPath))
dlError := func(message string) error {
return fmt.Errorf("%s: (%s)", message, C.GoString(C.dlerror()))
}
sharedLib := C.dlopen(cstrPath, C.int(C.RTLD_NOW))
if sharedLib == nil {
return nil, dlError("Can not load the Antithesis native library")
}
loadFunc := func(name string) (symbol unsafe.Pointer, err error) {
cstrName := C.CString(name)
defer C.free(unsafe.Pointer(cstrName))
if symbol = C.dlsym(sharedLib, cstrName); symbol == nil {
err = dlError(fmt.Sprintf("Can not access symbol %s", name))
}
return
}
fuzzJsonData, err := loadFunc("fuzz_json_data")
if err != nil {
return nil, err
}
fuzzFlush, err := loadFunc("fuzz_flush")
if err != nil {
return nil, err
}
fuzzGetRandom, err := loadFunc("fuzz_get_random")
if err != nil {
return nil, err
}
notifyCoverage, err := loadFunc("notify_coverage")
if err != nil {
return nil, err
}
initCoverage, err := loadFunc("init_coverage_module")
if err != nil {
return nil, err
}
return &voidstarHandler{fuzzJsonData, fuzzFlush, fuzzGetRandom, initCoverage, notifyCoverage}, nil
}
// If we have a file at `defaultNativeLibraryPath`, we load the shared library
// (and panic on any error encountered during load).
func init_in_antithesis() libHandler {
if _, err := os.Stat(defaultNativeLibraryPath); err == nil {
handler, err := openSharedLib(defaultNativeLibraryPath)
if err != nil {
panic(err)
}
return handler
}
return nil
}
@@ -0,0 +1,7 @@
//go:build enable_antithesis_sdk && (!linux || !amd64 || !cgo)
package internal
func init_in_antithesis() libHandler {
return nil
}
+23
View File
@@ -0,0 +1,23 @@
# Compiled Object files, Static and Dynamic libs (Shared Objects)
*.o
*.a
*.so
# Folders
_obj
_test
# Architecture specific extensions/prefixes
*.[568vq]
[568vq].out
*.cgo1.go
*.cgo2.c
_cgo_defun.c
_cgo_gotypes.go
_cgo_export.*
_testmain.go
*.exe
*.test
+46
View File
@@ -0,0 +1,46 @@
# Copyright (c) HashiCorp, Inc.
# SPDX-License-Identifier: MPL-2.0
linters:
fast: false
disable-all: true
enable:
- revive
- megacheck
- govet
- unconvert
- gas
- gocyclo
- dupl
- misspell
- unparam
- unused
- typecheck
- ineffassign
# - stylecheck
- exportloopref
- gocritic
- nakedret
- gosimple
- prealloc
# golangci-lint configuration file
linters-settings:
revive:
ignore-generated-header: true
severity: warning
rules:
- name: package-comments
severity: warning
disabled: true
- name: exported
severity: warning
disabled: false
arguments: ["checkPrivateReceivers", "disableStutteringCheck"]
issues:
exclude-use-default: false
exclude-rules:
- path: _test\.go
linters:
- dupl
+267
View File
@@ -0,0 +1,267 @@
// Copyright (c) HashiCorp, Inc.
// SPDX-License-Identifier: MPL-2.0
package lru
import (
"errors"
"sync"
"github.com/hashicorp/golang-lru/v2/simplelru"
)
const (
// Default2QRecentRatio is the ratio of the 2Q cache dedicated
// to recently added entries that have only been accessed once.
Default2QRecentRatio = 0.25
// Default2QGhostEntries is the default ratio of ghost
// entries kept to track entries recently evicted
Default2QGhostEntries = 0.50
)
// TwoQueueCache is a thread-safe fixed size 2Q cache.
// 2Q is an enhancement over the standard LRU cache
// in that it tracks both frequently and recently used
// entries separately. This avoids a burst in access to new
// entries from evicting frequently used entries. It adds some
// additional tracking overhead to the standard LRU cache, and is
// computationally about 2x the cost, and adds some metadata over
// head. The ARCCache is similar, but does not require setting any
// parameters.
type TwoQueueCache[K comparable, V any] struct {
size int
recentSize int
recentRatio float64
ghostRatio float64
recent simplelru.LRUCache[K, V]
frequent simplelru.LRUCache[K, V]
recentEvict simplelru.LRUCache[K, struct{}]
lock sync.RWMutex
}
// New2Q creates a new TwoQueueCache using the default
// values for the parameters.
func New2Q[K comparable, V any](size int) (*TwoQueueCache[K, V], error) {
return New2QParams[K, V](size, Default2QRecentRatio, Default2QGhostEntries)
}
// New2QParams creates a new TwoQueueCache using the provided
// parameter values.
func New2QParams[K comparable, V any](size int, recentRatio, ghostRatio float64) (*TwoQueueCache[K, V], error) {
if size <= 0 {
return nil, errors.New("invalid size")
}
if recentRatio < 0.0 || recentRatio > 1.0 {
return nil, errors.New("invalid recent ratio")
}
if ghostRatio < 0.0 || ghostRatio > 1.0 {
return nil, errors.New("invalid ghost ratio")
}
// Determine the sub-sizes
recentSize := int(float64(size) * recentRatio)
evictSize := int(float64(size) * ghostRatio)
// Allocate the LRUs
recent, err := simplelru.NewLRU[K, V](size, nil)
if err != nil {
return nil, err
}
frequent, err := simplelru.NewLRU[K, V](size, nil)
if err != nil {
return nil, err
}
recentEvict, err := simplelru.NewLRU[K, struct{}](evictSize, nil)
if err != nil {
return nil, err
}
// Initialize the cache
c := &TwoQueueCache[K, V]{
size: size,
recentSize: recentSize,
recentRatio: recentRatio,
ghostRatio: ghostRatio,
recent: recent,
frequent: frequent,
recentEvict: recentEvict,
}
return c, nil
}
// Get looks up a key's value from the cache.
func (c *TwoQueueCache[K, V]) Get(key K) (value V, ok bool) {
c.lock.Lock()
defer c.lock.Unlock()
// Check if this is a frequent value
if val, ok := c.frequent.Get(key); ok {
return val, ok
}
// If the value is contained in recent, then we
// promote it to frequent
if val, ok := c.recent.Peek(key); ok {
c.recent.Remove(key)
c.frequent.Add(key, val)
return val, ok
}
// No hit
return
}
// Add adds a value to the cache.
func (c *TwoQueueCache[K, V]) Add(key K, value V) {
c.lock.Lock()
defer c.lock.Unlock()
// Check if the value is frequently used already,
// and just update the value
if c.frequent.Contains(key) {
c.frequent.Add(key, value)
return
}
// Check if the value is recently used, and promote
// the value into the frequent list
if c.recent.Contains(key) {
c.recent.Remove(key)
c.frequent.Add(key, value)
return
}
// If the value was recently evicted, add it to the
// frequently used list
if c.recentEvict.Contains(key) {
c.ensureSpace(true)
c.recentEvict.Remove(key)
c.frequent.Add(key, value)
return
}
// Add to the recently seen list
c.ensureSpace(false)
c.recent.Add(key, value)
}
// ensureSpace is used to ensure we have space in the cache
func (c *TwoQueueCache[K, V]) ensureSpace(recentEvict bool) {
// If we have space, nothing to do
recentLen := c.recent.Len()
freqLen := c.frequent.Len()
if recentLen+freqLen < c.size {
return
}
// If the recent buffer is larger than
// the target, evict from there
if recentLen > 0 && (recentLen > c.recentSize || (recentLen == c.recentSize && !recentEvict)) {
k, _, _ := c.recent.RemoveOldest()
c.recentEvict.Add(k, struct{}{})
return
}
// Remove from the frequent list otherwise
c.frequent.RemoveOldest()
}
// Len returns the number of items in the cache.
func (c *TwoQueueCache[K, V]) Len() int {
c.lock.RLock()
defer c.lock.RUnlock()
return c.recent.Len() + c.frequent.Len()
}
// Resize changes the cache size.
func (c *TwoQueueCache[K, V]) Resize(size int) (evicted int) {
c.lock.Lock()
defer c.lock.Unlock()
// Recalculate the sub-sizes
recentSize := int(float64(size) * c.recentRatio)
evictSize := int(float64(size) * c.ghostRatio)
c.size = size
c.recentSize = recentSize
// ensureSpace
diff := c.recent.Len() + c.frequent.Len() - size
if diff < 0 {
diff = 0
}
for i := 0; i < diff; i++ {
c.ensureSpace(true)
}
// Reallocate the LRUs
c.recent.Resize(size)
c.frequent.Resize(size)
c.recentEvict.Resize(evictSize)
return diff
}
// Keys returns a slice of the keys in the cache.
// The frequently used keys are first in the returned slice.
func (c *TwoQueueCache[K, V]) Keys() []K {
c.lock.RLock()
defer c.lock.RUnlock()
k1 := c.frequent.Keys()
k2 := c.recent.Keys()
return append(k1, k2...)
}
// Values returns a slice of the values in the cache.
// The frequently used values are first in the returned slice.
func (c *TwoQueueCache[K, V]) Values() []V {
c.lock.RLock()
defer c.lock.RUnlock()
v1 := c.frequent.Values()
v2 := c.recent.Values()
return append(v1, v2...)
}
// Remove removes the provided key from the cache.
func (c *TwoQueueCache[K, V]) Remove(key K) {
c.lock.Lock()
defer c.lock.Unlock()
if c.frequent.Remove(key) {
return
}
if c.recent.Remove(key) {
return
}
if c.recentEvict.Remove(key) {
return
}
}
// Purge is used to completely clear the cache.
func (c *TwoQueueCache[K, V]) Purge() {
c.lock.Lock()
defer c.lock.Unlock()
c.recent.Purge()
c.frequent.Purge()
c.recentEvict.Purge()
}
// Contains is used to check if the cache contains a key
// without updating recency or frequency.
func (c *TwoQueueCache[K, V]) Contains(key K) bool {
c.lock.RLock()
defer c.lock.RUnlock()
return c.frequent.Contains(key) || c.recent.Contains(key)
}
// Peek is used to inspect the cache value of a key
// without updating recency or frequency.
func (c *TwoQueueCache[K, V]) Peek(key K) (value V, ok bool) {
c.lock.RLock()
defer c.lock.RUnlock()
if val, ok := c.frequent.Peek(key); ok {
return val, ok
}
return c.recent.Peek(key)
}
+364
View File
@@ -0,0 +1,364 @@
Copyright (c) 2014 HashiCorp, Inc.
Mozilla Public License, version 2.0
1. Definitions
1.1. "Contributor"
means each individual or legal entity that creates, contributes to the
creation of, or owns Covered Software.
1.2. "Contributor Version"
means the combination of the Contributions of others (if any) used by a
Contributor and that particular Contributor's Contribution.
1.3. "Contribution"
means Covered Software of a particular Contributor.
1.4. "Covered Software"
means Source Code Form to which the initial Contributor has attached the
notice in Exhibit A, the Executable Form of such Source Code Form, and
Modifications of such Source Code Form, in each case including portions
thereof.
1.5. "Incompatible With Secondary Licenses"
means
a. that the initial Contributor has attached the notice described in
Exhibit B to the Covered Software; or
b. that the Covered Software was made available under the terms of
version 1.1 or earlier of the License, but not also under the terms of
a Secondary License.
1.6. "Executable Form"
means any form of the work other than Source Code Form.
1.7. "Larger Work"
means a work that combines Covered Software with other material, in a
separate file or files, that is not Covered Software.
1.8. "License"
means this document.
1.9. "Licensable"
means having the right to grant, to the maximum extent possible, whether
at the time of the initial grant or subsequently, any and all of the
rights conveyed by this License.
1.10. "Modifications"
means any of the following:
a. any file in Source Code Form that results from an addition to,
deletion from, or modification of the contents of Covered Software; or
b. any new file in Source Code Form that contains any Covered Software.
1.11. "Patent Claims" of a Contributor
means any patent claim(s), including without limitation, method,
process, and apparatus claims, in any patent Licensable by such
Contributor that would be infringed, but for the grant of the License,
by the making, using, selling, offering for sale, having made, import,
or transfer of either its Contributions or its Contributor Version.
1.12. "Secondary License"
means either the GNU General Public License, Version 2.0, the GNU Lesser
General Public License, Version 2.1, the GNU Affero General Public
License, Version 3.0, or any later versions of those licenses.
1.13. "Source Code Form"
means the form of the work preferred for making modifications.
1.14. "You" (or "Your")
means an individual or a legal entity exercising rights under this
License. For legal entities, "You" includes any entity that controls, is
controlled by, or is under common control with You. For purposes of this
definition, "control" means (a) the power, direct or indirect, to cause
the direction or management of such entity, whether by contract or
otherwise, or (b) ownership of more than fifty percent (50%) of the
outstanding shares or beneficial ownership of such entity.
2. License Grants and Conditions
2.1. Grants
Each Contributor hereby grants You a world-wide, royalty-free,
non-exclusive license:
a. under intellectual property rights (other than patent or trademark)
Licensable by such Contributor to use, reproduce, make available,
modify, display, perform, distribute, and otherwise exploit its
Contributions, either on an unmodified basis, with Modifications, or
as part of a Larger Work; and
b. under Patent Claims of such Contributor to make, use, sell, offer for
sale, have made, import, and otherwise transfer either its
Contributions or its Contributor Version.
2.2. Effective Date
The licenses granted in Section 2.1 with respect to any Contribution
become effective for each Contribution on the date the Contributor first
distributes such Contribution.
2.3. Limitations on Grant Scope
The licenses granted in this Section 2 are the only rights granted under
this License. No additional rights or licenses will be implied from the
distribution or licensing of Covered Software under this License.
Notwithstanding Section 2.1(b) above, no patent license is granted by a
Contributor:
a. for any code that a Contributor has removed from Covered Software; or
b. for infringements caused by: (i) Your and any other third party's
modifications of Covered Software, or (ii) the combination of its
Contributions with other software (except as part of its Contributor
Version); or
c. under Patent Claims infringed by Covered Software in the absence of
its Contributions.
This License does not grant any rights in the trademarks, service marks,
or logos of any Contributor (except as may be necessary to comply with
the notice requirements in Section 3.4).
2.4. Subsequent Licenses
No Contributor makes additional grants as a result of Your choice to
distribute the Covered Software under a subsequent version of this
License (see Section 10.2) or under the terms of a Secondary License (if
permitted under the terms of Section 3.3).
2.5. Representation
Each Contributor represents that the Contributor believes its
Contributions are its original creation(s) or it has sufficient rights to
grant the rights to its Contributions conveyed by this License.
2.6. Fair Use
This License is not intended to limit any rights You have under
applicable copyright doctrines of fair use, fair dealing, or other
equivalents.
2.7. Conditions
Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted in
Section 2.1.
3. Responsibilities
3.1. Distribution of Source Form
All distribution of Covered Software in Source Code Form, including any
Modifications that You create or to which You contribute, must be under
the terms of this License. You must inform recipients that the Source
Code Form of the Covered Software is governed by the terms of this
License, and how they can obtain a copy of this License. You may not
attempt to alter or restrict the recipients' rights in the Source Code
Form.
3.2. Distribution of Executable Form
If You distribute Covered Software in Executable Form then:
a. such Covered Software must also be made available in Source Code Form,
as described in Section 3.1, and You must inform recipients of the
Executable Form how they can obtain a copy of such Source Code Form by
reasonable means in a timely manner, at a charge no more than the cost
of distribution to the recipient; and
b. You may distribute such Executable Form under the terms of this
License, or sublicense it under different terms, provided that the
license for the Executable Form does not attempt to limit or alter the
recipients' rights in the Source Code Form under this License.
3.3. Distribution of a Larger Work
You may create and distribute a Larger Work under terms of Your choice,
provided that You also comply with the requirements of this License for
the Covered Software. If the Larger Work is a combination of Covered
Software with a work governed by one or more Secondary Licenses, and the
Covered Software is not Incompatible With Secondary Licenses, this
License permits You to additionally distribute such Covered Software
under the terms of such Secondary License(s), so that the recipient of
the Larger Work may, at their option, further distribute the Covered
Software under the terms of either this License or such Secondary
License(s).
3.4. Notices
You may not remove or alter the substance of any license notices
(including copyright notices, patent notices, disclaimers of warranty, or
limitations of liability) contained within the Source Code Form of the
Covered Software, except that You may alter any license notices to the
extent required to remedy known factual inaccuracies.
3.5. Application of Additional Terms
You may choose to offer, and to charge a fee for, warranty, support,
indemnity or liability obligations to one or more recipients of Covered
Software. However, You may do so only on Your own behalf, and not on
behalf of any Contributor. You must make it absolutely clear that any
such warranty, support, indemnity, or liability obligation is offered by
You alone, and You hereby agree to indemnify every Contributor for any
liability incurred by such Contributor as a result of warranty, support,
indemnity or liability terms You offer. You may include additional
disclaimers of warranty and limitations of liability specific to any
jurisdiction.
4. Inability to Comply Due to Statute or Regulation
If it is impossible for You to comply with any of the terms of this License
with respect to some or all of the Covered Software due to statute,
judicial order, or regulation then You must: (a) comply with the terms of
this License to the maximum extent possible; and (b) describe the
limitations and the code they affect. Such description must be placed in a
text file included with all distributions of the Covered Software under
this License. Except to the extent prohibited by statute or regulation,
such description must be sufficiently detailed for a recipient of ordinary
skill to be able to understand it.
5. Termination
5.1. The rights granted under this License will terminate automatically if You
fail to comply with any of its terms. However, if You become compliant,
then the rights granted under this License from a particular Contributor
are reinstated (a) provisionally, unless and until such Contributor
explicitly and finally terminates Your grants, and (b) on an ongoing
basis, if such Contributor fails to notify You of the non-compliance by
some reasonable means prior to 60 days after You have come back into
compliance. Moreover, Your grants from a particular Contributor are
reinstated on an ongoing basis if such Contributor notifies You of the
non-compliance by some reasonable means, this is the first time You have
received notice of non-compliance with this License from such
Contributor, and You become compliant prior to 30 days after Your receipt
of the notice.
5.2. If You initiate litigation against any entity by asserting a patent
infringement claim (excluding declaratory judgment actions,
counter-claims, and cross-claims) alleging that a Contributor Version
directly or indirectly infringes any patent, then the rights granted to
You by any and all Contributors for the Covered Software under Section
2.1 of this License shall terminate.
5.3. In the event of termination under Sections 5.1 or 5.2 above, all end user
license agreements (excluding distributors and resellers) which have been
validly granted by You or Your distributors under this License prior to
termination shall survive termination.
6. Disclaimer of Warranty
Covered Software is provided under this License on an "as is" basis,
without warranty of any kind, either expressed, implied, or statutory,
including, without limitation, warranties that the Covered Software is free
of defects, merchantable, fit for a particular purpose or non-infringing.
The entire risk as to the quality and performance of the Covered Software
is with You. Should any Covered Software prove defective in any respect,
You (not any Contributor) assume the cost of any necessary servicing,
repair, or correction. This disclaimer of warranty constitutes an essential
part of this License. No use of any Covered Software is authorized under
this License except under this disclaimer.
7. Limitation of Liability
Under no circumstances and under no legal theory, whether tort (including
negligence), contract, or otherwise, shall any Contributor, or anyone who
distributes Covered Software as permitted above, be liable to You for any
direct, indirect, special, incidental, or consequential damages of any
character including, without limitation, damages for lost profits, loss of
goodwill, work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses, even if such party shall have been
informed of the possibility of such damages. This limitation of liability
shall not apply to liability for death or personal injury resulting from
such party's negligence to the extent applicable law prohibits such
limitation. Some jurisdictions do not allow the exclusion or limitation of
incidental or consequential damages, so this exclusion and limitation may
not apply to You.
8. Litigation
Any litigation relating to this License may be brought only in the courts
of a jurisdiction where the defendant maintains its principal place of
business and such litigation shall be governed by laws of that
jurisdiction, without reference to its conflict-of-law provisions. Nothing
in this Section shall prevent a party's ability to bring cross-claims or
counter-claims.
9. Miscellaneous
This License represents the complete agreement concerning the subject
matter hereof. If any provision of this License is held to be
unenforceable, such provision shall be reformed only to the extent
necessary to make it enforceable. Any law or regulation which provides that
the language of a contract shall be construed against the drafter shall not
be used to construe this License against a Contributor.
10. Versions of the License
10.1. New Versions
Mozilla Foundation is the license steward. Except as provided in Section
10.3, no one other than the license steward has the right to modify or
publish new versions of this License. Each version will be given a
distinguishing version number.
10.2. Effect of New Versions
You may distribute the Covered Software under the terms of the version
of the License under which You originally received the Covered Software,
or under the terms of any subsequent version published by the license
steward.
10.3. Modified Versions
If you create software not governed by this License, and you want to
create a new license for such software, you may create and use a
modified version of this License if you rename the license and remove
any references to the name of the license steward (except to note that
such modified license differs from this License).
10.4. Distributing Source Code Form that is Incompatible With Secondary
Licenses If You choose to distribute Source Code Form that is
Incompatible With Secondary Licenses under the terms of this version of
the License, the notice described in Exhibit B of this License must be
attached.
Exhibit A - Source Code Form License Notice
This Source Code Form is subject to the
terms of the Mozilla Public License, v.
2.0. If a copy of the MPL was not
distributed with this file, You can
obtain one at
http://mozilla.org/MPL/2.0/.
If it is not possible or desirable to put the notice in a particular file,
then You may include the notice in a location (such as a LICENSE file in a
relevant directory) where a recipient would be likely to look for such a
notice.
You may add additional accurate notices of copyright ownership.
Exhibit B - "Incompatible With Secondary Licenses" Notice
This Source Code Form is "Incompatible
With Secondary Licenses", as defined by
the Mozilla Public License, v. 2.0.
+79
View File
@@ -0,0 +1,79 @@
golang-lru
==========
This provides the `lru` package which implements a fixed-size
thread safe LRU cache. It is based on the cache in Groupcache.
Documentation
=============
Full docs are available on [Go Packages](https://pkg.go.dev/github.com/hashicorp/golang-lru/v2)
LRU cache example
=================
```go
package main
import (
"fmt"
"github.com/hashicorp/golang-lru/v2"
)
func main() {
l, _ := lru.New[int, any](128)
for i := 0; i < 256; i++ {
l.Add(i, nil)
}
if l.Len() != 128 {
panic(fmt.Sprintf("bad len: %v", l.Len()))
}
}
```
Expirable LRU cache example
===========================
```go
package main
import (
"fmt"
"time"
"github.com/hashicorp/golang-lru/v2/expirable"
)
func main() {
// make cache with 10ms TTL and 5 max keys
cache := expirable.NewLRU[string, string](5, nil, time.Millisecond*10)
// set value under key1.
cache.Add("key1", "val1")
// get value under key1
r, ok := cache.Get("key1")
// check for OK value
if ok {
fmt.Printf("value before expiration is found: %v, value: %q\n", ok, r)
}
// wait for cache to expire
time.Sleep(time.Millisecond * 12)
// get value under key1 after key expiration
r, ok = cache.Get("key1")
fmt.Printf("value after expiration is found: %v, value: %q\n", ok, r)
// set value under key2, would evict old entry because it is already expired.
cache.Add("key2", "val2")
fmt.Printf("Cache len: %d\n", cache.Len())
// Output:
// value before expiration is found: true, value: "val1"
// value after expiration is found: false, value: ""
// Cache len: 1
}
```
+24
View File
@@ -0,0 +1,24 @@
// Copyright (c) HashiCorp, Inc.
// SPDX-License-Identifier: MPL-2.0
// Package lru provides three different LRU caches of varying sophistication.
//
// Cache is a simple LRU cache. It is based on the LRU implementation in
// groupcache: https://github.com/golang/groupcache/tree/master/lru
//
// TwoQueueCache tracks frequently used and recently used entries separately.
// This avoids a burst of accesses from taking out frequently used entries, at
// the cost of about 2x computational overhead and some extra bookkeeping.
//
// ARCCache is an adaptive replacement cache. It tracks recent evictions as well
// as recent usage in both the frequent and recent caches. Its computational
// overhead is comparable to TwoQueueCache, but the memory overhead is linear
// with the size of the cache.
//
// ARC has been patented by IBM, so do not use it if that is problematic for
// your program. For this reason, it is in a separate go module contained within
// this repository.
//
// All caches in this package take locks while operating, and are therefore
// thread-safe for consumers.
package lru
+142
View File
@@ -0,0 +1,142 @@
// Copyright 2009 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE_list file.
package internal
import "time"
// Entry is an LRU Entry
type Entry[K comparable, V any] struct {
// Next and previous pointers in the doubly-linked list of elements.
// To simplify the implementation, internally a list l is implemented
// as a ring, such that &l.root is both the next element of the last
// list element (l.Back()) and the previous element of the first list
// element (l.Front()).
next, prev *Entry[K, V]
// The list to which this element belongs.
list *LruList[K, V]
// The LRU Key of this element.
Key K
// The Value stored with this element.
Value V
// The time this element would be cleaned up, optional
ExpiresAt time.Time
// The expiry bucket item was put in, optional
ExpireBucket uint8
}
// PrevEntry returns the previous list element or nil.
func (e *Entry[K, V]) PrevEntry() *Entry[K, V] {
if p := e.prev; e.list != nil && p != &e.list.root {
return p
}
return nil
}
// LruList represents a doubly linked list.
// The zero Value for LruList is an empty list ready to use.
type LruList[K comparable, V any] struct {
root Entry[K, V] // sentinel list element, only &root, root.prev, and root.next are used
len int // current list Length excluding (this) sentinel element
}
// Init initializes or clears list l.
func (l *LruList[K, V]) Init() *LruList[K, V] {
l.root.next = &l.root
l.root.prev = &l.root
l.len = 0
return l
}
// NewList returns an initialized list.
func NewList[K comparable, V any]() *LruList[K, V] { return new(LruList[K, V]).Init() }
// Length returns the number of elements of list l.
// The complexity is O(1).
func (l *LruList[K, V]) Length() int { return l.len }
// Back returns the last element of list l or nil if the list is empty.
func (l *LruList[K, V]) Back() *Entry[K, V] {
if l.len == 0 {
return nil
}
return l.root.prev
}
// lazyInit lazily initializes a zero List Value.
func (l *LruList[K, V]) lazyInit() {
if l.root.next == nil {
l.Init()
}
}
// insert inserts e after at, increments l.len, and returns e.
func (l *LruList[K, V]) insert(e, at *Entry[K, V]) *Entry[K, V] {
e.prev = at
e.next = at.next
e.prev.next = e
e.next.prev = e
e.list = l
l.len++
return e
}
// insertValue is a convenience wrapper for insert(&Entry{Value: v, ExpiresAt: ExpiresAt}, at).
func (l *LruList[K, V]) insertValue(k K, v V, expiresAt time.Time, at *Entry[K, V]) *Entry[K, V] {
return l.insert(&Entry[K, V]{Value: v, Key: k, ExpiresAt: expiresAt}, at)
}
// Remove removes e from its list, decrements l.len
func (l *LruList[K, V]) Remove(e *Entry[K, V]) V {
e.prev.next = e.next
e.next.prev = e.prev
e.next = nil // avoid memory leaks
e.prev = nil // avoid memory leaks
e.list = nil
l.len--
return e.Value
}
// move moves e to next to at.
func (l *LruList[K, V]) move(e, at *Entry[K, V]) {
if e == at {
return
}
e.prev.next = e.next
e.next.prev = e.prev
e.prev = at
e.next = at.next
e.prev.next = e
e.next.prev = e
}
// PushFront inserts a new element e with value v at the front of list l and returns e.
func (l *LruList[K, V]) PushFront(k K, v V) *Entry[K, V] {
l.lazyInit()
return l.insertValue(k, v, time.Time{}, &l.root)
}
// PushFrontExpirable inserts a new expirable element e with Value v at the front of list l and returns e.
func (l *LruList[K, V]) PushFrontExpirable(k K, v V, expiresAt time.Time) *Entry[K, V] {
l.lazyInit()
return l.insertValue(k, v, expiresAt, &l.root)
}
// MoveToFront moves element e to the front of list l.
// If e is not an element of l, the list is not modified.
// The element must not be nil.
func (l *LruList[K, V]) MoveToFront(e *Entry[K, V]) {
if e.list != l || l.root.next == e {
return
}
// see comment in List.Remove about initialization of l
l.move(e, &l.root)
}
+250
View File
@@ -0,0 +1,250 @@
// Copyright (c) HashiCorp, Inc.
// SPDX-License-Identifier: MPL-2.0
package lru
import (
"sync"
"github.com/hashicorp/golang-lru/v2/simplelru"
)
const (
// DefaultEvictedBufferSize defines the default buffer size to store evicted key/val
DefaultEvictedBufferSize = 16
)
// Cache is a thread-safe fixed size LRU cache.
type Cache[K comparable, V any] struct {
lru *simplelru.LRU[K, V]
evictedKeys []K
evictedVals []V
onEvictedCB func(k K, v V)
lock sync.RWMutex
}
// New creates an LRU of the given size.
func New[K comparable, V any](size int) (*Cache[K, V], error) {
return NewWithEvict[K, V](size, nil)
}
// NewWithEvict constructs a fixed size cache with the given eviction
// callback.
func NewWithEvict[K comparable, V any](size int, onEvicted func(key K, value V)) (c *Cache[K, V], err error) {
// create a cache with default settings
c = &Cache[K, V]{
onEvictedCB: onEvicted,
}
if onEvicted != nil {
c.initEvictBuffers()
onEvicted = c.onEvicted
}
c.lru, err = simplelru.NewLRU(size, onEvicted)
return
}
func (c *Cache[K, V]) initEvictBuffers() {
c.evictedKeys = make([]K, 0, DefaultEvictedBufferSize)
c.evictedVals = make([]V, 0, DefaultEvictedBufferSize)
}
// onEvicted save evicted key/val and sent in externally registered callback
// outside of critical section
func (c *Cache[K, V]) onEvicted(k K, v V) {
c.evictedKeys = append(c.evictedKeys, k)
c.evictedVals = append(c.evictedVals, v)
}
// Purge is used to completely clear the cache.
func (c *Cache[K, V]) Purge() {
var ks []K
var vs []V
c.lock.Lock()
c.lru.Purge()
if c.onEvictedCB != nil && len(c.evictedKeys) > 0 {
ks, vs = c.evictedKeys, c.evictedVals
c.initEvictBuffers()
}
c.lock.Unlock()
// invoke callback outside of critical section
if c.onEvictedCB != nil {
for i := 0; i < len(ks); i++ {
c.onEvictedCB(ks[i], vs[i])
}
}
}
// Add adds a value to the cache. Returns true if an eviction occurred.
func (c *Cache[K, V]) Add(key K, value V) (evicted bool) {
var k K
var v V
c.lock.Lock()
evicted = c.lru.Add(key, value)
if c.onEvictedCB != nil && evicted {
k, v = c.evictedKeys[0], c.evictedVals[0]
c.evictedKeys, c.evictedVals = c.evictedKeys[:0], c.evictedVals[:0]
}
c.lock.Unlock()
if c.onEvictedCB != nil && evicted {
c.onEvictedCB(k, v)
}
return
}
// Get looks up a key's value from the cache.
func (c *Cache[K, V]) Get(key K) (value V, ok bool) {
c.lock.Lock()
value, ok = c.lru.Get(key)
c.lock.Unlock()
return value, ok
}
// Contains checks if a key is in the cache, without updating the
// recent-ness or deleting it for being stale.
func (c *Cache[K, V]) Contains(key K) bool {
c.lock.RLock()
containKey := c.lru.Contains(key)
c.lock.RUnlock()
return containKey
}
// Peek returns the key value (or undefined if not found) without updating
// the "recently used"-ness of the key.
func (c *Cache[K, V]) Peek(key K) (value V, ok bool) {
c.lock.RLock()
value, ok = c.lru.Peek(key)
c.lock.RUnlock()
return value, ok
}
// ContainsOrAdd checks if a key is in the cache without updating the
// recent-ness or deleting it for being stale, and if not, adds the value.
// Returns whether found and whether an eviction occurred.
func (c *Cache[K, V]) ContainsOrAdd(key K, value V) (ok, evicted bool) {
var k K
var v V
c.lock.Lock()
if c.lru.Contains(key) {
c.lock.Unlock()
return true, false
}
evicted = c.lru.Add(key, value)
if c.onEvictedCB != nil && evicted {
k, v = c.evictedKeys[0], c.evictedVals[0]
c.evictedKeys, c.evictedVals = c.evictedKeys[:0], c.evictedVals[:0]
}
c.lock.Unlock()
if c.onEvictedCB != nil && evicted {
c.onEvictedCB(k, v)
}
return false, evicted
}
// PeekOrAdd checks if a key is in the cache without updating the
// recent-ness or deleting it for being stale, and if not, adds the value.
// Returns whether found and whether an eviction occurred.
func (c *Cache[K, V]) PeekOrAdd(key K, value V) (previous V, ok, evicted bool) {
var k K
var v V
c.lock.Lock()
previous, ok = c.lru.Peek(key)
if ok {
c.lock.Unlock()
return previous, true, false
}
evicted = c.lru.Add(key, value)
if c.onEvictedCB != nil && evicted {
k, v = c.evictedKeys[0], c.evictedVals[0]
c.evictedKeys, c.evictedVals = c.evictedKeys[:0], c.evictedVals[:0]
}
c.lock.Unlock()
if c.onEvictedCB != nil && evicted {
c.onEvictedCB(k, v)
}
return
}
// Remove removes the provided key from the cache.
func (c *Cache[K, V]) Remove(key K) (present bool) {
var k K
var v V
c.lock.Lock()
present = c.lru.Remove(key)
if c.onEvictedCB != nil && present {
k, v = c.evictedKeys[0], c.evictedVals[0]
c.evictedKeys, c.evictedVals = c.evictedKeys[:0], c.evictedVals[:0]
}
c.lock.Unlock()
if c.onEvictedCB != nil && present {
c.onEvictedCB(k, v)
}
return
}
// Resize changes the cache size.
func (c *Cache[K, V]) Resize(size int) (evicted int) {
var ks []K
var vs []V
c.lock.Lock()
evicted = c.lru.Resize(size)
if c.onEvictedCB != nil && evicted > 0 {
ks, vs = c.evictedKeys, c.evictedVals
c.initEvictBuffers()
}
c.lock.Unlock()
if c.onEvictedCB != nil && evicted > 0 {
for i := 0; i < len(ks); i++ {
c.onEvictedCB(ks[i], vs[i])
}
}
return evicted
}
// RemoveOldest removes the oldest item from the cache.
func (c *Cache[K, V]) RemoveOldest() (key K, value V, ok bool) {
var k K
var v V
c.lock.Lock()
key, value, ok = c.lru.RemoveOldest()
if c.onEvictedCB != nil && ok {
k, v = c.evictedKeys[0], c.evictedVals[0]
c.evictedKeys, c.evictedVals = c.evictedKeys[:0], c.evictedVals[:0]
}
c.lock.Unlock()
if c.onEvictedCB != nil && ok {
c.onEvictedCB(k, v)
}
return
}
// GetOldest returns the oldest entry
func (c *Cache[K, V]) GetOldest() (key K, value V, ok bool) {
c.lock.RLock()
key, value, ok = c.lru.GetOldest()
c.lock.RUnlock()
return
}
// Keys returns a slice of the keys in the cache, from oldest to newest.
func (c *Cache[K, V]) Keys() []K {
c.lock.RLock()
keys := c.lru.Keys()
c.lock.RUnlock()
return keys
}
// Values returns a slice of the values in the cache, from oldest to newest.
func (c *Cache[K, V]) Values() []V {
c.lock.RLock()
values := c.lru.Values()
c.lock.RUnlock()
return values
}
// Len returns the number of items in the cache.
func (c *Cache[K, V]) Len() int {
c.lock.RLock()
length := c.lru.Len()
c.lock.RUnlock()
return length
}
+29
View File
@@ -0,0 +1,29 @@
This license applies to simplelru/list.go
Copyright (c) 2009 The Go Authors. All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
* Neither the name of Google Inc. nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+177
View File
@@ -0,0 +1,177 @@
// Copyright (c) HashiCorp, Inc.
// SPDX-License-Identifier: MPL-2.0
package simplelru
import (
"errors"
"github.com/hashicorp/golang-lru/v2/internal"
)
// EvictCallback is used to get a callback when a cache entry is evicted
type EvictCallback[K comparable, V any] func(key K, value V)
// LRU implements a non-thread safe fixed size LRU cache
type LRU[K comparable, V any] struct {
size int
evictList *internal.LruList[K, V]
items map[K]*internal.Entry[K, V]
onEvict EvictCallback[K, V]
}
// NewLRU constructs an LRU of the given size
func NewLRU[K comparable, V any](size int, onEvict EvictCallback[K, V]) (*LRU[K, V], error) {
if size <= 0 {
return nil, errors.New("must provide a positive size")
}
c := &LRU[K, V]{
size: size,
evictList: internal.NewList[K, V](),
items: make(map[K]*internal.Entry[K, V]),
onEvict: onEvict,
}
return c, nil
}
// Purge is used to completely clear the cache.
func (c *LRU[K, V]) Purge() {
for k, v := range c.items {
if c.onEvict != nil {
c.onEvict(k, v.Value)
}
delete(c.items, k)
}
c.evictList.Init()
}
// Add adds a value to the cache. Returns true if an eviction occurred.
func (c *LRU[K, V]) Add(key K, value V) (evicted bool) {
// Check for existing item
if ent, ok := c.items[key]; ok {
c.evictList.MoveToFront(ent)
ent.Value = value
return false
}
// Add new item
ent := c.evictList.PushFront(key, value)
c.items[key] = ent
evict := c.evictList.Length() > c.size
// Verify size not exceeded
if evict {
c.removeOldest()
}
return evict
}
// Get looks up a key's value from the cache.
func (c *LRU[K, V]) Get(key K) (value V, ok bool) {
if ent, ok := c.items[key]; ok {
c.evictList.MoveToFront(ent)
return ent.Value, true
}
return
}
// Contains checks if a key is in the cache, without updating the recent-ness
// or deleting it for being stale.
func (c *LRU[K, V]) Contains(key K) (ok bool) {
_, ok = c.items[key]
return ok
}
// Peek returns the key value (or undefined if not found) without updating
// the "recently used"-ness of the key.
func (c *LRU[K, V]) Peek(key K) (value V, ok bool) {
var ent *internal.Entry[K, V]
if ent, ok = c.items[key]; ok {
return ent.Value, true
}
return
}
// Remove removes the provided key from the cache, returning if the
// key was contained.
func (c *LRU[K, V]) Remove(key K) (present bool) {
if ent, ok := c.items[key]; ok {
c.removeElement(ent)
return true
}
return false
}
// RemoveOldest removes the oldest item from the cache.
func (c *LRU[K, V]) RemoveOldest() (key K, value V, ok bool) {
if ent := c.evictList.Back(); ent != nil {
c.removeElement(ent)
return ent.Key, ent.Value, true
}
return
}
// GetOldest returns the oldest entry
func (c *LRU[K, V]) GetOldest() (key K, value V, ok bool) {
if ent := c.evictList.Back(); ent != nil {
return ent.Key, ent.Value, true
}
return
}
// Keys returns a slice of the keys in the cache, from oldest to newest.
func (c *LRU[K, V]) Keys() []K {
keys := make([]K, c.evictList.Length())
i := 0
for ent := c.evictList.Back(); ent != nil; ent = ent.PrevEntry() {
keys[i] = ent.Key
i++
}
return keys
}
// Values returns a slice of the values in the cache, from oldest to newest.
func (c *LRU[K, V]) Values() []V {
values := make([]V, len(c.items))
i := 0
for ent := c.evictList.Back(); ent != nil; ent = ent.PrevEntry() {
values[i] = ent.Value
i++
}
return values
}
// Len returns the number of items in the cache.
func (c *LRU[K, V]) Len() int {
return c.evictList.Length()
}
// Resize changes the cache size.
func (c *LRU[K, V]) Resize(size int) (evicted int) {
diff := c.Len() - size
if diff < 0 {
diff = 0
}
for i := 0; i < diff; i++ {
c.removeOldest()
}
c.size = size
return diff
}
// removeOldest removes the oldest item from the cache.
func (c *LRU[K, V]) removeOldest() {
if ent := c.evictList.Back(); ent != nil {
c.removeElement(ent)
}
}
// removeElement is used to remove a given list element from the cache
func (c *LRU[K, V]) removeElement(e *internal.Entry[K, V]) {
c.evictList.Remove(e)
delete(c.items, e.Key)
if c.onEvict != nil {
c.onEvict(e.Key, e.Value)
}
}
+46
View File
@@ -0,0 +1,46 @@
// Copyright (c) HashiCorp, Inc.
// SPDX-License-Identifier: MPL-2.0
// Package simplelru provides simple LRU implementation based on build-in container/list.
package simplelru
// LRUCache is the interface for simple LRU cache.
type LRUCache[K comparable, V any] interface {
// Adds a value to the cache, returns true if an eviction occurred and
// updates the "recently used"-ness of the key.
Add(key K, value V) bool
// Returns key's value from the cache and
// updates the "recently used"-ness of the key. #value, isFound
Get(key K) (value V, ok bool)
// Checks if a key exists in cache without updating the recent-ness.
Contains(key K) (ok bool)
// Returns key's value without updating the "recently used"-ness of the key.
Peek(key K) (value V, ok bool)
// Removes a key from the cache.
Remove(key K) bool
// Removes the oldest entry from cache.
RemoveOldest() (K, V, bool)
// Returns the oldest entry from the cache. #key, value, isFound
GetOldest() (K, V, bool)
// Returns a slice of the keys in the cache, from oldest to newest.
Keys() []K
// Values returns a slice of the values in the cache, from oldest to newest.
Values() []V
// Returns the number of items in the cache.
Len() int
// Clears all cache entries.
Purge()
// Resizes cache, returning number evicted
Resize(int) int
}
+30 -8
View File
@@ -7,7 +7,7 @@ This package provides various compression algorithms.
* Optimized [deflate](https://godoc.org/github.com/klauspost/compress/flate) packages which can be used as a dropin replacement for [gzip](https://godoc.org/github.com/klauspost/compress/gzip), [zip](https://godoc.org/github.com/klauspost/compress/zip) and [zlib](https://godoc.org/github.com/klauspost/compress/zlib).
* [snappy](https://github.com/klauspost/compress/tree/master/snappy) is a drop-in replacement for `github.com/golang/snappy` offering better compression and concurrent streams.
* [huff0](https://github.com/klauspost/compress/tree/master/huff0) and [FSE](https://github.com/klauspost/compress/tree/master/fse) implementations for raw entropy encoding.
* [gzhttp](https://github.com/klauspost/compress/tree/master/gzhttp) Provides client and server wrappers for handling gzipped requests efficiently.
* [gzhttp](https://github.com/klauspost/compress/tree/master/gzhttp) Provides client and server wrappers for handling gzipped/zstd HTTP requests efficiently.
* [pgzip](https://github.com/klauspost/pgzip) is a separate package that provides a very fast parallel gzip implementation.
[![Go Reference](https://pkg.go.dev/badge/klauspost/compress.svg)](https://pkg.go.dev/github.com/klauspost/compress?tab=subdirectories)
@@ -26,6 +26,22 @@ This package will support the current Go version and 2 versions back.
Use the links above for more information on each.
# changelog
* Jan 16th, 2026 [1.18.3](https://github.com/klauspost/compress/releases/tag/v1.18.3)
* Downstream CVE-2025-61728. See [golang/go#77102](https://github.com/golang/go/issues/77102).
* Dec 1st, 2025 - [1.18.2](https://github.com/klauspost/compress/releases/tag/v1.18.2)
* flate: Fix invalid encoding on level 9 with single value input in https://github.com/klauspost/compress/pull/1115
* flate: reduce stateless allocations by @RXamzin in https://github.com/klauspost/compress/pull/1106
* Oct 20, 2025 - [1.18.1](https://github.com/klauspost/compress/releases/tag/v1.18.1) - RETRACTED
* zstd: Add simple zstd EncodeTo/DecodeTo functions https://github.com/klauspost/compress/pull/1079
* zstd: Fix incorrect buffer size in dictionary encodes https://github.com/klauspost/compress/pull/1059
* s2: check for cap, not len of buffer in EncodeBetter/Best by @vdarulis in https://github.com/klauspost/compress/pull/1080
* zlib: Avoiding extra allocation in zlib.reader.Reset by @travelpolicy in https://github.com/klauspost/compress/pull/1086
* gzhttp: remove redundant err check in zstdReader by @ryanfowler in https://github.com/klauspost/compress/pull/1090
* flate: Faster load+store https://github.com/klauspost/compress/pull/1104
* flate: Simplify matchlen https://github.com/klauspost/compress/pull/1101
* flate: Use exact sizes for huffman tables https://github.com/klauspost/compress/pull/1103
* Feb 19th, 2025 - [1.18.0](https://github.com/klauspost/compress/releases/tag/v1.18.0)
* Add unsafe little endian loaders https://github.com/klauspost/compress/pull/1036
@@ -36,6 +52,9 @@ Use the links above for more information on each.
* flate: Fix matchlen L5+L6 https://github.com/klauspost/compress/pull/1049
* flate: Cleanup & reduce casts https://github.com/klauspost/compress/pull/1050
<details>
<summary>See changes to v1.17.x</summary>
* Oct 11th, 2024 - [1.17.11](https://github.com/klauspost/compress/releases/tag/v1.17.11)
* zstd: Fix extra CRC written with multiple Close calls https://github.com/klauspost/compress/pull/1017
* s2: Don't use stack for index tables https://github.com/klauspost/compress/pull/1014
@@ -102,7 +121,8 @@ https://github.com/klauspost/compress/pull/919 https://github.com/klauspost/comp
* s2: Do 2 overlapping match checks https://github.com/klauspost/compress/pull/839
* flate: Add amd64 assembly matchlen https://github.com/klauspost/compress/pull/837
* gzip: Copy bufio.Reader on Reset by @thatguystone in https://github.com/klauspost/compress/pull/860
</details>
<details>
<summary>See changes to v1.16.x</summary>
@@ -589,7 +609,7 @@ While the release has been extensively tested, it is recommended to testing when
# deflate usage
The packages are drop-in replacements for standard libraries. Simply replace the import path to use them:
The packages are drop-in replacements for standard library [deflate](https://godoc.org/github.com/klauspost/compress/flate), [gzip](https://godoc.org/github.com/klauspost/compress/gzip), [zip](https://godoc.org/github.com/klauspost/compress/zip), and [zlib](https://godoc.org/github.com/klauspost/compress/zlib). Simply replace the import path to use them:
Typical speed is about 2x of the standard library packages.
@@ -600,17 +620,15 @@ Typical speed is about 2x of the standard library packages.
| `archive/zip` | `github.com/klauspost/compress/zip` | [zip](https://pkg.go.dev/github.com/klauspost/compress/zip?tab=doc) |
| `compress/flate` | `github.com/klauspost/compress/flate` | [flate](https://pkg.go.dev/github.com/klauspost/compress/flate?tab=doc) |
* Optimized [deflate](https://godoc.org/github.com/klauspost/compress/flate) packages which can be used as a dropin replacement for [gzip](https://godoc.org/github.com/klauspost/compress/gzip), [zip](https://godoc.org/github.com/klauspost/compress/zip) and [zlib](https://godoc.org/github.com/klauspost/compress/zlib).
You may also be interested in [pgzip](https://github.com/klauspost/pgzip), which is a drop-in replacement for gzip, which support multithreaded compression on big files and the optimized [crc32](https://github.com/klauspost/crc32) package used by these packages.
You may also be interested in [pgzip](https://github.com/klauspost/pgzip), which is a drop in replacement for gzip, which support multithreaded compression on big files and the optimized [crc32](https://github.com/klauspost/crc32) package used by these packages.
The packages contains the same as the standard library, so you can use the godoc for that: [gzip](http://golang.org/pkg/compress/gzip/), [zip](http://golang.org/pkg/archive/zip/), [zlib](http://golang.org/pkg/compress/zlib/), [flate](http://golang.org/pkg/compress/flate/).
The packages implement the same API as the standard library, so you can use the original godoc documentation: [gzip](http://golang.org/pkg/compress/gzip/), [zip](http://golang.org/pkg/archive/zip/), [zlib](http://golang.org/pkg/compress/zlib/), [flate](http://golang.org/pkg/compress/flate/).
Currently there is only minor speedup on decompression (mostly CRC32 calculation).
Memory usage is typically 1MB for a Writer. stdlib is in the same range.
If you expect to have a lot of concurrently allocated Writers consider using
the stateless compress described below.
the stateless compression described below.
For compression performance, see: [this spreadsheet](https://docs.google.com/spreadsheets/d/1nuNE2nPfuINCZJRMt6wFWhKpToF95I47XjSsc-1rbPQ/edit?usp=sharing).
@@ -669,3 +687,7 @@ Here are other packages of good quality and pure Go (no cgo wrappers or autoconv
# license
This code is licensed under the same conditions as the original Go code. See LICENSE file.
+3 -1
View File
@@ -421,7 +421,9 @@ func (d *compressor) deflateLazy() {
d.h = newHuffmanEncoder(maxFlateBlockTokens)
}
var tmp [256]uint16
for _, v := range d.window[s.index:d.windowEnd] {
toIndex := d.window[s.index:d.windowEnd]
toIndex = toIndex[:min(len(toIndex), maxFlateBlockTokens)]
for _, v := range toIndex {
tmp[v]++
}
d.h.generate(tmp[:], 15)
+5 -4
View File
@@ -646,7 +646,7 @@ func (w *huffmanBitWriter) writeBlockDynamic(tokens *tokens, eof bool, input []b
w.lastHeader = 0
}
numLiterals, numOffsets := w.indexTokens(tokens, fillReuse && !sync)
numLiterals, numOffsets := w.indexTokens(tokens, true)
extraBits := 0
ssize, storable := w.storedSize(input)
@@ -781,7 +781,7 @@ func (w *huffmanBitWriter) fillTokens() {
// literalFreq and offsetFreq, and generates literalEncoding
// and offsetEncoding.
// The number of literal and offset tokens is returned.
func (w *huffmanBitWriter) indexTokens(t *tokens, filled bool) (numLiterals, numOffsets int) {
func (w *huffmanBitWriter) indexTokens(t *tokens, alwaysEOB bool) (numLiterals, numOffsets int) {
//copy(w.literalFreq[:], t.litHist[:])
*(*[256]uint16)(w.literalFreq[:]) = t.litHist
//copy(w.literalFreq[256:], t.extraHist[:])
@@ -791,9 +791,10 @@ func (w *huffmanBitWriter) indexTokens(t *tokens, filled bool) (numLiterals, num
if t.n == 0 {
return
}
if filled {
return maxNumLit, maxNumDist
if alwaysEOB {
w.literalFreq[endBlockMarker] = 1
}
// get the number of literals
numLiterals = len(w.literalFreq)
for w.literalFreq[numLiterals-1] == 0 {
+16 -4
View File
@@ -61,13 +61,19 @@ var bitWriterPool = sync.Pool{
},
}
// tokensPool contains tokens struct objects that can be reused
var tokensPool = sync.Pool{
New: func() any {
return &tokens{}
},
}
// StatelessDeflate allows compressing directly to a Writer without retaining state.
// When returning everything will be flushed.
// Up to 8KB of an optional dictionary can be given which is presumed to precede the block.
// Longer dictionaries will be truncated and will still produce valid output.
// Sending nil dictionary is perfectly fine.
func StatelessDeflate(out io.Writer, in []byte, eof bool, dict []byte) error {
var dst tokens
bw := bitWriterPool.Get().(*huffmanBitWriter)
bw.reset(out)
defer func() {
@@ -91,6 +97,12 @@ func StatelessDeflate(out io.Writer, in []byte, eof bool, dict []byte) error {
// For subsequent loops, keep shallow dict reference to avoid alloc+copy.
var inDict []byte
dst := tokensPool.Get().(*tokens)
dst.Reset()
defer func() {
tokensPool.Put(dst)
}()
for len(in) > 0 {
todo := in
if len(inDict) > 0 {
@@ -113,9 +125,9 @@ func StatelessDeflate(out io.Writer, in []byte, eof bool, dict []byte) error {
}
// Compress
if len(inDict) == 0 {
statelessEnc(&dst, todo, int16(len(dict)))
statelessEnc(dst, todo, int16(len(dict)))
} else {
statelessEnc(&dst, inDict[:maxStatelessDict+len(todo)], maxStatelessDict)
statelessEnc(dst, inDict[:maxStatelessDict+len(todo)], maxStatelessDict)
}
isEof := eof && len(in) == 0
@@ -129,7 +141,7 @@ func StatelessDeflate(out io.Writer, in []byte, eof bool, dict []byte) error {
// If we removed less than 1/16th, huffman compress the block.
bw.writeBlockHuff(isEof, uncompressed, len(in) == 0)
} else {
bw.writeBlockDynamic(&dst, isEof, uncompressed, len(in) == 0)
bw.writeBlockDynamic(dst, isEof, uncompressed, len(in) == 0)
}
if len(in) > 0 {
// Retain a dict if we have more
+19 -9
View File
@@ -39,9 +39,6 @@ type Decoder struct {
frame *frameDec
// Custom dictionaries.
dicts map[uint32]*dict
// streamWg is the waitgroup for all streams
streamWg sync.WaitGroup
}
@@ -101,12 +98,10 @@ func NewReader(r io.Reader, opts ...DOption) (*Decoder, error) {
d.current.err = ErrDecoderNilInput
}
// Transfer option dicts.
d.dicts = make(map[uint32]*dict, len(d.o.dicts))
for _, dc := range d.o.dicts {
d.dicts[dc.id] = dc
// Initialize dict map if needed.
if d.o.dicts == nil {
d.o.dicts = make(map[uint32]*dict)
}
d.o.dicts = nil
// Create decoders
d.decoders = make(chan *blockDec, d.o.concurrent)
@@ -238,6 +233,21 @@ func (d *Decoder) Reset(r io.Reader) error {
return nil
}
// ResetWithOptions will reset the decoder and apply the given options
// for the next stream or DecodeAll operation.
// Options are applied on top of the existing options.
// Some options cannot be changed on reset and will return an error.
func (d *Decoder) ResetWithOptions(r io.Reader, opts ...DOption) error {
d.o.resetOpt = true
defer func() { d.o.resetOpt = false }()
for _, o := range opts {
if err := o(&d.o); err != nil {
return err
}
}
return d.Reset(r)
}
// drainOutput will drain the output until errEndOfStream is sent.
func (d *Decoder) drainOutput() {
if d.current.cancel != nil {
@@ -930,7 +940,7 @@ decodeStream:
}
func (d *Decoder) setDict(frame *frameDec) (err error) {
dict, ok := d.dicts[frame.DictionaryID]
dict, ok := d.o.dicts[frame.DictionaryID]
if ok {
if debugDecoder {
println("setting dict", frame.DictionaryID)
+52 -8
View File
@@ -20,10 +20,11 @@ type decoderOptions struct {
concurrent int
maxDecodedSize uint64
maxWindowSize uint64
dicts []*dict
dicts map[uint32]*dict
ignoreChecksum bool
limitToCap bool
decodeBufsBelow int
resetOpt bool
}
func (o *decoderOptions) setDefault() {
@@ -42,8 +43,15 @@ func (o *decoderOptions) setDefault() {
// WithDecoderLowmem will set whether to use a lower amount of memory,
// but possibly have to allocate more while running.
// Cannot be changed with ResetWithOptions.
func WithDecoderLowmem(b bool) DOption {
return func(o *decoderOptions) error { o.lowMem = b; return nil }
return func(o *decoderOptions) error {
if o.resetOpt && b != o.lowMem {
return errors.New("WithDecoderLowmem cannot be changed on Reset")
}
o.lowMem = b
return nil
}
}
// WithDecoderConcurrency sets the number of created decoders.
@@ -53,18 +61,23 @@ func WithDecoderLowmem(b bool) DOption {
// inflight blocks.
// When decoding streams and setting maximum to 1,
// no async decoding will be done.
// The value supplied must be at least 0.
// When a value of 0 is provided GOMAXPROCS will be used.
// By default this will be set to 4 or GOMAXPROCS, whatever is lower.
// Cannot be changed with ResetWithOptions.
func WithDecoderConcurrency(n int) DOption {
return func(o *decoderOptions) error {
if n < 0 {
return errors.New("concurrency must be at least 1")
return errors.New("concurrency must be at least 0")
}
newVal := n
if n == 0 {
o.concurrent = runtime.GOMAXPROCS(0)
} else {
o.concurrent = n
newVal = runtime.GOMAXPROCS(0)
}
if o.resetOpt && newVal != o.concurrent {
return errors.New("WithDecoderConcurrency cannot be changed on Reset")
}
o.concurrent = newVal
return nil
}
}
@@ -73,6 +86,7 @@ func WithDecoderConcurrency(n int) DOption {
// non-streaming operations or maximum window size for streaming operations.
// This can be used to control memory usage of potentially hostile content.
// Maximum is 1 << 63 bytes. Default is 64GiB.
// Can be changed with ResetWithOptions.
func WithDecoderMaxMemory(n uint64) DOption {
return func(o *decoderOptions) error {
if n == 0 {
@@ -92,16 +106,20 @@ func WithDecoderMaxMemory(n uint64) DOption {
// "zstd --train" from the Zstandard reference implementation.
//
// If several dictionaries with the same ID are provided, the last one will be used.
// Can be changed with ResetWithOptions.
//
// [dictionary format]: https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md#dictionary-format
func WithDecoderDicts(dicts ...[]byte) DOption {
return func(o *decoderOptions) error {
if o.dicts == nil {
o.dicts = make(map[uint32]*dict)
}
for _, b := range dicts {
d, err := loadDict(b)
if err != nil {
return err
}
o.dicts = append(o.dicts, d)
o.dicts[d.id] = d
}
return nil
}
@@ -109,12 +127,16 @@ func WithDecoderDicts(dicts ...[]byte) DOption {
// WithDecoderDictRaw registers a dictionary that may be used by the decoder.
// The slice content can be arbitrary data.
// Can be changed with ResetWithOptions.
func WithDecoderDictRaw(id uint32, content []byte) DOption {
return func(o *decoderOptions) error {
if bits.UintSize > 32 && uint(len(content)) > dictMaxLength {
return fmt.Errorf("dictionary of size %d > 2GiB too large", len(content))
}
o.dicts = append(o.dicts, &dict{id: id, content: content, offsets: [3]int{1, 4, 8}})
if o.dicts == nil {
o.dicts = make(map[uint32]*dict)
}
o.dicts[id] = &dict{id: id, content: content, offsets: [3]int{1, 4, 8}}
return nil
}
}
@@ -124,6 +146,7 @@ func WithDecoderDictRaw(id uint32, content []byte) DOption {
// The Decoder will likely allocate more memory based on the WithDecoderLowmem setting.
// If WithDecoderMaxMemory is set to a lower value, that will be used.
// Default is 512MB, Maximum is ~3.75 TB as per zstandard spec.
// Can be changed with ResetWithOptions.
func WithDecoderMaxWindow(size uint64) DOption {
return func(o *decoderOptions) error {
if size < MinWindowSize {
@@ -141,6 +164,7 @@ func WithDecoderMaxWindow(size uint64) DOption {
// or any size set in WithDecoderMaxMemory.
// This can be used to limit decoding to a specific maximum output size.
// Disabled by default.
// Can be changed with ResetWithOptions.
func WithDecodeAllCapLimit(b bool) DOption {
return func(o *decoderOptions) error {
o.limitToCap = b
@@ -153,17 +177,37 @@ func WithDecodeAllCapLimit(b bool) DOption {
// This typically uses less allocations but will have the full decompressed object in memory.
// Note that DecodeAllCapLimit will disable this, as well as giving a size of 0 or less.
// Default is 128KiB.
// Cannot be changed with ResetWithOptions.
func WithDecodeBuffersBelow(size int) DOption {
return func(o *decoderOptions) error {
if o.resetOpt && size != o.decodeBufsBelow {
return errors.New("WithDecodeBuffersBelow cannot be changed on Reset")
}
o.decodeBufsBelow = size
return nil
}
}
// IgnoreChecksum allows to forcibly ignore checksum checking.
// Can be changed with ResetWithOptions.
func IgnoreChecksum(b bool) DOption {
return func(o *decoderOptions) error {
o.ignoreChecksum = b
return nil
}
}
// WithDecoderDictDelete removes dictionaries by ID.
// If no ids are passed, all dictionaries are deleted.
// Should be used with ResetWithOptions.
func WithDecoderDictDelete(ids ...uint32) DOption {
return func(o *decoderOptions) error {
if len(ids) == 0 {
clear(o.dicts)
}
for _, id := range ids {
delete(o.dicts, id)
}
return nil
}
}
+16
View File
@@ -131,6 +131,22 @@ func (e *Encoder) Reset(w io.Writer) {
s.frameContentSize = 0
}
// ResetWithOptions will re-initialize the writer and apply the given options
// as a new, independent stream.
// Options are applied on top of the existing options.
// Some options cannot be changed on reset and will return an error.
func (e *Encoder) ResetWithOptions(w io.Writer, opts ...EOption) error {
e.o.resetOpt = true
defer func() { e.o.resetOpt = false }()
for _, o := range opts {
if err := o(&e.o); err != nil {
return err
}
}
e.Reset(w)
return nil
}
// ResetContentSize will reset and set a content size for the next stream.
// If the bytes written does not match the size given an error will be returned
// when calling Close().
+41 -3
View File
@@ -14,6 +14,7 @@ type EOption func(*encoderOptions) error
// options retains accumulated state of multiple options.
type encoderOptions struct {
resetOpt bool
concurrent int
level EncoderLevel
single *bool
@@ -71,19 +72,28 @@ func (o encoderOptions) encoder() encoder {
// WithEncoderCRC will add CRC value to output.
// Output will be 4 bytes larger.
// Can be changed with ResetWithOptions.
func WithEncoderCRC(b bool) EOption {
return func(o *encoderOptions) error { o.crc = b; return nil }
}
// WithEncoderConcurrency will set the concurrency,
// meaning the maximum number of encoders to run concurrently.
// The value supplied must be at least 1.
// The value supplied must be at least 0.
// When a value of 0 is provided GOMAXPROCS will be used.
// For streams, setting a value of 1 will disable async compression.
// By default this will be set to GOMAXPROCS.
// Cannot be changed with ResetWithOptions.
func WithEncoderConcurrency(n int) EOption {
return func(o *encoderOptions) error {
if n <= 0 {
return fmt.Errorf("concurrency must be at least 1")
if n < 0 {
return errors.New("concurrency must at least 0")
}
if n == 0 {
n = runtime.GOMAXPROCS(0)
}
if o.resetOpt && n != o.concurrent {
return errors.New("WithEncoderConcurrency cannot be changed on Reset")
}
o.concurrent = n
return nil
@@ -95,6 +105,7 @@ func WithEncoderConcurrency(n int) EOption {
// A larger value will enable better compression but allocate more memory and,
// for above-default values, take considerably longer.
// The default value is determined by the compression level and max 8MB.
// Cannot be changed with ResetWithOptions.
func WithWindowSize(n int) EOption {
return func(o *encoderOptions) error {
switch {
@@ -105,6 +116,9 @@ func WithWindowSize(n int) EOption {
case (n & (n - 1)) != 0:
return errors.New("window size must be a power of 2")
}
if o.resetOpt && n != o.windowSize {
return errors.New("WithWindowSize cannot be changed on Reset")
}
o.windowSize = n
o.customWindow = true
@@ -122,6 +136,7 @@ func WithWindowSize(n int) EOption {
// n must be > 0 and <= 1GB, 1<<30 bytes.
// The padded area will be filled with data from crypto/rand.Reader.
// If `EncodeAll` is used with data already in the destination, the total size will be multiple of this.
// Can be changed with ResetWithOptions.
func WithEncoderPadding(n int) EOption {
return func(o *encoderOptions) error {
if n <= 0 {
@@ -215,12 +230,16 @@ func (e EncoderLevel) String() string {
}
// WithEncoderLevel specifies a predefined compression level.
// Cannot be changed with ResetWithOptions.
func WithEncoderLevel(l EncoderLevel) EOption {
return func(o *encoderOptions) error {
switch {
case l <= speedNotSet || l >= speedLast:
return fmt.Errorf("unknown encoder level")
}
if o.resetOpt && l != o.level {
return errors.New("WithEncoderLevel cannot be changed on Reset")
}
o.level = l
if !o.customWindow {
switch o.level {
@@ -248,6 +267,7 @@ func WithEncoderLevel(l EncoderLevel) EOption {
// WithZeroFrames will encode 0 length input as full frames.
// This can be needed for compatibility with zstandard usage,
// but is not needed for this package.
// Can be changed with ResetWithOptions.
func WithZeroFrames(b bool) EOption {
return func(o *encoderOptions) error {
o.fullZero = b
@@ -259,6 +279,7 @@ func WithZeroFrames(b bool) EOption {
// Disabling this will skip incompressible data faster, but in cases with no matches but
// skewed character distribution compression is lost.
// Default value depends on the compression level selected.
// Can be changed with ResetWithOptions.
func WithAllLitEntropyCompression(b bool) EOption {
return func(o *encoderOptions) error {
o.customALEntropy = true
@@ -270,6 +291,7 @@ func WithAllLitEntropyCompression(b bool) EOption {
// WithNoEntropyCompression will always skip entropy compression of literals.
// This can be useful if content has matches, but unlikely to benefit from entropy
// compression. Usually the slight speed improvement is not worth enabling this.
// Can be changed with ResetWithOptions.
func WithNoEntropyCompression(b bool) EOption {
return func(o *encoderOptions) error {
o.noEntropy = b
@@ -287,6 +309,7 @@ func WithNoEntropyCompression(b bool) EOption {
// This is only a recommendation, each decoder is free to support higher or lower limits, depending on local limitations.
// If this is not specified, block encodes will automatically choose this based on the input size and the window size.
// This setting has no effect on streamed encodes.
// Can be changed with ResetWithOptions.
func WithSingleSegment(b bool) EOption {
return func(o *encoderOptions) error {
o.single = &b
@@ -298,8 +321,12 @@ func WithSingleSegment(b bool) EOption {
// slower encoding speed.
// This will not change the window size which is the primary function for reducing
// memory usage. See WithWindowSize.
// Cannot be changed with ResetWithOptions.
func WithLowerEncoderMem(b bool) EOption {
return func(o *encoderOptions) error {
if o.resetOpt && b != o.lowMem {
return errors.New("WithLowerEncoderMem cannot be changed on Reset")
}
o.lowMem = b
return nil
}
@@ -311,6 +338,7 @@ func WithLowerEncoderMem(b bool) EOption {
// "zstd --train" from the Zstandard reference implementation.
//
// The encoder *may* choose to use no dictionary instead for certain payloads.
// Can be changed with ResetWithOptions.
//
// [dictionary format]: https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md#dictionary-format
func WithEncoderDict(dict []byte) EOption {
@@ -328,6 +356,7 @@ func WithEncoderDict(dict []byte) EOption {
//
// The slice content may contain arbitrary data. It will be used as an initial
// history.
// Can be changed with ResetWithOptions.
func WithEncoderDictRaw(id uint32, content []byte) EOption {
return func(o *encoderOptions) error {
if bits.UintSize > 32 && uint(len(content)) > dictMaxLength {
@@ -337,3 +366,12 @@ func WithEncoderDictRaw(id uint32, content []byte) EOption {
return nil
}
}
// WithEncoderDictDelete clears the dictionary, so no dictionary will be used.
// Should be used with ResetWithOptions.
func WithEncoderDictDelete() EOption {
return func(o *encoderOptions) error {
o.dict = nil
return nil
}
}
+9 -6
View File
@@ -152,19 +152,22 @@ type Mapping map[Subject][]WeightedMapping
func (m *Mapping) Validate(vr *ValidationResults) {
for ubFrom, wm := range (map[Subject][]WeightedMapping)(*m) {
ubFrom.Validate(vr)
perCluster := make(map[string]uint8)
total := uint8(0)
perCluster := make(map[string]uint32)
total := uint32(0)
for _, e := range wm {
e.Subject.Validate(vr)
if e.GetWeight() > 100 {
vr.AddError("Mapping %q has a weight %d that exceeds 100", ubFrom, e.GetWeight())
}
if e.Cluster != "" {
t := perCluster[e.Cluster]
t += e.Weight
t += uint32(e.GetWeight())
perCluster[e.Cluster] = t
if t > 100 {
vr.AddError("Mapping %q in cluster %q exceeds 100%% among all of it's weighted to mappings", ubFrom, e.Cluster)
}
} else {
total += e.GetWeight()
total += uint32(e.GetWeight())
}
}
if total > 100 {
@@ -286,7 +289,7 @@ func (a *Account) Validate(acct *AccountClaims, vr *ValidationResults) {
tvr := CreateValidationResults()
a.Trace.Destination.Validate(tvr)
if !tvr.IsEmpty() {
vr.AddError(fmt.Sprintf("the account Trace.Destination %s", tvr.Issues[0].Description))
vr.AddError("the account Trace.Destination %s", tvr.Issues[0].Description)
}
if a.Trace.Destination.HasWildCards() {
vr.AddError("the account Trace.Destination subject %q is not a valid publish subject", a.Trace.Destination)
@@ -325,7 +328,7 @@ func (a *Account) Validate(acct *AccountClaims, vr *ValidationResults) {
a.Info.Validate(vr)
if err := a.ClusterTraffic.Valid(); err != nil {
vr.AddError(err.Error())
vr.AddError("%s", err.Error())
}
}
+15
View File
@@ -63,6 +63,9 @@ func formatJwt(kind string, jwtString string) ([]byte, error) {
func DecorateSeed(seed []byte) ([]byte, error) {
w := bytes.NewBuffer(nil)
ts := bytes.TrimSpace(seed)
if len(ts) < 2 {
return nil, errors.New("seed is too short")
}
pre := string(ts[0:2])
kind := ""
switch pre {
@@ -138,6 +141,18 @@ func FormatUserConfig(jwtString string, seed []byte) ([]byte, error) {
return nil, fmt.Errorf("nkey seed is not an user seed")
}
kp, err := nkeys.FromSeed(seed)
if err != nil {
return nil, err
}
pk, err := kp.PublicKey()
if err != nil {
return nil, err
}
if pk != gc.Claims().Subject {
return nil, fmt.Errorf("nkey seed does not match the jwt subject")
}
d, err := DecorateSeed(seed)
if err != nil {
return nil, err
+9
View File
@@ -26,6 +26,12 @@ import (
const libVersion = 2
// MaxTokenSize is the maximum size of a JWT token in bytes
const MaxTokenSize = 1024 * 1024 // 1MB
// ErrTokenTooLarge is returned when a token exceeds MaxTokenSize
var ErrTokenTooLarge = errors.New("token too large")
type identifier struct {
Type ClaimType `json:"type,omitempty"`
GenericFields `json:"nats,omitempty"`
@@ -56,6 +62,9 @@ type v1ClaimsDataDeletedFields struct {
// doesn't match the expected algorithm, or the claim is
// not valid or verification fails an error is returned.
func Decode(token string) (Claims, error) {
if len(token) > MaxTokenSize {
return nil, fmt.Errorf("token size %d exceeds maximum of %d bytes: %w", len(token), MaxTokenSize, ErrTokenTooLarge)
}
// must have 3 chunks
chunks := strings.Split(token, ".")
if len(chunks) != 3 {
+12 -7
View File
@@ -126,7 +126,8 @@ type Imports []*Import
// Validate checks if an import is valid for the wrapping account
func (i *Imports) Validate(acctPubKey string, vr *ValidationResults) {
toSet := make(map[Subject]struct{}, len(*i))
// Group subjects by account to check for overlaps only within the same account
subsByAcct := make(map[string]map[Subject]struct{}, len(*i))
for _, v := range *i {
if v == nil {
vr.AddError("null import is not allowed")
@@ -140,15 +141,19 @@ func (i *Imports) Validate(acctPubKey string, vr *ValidationResults) {
if sub == "" {
sub = v.Subject
}
for k := range toSet {
if sub.IsContainedIn(k) || k.IsContainedIn(sub) {
vr.AddError("overlapping subject namespace for %q and %q", sub, k)
// Check for overlapping subjects only within the same account
for subOther := range subsByAcct[v.Account] {
if sub.IsContainedIn(subOther) || subOther.IsContainedIn(sub) {
vr.AddError("overlapping subject namespace for %q and %q in same account %q", sub, subOther, v.Account)
}
}
if _, ok := toSet[sub]; ok {
vr.AddError("overlapping subject namespace for %q", v.To)
if subsByAcct[v.Account] == nil {
subsByAcct[v.Account] = make(map[Subject]struct{}, len(*i))
}
toSet[sub] = struct{}{}
if _, ok := subsByAcct[v.Account][sub]; ok {
vr.AddError("overlapping subject namespace for %q in account %q", sub, v.Account)
}
subsByAcct[v.Account][sub] = struct{}{}
}
v.Validate(acctPubKey, vr)
}
+2 -2
View File
@@ -71,12 +71,12 @@ func ParseServerVersion(version string) (int, int, int, error) {
// Validate checks the validity of the operators contents
func (o *Operator) Validate(vr *ValidationResults) {
if err := o.validateAccountServerURL(); err != nil {
vr.AddError(err.Error())
vr.AddError("%s", err.Error())
}
for _, v := range o.validateOperatorServiceURLs() {
if v != nil {
vr.AddError(v.Error())
vr.AddError("%s", v.Error())
}
}
+56 -39
View File
@@ -60,6 +60,9 @@ type parser struct {
// pedantic reports error when configuration is not correct.
pedantic bool
// Tracks environment variable references, to avoid cycles
envVarReferences map[string]bool
}
// Parse will return a map of keys to any, although concrete types
@@ -111,42 +114,28 @@ func ParseFileWithChecks(fp string) (map[string]any, error) {
return p.mapping, nil
}
// cleanupUsedEnvVars will recursively remove all already used
// environment variables which might be in the parsed tree.
func cleanupUsedEnvVars(m map[string]any) {
for k, v := range m {
t := v.(*token)
if t.usedVariable {
delete(m, k)
continue
}
// Cleanup any other env var that is still in the map.
if tm, ok := t.value.(map[string]any); ok {
cleanupUsedEnvVars(tm)
}
// configDigest returns a digest for the parsed config.
func configDigest(m map[string]any) (string, error) {
digest := sha256.New()
e := json.NewEncoder(digest)
if err := e.Encode(m); err != nil {
return _EMPTY_, err
}
return fmt.Sprintf("sha256:%x", digest.Sum(nil)), nil
}
// ParseFileWithChecksDigest returns the processed config and a digest
// that represents the configuration.
func ParseFileWithChecksDigest(fp string) (map[string]any, string, error) {
data, err := os.ReadFile(fp)
m, err := ParseFileWithChecks(fp)
if err != nil {
return nil, _EMPTY_, err
}
p, err := parse(string(data), fp, true)
digest, err := configDigest(m)
if err != nil {
return nil, _EMPTY_, err
}
// Filter out any environment variables before taking the digest.
cleanupUsedEnvVars(p.mapping)
digest := sha256.New()
e := json.NewEncoder(digest)
err = e.Encode(p.mapping)
if err != nil {
return nil, _EMPTY_, err
}
return p.mapping, fmt.Sprintf("sha256:%x", digest.Sum(nil)), nil
return m, digest, nil
}
type token struct {
@@ -180,16 +169,37 @@ func (t *token) Position() int {
return t.item.pos
}
func parse(data, fp string, pedantic bool) (p *parser, err error) {
p = &parser{
mapping: make(map[string]any),
lx: lex(data),
ctxs: make([]any, 0, 4),
keys: make([]string, 0, 4),
ikeys: make([]item, 0, 4),
fp: filepath.Dir(fp),
pedantic: pedantic,
func newParser(data, fp string, pedantic bool) *parser {
return &parser{
mapping: make(map[string]any),
lx: lex(data),
ctxs: make([]any, 0, 4),
keys: make([]string, 0, 4),
ikeys: make([]item, 0, 4),
fp: filepath.Dir(fp),
pedantic: pedantic,
envVarReferences: make(map[string]bool),
}
}
func parse(data, fp string, pedantic bool) (*parser, error) {
p := newParser(data, fp, pedantic)
if err := p.parse(fp); err != nil {
return nil, err
}
return p, nil
}
func parseEnv(data string, parent *parser) (*parser, error) {
p := newParser(data, "", false)
p.envVarReferences = parent.envVarReferences
if err := p.parse(""); err != nil {
return nil, err
}
return p, nil
}
func (p *parser) parse(fp string) error {
p.pushContext(p.mapping)
var prevItem item
@@ -199,16 +209,16 @@ func parse(data, fp string, pedantic bool) (p *parser, err error) {
// Here we allow the final character to be a bracket '}'
// in order to support JSON like configurations.
if prevItem.typ == itemKey && prevItem.val != mapEndString {
return nil, fmt.Errorf("config is invalid (%s:%d:%d)", fp, it.line, it.pos)
return fmt.Errorf("config is invalid (%s:%d:%d)", fp, it.line, it.pos)
}
break
}
prevItem = it
if err := p.processItem(it, fp); err != nil {
return nil, err
return err
}
}
return p, nil
return nil
}
func (p *parser) next() item {
@@ -453,11 +463,18 @@ func (p *parser) lookupVariable(varReference string) (any, bool, error) {
}
// If we are here, we have exhausted our context maps and still not found anything.
// Parse from the environment.
// Detect reference cycles
if p.envVarReferences[varReference] {
return nil, false, fmt.Errorf("variable reference cycle for '%s'", varReference)
}
p.envVarReferences[varReference] = true
defer delete(p.envVarReferences, varReference)
// Parse from the environment
if vStr, ok := os.LookupEnv(varReference); ok {
// Everything we get here will be a string value, so we need to process as a parser would.
if vmap, err := Parse(fmt.Sprintf("%s=%s", pkey, vStr)); err == nil {
v, ok := vmap[pkey]
if subp, err := parseEnv(fmt.Sprintf("%s=%s", pkey, vStr), p); err == nil {
v, ok := subp.mapping[pkey]
return v, ok, nil
} else {
return nil, false, err
+10 -10
View File
@@ -235,19 +235,19 @@ func (d *DN) RDNsMatch(other *DN) bool {
if len(d.RDNs) != len(other.RDNs) {
return false
}
CheckNextRDN:
matched := make([]bool, len(other.RDNs))
for _, irdn := range d.RDNs {
for _, ordn := range other.RDNs {
if (len(irdn.Attributes) == len(ordn.Attributes)) &&
(irdn.hasAllAttributes(ordn.Attributes) && ordn.hasAllAttributes(irdn.Attributes)) {
// Found the RDN, check if next one matches.
continue CheckNextRDN
found := false
for j, ordn := range other.RDNs {
if !matched[j] && irdn.Equal(ordn) {
matched[j] = true
found = true
break
}
}
// Could not find a matching individual RDN, auth fails.
return false
if !found {
return false
}
}
return true
}
+21 -10
View File
@@ -138,6 +138,12 @@ type sconns struct {
leafs int32
}
// clampInt64ToInt32 safely converts an int64 limit to int32,
// clamping values to the [math.MinInt32, math.MaxInt32] range.
func clampInt64ToInt32(v int64) int32 {
return int32(max(math.MinInt32, min(math.MaxInt32, v)))
}
// Import stream mapping struct
type streamImport struct {
acc *Account
@@ -299,6 +305,7 @@ func (a *Account) shallowCopy(na *Account) {
na.Nkey = a.Nkey
na.Issuer = a.Issuer
na.traceDest, na.traceDestSampling = a.traceDest, a.traceDestSampling
na.nrgAccount = a.nrgAccount
if a.imports.streams != nil {
na.imports.streams = make([]*streamImport, 0, len(a.imports.streams))
@@ -1609,10 +1616,12 @@ func (a *Account) checkServiceImportsForCycles(from string, visited map[string]b
}
// Push ourselves and check si.acc
visited[a.Name] = true
if subjectIsSubsetMatch(si.from, from) {
from = si.from
// Make a copy to not overwrite the passed value.
f := from
if subjectIsSubsetMatch(si.from, f) {
f = si.from
}
if err := si.acc.checkServiceImportsForCycles(from, visited); err != nil {
if err := si.acc.checkServiceImportsForCycles(f, visited); err != nil {
return err
}
a.mu.RLock()
@@ -1667,10 +1676,12 @@ func (a *Account) checkStreamImportsForCycles(to string, visited map[string]bool
}
// Push ourselves and check si.acc
visited[a.Name] = true
if subjectIsSubsetMatch(si.to, to) {
to = si.to
// Make a copy to not overwrite the passed value.
t := to
if subjectIsSubsetMatch(si.to, t) {
t = si.to
}
if err := si.acc.checkStreamImportsForCycles(to, visited); err != nil {
if err := si.acc.checkStreamImportsForCycles(t, visited); err != nil {
return err
}
a.mu.RLock()
@@ -3715,10 +3726,10 @@ func (s *Server) updateAccountClaimsWithRefresh(a *Account, ac *jwt.AccountClaim
// Now do limits if they are present.
a.mu.Lock()
a.msubs = int32(ac.Limits.Subs)
a.mpay = int32(ac.Limits.Payload)
a.mconns = int32(ac.Limits.Conn)
a.mleafs = int32(ac.Limits.LeafNodeConn)
a.msubs = clampInt64ToInt32(ac.Limits.Subs)
a.mpay = clampInt64ToInt32(ac.Limits.Payload)
a.mconns = clampInt64ToInt32(ac.Limits.Conn)
a.mleafs = clampInt64ToInt32(ac.Limits.LeafNodeConn)
a.disallowBearer = ac.Limits.DisallowBearer
// Check for any revocations
if len(ac.Revocations) > 0 {
+37 -8
View File
@@ -421,7 +421,9 @@ func (c *client) matchesPinnedCert(tlsPinnedCerts PinnedCertSet) bool {
}
var (
mustacheRE = regexp.MustCompile(`{{2}([^}]+)}{2}`)
mustacheRE = regexp.MustCompile(`{{2}([^}]+)}{2}`)
maxPermTemplateSubjectExpansions = 4096
errPermTemplateExpansionLimit error = fmt.Errorf("template expansion exceeds limit")
)
func processUserPermissionsTemplate(lim jwt.UserPermissionLimits, ujwt *jwt.UserClaims, acc *Account) (jwt.UserPermissionLimits, error) {
@@ -456,11 +458,11 @@ func processUserPermissionsTemplate(lim jwt.UserPermissionLimits, ujwt *jwt.User
return p
}
isTag := func(op string) []string {
if strings.EqualFold("tag(", op[:4]) && strings.HasSuffix(op, ")") {
if len(op) >= 4 && strings.EqualFold("tag(", op[:4]) && strings.HasSuffix(op, ")") {
v := strings.TrimPrefix(op, "tag(")
v = strings.TrimSuffix(v, ")")
return []string{"tag", v}
} else if strings.EqualFold("account-tag(", op[:12]) && strings.HasSuffix(op, ")") {
} else if len(op) >= 12 && strings.EqualFold("account-tag(", op[:12]) && strings.HasSuffix(op, ")") {
v := strings.TrimPrefix(op, "account-tag(")
v = strings.TrimSuffix(v, ")")
return []string{"account-tag", v}
@@ -529,7 +531,7 @@ func processUserPermissionsTemplate(lim jwt.UserPermissionLimits, ujwt *jwt.User
// generate an invalid subject?
values[tokenNum] = []string{" "}
}
} else if failOnBadSubject {
} else {
return nil, fmt.Errorf("template operation in %q: %q is not defined", list[i], op)
}
}
@@ -544,6 +546,20 @@ func processUserPermissionsTemplate(lim jwt.UserPermissionLimits, ujwt *jwt.User
return nil, fmt.Errorf("generated invalid subject")
}
} else {
expCount := 1
for _, v := range values {
if len(v) == 0 {
expCount = 0
break
}
if expCount > maxPermTemplateSubjectExpansions/len(v) {
return nil, fmt.Errorf("%w: %d", errPermTemplateExpansionLimit, maxPermTemplateSubjectExpansions)
}
expCount *= len(v)
}
if len(emittedList) > maxPermTemplateSubjectExpansions-expCount {
return nil, fmt.Errorf("%w: %d", errPermTemplateExpansionLimit, maxPermTemplateSubjectExpansions)
}
a := nArrayCartesianProduct(values...)
for _, aa := range a {
subj := list[i]
@@ -588,6 +604,7 @@ func processUserPermissionsTemplate(lim jwt.UserPermissionLimits, ujwt *jwt.User
func (s *Server) processClientOrLeafAuthentication(c *client, opts *Options) (authorized bool) {
var (
nkey *NkeyUser
ujwt string
juc *jwt.UserClaims
acc *Account
user *User
@@ -782,16 +799,23 @@ func (s *Server) processClientOrLeafAuthentication(c *client, opts *Options) (au
// Check if we have trustedKeys defined in the server. If so we require a user jwt.
if s.trustedKeys != nil {
if c.opts.JWT == _EMPTY_ && opts.DefaultSentinel != _EMPTY_ {
c.opts.JWT = opts.DefaultSentinel
ujwt = c.opts.JWT
if ujwt == _EMPTY_ && c.isMqtt() {
// For MQTT, we pass the password as the JWT too, but do so here so it's not
// publicly exposed in the client options if it isn't a JWT.
ujwt = c.opts.Password
}
if c.opts.JWT == _EMPTY_ {
if ujwt == _EMPTY_ && opts.DefaultSentinel != _EMPTY_ {
c.opts.JWT = opts.DefaultSentinel
ujwt = c.opts.JWT
}
if ujwt == _EMPTY_ {
s.mu.Unlock()
c.Debugf("Authentication requires a user JWT")
return false
}
// So we have a valid user jwt here.
juc, err = jwt.DecodeUserClaims(c.opts.JWT)
juc, err = jwt.DecodeUserClaims(ujwt)
if err != nil {
s.mu.Unlock()
c.Debugf("User JWT not valid: %v", err)
@@ -1061,6 +1085,11 @@ func (s *Server) processClientOrLeafAuthentication(c *client, opts *Options) (au
// Hold onto the user's public key.
c.mu.Lock()
c.pubKey = juc.Subject
// If this is a MQTT client, we purposefully didn't populate the JWT as it could contain
// a password or token. Now we know it's a valid JWT, we can populate it.
if c.isMqtt() {
c.opts.JWT = ujwt
}
c.tags = juc.Tags
c.nameTag = juc.Name
c.mu.Unlock()
+15 -8
View File
@@ -32,6 +32,14 @@ const (
AuthRequestXKeyHeader = "Nats-Server-Xkey"
)
func titleCase(m string) string {
r := []rune(m)
if len(r) == 0 {
return _EMPTY_
}
return string(append([]rune{unicode.ToUpper(r[0])}, r[1:]...))
}
// Process a callout on this client's behalf.
func (s *Server) processClientOrLeafCallout(c *client, opts *Options, proxyRequired, trustedProxy bool) (authorized bool, errStr string) {
isOperatorMode := len(opts.TrustedKeys) > 0
@@ -50,6 +58,13 @@ func (s *Server) processClientOrLeafCallout(c *client, opts *Options, proxyRequi
} else {
acc = c.acc
}
if acc == nil {
// FIX for https://github.com/nats-io/nats-server/issues/7841
// hand rolled creds on leafnode became crasher here
errStr = fmt.Sprintf("%s not mapped to a callout account", c.kindString())
s.Warnf(errStr)
return false, errStr
}
// Check if we have been requested to encrypt.
var xkp nkeys.KeyPair
@@ -66,9 +81,6 @@ func (s *Server) processClientOrLeafCallout(c *client, opts *Options, proxyRequi
xkp, xkey = s.xkp, s.info.XKey
}
// FIXME: so things like the server ID that get assigned, are used as a sort of nonce - but
// reality is that the keypair here, is generated, so the response generated a JWT has to be
// this user - no replay possible
// Create a keypair for the user. We will expect this public user to be in the signed response.
// This prevents replay attacks.
ukp, _ := nkeys.CreateUser()
@@ -234,11 +246,6 @@ func (s *Server) processClientOrLeafCallout(c *client, opts *Options, proxyRequi
}
processReply := func(_ *subscription, rc *client, racc *Account, subject, reply string, rmsg []byte) {
titleCase := func(m string) string {
r := []rune(m)
return string(append([]rune{unicode.ToUpper(r[0])}, r[1:]...))
}
arc, err := decodeResponse(rc, rmsg, racc)
if err != nil {
c.authViolation()
+1 -1
View File
@@ -1,4 +1,4 @@
// Copyright 2023-2024 The NATS Authors
// Copyright 2023-2025 The NATS Authors
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
+254 -116
View File
@@ -1,4 +1,4 @@
// Copyright 2012-2025 The NATS Authors
// Copyright 2012-2026 The NATS Authors
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
@@ -23,6 +23,7 @@ import (
"errors"
"fmt"
"io"
"math"
"math/rand"
"net"
"net/http"
@@ -35,8 +36,6 @@ import (
"sync/atomic"
"time"
"slices"
"github.com/klauspost/compress/s2"
"github.com/nats-io/jwt/v2"
"github.com/nats-io/nats-server/v2/internal/fastrand"
@@ -154,7 +153,6 @@ const (
compressionNegotiated // Marks if this connection has negotiated compression level with remote.
didTLSFirst // Marks if this connection requested and was accepted doing the TLS handshake first (prior to INFO).
isSlowConsumer // Marks connection as a slow consumer.
firstPong // Marks if this is the first PONG received
)
// set the flag (would be equivalent to set the boolean to true)
@@ -870,6 +868,11 @@ func (c *client) registerWithAccount(acc *Account) error {
}
c.mu.Lock()
// This check does not apply to SYSTEM or JETSTREAM or ACCOUNT clients (because they don't have a `nc`...)
if c.isClosed() && !isInternalClient(c.kind) {
c.mu.Unlock()
return ErrConnectionClosed
}
kind := c.kind
srv := c.srv
c.acc = acc
@@ -929,14 +932,14 @@ func (c *client) applyAccountLimits() {
c.msubs = jwt.NoLimit
if c.opts.JWT != _EMPTY_ { // user jwt implies account
if uc, _ := jwt.DecodeUserClaims(c.opts.JWT); uc != nil {
atomic.StoreInt32(&c.mpay, int32(uc.Limits.Payload))
c.msubs = int32(uc.Limits.Subs)
atomic.StoreInt32(&c.mpay, clampInt64ToInt32(uc.Limits.Payload))
c.msubs = clampInt64ToInt32(uc.Limits.Subs)
if uc.IssuerAccount != _EMPTY_ && uc.IssuerAccount != uc.Issuer {
if scope, ok := c.acc.signingKeys[uc.Issuer]; ok {
if userScope, ok := scope.(*jwt.UserScope); ok {
// if signing key disappeared or changed and we don't get here, the client will be disconnected
c.mpay = int32(userScope.Template.Limits.Payload)
c.msubs = int32(userScope.Template.Limits.Subs)
c.mpay = clampInt64ToInt32(userScope.Template.Limits.Payload)
c.msubs = clampInt64ToInt32(userScope.Template.Limits.Subs)
}
}
}
@@ -1354,6 +1357,13 @@ func (c *client) flushClients(budget time.Duration) time.Time {
return last
}
func (c *client) resetReadLoopStallTime() {
if c.in.tst >= stallClientMaxDuration {
c.rateLimitFormatWarnf("Producer was stalled for a total of %v", c.in.tst.Round(time.Millisecond))
}
c.in.tst = 0
}
// readLoop is the main socket read functionality.
// Runs in its own Go routine.
func (c *client) readLoop(pre []byte) {
@@ -1431,21 +1441,6 @@ func (c *client) readLoop(pre []byte) {
return
}
}
if ws {
bufs, err = c.wsRead(wsr, reader, b[:n])
if bufs == nil && err != nil {
if err != io.EOF {
c.Errorf("read error: %v", err)
}
c.closeConnection(closedStateForErr(err))
return
} else if bufs == nil {
continue
}
} else {
bufs[0] = b[:n]
}
// Check if the account has mappings and if so set the local readcache flag.
// We check here to make sure any changes such as config reload are reflected here.
if c.kind == CLIENT || c.kind == LEAF {
@@ -1463,17 +1458,32 @@ func (c *client) readLoop(pre []byte) {
c.in.bytes = 0
c.in.subs = 0
if ws {
err = c.wsReadAndParse(wsr, reader, b[:n])
if err != nil {
// Match the normal parse path: any already-buffered deliveries
// need their pending flush signals drained before we close.
c.flushClients(0)
if err != io.EOF {
c.Errorf("read error: %v", err)
}
c.closeConnection(closedStateForErr(err))
return
}
c.resetReadLoopStallTime()
goto postParse
} else {
bufs[0] = b[:n]
}
// Main call into parser for inbound data. This will generate callouts
// to process messages, etc.
for i := 0; i < len(bufs); i++ {
if err := c.parse(bufs[i]); err != nil {
if err == ErrMinimumVersionRequired {
// Special case here, currently only for leaf node connections.
// When process the CONNECT protocol, if the minimum version
// required was not met, an error was printed and sent back to
// the remote, and connection was closed after a certain delay
// (to avoid "rapid" reconnection from the remote).
// We don't need to do any of the things below, simply return.
// processLeafConnect() already sent the rejection and closed
// the connection, so there is nothing else to do here.
return
}
if dur := time.Since(c.in.start); dur >= readLoopReportThreshold {
@@ -1490,13 +1500,10 @@ func (c *client) readLoop(pre []byte) {
}
return
}
// Clear total stalled time here.
if c.in.tst >= stallClientMaxDuration {
c.rateLimitFormatWarnf("Producer was stalled for a total of %v", c.in.tst.Round(time.Millisecond))
}
c.in.tst = 0
c.resetReadLoopStallTime()
}
postParse:
// If we are a ROUTER/LEAF and have processed an INFO, it is possible that
// we are asked to switch to compression now.
if checkCompress && c.in.flags.isSet(switchToCompression) {
@@ -1687,9 +1694,11 @@ func (c *client) flushOutbound() bool {
cw.Reset(&bb)
for _, buf := range collapsed {
if _, err = cw.Write(buf); err != nil {
break
if err == nil {
_, err = cw.Write(buf)
}
// Return always after consumed or error.
nbPoolPut(buf)
}
if err == nil {
err = cw.Close()
@@ -2114,41 +2123,37 @@ func (c *client) processErr(errStr string) {
}
}
// Password pattern matcher.
var passPat = regexp.MustCompile(`"?\s*pass\S*?"?\s*[:=]\s*"?(([^",\r\n}])*)`)
var tokenPat = regexp.MustCompile(`"?\s*auth_token\S*?"?\s*[:=]\s*"?(([^",\r\n}])*)`)
// Matcher for pass/password and auth_token fields.
var prefixAuthPat = regexp.MustCompile(`"?\s*(?:auth_token\S*?|pass\S*?)"?\s*[:=]\s*"?([^",\r\n}]*)`)
// Exact matcher for fields sig, proxy_sig and nkey.
// Overlapping field "sig" does not match inside "proxy_sig".
var exactAuthPat = regexp.MustCompile(`(?:^|[^A-Za-z0-9_])"?\s*(?:proxy_sig|nkey|sig)"?\s*[:=]\s*"?([^",\r\n}]*)`)
// removeSecretsFromTrace removes any notion of passwords/tokens from trace
// messages for logging.
func removeSecretsFromTrace(arg []byte) []byte {
buf := redact("pass", passPat, arg)
return redact("auth_token", tokenPat, buf)
buf := redact(prefixAuthPat, arg)
return redact(exactAuthPat, buf)
}
func redact(name string, pat *regexp.Regexp, proto []byte) []byte {
if !bytes.Contains(proto, []byte(name)) {
func redact(pat *regexp.Regexp, proto []byte) []byte {
m := pat.FindAllSubmatchIndex(proto, -1)
if len(m) == 0 {
return proto
}
// Take a copy of the connect proto just for the trace message.
var _arg [4096]byte
buf := append(_arg[:0], proto...)
m := pat.FindAllSubmatchIndex(buf, -1)
if len(m) == 0 {
return proto
}
redactedPass := []byte("[REDACTED]")
for _, i := range m {
if len(i) < 4 {
for i := len(m) - 1; i >= 0; i-- {
match := m[i]
if len(match) < 4 {
continue
}
start := i[2]
end := i[3]
start, end := match[2], match[3]
// Replace value substring.
buf = append(buf[:start], append(redactedPass, buf[end:]...)...)
break
}
return buf
}
@@ -2684,11 +2689,9 @@ func (c *client) processPong() {
c.rtt = computeRTT(c.rttStart)
srv := c.srv
reorderGWs := c.kind == GATEWAY && c.gw.outbound
firstPong := c.flags.setIfNotSet(firstPong)
var ri *routeInfo
// When receiving the first PONG, for a route with pooling, we may be
// instructed to start a new route.
if firstPong && c.kind == ROUTER && c.route != nil {
// For a route with pooling, we may be instructed to start a new route.
if c.kind == ROUTER && c.route != nil && c.route.startNewRoute != nil {
ri = c.route.startNewRoute
c.route.startNewRoute = nil
}
@@ -2732,9 +2735,12 @@ func (c *client) updateS2AutoCompressionLevel(co *CompressionOpts, compression *
}
// Will return the parts from the raw wire msg.
// We return the `hdr` as a slice that is capped to the length of the headers
// so that if the caller later tries to append to the returned header slice it
// does not affect the message content.
func (c *client) msgParts(data []byte) (hdr []byte, msg []byte) {
if c != nil && c.pa.hdr > 0 {
return data[:c.pa.hdr], data[c.pa.hdr:]
return data[:c.pa.hdr:c.pa.hdr], data[c.pa.hdr:]
}
return nil, data
}
@@ -2803,9 +2809,10 @@ func (c *client) processHeaderPub(arg, remaining []byte) error {
// look for the tracing header and if found, we will generate a
// trace event with the max payload ingress error.
// Do this only for CLIENT connections.
if c.kind == CLIENT && len(remaining) > 0 {
if td := getHeader(MsgTraceDest, remaining); len(td) > 0 {
c.initAndSendIngressErrEvent(remaining, string(td), ErrMaxPayload)
if c.kind == CLIENT && c.pa.hdr > 0 && len(remaining) > 0 {
hdr := remaining[:min(len(remaining), c.pa.hdr)]
if td, ok := c.allowedMsgTraceDest(hdr, false); ok && td != _EMPTY_ {
c.initAndSendIngressErrEvent(hdr, td, ErrMaxPayload)
}
}
c.maxPayloadViolation(c.pa.size, maxPayload)
@@ -3022,7 +3029,7 @@ func (c *client) processSubEx(subject, queue, bsid []byte, cb msgHandler, noForw
return sub, nil
}
if err := c.addShadowSubscriptions(acc, sub, true); err != nil {
if err := c.addShadowSubscriptions(acc, sub); err != nil {
c.Errorf(err.Error())
}
@@ -3052,10 +3059,7 @@ type ime struct {
// If the client's account has stream imports and there are matches for this
// subscription's subject, then add shadow subscriptions in the other accounts
// that export this subject.
//
// enact=false allows MQTT clients to get the list of shadow subscriptions
// without enacting them, in order to first obtain matching "retained" messages.
func (c *client) addShadowSubscriptions(acc *Account, sub *subscription, enact bool) error {
func (c *client) addShadowSubscriptions(acc *Account, sub *subscription) error {
if acc == nil {
return ErrMissingAccount
}
@@ -3158,7 +3162,7 @@ func (c *client) addShadowSubscriptions(acc *Account, sub *subscription, enact b
for i := 0; i < len(ims); i++ {
ime := &ims[i]
// We will create a shadow subscription.
nsub, err := c.addShadowSub(sub, ime, enact)
nsub, err := c.addShadowSub(sub, ime)
if err != nil {
return err
}
@@ -3175,7 +3179,7 @@ func (c *client) addShadowSubscriptions(acc *Account, sub *subscription, enact b
}
// Add in the shadow subscription.
func (c *client) addShadowSub(sub *subscription, ime *ime, enact bool) (*subscription, error) {
func (c *client) addShadowSub(sub *subscription, ime *ime) (*subscription, error) {
c.mu.Lock()
nsub := *sub // copy
c.mu.Unlock()
@@ -3203,10 +3207,6 @@ func (c *client) addShadowSub(sub *subscription, ime *ime, enact bool) (*subscri
}
// Else use original subject
if !enact {
return &nsub, nil
}
c.Debugf("Creating import subscription on %q from account %q", nsub.subject, im.acc.Name)
if err := im.acc.sl.Insert(&nsub); err != nil {
@@ -3237,7 +3237,7 @@ func (c *client) canSubscribe(subject string, optQueue ...string) bool {
return true
}
allowed := true
allowed, checkAllow := true, true
// Optional queue group.
var queue string
@@ -3245,8 +3245,16 @@ func (c *client) canSubscribe(subject string, optQueue ...string) bool {
queue = optQueue[0]
}
// For CLIENT connections that are MQTT we will implicitly allow anything that starts with
// the "$MQTT.sub." or "$MQTT.deliver.pubrel." prefix. For other types of connections, we
// will implicitly allow anything that starts with the full "$MQTT." prefix. However,
// we don't just return here, we skip the check for "allow" but will check "deny".
if (c.isMqtt() && (strings.HasPrefix(subject, mqttSubPrefix) || strings.HasPrefix(subject, mqttPubRelDeliverySubjectPrefix))) ||
(c.kind != CLIENT && strings.HasPrefix(subject, mqttPrefix)) {
checkAllow = false
}
// Check allow list. If no allow list that means all are allowed. Deny can overrule.
if c.perms.sub.allow != nil {
if checkAllow && c.perms.sub.allow != nil {
r := c.perms.sub.allow.Match(subject)
allowed = len(r.psubs) > 0
if queue != _EMPTY_ && len(r.qsubs) > 0 {
@@ -3338,7 +3346,7 @@ func (c *client) unsubscribe(acc *Account, sub *subscription, force, remove bool
sub.shadow = nil
if len(shadowSubs) > 0 {
isSpokeLeaf = c.isSpokeLeafNode()
updateRoute = !isSpokeLeaf && (c.kind == CLIENT || c.kind == SYSTEM || c.kind == LEAF) && c.srv != nil
updateRoute = !isSpokeLeaf && (c.kind == CLIENT || c.kind == SYSTEM || c.kind == LEAF || c.kind == JETSTREAM) && c.srv != nil
}
sub.close()
c.mu.Unlock()
@@ -4052,6 +4060,41 @@ func (c *client) pubAllowed(subject string) bool {
return c.pubAllowedFullCheck(subject, true, false)
}
// allowedMsgTraceDest returns the trace destination if present and authorized.
// It only considers static publish permissions and does not consume dynamic
// reply permissions because the client is not publishing the trace event itself.
func (c *client) allowedMsgTraceDest(hdr []byte, hasLock bool) (string, bool) {
if len(hdr) == 0 {
return _EMPTY_, true
}
td := sliceHeader(MsgTraceDest, hdr)
if len(td) == 0 {
return _EMPTY_, true
}
dest := bytesToString(td)
if c.kind == CLIENT {
if hasGWRoutedReplyPrefix(td) {
return dest, false
}
var acc *Account
var srv *Server
if !hasLock {
c.mu.Lock()
}
acc, srv = c.acc, c.srv
if !hasLock {
c.mu.Unlock()
}
if bytes.HasPrefix(td, clientNRGPrefix) && srv != nil && acc != srv.SystemAccount() {
return dest, false
}
}
if c.perms != nil && (c.perms.pub.allow != nil || c.perms.pub.deny != nil) && !c.pubAllowedFullCheck(dest, false, hasLock) {
return dest, false
}
return dest, true
}
// pubAllowedFullCheck checks on all publish permissioning depending
// on the flag for dynamic reply permissions.
func (c *client) pubAllowedFullCheck(subject string, fullCheck, hasLock bool) bool {
@@ -4063,9 +4106,15 @@ func (c *client) pubAllowedFullCheck(subject string, fullCheck, hasLock bool) bo
if ok {
return v.(bool)
}
allowed := true
allowed, checkAllow := true, true
// For any connections, other than CLIENT, we will implicitly allow anything that
// starts with the "$MQTT." prefix. However, we don't just return here,
// we skip the check for "allow" but will check "deny".
if c.kind != CLIENT && strings.HasPrefix(subject, mqttPrefix) {
checkAllow = false
}
// Cache miss, check allow then deny as needed.
if c.perms.pub.allow != nil {
if checkAllow && c.perms.pub.allow != nil {
np, _ := c.perms.pub.allow.NumInterest(subject)
allowed = np != 0
}
@@ -4181,10 +4230,19 @@ func (c *client) processInboundClientMsg(msg []byte) (bool, bool) {
genidAddr := &acc.sl.genid
// Check pub permissions
if c.perms != nil && (c.perms.pub.allow != nil || c.perms.pub.deny != nil) && !c.pubAllowedFullCheck(string(c.pa.subject), true, true) {
c.mu.Unlock()
c.pubPermissionViolation(c.pa.subject)
return false, true
if c.perms != nil && (c.perms.pub.allow != nil || c.perms.pub.deny != nil) {
if !c.pubAllowedFullCheck(string(c.pa.subject), true, true) {
c.mu.Unlock()
c.pubPermissionViolation(c.pa.subject)
return false, true
}
}
if c.pa.hdr > 0 {
if td, ok := c.allowedMsgTraceDest(msg[:c.pa.hdr], true); !ok {
c.mu.Unlock()
c.pubPermissionViolation(stringToBytes(td))
return false, true
}
}
c.mu.Unlock()
@@ -4384,28 +4442,43 @@ func (c *client) setupResponseServiceImport(acc *Account, si *serviceImport, tra
return rsi
}
// Will remove a header if present.
func removeHeaderIfPresent(hdr []byte, key string) []byte {
start := getHeaderKeyIndex(key, hdr)
// key can't be first and we want to check that it is preceded by a '\n'
if start < 1 || hdr[start-1] != '\n' {
// Will remove a status and description from the header if present.
func removeHeaderStatusIfPresent(hdr []byte) []byte {
k := []byte("NATS/1.0")
kl, i := len(k), bytes.IndexByte(hdr, '\r')
if !bytes.HasPrefix(hdr, k) || i <= kl {
return hdr
}
index := start + len(key)
if index >= len(hdr) || hdr[index] != ':' {
return hdr
}
end := bytes.Index(hdr[start:], []byte(_CRLF_))
if end < 0 {
return hdr
}
hdr = append(hdr[:start], hdr[start+end+len(_CRLF_):]...)
if len(hdr) <= len(emptyHdrLine) {
hdr = append(hdr[:kl], hdr[i:]...)
if len(hdr) == len(emptyHdrLine) {
return nil
}
return hdr
}
// Will remove a header if present.
func removeHeaderIfPresent(hdr []byte, key string) []byte {
for {
start := getHeaderKeyIndex(key, hdr)
// key can't be first and we want to check that it is preceded by a '\n'
if start < 1 || hdr[start-1] != '\n' {
return hdr
}
index := start + len(key)
if index >= len(hdr) || hdr[index] != ':' {
return hdr
}
end := bytes.Index(hdr[start:], []byte(_CRLF_))
if end < 0 {
return hdr
}
hdr = append(hdr[:start], hdr[start+end+len(_CRLF_):]...)
if len(hdr) <= len(emptyHdrLine) {
return nil
}
}
}
func removeHeaderIfPrefixPresent(hdr []byte, prefix string) []byte {
var index int
for {
@@ -4560,6 +4633,19 @@ func getHeaderKeyIndex(key string, hdr []byte) int {
}
}
// setHeader will replace the value of the first existing key `key`
// with the given value `val`, or add this new key at the end of
// the headers.
//
// Note: If the key does not exist, or if it exists but the new value
// would make the resulting byte slice larger than the original one,
// a new byte slice is returned and the original is left untouched.
// This is to prevent situations where caller may have a `hdr` and
// `msg` that are the parts of an underlying buffer. Extending the
// `hdr` would otherwise overwrite the `msg` part.
//
// If the new value is smaller, then the original `hdr` byte slice
// is modified.
func setHeader(key, val string, hdr []byte) []byte {
start := getHeaderKeyIndex(key, hdr)
if start >= 0 {
@@ -4574,15 +4660,45 @@ func setHeader(key, val string, hdr []byte) []byte {
return hdr // malformed headers
}
valEnd += valStart
suffix := slices.Clone(hdr[valEnd:])
newHdr := append(hdr[:valStart], val...)
return append(newHdr, suffix...)
// Length of the existing value (before the `\r`)
oldValLen := valEnd - valStart
// This is how many extra bytes we need for the new value.
// If <= 0, it means that we need less and so will reuse the `hdr` buffer.
if extra := len(val) - oldValLen; extra > 0 {
// Check that we don't overflow an "int".
if rem := math.MaxInt - hdrLen; rem < extra {
// We don't grow, and return the existing header.
return hdr
}
// The new size is the old size plus the extra bytes.
newHdrSize := hdrLen + extra
newHdr := make([]byte, newHdrSize)
// Copy the parts from `hdr` and `val` into the new buffer.
n := copy(newHdr, hdr[:valStart])
n += copy(newHdr[n:], val)
copy(newHdr[n:], hdr[valEnd:])
return newHdr
}
// We can write in place since it fits in the existing `hdr` buffer.
n := copy(hdr[valStart:], val)
n += copy(hdr[valStart+n:], hdr[valEnd:])
hdr = hdr[:valStart+n]
return hdr
}
if len(hdr) > 0 && bytes.HasSuffix(hdr, []byte("\r\n")) {
hdr = hdr[:len(hdr)-2]
val += "\r\n"
}
return fmt.Appendf(hdr, "%s: %s\r\n", key, val)
// Create the new buffer based on length of existing one and
// length of the new "<key>: <value>\r\n". Protect against "int" overflow.
newSize := uint64(len(hdr)) + uint64(len(key)) + 1 + 1 + uint64(len(val)) + 2
if newSize > uint64(math.MaxInt) {
// We don't grow, and return the existing header.
return hdr
}
newHdr := make([]byte, 0, int(newSize))
newHdr = append(newHdr, hdr...)
return fmt.Appendf(newHdr, "%s: %s\r\n", key, val)
}
// For bytes.HasPrefix below.
@@ -4697,16 +4813,33 @@ func (c *client) processServiceImport(si *serviceImport, acc *Account, msg []byt
if !isResponse {
isSysImport := siAcc == c.srv.SystemAccount()
var ci *ClientInfo
if hadPrevSi && c.pa.hdr >= 0 {
var cis ClientInfo
if err := json.Unmarshal(sliceHeader(ClientInfoHdr, msg[:c.pa.hdr]), &cis); err == nil {
ci = &cis
var cis *ClientInfo
if c.pa.hdr >= 0 {
var hci ClientInfo
if err := json.Unmarshal(sliceHeader(ClientInfoHdr, msg[:c.pa.hdr]), &hci); err == nil {
cis = &hci
}
}
if c.kind == LEAF && c.pa.hdr >= 0 && len(sliceHeader(ClientInfoHdr, msg[:c.pa.hdr])) > 0 {
// Leaf nodes may forward a Nats-Request-Info from a remote domain,
// but the local server must replace it with the identity of the
// authenticated leaf connection instead of trusting forwarded values.
ci = c.getClientInfo(share)
if hadPrevSi {
ci.Service = acc.Name
// Check if we are moving into a share details account from a non-shared
// and add in server and cluster details.
if !share && (si.share || isSysImport) {
c.addServerAndClusterInfo(ci)
}
} else if !share && isSysImport {
c.addServerAndClusterInfo(ci)
}
} else if hadPrevSi && cis != nil {
ci = cis
ci.Service = acc.Name
// Check if we are moving into a share details account from a non-shared
// and add in server and cluster details.
if !share && (si.share || isSysImport) {
c.addServerAndClusterInfo(ci)
}
} else if c.kind != LEAF || c.pa.hdr < 0 || len(sliceHeader(ClientInfoHdr, msg[:c.pa.hdr])) == 0 {
ci = c.getClientInfo(share)
@@ -4714,12 +4847,6 @@ func (c *client) processServiceImport(si *serviceImport, acc *Account, msg []byt
if !share && isSysImport {
c.addServerAndClusterInfo(ci)
}
} else if c.kind == LEAF && (si.share || isSysImport) {
// We have a leaf header here for ci, augment as above.
ci = c.getClientInfo(si.share)
if !si.share && isSysImport {
c.addServerAndClusterInfo(ci)
}
}
// Set clientInfo if present.
if ci != nil {
@@ -5341,8 +5468,10 @@ sendToRoutesOrLeafs:
// If we do have a deliver subject we need to do something with it.
// Again this is when JetStream (but possibly others) wants the system
// to rewrite the delivered subject. The way we will do that is place it
// at the end of the reply subject if it exists.
if len(deliver) > 0 && len(reply) > 0 {
// at the end of the reply subject if it exists. But only if this wasn't
// already performed, otherwise we'd end up with a duplicate '@' suffix
// resulting in a protocol error.
if len(deliver) > 0 && len(reply) > 0 && !remapped {
reply = append(reply, '@')
reply = append(reply, deliver...)
}
@@ -5501,6 +5630,9 @@ func (c *client) processPingTimer() {
if c.kind == ROUTER && opts.Cluster.PingInterval > 0 {
pingInterval = opts.Cluster.PingInterval
}
if c.isWebsocket() && opts.Websocket.PingInterval > 0 {
pingInterval = opts.Websocket.PingInterval
}
pingInterval = adjustPingInterval(c.kind, pingInterval)
now := time.Now()
needRTT := c.rtt == 0 || now.Sub(c.rttStart) > DEFAULT_RTT_MEASUREMENT_INTERVAL
@@ -5583,6 +5715,9 @@ func (c *client) setPingTimer() {
if c.kind == ROUTER && opts.Cluster.PingInterval > 0 {
d = opts.Cluster.PingInterval
}
if c.isWebsocket() && opts.Websocket.PingInterval > 0 {
d = opts.Websocket.PingInterval
}
d = adjustPingInterval(c.kind, d)
c.ping.tmr = time.AfterFunc(d, c.processPingTimer)
}
@@ -5788,7 +5923,7 @@ func (c *client) processSubsOnConfigReload(awcsti map[string]struct{}) {
oldShadows := sub.shadow
sub.shadow = nil
c.mu.Unlock()
c.addShadowSubscriptions(acc, sub, true)
c.addShadowSubscriptions(acc, sub)
for _, nsub := range oldShadows {
nsub.im.acc.sl.Remove(nsub)
}
@@ -6617,6 +6752,9 @@ func (c *client) setFirstPingTimer() {
if c.kind == ROUTER && opts.Cluster.PingInterval > 0 {
d = opts.Cluster.PingInterval
}
if c.isWebsocket() && opts.Websocket.PingInterval > 0 {
d = opts.Websocket.PingInterval
}
if !opts.DisableShortFirstPing {
if c.kind != CLIENT {
if d > firstPingInterval {
+29 -22
View File
@@ -131,19 +131,22 @@ func detectProxyProtoVersion(conn net.Conn) (version int, header []byte, err err
// readProxyProtoV1Header parses PROXY protocol v1 text format.
// Expects the "PROXY " prefix (6 bytes) to have already been consumed.
func readProxyProtoV1Header(conn net.Conn) (*proxyProtoAddr, error) {
// Returns any bytes that were read past the trailing CRLF so the caller can
// replay them into the next protocol layer.
func readProxyProtoV1Header(conn net.Conn) (*proxyProtoAddr, []byte, error) {
// Read rest of line (max 107 bytes total, already read 6)
maxRemaining := proxyProtoV1MaxLineLen - 6
// Read up to maxRemaining bytes at once (more efficient than byte-by-byte)
buf := make([]byte, maxRemaining)
var line []byte
var remaining []byte
for len(line) < maxRemaining {
// Read available data
n, err := conn.Read(buf[len(line):])
if err != nil {
return nil, fmt.Errorf("failed to read v1 line: %w", err)
return nil, nil, fmt.Errorf("failed to read v1 line: %w", err)
}
line = buf[:len(line)+n]
@@ -151,7 +154,8 @@ func readProxyProtoV1Header(conn net.Conn) (*proxyProtoAddr, error) {
// Look for CRLF in what we've read so far
for i := 0; i < len(line)-1; i++ {
if line[i] == '\r' && line[i+1] == '\n' {
// Found CRLF - extract just the line portion
// Found CRLF - keep any over-read bytes for the client parser.
remaining = append(remaining, line[i+2:]...)
line = line[:i]
goto foundCRLF
}
@@ -159,7 +163,7 @@ func readProxyProtoV1Header(conn net.Conn) (*proxyProtoAddr, error) {
}
// Exceeded max length without finding CRLF
return nil, fmt.Errorf("%w: v1 line too long", errProxyProtoInvalid)
return nil, nil, fmt.Errorf("%w: v1 line too long", errProxyProtoInvalid)
foundCRLF:
// Get parts from the protocol
@@ -167,17 +171,17 @@ foundCRLF:
// Validate format
if len(parts) < 1 {
return nil, fmt.Errorf("%w: invalid v1 format", errProxyProtoInvalid)
return nil, nil, fmt.Errorf("%w: invalid v1 format", errProxyProtoInvalid)
}
// Handle UNKNOWN (health check, like v2 LOCAL)
if parts[0] == proxyProtoV1Unknown {
return nil, nil
return nil, remaining, nil
}
// Must have exactly 5 parts: protocol, src-ip, dst-ip, src-port, dst-port
if len(parts) != 5 {
return nil, fmt.Errorf("%w: invalid v1 format", errProxyProtoInvalid)
return nil, nil, fmt.Errorf("%w: invalid v1 format", errProxyProtoInvalid)
}
protocol := parts[0]
@@ -185,29 +189,29 @@ foundCRLF:
dstIP := net.ParseIP(parts[2])
if srcIP == nil || dstIP == nil {
return nil, fmt.Errorf("%w: invalid address", errProxyProtoInvalid)
return nil, nil, fmt.Errorf("%w: invalid address", errProxyProtoInvalid)
}
// Parse ports
srcPort, err := strconv.ParseUint(parts[3], 10, 16)
if err != nil {
return nil, fmt.Errorf("invalid source port: %w", err)
return nil, nil, fmt.Errorf("invalid source port: %w", err)
}
dstPort, err := strconv.ParseUint(parts[4], 10, 16)
if err != nil {
return nil, fmt.Errorf("invalid dest port: %w", err)
return nil, nil, fmt.Errorf("invalid dest port: %w", err)
}
// Validate protocol matches IP version
if protocol == proxyProtoV1TCP4 && srcIP.To4() == nil {
return nil, fmt.Errorf("%w: TCP4 with IPv6 address", errProxyProtoInvalid)
return nil, nil, fmt.Errorf("%w: TCP4 with IPv6 address", errProxyProtoInvalid)
}
if protocol == proxyProtoV1TCP6 && srcIP.To4() != nil {
return nil, fmt.Errorf("%w: TCP6 with IPv4 address", errProxyProtoInvalid)
return nil, nil, fmt.Errorf("%w: TCP6 with IPv4 address", errProxyProtoInvalid)
}
if protocol != proxyProtoV1TCP4 && protocol != proxyProtoV1TCP6 {
return nil, fmt.Errorf("%w: invalid protocol %s", errProxyProtoInvalid, protocol)
return nil, nil, fmt.Errorf("%w: invalid protocol %s", errProxyProtoInvalid, protocol)
}
return &proxyProtoAddr{
@@ -215,25 +219,27 @@ foundCRLF:
srcPort: uint16(srcPort),
dstIP: dstIP,
dstPort: uint16(dstPort),
}, nil
}, remaining, nil
}
// readProxyProtoHeader reads and parses PROXY protocol (v1 or v2) from the connection.
// Automatically detects version and routes to appropriate parser.
// If the command is LOCAL/UNKNOWN (health check), it returns nil for addr and no error.
// If the command is PROXY, it returns the parsed address information.
// It also returns any bytes that were read past the v1 header terminator so the
// caller can replay them into the normal client parser.
// The connection must be fresh (no data read yet).
func readProxyProtoHeader(conn net.Conn) (*proxyProtoAddr, error) {
func readProxyProtoHeader(conn net.Conn) (*proxyProtoAddr, []byte, error) {
// Set read deadline to prevent hanging on slow/malicious clients
if err := conn.SetReadDeadline(time.Now().Add(proxyProtoReadTimeout)); err != nil {
return nil, err
return nil, nil, err
}
defer conn.SetReadDeadline(time.Time{})
// Detect version
version, firstBytes, err := detectProxyProtoVersion(conn)
if err != nil {
return nil, err
return nil, nil, err
}
switch version {
@@ -244,25 +250,26 @@ func readProxyProtoHeader(conn net.Conn) (*proxyProtoAddr, error) {
// Read rest of v2 signature (bytes 6-11, total 6 more bytes)
remaining := make([]byte, 6)
if _, err := io.ReadFull(conn, remaining); err != nil {
return nil, fmt.Errorf("failed to read v2 signature: %w", err)
return nil, nil, fmt.Errorf("failed to read v2 signature: %w", err)
}
// Verify full signature
fullSig := string(firstBytes) + string(remaining)
if fullSig != proxyProtoV2Sig {
return nil, fmt.Errorf("%w: invalid signature", errProxyProtoInvalid)
return nil, nil, fmt.Errorf("%w: invalid signature", errProxyProtoInvalid)
}
// Read rest of header: ver/cmd, fam/proto, addr-len (4 bytes)
header := make([]byte, 4)
if _, err := io.ReadFull(conn, header); err != nil {
return nil, fmt.Errorf("failed to read v2 header: %w", err)
return nil, nil, fmt.Errorf("failed to read v2 header: %w", err)
}
// Continue with parsing
return parseProxyProtoV2Header(conn, header)
addr, err := parseProxyProtoV2Header(conn, header)
return addr, nil, err
default:
return nil, fmt.Errorf("unsupported PROXY protocol version: %d", version)
return nil, nil, fmt.Errorf("unsupported PROXY protocol version: %d", version)
}
}
+1 -1
View File
@@ -66,7 +66,7 @@ func init() {
const (
// VERSION is the current version for the server.
VERSION = "2.12.2"
VERSION = "2.12.6"
// PROTO is the currently supported protocol.
// 0 was the original
+187 -123
View File
@@ -1,4 +1,4 @@
// Copyright 2019-2025 The NATS Authors
// Copyright 2019-2026 The NATS Authors
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
@@ -42,7 +42,6 @@ import (
const (
JSPullRequestPendingMsgs = "Nats-Pending-Messages"
JSPullRequestPendingBytes = "Nats-Pending-Bytes"
JSPullRequestWrongPinID = "NATS/1.0 423 Nats-Wrong-Pin-Id\r\n\r\n"
JSPullRequestNatsPinId = "Nats-Pin-Id"
)
@@ -512,7 +511,7 @@ type consumer struct {
// Details described in ADR-42.
// currentPinId is the current nuid for the pinned consumer.
// If the Consumer is running in `PriorityPinnedClient` mode, server will
// If the Consumer is running in `PriorityPinnedClient` mode, server will
// pick up a new nuid and assign it to first pending pull request.
currentPinId string
/// pinnedTtl is the remaining time before the current PinId expires.
@@ -825,7 +824,7 @@ func checkConsumerCfg(
return NewJSStreamInvalidConfigError(ErrBadSubject)
}
for inner, ssubject := range subjectFilters {
if inner != outer && SubjectsCollide(subject, ssubject) {
if inner != outer && subjectIsSubsetMatch(subject, ssubject) {
return NewJSConsumerOverlappingSubjectFiltersError()
}
}
@@ -963,7 +962,7 @@ func (mset *stream) addConsumerWithAssignment(config *ConsumerConfig, oname stri
}
mset.mu.RLock()
s, jsa, cfg, acc := mset.srv, mset.jsa, mset.cfg, mset.acc
s, js, jsa, cfg, acc := mset.srv, mset.js, mset.jsa, mset.cfg, mset.acc
mset.mu.RUnlock()
// If we do not have the consumer currently assigned to us in cluster mode we will proceed but warn.
@@ -1030,11 +1029,11 @@ func (mset *stream) addConsumerWithAssignment(config *ConsumerConfig, oname stri
}
if cName != _EMPTY_ {
if eo, ok := mset.consumers[cName]; ok {
mset.mu.Unlock()
if action == ActionCreate {
ocfg := eo.config()
copyConsumerMetadata(config, &ocfg)
if !reflect.DeepEqual(config, &ocfg) {
mset.mu.Unlock()
return nil, NewJSConsumerAlreadyExistsError()
}
}
@@ -1042,9 +1041,11 @@ func (mset *stream) addConsumerWithAssignment(config *ConsumerConfig, oname stri
if cfg.Retention == WorkQueuePolicy {
subjects := gatherSubjectFilters(config.FilterSubject, config.FilterSubjects)
if !mset.partitionUnique(cName, subjects) {
mset.mu.Unlock()
return nil, NewJSConsumerWQConsumerNotUniqueError()
}
}
mset.mu.Unlock()
err := eo.updateConfig(config)
if err == nil {
return eo, nil
@@ -1057,17 +1058,22 @@ func (mset *stream) addConsumerWithAssignment(config *ConsumerConfig, oname stri
return nil, NewJSConsumerDoesNotExistError()
}
// Check for any limits, if the config for the consumer sets a limit we check against that
// but if not we use the value from account limits, if account limits is more restrictive
// than stream config we prefer the account limits to handle cases where account limits are
// updated during the lifecycle of the stream
maxc := cfg.MaxConsumers
if maxc <= 0 || (selectedLimits.MaxConsumers > 0 && selectedLimits.MaxConsumers < maxc) {
maxc = selectedLimits.MaxConsumers
}
if maxc > 0 && mset.numPublicConsumers() >= maxc {
mset.mu.Unlock()
return nil, NewJSMaximumConsumersLimitError()
// If we're clustered we've already done this check, only do this if we're a standalone server.
// But if we're standalone, only enforce if we're not recovering, since the MaxConsumers could've
// been updated while we already had more consumers on disk.
if !s.JetStreamIsClustered() && s.standAloneMode() && !isRecovering {
// Check for any limits, if the config for the consumer sets a limit we check against that
// but if not we use the value from account limits, if account limits is more restrictive
// than stream config we prefer the account limits to handle cases where account limits are
// updated during the lifecycle of the stream
maxc := cfg.MaxConsumers
if maxc <= 0 || (selectedLimits.MaxConsumers > 0 && selectedLimits.MaxConsumers < maxc) {
maxc = selectedLimits.MaxConsumers
}
if maxc > 0 && mset.numPublicConsumers() >= maxc {
mset.mu.Unlock()
return nil, NewJSMaximumConsumersLimitError()
}
}
// Check on stream type conflicts with WorkQueues.
@@ -1134,6 +1140,13 @@ func (mset *stream) addConsumerWithAssignment(config *ConsumerConfig, oname stri
created: time.Now().UTC(),
}
// Add created timestamp used for the store, must match that of the consumer assignment if it exists.
if ca != nil {
js.mu.RLock()
o.created = ca.Created
js.mu.RUnlock()
}
// Bind internal client to the user account.
o.client.registerWithAccount(a)
// Bind to the system account.
@@ -1186,7 +1199,7 @@ func (mset *stream) addConsumerWithAssignment(config *ConsumerConfig, oname stri
// Setup our storage if not a direct consumer.
if !config.Direct {
store, err := mset.store.ConsumerStore(o.name, config)
store, err := mset.store.ConsumerStore(o.name, o.created, config)
if err != nil {
mset.mu.Unlock()
o.deleteWithoutAdvisory()
@@ -1206,14 +1219,13 @@ func (mset *stream) addConsumerWithAssignment(config *ConsumerConfig, oname stri
// If we have multiple filter subjects, create a sublist which we will use
// in calling store.LoadNextMsgMulti.
if len(o.cfg.FilterSubjects) > 0 {
o.filters = gsl.NewSublist[struct{}]()
for _, filter := range o.cfg.FilterSubjects {
o.filters.Insert(filter, struct{}{})
}
} else {
// Make sure this is nil otherwise.
if len(o.subjf) <= 1 {
o.filters = nil
} else {
o.filters = gsl.NewSublist[struct{}]()
for _, filter := range o.subjf {
o.filters.Insert(filter.subject, struct{}{})
}
}
if o.store != nil && o.store.HasState() {
@@ -1393,8 +1405,12 @@ func (o *consumer) monitorQuitC() <-chan struct{} {
if o == nil {
return nil
}
o.mu.RLock()
defer o.mu.RUnlock()
o.mu.Lock()
defer o.mu.Unlock()
// Recreate if a prior monitor routine was stopped.
if o.mqch == nil {
o.mqch = make(chan struct{})
}
return o.mqch
}
@@ -1535,7 +1551,6 @@ func (o *consumer) setLeader(isLeader bool) {
if o.cfg.AckPolicy != AckNone {
if o.ackSub, err = o.subscribeInternal(o.ackSubj, o.pushAck); err != nil {
o.mu.Unlock()
o.deleteWithoutAdvisory()
return
}
}
@@ -1544,7 +1559,6 @@ func (o *consumer) setLeader(isLeader bool) {
// Will error if wrong mode to provide feedback to users.
if o.reqSub, err = o.subscribeInternal(o.nextMsgSubj, o.processNextMsgReq); err != nil {
o.mu.Unlock()
o.deleteWithoutAdvisory()
return
}
@@ -1554,7 +1568,6 @@ func (o *consumer) setLeader(isLeader bool) {
fcsubj := fmt.Sprintf(jsFlowControl, stream, o.name)
if o.fcSub, err = o.subscribeInternal(fcsubj, o.processFlowControl); err != nil {
o.mu.Unlock()
o.deleteWithoutAdvisory()
return
}
}
@@ -1680,6 +1693,7 @@ func (o *consumer) setLeader(isLeader bool) {
} else if o.srv.gateway.enabled {
stopAndClearTimer(&o.gwdtmr)
}
o.unassignPinId()
// If we were the leader make sure to drain queued up acks.
if wasLeader {
o.ackMsgs.drain()
@@ -2039,6 +2053,7 @@ func (o *consumer) deleteNotActive() {
if o.srv != nil {
qch = o.srv.quitCh
}
oqch := o.qch
o.mu.Unlock()
if js != nil {
cqch = js.clusterQuitC()
@@ -2087,6 +2102,9 @@ func (o *consumer) deleteNotActive() {
return
case <-cqch:
return
case <-oqch:
// The consumer has stopped already, likely by an earlier delete proposal being applied.
return
}
js.mu.RLock()
if js.shuttingDown {
@@ -2394,7 +2412,8 @@ func (o *consumer) updateConfig(cfg *ConsumerConfig) error {
// Check for Subject Filters update.
newSubjects := gatherSubjectFilters(cfg.FilterSubject, cfg.FilterSubjects)
if !subjectSliceEqual(newSubjects, o.subjf.subjects()) {
updatedFilters := !subjectSliceEqual(newSubjects, o.subjf.subjects())
if updatedFilters {
newSubjf := make(subjectFilters, 0, len(newSubjects))
for _, newFilter := range newSubjects {
fs := &subjectFilter{
@@ -2433,15 +2452,17 @@ func (o *consumer) updateConfig(cfg *ConsumerConfig) error {
// Allowed but considered no-op, [Description, SampleFrequency, MaxWaiting, HeadersOnly]
o.cfg = *cfg
// Cleanup messages that lost interest.
if o.retention == InterestPolicy {
o.mu.Unlock()
o.cleanupNoInterestMessages(o.mset, false)
o.mu.Lock()
}
if updatedFilters {
// Cleanup messages that lost interest.
if o.retention == InterestPolicy {
o.mu.Unlock()
o.cleanupNoInterestMessages(o.mset, false)
o.mu.Lock()
}
// Re-calculate num pending on update.
o.streamNumPending()
// Re-calculate num pending on update.
o.streamNumPending()
}
return nil
}
@@ -2831,14 +2852,10 @@ func (o *consumer) releaseAnyPendingRequests(isAssigned bool) {
if o.mset == nil || o.outq == nil || o.waiting.len() == 0 {
return
}
var hdr []byte
if !isAssigned {
hdr = []byte("NATS/1.0 409 Consumer Deleted\r\n\r\n")
}
wq := o.waiting
for wr := wq.head; wr != nil; {
if hdr != nil {
if !isAssigned {
hdr := []byte("NATS/1.0 409 Consumer Deleted\r\n\r\n")
o.outq.send(newJSPubMsg(wr.reply, _EMPTY_, _EMPTY_, hdr, nil, nil, 0))
}
next := wr.next
@@ -3042,6 +3059,11 @@ func (o *consumer) setStoreState(state *ConsumerState) error {
err := o.store.Update(state)
if err == nil {
o.applyState(state)
} else if err == ErrStoreOldUpdate {
// Our store already has a newer state, which is normal during recovery
// when the consumer was loaded from disk before the meta snapshot state
// was applied.
return nil
}
return err
}
@@ -3133,6 +3155,12 @@ func (o *consumer) infoWithSnapAndReply(snap bool, reply string) *ConsumerInfo {
})
}
np, err := o.checkNumPending()
if err != nil {
o.mu.Unlock()
return nil
}
cfg := o.cfg
info := &ConsumerInfo{
Stream: o.stream,
@@ -3149,7 +3177,7 @@ func (o *consumer) infoWithSnapAndReply(snap bool, reply string) *ConsumerInfo {
},
NumAckPending: len(o.pending),
NumRedelivered: len(o.rdc),
NumPending: o.checkNumPending(),
NumPending: np,
PushBound: o.isPushMode() && o.active,
TimeStamp: time.Now().UTC(),
PriorityGroups: priorityGroups,
@@ -3900,7 +3928,12 @@ func (o *consumer) setPinnedTimer(priorityGroup string) {
} else {
o.pinnedTtl = time.AfterFunc(o.cfg.PinnedTTL, func() {
o.mu.Lock()
o.currentPinId = _EMPTY_
// Skip if already unset.
if o.currentPinId == _EMPTY_ {
o.mu.Unlock()
return
}
o.unassignPinId()
o.sendUnpinnedAdvisoryLocked(priorityGroup, "timeout")
o.mu.Unlock()
o.signalNewMessages()
@@ -3908,6 +3941,28 @@ func (o *consumer) setPinnedTimer(priorityGroup string) {
}
}
// Lock should be held.
func (o *consumer) assignNewPinId(wr *waitingRequest) {
if wr.priorityGroup == nil || wr.priorityGroup.Group == _EMPTY_ {
return
}
o.currentPinId = nuid.Next()
o.pinnedTS = time.Now().UTC()
wr.priorityGroup.Id = o.currentPinId
o.setPinnedTimer(wr.priorityGroup.Group)
o.sendPinnedAdvisoryLocked(wr.priorityGroup.Group)
}
// Lock should be held.
func (o *consumer) unassignPinId() {
o.currentPinId = _EMPTY_
o.pinnedTS = time.Time{}
if o.pinnedTtl != nil {
o.pinnedTtl.Stop()
o.pinnedTtl = nil
}
}
// Return next waiting request. This will check for expirations but not noWait or interest.
// That will be handled by processWaiting.
// Lock should be held.
@@ -3918,11 +3973,6 @@ func (o *consumer) nextWaiting(sz int) *waitingRequest {
// Check if server needs to assign a new pin id.
needNewPin := o.currentPinId == _EMPTY_ && o.cfg.PriorityPolicy == PriorityPinnedClient
// As long as we support only one priority group, we can capture that group here and reuse it.
var priorityGroup string
if len(o.cfg.PriorityGroups) > 0 {
priorityGroup = o.cfg.PriorityGroups[0]
}
numCycled := 0
for wr := o.waiting.peek(); !o.waiting.isEmpty(); wr = o.waiting.peek() {
@@ -3956,15 +4006,12 @@ func (o *consumer) nextWaiting(sz int) *waitingRequest {
if wr.expires.IsZero() || time.Now().Before(wr.expires) {
if needNewPin {
if wr.priorityGroup.Id == _EMPTY_ {
o.currentPinId = nuid.Next()
o.pinnedTS = time.Now().UTC()
wr.priorityGroup.Id = o.currentPinId
o.setPinnedTimer(priorityGroup)
o.assignNewPinId(wr)
} else {
// There is pin id set, but not a matching one. Send a notification to the client and remove the request.
// Probably this is the old pin id.
o.outq.send(newJSPubMsg(wr.reply, _EMPTY_, _EMPTY_, []byte(JSPullRequestWrongPinID), nil, nil, 0))
hdr := fmt.Appendf(nil, "NATS/1.0 423 Nats-Wrong-Pin-Id\r\n%s: %d\r\n%s: %d\r\n\r\n", JSPullRequestPendingMsgs, wr.n, JSPullRequestPendingBytes, wr.b)
o.outq.send(newJSPubMsg(wr.reply, _EMPTY_, _EMPTY_, hdr, nil, nil, 0))
o.waiting.removeCurrent()
if o.node != nil {
o.removeClusterPendingRequest(wr.reply)
@@ -3985,7 +4032,8 @@ func (o *consumer) nextWaiting(sz int) *waitingRequest {
continue
} else {
// There is pin id set, but not a matching one. Send a notification to the client and remove the request.
o.outq.send(newJSPubMsg(wr.reply, _EMPTY_, _EMPTY_, []byte(JSPullRequestWrongPinID), nil, nil, 0))
hdr := fmt.Appendf(nil, "NATS/1.0 423 Nats-Wrong-Pin-Id\r\n%s: %d\r\n%s: %d\r\n\r\n", JSPullRequestPendingMsgs, wr.n, JSPullRequestPendingBytes, wr.b)
o.outq.send(newJSPubMsg(wr.reply, _EMPTY_, _EMPTY_, hdr, nil, nil, 0))
o.waiting.removeCurrent()
if o.node != nil {
o.removeClusterPendingRequest(wr.reply)
@@ -3997,9 +4045,13 @@ func (o *consumer) nextWaiting(sz int) *waitingRequest {
if o.cfg.PriorityPolicy == PriorityOverflow {
if wr.priorityGroup != nil &&
// If both limits are zero we don't cycle and the request will be fulfilled.
(wr.priorityGroup.MinPending > 0 || wr.priorityGroup.MinAckPending > 0) &&
// We need to check o.npc+1, because before calling nextWaiting, we do o.npc--
(wr.priorityGroup.MinPending > 0 && wr.priorityGroup.MinPending > o.npc+1 ||
wr.priorityGroup.MinAckPending > 0 && wr.priorityGroup.MinAckPending > int64(len(o.pending))) {
// If one OR the other limit is exceeded, we want to fulfill the request.
// This is an inverted check. For clarity, we check the positive condition and negate.
!((wr.priorityGroup.MinPending > 0 && wr.priorityGroup.MinPending <= o.npc+1) ||
(wr.priorityGroup.MinAckPending > 0 && wr.priorityGroup.MinAckPending <= int64(len(o.pending)))) {
o.waiting.cycle()
numCycled++
// We're done cycling through the requests.
@@ -4010,19 +4062,10 @@ func (o *consumer) nextWaiting(sz int) *waitingRequest {
}
}
if wr.acc.sl.HasInterest(wr.interest) {
if needNewPin {
o.sendPinnedAdvisoryLocked(priorityGroup)
}
return o.waiting.popOrPopAndRequeue(o.cfg.PriorityPolicy)
} else if time.Since(wr.received) < defaultGatewayRecentSubExpiration && (o.srv.leafNodeEnabled || o.srv.gateway.enabled) {
if needNewPin {
o.sendPinnedAdvisoryLocked(priorityGroup)
}
return o.waiting.popOrPopAndRequeue(o.cfg.PriorityPolicy)
} else if o.srv.gateway.enabled && o.srv.hasGatewayInterest(wr.acc.Name, wr.interest) {
if needNewPin {
o.sendPinnedAdvisoryLocked(priorityGroup)
}
return o.waiting.popOrPopAndRequeue(o.cfg.PriorityPolicy)
}
} else {
@@ -4180,15 +4223,7 @@ func (o *consumer) processNextMsgRequest(reply string, msg []byte) {
sendErr(400, "Bad Request - Priority Group missing")
return
}
found := false
for _, group := range o.cfg.PriorityGroups {
if group == priorityGroup.Group {
found = true
break
}
}
if !found {
if !slices.Contains(o.cfg.PriorityGroups, priorityGroup.Group) {
sendErr(400, "Bad Request - Invalid Priority Group")
return
}
@@ -4426,6 +4461,8 @@ func (o *consumer) getNextMsg() (*jsPubMsg, uint64, error) {
// scheduled for redelivery, but it has been removed from the stream.
// o.processTerm is called in a goroutine so could run after we get here.
// That will correct the pending state and delivery/ack floors, so just skip here.
pmsg.returnToPool()
pmsg = nil
continue
}
return pmsg, dc, err
@@ -4453,6 +4490,7 @@ func (o *consumer) getNextMsg() (*jsPubMsg, uint64, error) {
sm, err := o.mset.store.LoadMsg(seq, &pmsg.StoreMsg)
if sm == nil || err != nil {
pmsg.returnToPool()
pmsg = nil
}
o.sseq++
return pmsg, 1, err
@@ -4964,6 +5002,7 @@ func (o *consumer) loopAndGatherMsgs(qch chan struct{}) {
o.addToRedeliverQueue(pmsg.seq)
}
pmsg.returnToPool()
pmsg = nil
goto waitForMsgs
}
@@ -4974,6 +5013,7 @@ func (o *consumer) loopAndGatherMsgs(qch chan struct{}) {
select {
case <-qch:
pmsg.returnToPool()
pmsg = nil
return
case <-time.After(delay):
}
@@ -4994,6 +5034,7 @@ func (o *consumer) loopAndGatherMsgs(qch chan struct{}) {
select {
case <-qch:
pmsg.returnToPool()
pmsg = nil
return
case <-time.After(delay):
}
@@ -5097,17 +5138,22 @@ func (o *consumer) setMaxPendingBytes(limit int) {
// The race is a getNextMsg skips a deleted msg, and then the decStreamPending call fires.
// This does some quick sanity checks to see if we should re-calculate num pending.
// Lock should be held.
func (o *consumer) checkNumPending() uint64 {
func (o *consumer) checkNumPending() (uint64, error) {
if o.mset != nil && o.mset.store != nil {
var state StreamState
o.mset.store.FastState(&state)
npc := o.numPending()
if o.sseq > state.LastSeq && npc > 0 || npc > state.Msgs {
// Re-calculate.
o.streamNumPending()
// Make sure we can't report more messages than there are.
// TODO(nat): It's not great that this means consumer info has side effects,
// since we can't know whether anyone will call it or not. The previous num
// pending calculation that this replaces had the same problem though.
if o.sseq > state.LastSeq {
o.npc = 0
} else if npc > 0 {
o.npc = int64(min(npc, state.Msgs, state.LastSeq-o.sseq+1))
}
}
return o.numPending()
return o.numPending(), nil
}
// Lock should be held.
@@ -5134,7 +5180,7 @@ func (o *consumer) checkNumPendingOnEOF() {
}
// Call into streamNumPending after acquiring the consumer lock.
func (o *consumer) streamNumPendingLocked() uint64 {
func (o *consumer) streamNumPendingLocked() (uint64, error) {
o.mu.Lock()
defer o.mu.Unlock()
return o.streamNumPending()
@@ -5143,22 +5189,25 @@ func (o *consumer) streamNumPendingLocked() uint64 {
// Will force a set from the stream store of num pending.
// Depends on delivery policy, for last per subject we calculate differently.
// Lock should be held.
func (o *consumer) streamNumPending() uint64 {
func (o *consumer) streamNumPending() (uint64, error) {
if o.mset == nil || o.mset.store == nil {
o.npc, o.npf = 0, 0
return 0
return 0, nil
}
npc, npf, err := o.calculateNumPending()
if err != nil {
return 0, err
}
npc, npf := o.calculateNumPending()
o.npc, o.npf = int64(npc), npf
return o.numPending()
return o.numPending(), nil
}
// Will calculate num pending but only requires a read lock.
// Depends on delivery policy, for last per subject we calculate differently.
// At least RLock should be held.
func (o *consumer) calculateNumPending() (npc, npf uint64) {
func (o *consumer) calculateNumPending() (npc, npf uint64, err error) {
if o.mset == nil || o.mset.store == nil {
return 0, 0
return 0, 0, nil
}
isLastPerSubject := o.cfg.DeliverPolicy == DeliverLastPerSubject
@@ -5349,6 +5398,15 @@ func (o *consumer) trackPending(sseq, dseq uint64) {
o.pending = make(map[uint64]*Pending)
}
now := time.Now()
if p, ok := o.pending[sseq]; ok {
// Update timestamp but keep original consumer delivery sequence.
// So do not update p.Sequence.
p.Timestamp = now.UnixNano()
} else {
o.pending[sseq] = &Pending{dseq, now.UnixNano()}
}
// We could have a backoff that set a timer higher than what we need for this message.
// In that case, reset to lowest backoff required for a message redelivery.
minDelay := o.ackWait(0)
@@ -5361,18 +5419,10 @@ func (o *consumer) trackPending(sseq, dseq uint64) {
}
minDelay = o.ackWait(o.cfg.BackOff[bi])
}
minDeadline := time.Now().Add(minDelay)
minDeadline := now.Add(minDelay)
if o.ptmr == nil || o.ptmrEnd.After(minDeadline) {
o.resetPtmr(minDelay)
}
if p, ok := o.pending[sseq]; ok {
// Update timestamp but keep original consumer delivery sequence.
// So do not update p.Sequence.
p.Timestamp = time.Now().UnixNano()
} else {
o.pending[sseq] = &Pending{dseq, time.Now().UnixNano()}
}
}
// Credit back a failed delivery.
@@ -6445,6 +6495,10 @@ func (o *consumer) checkStateForInterestStream(ss *StreamState) error {
if asflr&(1<<63) != 0 {
return errAckFloorInvalid
}
dflr := asflr
if len(state.Pending) > 0 && state.Delivered.Stream > dflr {
dflr = state.Delivered.Stream
}
// Check if the underlying stream's last sequence is less than our floor.
// This can happen if the stream has been reset and has not caught up yet.
@@ -6463,7 +6517,7 @@ func (o *consumer) checkStateForInterestStream(ss *StreamState) error {
}
var retryAsflr uint64
for seq = fseq; asflr > 0 && seq <= asflr; seq++ {
for seq = fseq; dflr > 0 && seq <= dflr; seq++ {
if filters != nil {
_, nseq, err = store.LoadNextMsgMulti(filters, seq, &smv)
} else {
@@ -6473,19 +6527,43 @@ func (o *consumer) checkStateForInterestStream(ss *StreamState) error {
if nseq > seq {
seq = nseq
}
// Only ack though if no error and seq <= ack floor.
if err == nil && seq <= asflr {
didRemove := mset.ackMsg(o, seq)
// Removing the message could fail. For example if clustered since we need to propose it.
// Overwrite retry floor (only the first time) to allow us to check next time if the removal was successful.
if didRemove && retryAsflr == 0 {
retryAsflr = seq
if err == nil {
// Only ack though if no error and seq <= ack floor.
if seq <= asflr {
didRemove := mset.ackMsg(o, seq)
// Removing the message could fail. For example if clustered since we need to propose it.
// Overwrite retry floor (only the first time) to allow us to check next time if the removal was successful.
if didRemove && retryAsflr == 0 {
retryAsflr = seq
}
} else if seq <= dflr {
// Store the first entry above our ack floor, so we don't need to look it up again on retryAsflr=0.
if retryAsflr == 0 {
retryAsflr = seq
}
// If we have pending, we will need to walk through to delivered in case we missed any of those acks as well.
if _, ok := state.Pending[seq]; !ok {
// The filters are already taken into account,
mset.ackMsg(o, seq)
}
}
} else if err == ErrStoreEOF {
break
}
}
// If retry floor was not overwritten, set to ack floor+1, we don't need to account for any retries below it.
// However, our ack floor may be lower than the next message we can receive, so we correct it upward if needed.
if retryAsflr == 0 {
retryAsflr = asflr + 1
if filters != nil {
_, nseq, err = store.LoadNextMsgMulti(filters, asflr+1, &smv)
} else {
_, nseq, err = store.LoadNextMsg(filter, wc, asflr+1, &smv)
}
if err == nil {
retryAsflr = max(asflr+1, nseq)
} else if err == ErrStoreEOF {
retryAsflr = ss.LastSeq + 1
}
}
o.mu.Lock()
@@ -6494,21 +6572,7 @@ func (o *consumer) checkStateForInterestStream(ss *StreamState) error {
if retryAsflr > o.chkflr {
o.chkflr = retryAsflr
}
// See if we need to process this update if our parent stream is not a limits policy stream.
state, _ = o.store.State()
o.mu.Unlock()
// If we have pending, we will need to walk through to delivered in case we missed any of those acks as well.
if state != nil && len(state.Pending) > 0 && state.AckFloor.Stream > 0 {
for seq := state.AckFloor.Stream + 1; seq <= state.Delivered.Stream; seq++ {
if _, ok := state.Pending[seq]; !ok {
// Want to call needAck since it is filter aware.
if o.needAck(seq, _EMPTY_) {
mset.ackMsg(o, seq)
}
}
}
}
return nil
}
+6
View File
@@ -215,6 +215,9 @@ var (
// ErrMinimumVersionRequired is returned when a connection is not at the minimum version required.
ErrMinimumVersionRequired = errors.New("minimum version required")
// ErrLeafNodeMinVersionRejected is the leafnode protocol error prefix used
// when rejecting a remote due to leafnodes.min_version.
ErrLeafNodeMinVersionRejected = errors.New("connection rejected since minimum version required is")
// ErrInvalidMappingDestination is used for all subject mapping destination errors
ErrInvalidMappingDestination = errors.New("invalid mapping destination")
@@ -251,6 +254,9 @@ type mappingDestinationErr struct {
}
func (e *mappingDestinationErr) Error() string {
if e.token == _EMPTY_ {
return e.err.Error()
}
return fmt.Sprintf("%s in %s", e.err, e.token)
}
+10
View File
@@ -1998,5 +1998,15 @@
"help": "",
"url": "",
"deprecates": ""
},
{
"constant": "JSClusterServerMemberChangeInflightErr",
"code": 400,
"error_code": 10202,
"description": "cluster member change is in progress",
"comment": "",
"help": "",
"url": "",
"deprecates": ""
}
]
+15
View File
@@ -1500,6 +1500,8 @@ func (s *Server) initEventTracking() {
type UserInfo struct {
UserID string `json:"user"`
Account string `json:"account"`
AccountName string `json:"account_name,omitempty"`
UserName string `json:"user_name,omitempty"`
Permissions *Permissions `json:"permissions,omitempty"`
Expires time.Duration `json:"expires,omitempty"`
}
@@ -1519,9 +1521,22 @@ func (s *Server) userInfoReq(sub *subscription, c *client, _ *Account, subject,
return
}
// Look up the requester's account directly from ci.Account rather than
// using the acc returned by getRequestInfo, which may resolve to the
// service account (ci.Service) when the request arrives via a chained
// service import.
var accountName string
if ci.Account != _EMPTY_ {
if reqAcc, _ := s.LookupAccount(ci.Account); reqAcc != nil {
accountName = reqAcc.getNameTag()
}
}
response.Data = &UserInfo{
UserID: ci.User,
Account: ci.Account,
AccountName: accountName,
UserName: ci.NameTag,
Permissions: c.publicPermissions(),
Expires: c.claimExpiration(),
}
File diff suppressed because it is too large. Load diff
+49 -86
View File
@@ -17,9 +17,6 @@ import (
"errors"
"strings"
"sync"
"unsafe"
"github.com/nats-io/nats-server/v2/server/stree"
)
// Sublist is a routing mechanism to handle subject distribution and
@@ -51,6 +48,11 @@ var (
// unnecessary allocations.
type SimpleSublist = GenericSublist[struct{}]
// NewSimpleSublist will create a simple sublist.
func NewSimpleSublist() *SimpleSublist {
return &GenericSublist[struct{}]{root: newLevel[struct{}]()}
}
// A GenericSublist stores and efficiently retrieves subscriptions.
type GenericSublist[T comparable] struct {
sync.RWMutex
@@ -246,7 +248,9 @@ func matchLevelForAny[T comparable](l *level[T], toks []string, np *int) bool {
if np != nil {
*np += len(n.subs)
}
return len(n.subs) > 0
if len(n.subs) > 0 {
return true
}
}
if pwc != nil {
if np != nil {
@@ -365,6 +369,36 @@ func (s *GenericSublist[T]) Remove(subject string, value T) error {
return s.remove(subject, value, true)
}
// HasInterestStartingIn is a helper for subject tree intersection.
func (s *GenericSublist[T]) HasInterestStartingIn(subj string) bool {
s.RLock()
defer s.RUnlock()
var _tokens [64]string
tokens := tokenizeSubjectIntoSlice(_tokens[:0], subj)
return hasInterestStartingIn(s.root, tokens)
}
func hasInterestStartingIn[T comparable](l *level[T], tokens []string) bool {
if l == nil {
return false
}
if len(tokens) == 0 {
return true
}
token := tokens[0]
if l.fwc != nil {
return true
}
found := false
if pwc := l.pwc; pwc != nil {
found = found || hasInterestStartingIn(pwc.next, tokens[1:])
}
if n := l.nodes[token]; n != nil {
found = found || hasInterestStartingIn(n.next, tokens[1:])
}
return found
}
// pruneNode is used to prune an empty node from the tree.
func (l *level[T]) pruneNode(n *node[T], t string) {
if n == nil {
@@ -458,86 +492,15 @@ func visitLevel[T comparable](l *level[T], depth int) int {
return maxDepth
}
// IntersectStree will match all items in the given subject tree that
// have interest expressed in the given sublist. The callback will only be called
// once for each subject, regardless of overlapping subscriptions in the sublist.
func IntersectStree[T1 any, T2 comparable](st *stree.SubjectTree[T1], sl *GenericSublist[T2], cb func(subj []byte, entry *T1)) {
var _subj [255]byte
intersectStree(st, sl.root, _subj[:0], cb)
}
func intersectStree[T1 any, T2 comparable](st *stree.SubjectTree[T1], r *level[T2], subj []byte, cb func(subj []byte, entry *T1)) {
nsubj := subj
if len(nsubj) > 0 {
nsubj = append(subj, '.')
}
if r.fwc != nil {
// We've reached a full wildcard, do a FWC match on the stree at this point
// and don't keep iterating downward.
nsubj := append(nsubj, '>')
st.Match(nsubj, cb)
return
}
if r.pwc != nil {
// We've found a partial wildcard. We'll keep iterating downwards, but first
// check whether there's interest at this level (without triggering dupes) and
// match if so.
var done bool
nsubj := append(nsubj, '*')
if len(r.pwc.subs) > 0 {
st.Match(nsubj, cb)
done = true
}
if r.pwc.next.numNodes() > 0 {
intersectStree(st, r.pwc.next, nsubj, cb)
}
if done {
return
}
}
// Normal node with subject literals, keep iterating.
for t, n := range r.nodes {
if r.pwc != nil && r.pwc.next.numNodes() > 0 && n.next.numNodes() > 0 {
// A wildcard at the next level will already visit these descendents
// so skip so we don't callback the same subject more than once.
continue
}
nsubj := append(nsubj, t...)
if len(n.subs) > 0 {
if subjectHasWildcard(bytesToString(nsubj)) {
st.Match(nsubj, cb)
} else {
if e, ok := st.Find(nsubj); ok {
cb(nsubj, e)
}
}
}
if n.next.numNodes() > 0 {
intersectStree(st, n.next, nsubj, cb)
}
}
}
// Determine if a subject has any wildcard tokens.
func subjectHasWildcard(subject string) bool {
// This one exits earlier then !subjectIsLiteral(subject)
for i, c := range subject {
if c == pwc || c == fwc {
if (i == 0 || subject[i-1] == btsep) &&
(i+1 == len(subject) || subject[i+1] == btsep) {
return true
}
}
}
return false
}
// Note this will avoid a copy of the data used for the string, but it will also reference the existing slice's data pointer.
// So this should be used sparingly when we know the encompassing byte slice's lifetime is the same.
func bytesToString(b []byte) string {
if len(b) == 0 {
return _EMPTY_
}
p := unsafe.SliceData(b)
return unsafe.String(p, len(b))
// use similar to append. meaning, the updated slice will be returned
func tokenizeSubjectIntoSlice(tts []string, subject string) []string {
start := 0
for i := 0; i < len(subject); i++ {
if subject[i] == btsep {
tts = append(tts, subject[start:i])
start = i + 1
}
}
tts = append(tts, subject[start:])
return tts
}
+17 -18
View File
@@ -1,4 +1,4 @@
// Copyright 2019-2025 The NATS Authors
// Copyright 2019-2026 The NATS Authors
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
@@ -1142,6 +1142,12 @@ func (a *Account) EnableJetStream(limits map[string]JetStreamAccountLimits, tq c
}
js.mu.Lock()
// Accounts get reset to nil on shutdown, since we re-acquire the locks here, we need to check again.
if js.accounts == nil {
js.mu.Unlock()
return NewJSNotEnabledError()
}
if jsa, ok := js.accounts[a.Name]; ok {
a.mu.Lock()
a.js = jsa
@@ -1370,7 +1376,7 @@ func (a *Account) EnableJetStream(limits map[string]JetStreamAccountLimits, tq c
}
obs, err := mset.addConsumerWithAssignment(&cfg.ConsumerConfig, _EMPTY_, nil, true, ActionCreateOrUpdate, false)
if err != nil {
s.Warnf(" Error adding consumer %q: %v", cfg.Name, err)
s.Warnf(" Error adding consumer '%s > %s > %s': %v", a.Name, mset.name(), cfg.Name, err)
continue
}
if isEphemeral {
@@ -1379,9 +1385,6 @@ func (a *Account) EnableJetStream(limits map[string]JetStreamAccountLimits, tq c
if !cfg.Created.IsZero() {
obs.setCreatedTime(cfg.Created)
}
if err != nil {
s.Warnf(" Error restoring consumer %q state: %v", cfg.Name, err)
}
}
}
@@ -1560,7 +1563,7 @@ func (a *Account) EnableJetStream(limits map[string]JetStreamAccountLimits, tq c
}
// Add in the stream.
mset, err := a.addStream(&cfg.StreamConfig)
mset, err := a.recoverStream(&cfg.StreamConfig)
if err != nil {
s.Warnf(" Error recreating stream %q: %v", cfg.Name, err)
// If we removed a keyfile from above make sure to put it back.
@@ -2362,8 +2365,10 @@ func tierName(replicas int) string {
}
func isSameTier(cfgA, cfgB *StreamConfig) bool {
a := max(1, cfgA.Replicas)
b := max(1, cfgB.Replicas)
// TODO (mh) this is where we could select based off a placement tag as well "qos:tier"
return cfgA.Replicas == cfgB.Replicas
return a == b
}
func (jsa *jsAccount) jetStreamAndClustered() (*jetStream, bool) {
@@ -2438,17 +2443,11 @@ func (jsa *jsAccount) wouldExceedLimits(storeType StorageType, tierName string,
// Since tiers are flat we need to scale limit up by replicas when checking.
if storeType == MemoryStorage {
totalMem := inUse.total.mem + (int64(memStoreMsgSize(subj, hdr, msg)) * r)
if selectedLimits.MemoryMaxStreamBytes > 0 && totalMem > selectedLimits.MemoryMaxStreamBytes*lr {
return true, nil
}
if selectedLimits.MaxMemory >= 0 && totalMem > selectedLimits.MaxMemory*lr {
return true, nil
}
} else {
totalStore := inUse.total.store + (int64(fileStoreMsgSize(subj, hdr, msg)) * r)
if selectedLimits.StoreMaxStreamBytes > 0 && totalStore > selectedLimits.StoreMaxStreamBytes*lr {
return true, nil
}
if selectedLimits.MaxStore >= 0 && totalStore > selectedLimits.MaxStore*lr {
return true, nil
}
@@ -2487,25 +2486,25 @@ func (js *jetStream) checkBytesLimits(selectedLimits *JetStreamAccountLimits, ad
if addBytes < 0 {
addBytes = 1
}
totalBytes := addBytes + maxBytesOffset
totalBytes := addSaturate(addBytes, maxBytesOffset)
switch storage {
case MemoryStorage:
// Account limits defined.
if selectedLimits.MaxMemory >= 0 && currentRes+totalBytes > selectedLimits.MaxMemory {
if selectedLimits.MaxMemory >= 0 && (currentRes > selectedLimits.MaxMemory || totalBytes > selectedLimits.MaxMemory-currentRes) {
return NewJSMemoryResourcesExceededError()
}
// Check if this server can handle request.
if checkServer && js.memReserved+totalBytes > js.config.MaxMemory {
if checkServer && (js.memReserved > js.config.MaxMemory || totalBytes > js.config.MaxMemory-js.memReserved) {
return NewJSMemoryResourcesExceededError()
}
case FileStorage:
// Account limits defined.
if selectedLimits.MaxStore >= 0 && currentRes+totalBytes > selectedLimits.MaxStore {
if selectedLimits.MaxStore >= 0 && (currentRes > selectedLimits.MaxStore || totalBytes > selectedLimits.MaxStore-currentRes) {
return NewJSStorageResourcesExceededError()
}
// Check if this server can handle request.
if checkServer && js.storeReserved+totalBytes > js.config.MaxStore {
if checkServer && (js.storeReserved > js.config.MaxStore || totalBytes > js.config.MaxStore-js.storeReserved) {
return NewJSStorageResourcesExceededError()
}
}
File diff suppressed because it is too large. Load diff
+35 -30
View File
@@ -611,41 +611,46 @@ func checkMsgHeadersPreClusteredProposal(
// We need to deny here otherwise we'd need to bump CLFS, and it could succeed on some
// peers and not others depending on consumer ack state (if interest policy).
// So we deny here, if we allow that means we know it would succeed on every peer.
if discard == DiscardNew && (maxMsgs > 0 || maxBytes > 0) {
// Error if over DiscardNew per subject threshold.
if discardNewPer {
totalMsgsForSubject := i.ops
if i, ok = mset.inflight[subject]; ok {
totalMsgsForSubject += i.ops
if discard == DiscardNew {
if maxMsgs > 0 || maxBytes > 0 {
// Track usual max msgs/bytes thresholds for DiscardNew.
var state StreamState
mset.store.FastState(&state)
totalMsgs := state.Msgs
totalBytes := state.Bytes
for _, i = range mset.inflight {
totalMsgs += i.ops
totalBytes += i.bytes
}
if maxMsgsPer > 0 && totalMsgsForSubject > uint64(maxMsgsPer) {
err = ErrMaxMsgsPerSubject
for _, i = range diff.inflight {
totalMsgs += i.ops
totalBytes += i.bytes
}
if maxMsgs > 0 && totalMsgs > uint64(maxMsgs) {
err = ErrMaxMsgs
} else if maxBytes > 0 && totalBytes > uint64(maxBytes) {
err = ErrMaxBytes
}
if err != nil {
return hdr, msg, 0, NewJSStreamStoreFailedError(err, Unless(err)), err
}
}
// Track usual max msgs/bytes thresholds for DiscardNew.
var state StreamState
mset.store.FastState(&state)
totalMsgs := state.Msgs
totalBytes := state.Bytes
for _, i = range mset.inflight {
totalMsgs += i.ops
totalBytes += i.bytes
}
for _, i = range diff.inflight {
totalMsgs += i.ops
totalBytes += i.bytes
}
if maxMsgs > 0 && totalMsgs > uint64(maxMsgs) {
err = ErrMaxMsgs
} else if maxBytes > 0 && totalBytes > uint64(maxBytes) {
err = ErrMaxBytes
}
if err != nil {
return hdr, msg, 0, NewJSStreamStoreFailedError(err, Unless(err)), err
// Similarly, check DiscardNew per-subject threshold to not need to bump CLFS.
if discardNewPer && maxMsgsPer > 0 {
// Get the current total for this subject.
totalMsgsForSubject := mset.store.SubjectsTotals(subject)[subject]
// Add inflight count in this batch and for this stream.
totalMsgsForSubject += i.ops
if i, ok = mset.inflight[subject]; ok {
totalMsgsForSubject += i.ops
}
if totalMsgsForSubject > uint64(maxMsgsPer) {
err = ErrMaxMsgsPerSubject
return hdr, msg, 0, NewJSStreamStoreFailedError(err, Unless(err)), err
}
}
}
File diff suppressed because it is too large. Load diff
@@ -59,6 +59,9 @@ const (
// JSClusterRequiredErr JetStream clustering support required
JSClusterRequiredErr ErrorIdentifier = 10010
// JSClusterServerMemberChangeInflightErr cluster member change is in progress
JSClusterServerMemberChangeInflightErr ErrorIdentifier = 10202
// JSClusterServerNotMemberErr server is not a member of the cluster
JSClusterServerNotMemberErr ErrorIdentifier = 10044
@@ -626,6 +629,7 @@ var (
JSClusterNotLeaderErr: {Code: 500, ErrCode: 10009, Description: "JetStream cluster can not handle request"},
JSClusterPeerNotMemberErr: {Code: 400, ErrCode: 10040, Description: "peer not a member"},
JSClusterRequiredErr: {Code: 503, ErrCode: 10010, Description: "JetStream clustering support required"},
JSClusterServerMemberChangeInflightErr: {Code: 400, ErrCode: 10202, Description: "cluster member change is in progress"},
JSClusterServerNotMemberErr: {Code: 400, ErrCode: 10044, Description: "server is not a member of the cluster"},
JSClusterTagsErr: {Code: 400, ErrCode: 10011, Description: "tags placement not supported for operation"},
JSClusterUnSupportFeatureErr: {Code: 503, ErrCode: 10036, Description: "not currently supported in clustered mode"},
@@ -1031,6 +1035,16 @@ func NewJSClusterRequiredError(opts ...ErrorOption) *ApiError {
return ApiErrors[JSClusterRequiredErr]
}
// NewJSClusterServerMemberChangeInflightError creates a new JSClusterServerMemberChangeInflightErr error: "cluster member change is in progress"
func NewJSClusterServerMemberChangeInflightError(opts ...ErrorOption) *ApiError {
eopts := parseOpts(opts)
if ae, ok := eopts.err.(*ApiError); ok {
return ae
}
return ApiErrors[JSClusterServerMemberChangeInflightErr]
}
// NewJSClusterServerNotMemberError creates a new JSClusterServerNotMemberErr error: "server is not a member of the cluster"
func NewJSClusterServerNotMemberError(opts ...ErrorOption) *ApiError {
eopts := parseOpts(opts)
+1 -1
View File
@@ -17,7 +17,7 @@ import "strconv"
const (
// JSApiLevel is the maximum supported JetStream API level for this server.
JSApiLevel int = 2
JSApiLevel int = 3
JSRequiredLevelMetadataKey = "_nats.req.level"
JSServerVersionMetadataKey = "_nats.ver"
+38 -23
View File
@@ -63,9 +63,9 @@ const (
// LEAF connection as opposed to a CLIENT.
leafNodeWSPath = "/leafnode"
// This is the time the server will wait, when receiving a CONNECT,
// before closing the connection if the required minimum version is not met.
leafNodeWaitBeforeClose = 5 * time.Second
// When a soliciting leafnode is rejected because it does not meet the
// configured minimum version, delay the next reconnect attempt by this long.
leafNodeMinVersionReconnectDelay = 5 * time.Second
)
type leaf struct {
@@ -691,9 +691,8 @@ func (s *Server) connectToRemoteLeafNode(remote *leafNodeCfg, firstConnect bool)
} else {
s.Debugf("Trying to connect as leafnode to remote server on %q%s", rURL.Host, ipStr)
// Check if proxy is configured first, then check if URL supports it
if proxyURL != _EMPTY_ && isWSURL(rURL) {
// Use proxy for WebSocket connections - use original hostname, resolved IP for connection
// Check if proxy is configured
if proxyURL != _EMPTY_ {
targetHost := rURL.Host
// If URL doesn't include port, add the default port for the scheme
if rURL.Port() == _EMPTY_ {
@@ -2082,17 +2081,11 @@ func (c *client) processLeafNodeConnect(s *Server, arg []byte, lang string) erro
if mv := s.getOpts().LeafNode.MinVersion; mv != _EMPTY_ {
major, minor, update, _ := versionComponents(mv)
if !versionAtLeast(proto.Version, major, minor, update) {
// We are going to send back an INFO because otherwise recent
// versions of the remote server would simply break the connection
// after 2 seconds if not receiving it. Instead, we want the
// other side to just "stall" until we finish waiting for the holding
// period and close the connection below.
// Send back an INFO so recent remote servers process the rejection
// cleanly, then close immediately. The soliciting side applies the
// reconnect delay when it processes the error.
s.sendPermsAndAccountInfo(c)
c.sendErrAndErr(fmt.Sprintf("connection rejected since minimum version required is %q", mv))
select {
case <-c.srv.quitCh:
case <-time.After(leafNodeWaitBeforeClose):
}
c.sendErrAndErr(fmt.Sprintf("%s %q", ErrLeafNodeMinVersionRejected, mv))
c.closeConnection(MinimumVersionRequired)
return ErrMinimumVersionRequired
}
@@ -2760,6 +2753,14 @@ func (c *client) processLeafSub(argo []byte) (err error) {
}
acc := c.acc
// Guard against LS+ arriving before CONNECT has been processed, which
// can happen when compression is enabled.
if acc == nil {
c.mu.Unlock()
c.sendErr("Authorization Violation")
c.closeConnection(ProtocolViolation)
return nil
}
// Check if we have a loop.
ldsPrefix := bytes.HasPrefix(sub.subject, []byte(leafNodeLoopDetectionSubjectPrefix))
@@ -2832,7 +2833,7 @@ func (c *client) processLeafSub(argo []byte) (err error) {
// Only add in shadow subs if a new sub or qsub.
if osub == nil {
if err := c.addShadowSubscriptions(acc, sub, true); err != nil {
if err := c.addShadowSubscriptions(acc, sub); err != nil {
c.Errorf(err.Error())
}
}
@@ -2876,7 +2877,6 @@ func (c *client) processLeafUnsub(arg []byte) error {
// Indicate any activity, so pub and sub or unsubs.
c.in.subs++
acc := c.acc
srv := c.srv
c.mu.Lock()
@@ -2885,6 +2885,15 @@ func (c *client) processLeafUnsub(arg []byte) error {
return nil
}
acc := c.acc
// Guard against LS- arriving before CONNECT has been processed.
if acc == nil {
c.mu.Unlock()
c.sendErr("Authorization Violation")
c.closeConnection(ProtocolViolation)
return nil
}
spoke := c.isSpokeLeafNode()
// We store local subs by account and subject and optionally queue name.
// LS- will have the arg exactly as the key.
@@ -2916,8 +2925,7 @@ func (c *client) processLeafUnsub(arg []byte) error {
func (c *client) processLeafHeaderMsgArgs(arg []byte) error {
// Unroll splitArgs to avoid runtime/heap issues
a := [MAX_MSG_ARGS][]byte{}
args := a[:0]
args := c.argsa[:0]
start := -1
for i, b := range arg {
switch b {
@@ -3000,8 +3008,7 @@ func (c *client) processLeafHeaderMsgArgs(arg []byte) error {
func (c *client) processLeafMsgArgs(arg []byte) error {
// Unroll splitArgs to avoid runtime/heap issues
a := [MAX_MSG_ARGS][]byte{}
args := a[:0]
args := c.argsa[:0]
start := -1
for i, b := range arg {
switch b {
@@ -3181,6 +3188,11 @@ func (c *client) leafProcessErr(errStr string) {
c.Errorf("Leafnode connection dropped with same cluster name error. Delaying attempt to reconnect for %v", delay)
return
}
if strings.Contains(errStr, ErrLeafNodeMinVersionRejected.Error()) {
_, delay := c.setLeafConnectDelayIfSoliciting(leafNodeMinVersionReconnectDelay)
c.Errorf("Leafnode connection dropped due to minimum version requirement. Delaying attempt to reconnect for %v", delay)
return
}
// We will look for Loop detected error coming from the other side.
// If we solicit, set the connect delay.
@@ -3203,7 +3215,10 @@ func (c *client) setLeafConnectDelayIfSoliciting(delay time.Duration) (string, t
}
c.leaf.remote.setConnectDelay(delay)
}
accName := c.acc.Name
var accName string
if c.acc != nil {
accName = c.acc.Name
}
c.mu.Unlock()
return accName, delay
}
+194 -54
View File
@@ -19,7 +19,6 @@ import (
"fmt"
"math"
"slices"
"sort"
"sync"
"time"
@@ -445,7 +444,6 @@ func (ms *memStore) RegisterProcessJetStreamMsg(cb ProcessJetStreamMsgHandler) {
// GetSeqFromTime looks for the first sequence number that has the message
// with >= timestamp.
// FIXME(dlc) - inefficient.
func (ms *memStore) GetSeqFromTime(t time.Time) uint64 {
ts := t.UnixNano()
ms.mu.RLock()
@@ -469,18 +467,57 @@ func (ms *memStore) GetSeqFromTime(t time.Time) uint64 {
last := lmsg.ts
if ts == last {
return ms.state.LastSeq
return lmsg.seq
}
if ts > last {
return ms.state.LastSeq + 1
}
index := sort.Search(len(ms.msgs), func(i int) bool {
if msg := ms.msgs[ms.state.FirstSeq+uint64(i)]; msg != nil {
return msg.ts >= ts
var (
cts int64
cseq uint64
off uint64
)
// Using a binary search, but need to be aware of interior deletes.
fseq := ms.state.FirstSeq
lseq := ms.state.LastSeq
seq := lseq + 1
loop:
for fseq <= lseq {
mid := fseq + (lseq-fseq)/2
off = 0
// Potentially skip over gaps. We keep the original middle but keep track of a
// potential delete range with an offset.
for {
msg := ms.msgs[mid+off]
if msg == nil {
off++
if mid+off <= lseq {
continue
} else {
// Continue search to the left. Purposely ignore the skipped deletes here.
lseq = mid - 1
continue loop
}
}
cts = msg.ts
cseq = msg.seq
break
}
return false
})
return uint64(index) + ms.state.FirstSeq
if cts >= ts {
seq = cseq
if mid == fseq {
break
}
// Continue search to the left.
lseq = mid - 1
} else {
// Continue search to the right (potentially skipping over interior deletes).
fseq = mid + off + 1
}
}
return seq
}
// FilteredState will return the SimpleState associated with the filtered subject and a proposed starting sequence.
@@ -749,6 +786,10 @@ func (ms *memStore) allLastSeqsLocked() ([]uint64, error) {
seqs := make([]uint64, 0, ms.fss.Size())
ms.fss.IterFast(func(subj []byte, ss *SimpleState) bool {
// Check if we need to recalculate. We only care about the last sequence.
if ss.lastNeedsUpdate {
ms.recalculateForSubj(bytesToString(subj), ss)
}
seqs = append(seqs, ss.Last)
return true
})
@@ -766,6 +807,7 @@ func (ms *memStore) filterIsAll(filters []string) bool {
}
// Sort so we can compare.
slices.Sort(filters)
slices.Sort(ms.cfg.Subjects)
for i, subj := range filters {
if !subjectIsSubsetMatch(ms.cfg.Subjects[i], subj) {
return false
@@ -777,8 +819,8 @@ func (ms *memStore) filterIsAll(filters []string) bool {
// MultiLastSeqs will return a sorted list of sequences that match all subjects presented in filters.
// We will not exceed the maxSeq, which if 0 becomes the store's last sequence.
func (ms *memStore) MultiLastSeqs(filters []string, maxSeq uint64, maxAllowed int) ([]uint64, error) {
ms.mu.RLock()
defer ms.mu.RUnlock()
ms.mu.Lock()
defer ms.mu.Unlock()
if len(ms.msgs) == 0 {
return nil, nil
@@ -806,6 +848,9 @@ func (ms *memStore) MultiLastSeqs(filters []string, maxSeq uint64, maxAllowed in
for _, filter := range filters {
ms.fss.Match(stringToBytes(filter), func(subj []byte, ss *SimpleState) {
if ss.lastNeedsUpdate {
ms.recalculateForSubj(bytesToString(subj), ss)
}
if ss.Last <= maxSeq {
addIfNotDupe(ss.Last)
} else if ss.Msgs > 1 {
@@ -861,17 +906,17 @@ func (ms *memStore) subjectsTotalsLocked(filterSubject string) map[string]uint64
}
// NumPending will return the number of pending messages matching the filter subject starting at sequence.
func (ms *memStore) NumPending(sseq uint64, filter string, lastPerSubject bool) (total, validThrough uint64) {
func (ms *memStore) NumPending(sseq uint64, filter string, lastPerSubject bool) (total, validThrough uint64, err error) {
// This needs to be a write lock, as filteredStateLocked can mutate the per-subject state.
ms.mu.Lock()
defer ms.mu.Unlock()
ss := ms.filteredStateLocked(sseq, filter, lastPerSubject)
return ss.Msgs, ms.state.LastSeq
return ss.Msgs, ms.state.LastSeq, nil
}
// NumPending will return the number of pending messages matching any subject in the sublist starting at sequence.
func (ms *memStore) NumPendingMulti(sseq uint64, sl *gsl.SimpleSublist, lastPerSubject bool) (total, validThrough uint64) {
func (ms *memStore) NumPendingMulti(sseq uint64, sl *gsl.SimpleSublist, lastPerSubject bool) (total, validThrough uint64, err error) {
if sl == nil {
return ms.NumPending(sseq, fwcs, lastPerSubject)
}
@@ -886,7 +931,7 @@ func (ms *memStore) NumPendingMulti(sseq uint64, sl *gsl.SimpleSublist, lastPerS
}
// If past the end no results.
if sseq > ms.state.LastSeq {
return 0, ms.state.LastSeq
return 0, ms.state.LastSeq, nil
}
update := func(fss *SimpleState) {
@@ -906,7 +951,7 @@ func (ms *memStore) NumPendingMulti(sseq uint64, sl *gsl.SimpleSublist, lastPerS
var havePartial bool
var totalSkipped uint64
// We will track start and end sequences as we go.
gsl.IntersectStree[SimpleState](ms.fss, sl, func(subj []byte, fss *SimpleState) {
stree.IntersectGSL[SimpleState](ms.fss, sl, func(subj []byte, fss *SimpleState) {
if fss.firstNeedsUpdate || fss.lastNeedsUpdate {
ms.recalculateForSubj(bytesToString(subj), fss)
}
@@ -924,7 +969,7 @@ func (ms *memStore) NumPendingMulti(sseq uint64, sl *gsl.SimpleSublist, lastPerS
// If we did not encounter any partials we can return here.
if !havePartial {
return ss.Msgs, ms.state.LastSeq
return ss.Msgs, ms.state.LastSeq, nil
}
// If we are here we need to scan the msgs.
@@ -1015,7 +1060,7 @@ func (ms *memStore) NumPendingMulti(sseq uint64, sl *gsl.SimpleSublist, lastPerS
ss.Msgs -= adjust
}
return ss.Msgs, ms.state.LastSeq
return ss.Msgs, ms.state.LastSeq, nil
}
// Will check the msg limit for this tracked subject.
@@ -1463,6 +1508,12 @@ func (ms *memStore) compact(seq uint64) (uint64, error) {
var purged, bytes uint64
ms.mu.Lock()
// Short-circuit if the store was already compacted past this point.
if ms.state.FirstSeq > seq {
ms.mu.Unlock()
return purged, nil
}
cb := ms.scb
if seq <= ms.state.LastSeq {
fseq := ms.state.FirstSeq
@@ -1621,7 +1672,8 @@ func (ms *memStore) SubjectForSeq(seq uint64) (string, error) {
return _EMPTY_, ErrStoreMsgNotFound
}
if sm, ok := ms.msgs[seq]; ok {
return sm.subj, nil
// Copy the subject, as it's used elsewhere, and we've released the lock in the meantime.
return copyString(sm.subj), nil
}
return _EMPTY_, ErrStoreMsgNotFound
}
@@ -1673,6 +1725,10 @@ func (ms *memStore) LoadLastMsg(subject string, smp *StoreMsg) (*StoreMsg, error
} else if subjectIsLiteral(subject) {
var ss *SimpleState
if ss, ok = ms.fss.Find(stringToBytes(subject)); ok && ss.Msgs > 0 {
// Check if we need to recalculate. We only care about the last sequence.
if ss.lastNeedsUpdate {
ms.recalculateForSubj(subject, ss)
}
sm, ok = ms.msgs[ss.Last]
}
} else if ss := ms.filteredStateLocked(1, subject, true); ss.Msgs > 0 {
@@ -1731,6 +1787,78 @@ func (ms *memStore) LoadNextMsg(filter string, wc bool, start uint64, smp *Store
return ms.loadNextMsgLocked(filter, wc, start, smp)
}
// Find sequence bounds matching a wildcard filter from ms.fss.
// Returns (first, last, true) if there is at least one matching
// subject at or after start (start <= first <= last).
// Returns (0, 0, false) if the subject does not exist or has no
// messages at or after start.
// Lock should be held.
func (ms *memStore) nextWildcardMatchLocked(filter string, start uint64) (uint64, uint64, bool) {
found := false
first, last := ms.state.LastSeq, uint64(0)
ms.fss.MatchUntil(stringToBytes(filter), func(subj []byte, ss *SimpleState) bool {
ms.recalculateForSubj(string(subj), ss)
// Skip matches that are below our starting sequence
if start > ss.Last {
return true
}
// A match was found, adjust the bounds accordingly
found = true
if ss.First < first {
first = ss.First
}
if ss.Last > last {
last = ss.Last
}
// If first > start, there may be more matches between
// start and first, in which case we keep searching.
// If not, we have a match between start and last, we
// can break out of the search.
// This could be further optimized: if first and start
// are "close", we could just extend the linear search,
// especially if we know that the remaining ms.fss to
// explore is large.
return first > start
})
if !found {
return 0, 0, false
}
return max(first, start), last, found
}
// Find sequence bounds matching a literal filter from ms.fss.
// Returns (first, last, true) if there is a matching literal
// subject at or after start (start <= first <= last).
// Returns (0, 0, false) if the subject does not exist or has no
// messages at or after start.
// Lock should be held.
func (ms *memStore) nextLiteralMatchLocked(filter string, start uint64) (uint64, uint64, bool) {
ss, ok := ms.fss.Find(stringToBytes(filter))
if !ok {
return 0, 0, false
}
ms.recalculateForSubj(filter, ss)
if start > ss.Last {
return 0, 0, false
}
return max(start, ss.First), ss.Last, true
}
// Returns true if LoadNextMsg should perform a linear scan,
// false if it should use the subject tree to try to reduce
// the search space.
// Lock should be held.
func (ms *memStore) shouldLinearScan(filter string, wc bool, start uint64) bool {
// Skip scan of ms.fss if number of messages in the block are less than
// 1/2 the number of subjects in ms.fss. Or we have a wc and lots of fss entries.
const linearScanMaxFSS = 256
isAll := filter == fwcs
return isAll || 2*int(ms.state.LastSeq-start) < ms.fss.Size() || (wc && ms.fss.Size() > linearScanMaxFSS)
}
// Lock should be held.
func (ms *memStore) loadNextMsgLocked(filter string, wc bool, start uint64, smp *StoreMsg) (*StoreMsg, uint64, error) {
if start < ms.state.FirstSeq {
@@ -1747,46 +1875,24 @@ func (ms *memStore) loadNextMsgLocked(filter string, wc bool, start uint64, smp
}
isAll := filter == fwcs
// Skip scan of ms.fss if number of messages in the block are less than
// 1/2 the number of subjects in ms.fss. Or we have a wc and lots of fss entries.
const linearScanMaxFSS = 256
doLinearScan := isAll || 2*int(ms.state.LastSeq-start) < ms.fss.Size() || (wc && ms.fss.Size() > linearScanMaxFSS)
// Initial setup.
fseq, lseq := start, ms.state.LastSeq
if !doLinearScan {
subs := []string{filter}
if wc || isAll {
subs = subs[:0]
ms.fss.Match(stringToBytes(filter), func(subj []byte, val *SimpleState) {
subs = append(subs, string(subj))
})
if !ms.shouldLinearScan(filter, wc, start) {
var found bool
if wc {
fseq, lseq, found = ms.nextWildcardMatchLocked(filter, start)
} else {
fseq, lseq, found = ms.nextLiteralMatchLocked(filter, start)
}
fseq, lseq = ms.state.LastSeq, uint64(0)
for _, subj := range subs {
ss, ok := ms.fss.Find(stringToBytes(subj))
if !ok {
continue
}
if ss.firstNeedsUpdate || ss.lastNeedsUpdate {
ms.recalculateForSubj(subj, ss)
}
if ss.First < fseq {
fseq = ss.First
}
if ss.Last > lseq {
lseq = ss.Last
}
}
if fseq < start {
fseq = start
if !found {
return nil, ms.state.LastSeq, ErrStoreEOF
}
}
eq := subjectsEqual
if wc {
eq = subjectIsSubsetMatch
eq = matchLiteral
}
for nseq := fseq; nseq <= lseq; nseq++ {
@@ -1828,6 +1934,40 @@ func (ms *memStore) LoadPrevMsg(start uint64, smp *StoreMsg) (sm *StoreMsg, err
return nil, ErrStoreEOF
}
// LoadPrevMsgMulti will find the previous message matching any entry in the sublist.
func (ms *memStore) LoadPrevMsgMulti(sl *gsl.SimpleSublist, start uint64, smp *StoreMsg) (sm *StoreMsg, skip uint64, err error) {
// TODO(dlc) - for now simple linear walk to get started.
ms.mu.RLock()
defer ms.mu.RUnlock()
if start > ms.state.LastSeq {
start = ms.state.LastSeq
}
// If past the start no results.
if start < ms.state.FirstSeq || ms.state.Msgs == 0 {
return nil, ms.state.FirstSeq, ErrStoreEOF
}
// Initial setup.
fseq, lseq := start, ms.state.FirstSeq
for nseq := fseq; nseq >= lseq; nseq-- {
sm, ok := ms.msgs[nseq]
if !ok {
continue
}
if sl.HasInterest(sm.subj) {
if smp == nil {
smp = new(StoreMsg)
}
sm.copy(smp)
return smp, nseq, nil
}
}
return nil, ms.state.LastSeq, ErrStoreEOF
}
// RemoveMsg will remove the message from this store.
// Will return the number of bytes removed.
func (ms *memStore) RemoveMsg(seq uint64) (bool, error) {
@@ -2036,8 +2176,8 @@ func (ms *memStore) FastState(state *StreamState) {
}
func (ms *memStore) State() StreamState {
ms.mu.RLock()
defer ms.mu.RUnlock()
ms.mu.Lock()
defer ms.mu.Unlock()
state := ms.state
state.Consumers = ms.consumers
@@ -2129,7 +2269,7 @@ type consumerMemStore struct {
closed bool
}
func (ms *memStore) ConsumerStore(name string, cfg *ConsumerConfig) (ConsumerStore, error) {
func (ms *memStore) ConsumerStore(name string, _ time.Time, cfg *ConsumerConfig) (ConsumerStore, error) {
if ms == nil {
return nil, fmt.Errorf("memstore is nil")
}
@@ -2259,7 +2399,7 @@ func (o *consumerMemStore) Update(state *ConsumerState) error {
// Check to see if this is an outdated update.
if state.Delivered.Consumer < o.state.Delivered.Consumer || state.AckFloor.Stream < o.state.AckFloor.Stream {
return fmt.Errorf("old update ignored")
return ErrStoreOldUpdate
}
o.state.Delivered = state.Delivered
+72 -38
View File
@@ -1,4 +1,4 @@
// Copyright 2013-2025 The NATS Authors
// Copyright 2013-2026 The NATS Authors
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
@@ -500,31 +500,31 @@ func (s *Server) Connz(opts *ConnzOptions) (*Connz, error) {
switch sortOpt {
case ByCid, ByStart:
sort.Sort(byCid{pconns})
sort.Sort(SortByCid{pconns})
case BySubs:
sort.Sort(sort.Reverse(bySubs{pconns}))
sort.Sort(sort.Reverse(SortBySubs{pconns}))
case ByPending:
sort.Sort(sort.Reverse(byPending{pconns}))
sort.Sort(sort.Reverse(SortByPending{pconns}))
case ByOutMsgs:
sort.Sort(sort.Reverse(byOutMsgs{pconns}))
sort.Sort(sort.Reverse(SortByOutMsgs{pconns}))
case ByInMsgs:
sort.Sort(sort.Reverse(byInMsgs{pconns}))
sort.Sort(sort.Reverse(SortByInMsgs{pconns}))
case ByOutBytes:
sort.Sort(sort.Reverse(byOutBytes{pconns}))
sort.Sort(sort.Reverse(SortByOutBytes{pconns}))
case ByInBytes:
sort.Sort(sort.Reverse(byInBytes{pconns}))
sort.Sort(sort.Reverse(SortByInBytes{pconns}))
case ByLast:
sort.Sort(sort.Reverse(byLast{pconns}))
sort.Sort(sort.Reverse(SortByLast{pconns}))
case ByIdle:
sort.Sort(sort.Reverse(byIdle{pconns, c.Now}))
sort.Sort(sort.Reverse(SortByIdle{pconns, c.Now}))
case ByUptime:
sort.Sort(byUptime{pconns, time.Now()})
sort.Sort(SortByUptime{pconns, time.Now()})
case ByStop:
sort.Sort(sort.Reverse(byStop{pconns}))
sort.Sort(sort.Reverse(SortByStop{pconns}))
case ByReason:
sort.Sort(byReason{pconns})
sort.Sort(SortByReason{pconns})
case ByRTT:
sort.Sort(sort.Reverse(byRTT{pconns}))
sort.Sort(sort.Reverse(SortByRTT{pconns}))
}
minoff := c.Offset
@@ -1279,6 +1279,7 @@ type Varz struct {
SlowConsumersStats *SlowConsumersStats `json:"slow_consumer_stats"` // SlowConsumersStats are statistics about all detected Slow Consumer
StaleConnectionStats *StaleConnectionStats `json:"stale_connection_stats,omitempty"` // StaleConnectionStats are statistics about all detected Stale Connections
Proxies *ProxiesOptsVarz `json:"proxies,omitempty"` // Proxies hold information about network proxy devices
TLSCertNotAfter time.Time `json:"tls_cert_not_after,omitzero"` // TLSCertNotAfter is the expiration date of the TLS certificate of this server
}
// JetStreamVarz contains basic runtime information about jetstream
@@ -1291,34 +1292,36 @@ type JetStreamVarz struct {
// ClusterOptsVarz contains monitoring cluster information
type ClusterOptsVarz struct {
Name string `json:"name,omitempty"` // Name is the configured cluster name
Host string `json:"addr,omitempty"` // Host is the host the cluster listens on for connections
Port int `json:"cluster_port,omitempty"` // Port is the port the cluster listens on for connections
AuthTimeout float64 `json:"auth_timeout,omitempty"` // AuthTimeout is the time cluster connections have to complete authentication
URLs []string `json:"urls,omitempty"` // URLs is the list of cluster URLs
TLSTimeout float64 `json:"tls_timeout,omitempty"` // TLSTimeout is how long TLS operations have to complete
TLSRequired bool `json:"tls_required,omitempty"` // TLSRequired indicates if TLS is required for connections
TLSVerify bool `json:"tls_verify,omitempty"` // TLSVerify indicates if full verification of TLS connections is performed
PoolSize int `json:"pool_size,omitempty"` // PoolSize is the configured route connection pool size
WriteDeadline time.Duration `json:"write_deadline,omitempty"` // WriteDeadline is the maximum time writes to sockets have to complete
WriteTimeout string `json:"write_timeout,omitempty"` // WriteTimeout is the closure policy for write deadline errors
Name string `json:"name,omitempty"` // Name is the configured cluster name
Host string `json:"addr,omitempty"` // Host is the host the cluster listens on for connections
Port int `json:"cluster_port,omitempty"` // Port is the port the cluster listens on for connections
AuthTimeout float64 `json:"auth_timeout,omitempty"` // AuthTimeout is the time cluster connections have to complete authentication
URLs []string `json:"urls,omitempty"` // URLs is the list of cluster URLs
TLSTimeout float64 `json:"tls_timeout,omitempty"` // TLSTimeout is how long TLS operations have to complete
TLSRequired bool `json:"tls_required,omitempty"` // TLSRequired indicates if TLS is required for connections
TLSVerify bool `json:"tls_verify,omitempty"` // TLSVerify indicates if full verification of TLS connections is performed
PoolSize int `json:"pool_size,omitempty"` // PoolSize is the configured route connection pool size
WriteDeadline time.Duration `json:"write_deadline,omitempty"` // WriteDeadline is the maximum time writes to sockets have to complete
WriteTimeout string `json:"write_timeout,omitempty"` // WriteTimeout is the closure policy for write deadline errors
TLSCertNotAfter time.Time `json:"tls_cert_not_after,omitzero"` // TLSCertNotAfter is the expiration date of the TLS certificate
}
// GatewayOptsVarz contains monitoring gateway information
type GatewayOptsVarz struct {
Name string `json:"name,omitempty"` // Name is the configured cluster name
Host string `json:"host,omitempty"` // Host is the host the gateway listens on for connections
Port int `json:"port,omitempty"` // Port is the post gateway connections listens on
AuthTimeout float64 `json:"auth_timeout,omitempty"` // AuthTimeout is the time cluster connections have to complete authentication
TLSTimeout float64 `json:"tls_timeout,omitempty"` // TLSTimeout is how long TLS operations have to complete
TLSRequired bool `json:"tls_required,omitempty"` // TLSRequired indicates if TLS is required for connections
TLSVerify bool `json:"tls_verify,omitempty"` // TLSVerify indicates if full verification of TLS connections is performed
Advertise string `json:"advertise,omitempty"` // Advertise is the URL advertised to remote gateway clients
ConnectRetries int `json:"connect_retries,omitempty"` // ConnectRetries is how many connection attempts the route will make
Gateways []RemoteGatewayOptsVarz `json:"gateways,omitempty"` // Gateways is state of configured gateway remotes
RejectUnknown bool `json:"reject_unknown,omitempty"` // RejectUnknown indicates if unknown cluster connections will be rejected
WriteDeadline time.Duration `json:"write_deadline,omitempty"` // WriteDeadline is the maximum time writes to sockets have to complete
WriteTimeout string `json:"write_timeout,omitempty"` // WriteTimeout is the closure policy for write deadline errors
Name string `json:"name,omitempty"` // Name is the configured cluster name
Host string `json:"host,omitempty"` // Host is the host the gateway listens on for connections
Port int `json:"port,omitempty"` // Port is the post gateway connections listens on
AuthTimeout float64 `json:"auth_timeout,omitempty"` // AuthTimeout is the time cluster connections have to complete authentication
TLSTimeout float64 `json:"tls_timeout,omitempty"` // TLSTimeout is how long TLS operations have to complete
TLSRequired bool `json:"tls_required,omitempty"` // TLSRequired indicates if TLS is required for connections
TLSVerify bool `json:"tls_verify,omitempty"` // TLSVerify indicates if full verification of TLS connections is performed
Advertise string `json:"advertise,omitempty"` // Advertise is the URL advertised to remote gateway clients
ConnectRetries int `json:"connect_retries,omitempty"` // ConnectRetries is how many connection attempts the route will make
Gateways []RemoteGatewayOptsVarz `json:"gateways,omitempty"` // Gateways is state of configured gateway remotes
RejectUnknown bool `json:"reject_unknown,omitempty"` // RejectUnknown indicates if unknown cluster connections will be rejected
WriteDeadline time.Duration `json:"write_deadline,omitempty"` // WriteDeadline is the maximum time writes to sockets have to complete
WriteTimeout string `json:"write_timeout,omitempty"` // WriteTimeout is the closure policy for write deadline errors
TLSCertNotAfter time.Time `json:"tls_cert_not_after,omitzero"` // TLSCertNotAfter is the expiration date of the TLS certificaet
}
// RemoteGatewayOptsVarz contains monitoring remote gateway information
@@ -1340,6 +1343,7 @@ type LeafNodeOptsVarz struct {
TLSOCSPPeerVerify bool `json:"tls_ocsp_peer_verify,omitempty"` // TLSOCSPPeerVerify indicates if OCSP verification will be performed
WriteDeadline time.Duration `json:"write_deadline,omitempty"` // WriteDeadline is the maximum time writes to sockets have to complete
WriteTimeout string `json:"write_timeout,omitempty"` // WriteTimeout is the closure policy for write deadline errors
TLSCertNotAfter time.Time `json:"tls_cert_not_after,omitzero"` // TLSCertNotAfter is the expiration date of the TLS certificate
}
// DenyRules Contains lists of subjects not allowed to be imported/exported
@@ -1370,6 +1374,7 @@ type MQTTOptsVarz struct {
AckWait time.Duration `json:"ack_wait,omitempty"` // AckWait is how long the internal JetStream state store will allow acks to complete
MaxAckPending uint16 `json:"max_ack_pending,omitempty"` // MaxAckPending is how many outstanding acks the internal JetStream state store will allow
TLSOCSPPeerVerify bool `json:"tls_ocsp_peer_verify,omitempty"` // TLSOCSPPeerVerify indicates if OCSP verification will be done
TLSCertNotAfter time.Time `json:"tls_cert_not_after,omitzero"` // TLSCertNotAfter is the expiration date of the TLS certificate
}
// WebsocketOptsVarz contains monitoring websocket information
@@ -1388,6 +1393,7 @@ type WebsocketOptsVarz struct {
AllowedOrigins []string `json:"allowed_origins,omitempty"` // AllowedOrigins list of configured trusted origins
Compression bool `json:"compression,omitempty"` // Compression indicates if compression is supported
TLSOCSPPeerVerify bool `json:"tls_ocsp_peer_verify,omitempty"` // TLSOCSPPeerVerify indicates if OCSP verification will be done
TLSCertNotAfter time.Time `json:"tls_cert_not_after,omitzero"` // TLSCertNotAfter is the expiration date of the TLS certificate
}
// OCSPResponseCacheVarz contains OCSP response cache information
@@ -1454,6 +1460,22 @@ func myUptime(d time.Duration) string {
return fmt.Sprintf("%ds", tsecs)
}
func tlsCertNotAfter(config *tls.Config) time.Time {
if config == nil || len(config.Certificates) == 0 {
return time.Time{}
}
cert := config.Certificates[0]
leaf := cert.Leaf
if leaf == nil {
var err error
leaf, err = x509.ParseCertificate(cert.Certificate[0])
if err != nil {
return time.Time{}
}
}
return leaf.NotAfter
}
// HandleRoot will show basic info and links to others handlers.
func (s *Server) HandleRoot(w http.ResponseWriter, r *http.Request) {
// This feels dumb to me, but is required: https://code.google.com/p/go/issues/detail?id=4799
@@ -1779,6 +1801,13 @@ func (s *Server) updateVarzConfigReloadableFields(v *Varz) {
v.TLSOCSPPeerVerify = s.ocspPeerVerify && v.TLSRequired && s.opts.tlsConfigOpts != nil && s.opts.tlsConfigOpts.OCSPPeerConfig != nil && s.opts.tlsConfigOpts.OCSPPeerConfig.Verify
v.TLSCertNotAfter = tlsCertNotAfter(opts.TLSConfig)
v.Cluster.TLSCertNotAfter = tlsCertNotAfter(opts.Cluster.TLSConfig)
v.Gateway.TLSCertNotAfter = tlsCertNotAfter(opts.Gateway.TLSConfig)
v.LeafNode.TLSCertNotAfter = tlsCertNotAfter(opts.LeafNode.TLSConfig)
v.MQTT.TLSCertNotAfter = tlsCertNotAfter(opts.MQTT.TLSConfig)
v.Websocket.TLSCertNotAfter = tlsCertNotAfter(opts.Websocket.TLSConfig)
if opts.Proxies != nil {
if v.Proxies == nil {
v.Proxies = &ProxiesOptsVarz{}
@@ -3982,6 +4011,11 @@ func (s *Server) healthz(opts *HealthzOptions) *HealthStatus {
return health
}
// Healthz returns the health status of the server.
func (s *Server) Healthz(opts *HealthzOptions) *HealthStatus {
return s.healthz(opts)
}
type ExpvarzStatus struct {
Memstats json.RawMessage `json:"memstats"`
Cmdline json.RawMessage `json:"cmdline"`
+26 -26
View File
@@ -50,64 +50,64 @@ const (
// Individual sort options provide the Less for sort.Interface. Len and Swap are on cList.
// CID
type byCid struct{ ConnInfos }
type SortByCid struct{ ConnInfos }
func (l byCid) Less(i, j int) bool { return l.ConnInfos[i].Cid < l.ConnInfos[j].Cid }
func (l SortByCid) Less(i, j int) bool { return l.ConnInfos[i].Cid < l.ConnInfos[j].Cid }
// Number of Subscriptions
type bySubs struct{ ConnInfos }
type SortBySubs struct{ ConnInfos }
func (l bySubs) Less(i, j int) bool { return l.ConnInfos[i].NumSubs < l.ConnInfos[j].NumSubs }
func (l SortBySubs) Less(i, j int) bool { return l.ConnInfos[i].NumSubs < l.ConnInfos[j].NumSubs }
// Pending Bytes
type byPending struct{ ConnInfos }
type SortByPending struct{ ConnInfos }
func (l byPending) Less(i, j int) bool { return l.ConnInfos[i].Pending < l.ConnInfos[j].Pending }
func (l SortByPending) Less(i, j int) bool { return l.ConnInfos[i].Pending < l.ConnInfos[j].Pending }
// Outbound Msgs
type byOutMsgs struct{ ConnInfos }
type SortByOutMsgs struct{ ConnInfos }
func (l byOutMsgs) Less(i, j int) bool { return l.ConnInfos[i].OutMsgs < l.ConnInfos[j].OutMsgs }
func (l SortByOutMsgs) Less(i, j int) bool { return l.ConnInfos[i].OutMsgs < l.ConnInfos[j].OutMsgs }
// Inbound Msgs
type byInMsgs struct{ ConnInfos }
type SortByInMsgs struct{ ConnInfos }
func (l byInMsgs) Less(i, j int) bool { return l.ConnInfos[i].InMsgs < l.ConnInfos[j].InMsgs }
func (l SortByInMsgs) Less(i, j int) bool { return l.ConnInfos[i].InMsgs < l.ConnInfos[j].InMsgs }
// Outbound Bytes
type byOutBytes struct{ ConnInfos }
type SortByOutBytes struct{ ConnInfos }
func (l byOutBytes) Less(i, j int) bool { return l.ConnInfos[i].OutBytes < l.ConnInfos[j].OutBytes }
func (l SortByOutBytes) Less(i, j int) bool { return l.ConnInfos[i].OutBytes < l.ConnInfos[j].OutBytes }
// Inbound Bytes
type byInBytes struct{ ConnInfos }
type SortByInBytes struct{ ConnInfos }
func (l byInBytes) Less(i, j int) bool { return l.ConnInfos[i].InBytes < l.ConnInfos[j].InBytes }
func (l SortByInBytes) Less(i, j int) bool { return l.ConnInfos[i].InBytes < l.ConnInfos[j].InBytes }
// Last Activity
type byLast struct{ ConnInfos }
type SortByLast struct{ ConnInfos }
func (l byLast) Less(i, j int) bool {
func (l SortByLast) Less(i, j int) bool {
return l.ConnInfos[i].LastActivity.UnixNano() < l.ConnInfos[j].LastActivity.UnixNano()
}
// Idle time
type byIdle struct {
type SortByIdle struct {
ConnInfos
now time.Time
}
func (l byIdle) Less(i, j int) bool {
func (l SortByIdle) Less(i, j int) bool {
return l.now.Sub(l.ConnInfos[i].LastActivity) < l.now.Sub(l.ConnInfos[j].LastActivity)
}
// Uptime
type byUptime struct {
type SortByUptime struct {
ConnInfos
now time.Time
}
func (l byUptime) Less(i, j int) bool {
func (l SortByUptime) Less(i, j int) bool {
ci := l.ConnInfos[i]
cj := l.ConnInfos[j]
var upi, upj time.Duration
@@ -125,25 +125,25 @@ func (l byUptime) Less(i, j int) bool {
}
// Stop
type byStop struct{ ConnInfos }
type SortByStop struct{ ConnInfos }
func (l byStop) Less(i, j int) bool {
func (l SortByStop) Less(i, j int) bool {
ciStop := l.ConnInfos[i].Stop
cjStop := l.ConnInfos[j].Stop
return ciStop.Before(*cjStop)
}
// Reason
type byReason struct{ ConnInfos }
type SortByReason struct{ ConnInfos }
func (l byReason) Less(i, j int) bool {
func (l SortByReason) Less(i, j int) bool {
return l.ConnInfos[i].Reason < l.ConnInfos[j].Reason
}
// RTT - Default is descending
type byRTT struct{ ConnInfos }
type SortByRTT struct{ ConnInfos }
func (l byRTT) Less(i, j int) bool { return l.ConnInfos[i].rtt < l.ConnInfos[j].rtt }
func (l SortByRTT) Less(i, j int) bool { return l.ConnInfos[i].rtt < l.ConnInfos[j].rtt }
// IsValid determines if a sort option is valid
func (s SortOpt) IsValid() bool {
+499 -343
View File
File diff suppressed because it is too large. Load diff
+7
View File
@@ -367,6 +367,13 @@ func (c *client) initMsgTrace() *msgTrace {
}
}
dest = getHdrVal(MsgTraceDest)
if c.kind == CLIENT {
if td, ok := c.allowedMsgTraceDest(hdr, false); !ok {
return nil
} else if td != _EMPTY_ {
dest = td
}
}
// Check the destination to see if this is a valid public subject.
if !IsValidPublishSubject(dest) {
// We still have to return a msgTrace object (if traceOnly is set)
+31 -4
View File
@@ -389,6 +389,7 @@ type Options struct {
JetStreamRequestQueueLimit int64
JetStreamMetaCompact uint64
JetStreamMetaCompactSize uint64
JetStreamMetaCompactSync bool
StreamMaxBufferedMsgs int `json:"-"`
StreamMaxBufferedSize int64 `json:"-"`
StoreDir string `json:"-"`
@@ -595,6 +596,11 @@ type WebsocketOpts struct {
// time needed for the TLS Handshake.
HandshakeTimeout time.Duration
// How often to send pings to WebSocket clients. When set to a non-zero
// duration, this overrides the default PingInterval for WebSocket connections.
// If not set or zero, the server's default PingInterval will be used.
PingInterval time.Duration
// Headers to be added to the upgrade response.
// Useful for adding custom headers like Strict-Transport-Security.
Headers map[string]string
@@ -1263,7 +1269,9 @@ func (o *Options) processConfigFileLine(k string, v any, errors *[]error, warnin
case "proxy_protocol":
o.ProxyProtocol = v.(bool)
case "max_connections", "max_conn":
o.MaxConn = int(v.(int64))
if o.MaxConn = int(v.(int64)); o.MaxConn == 0 {
o.MaxConn = -1
}
case "max_traced_msg_len":
o.MaxTracedMsgLen = int(v.(int64))
case "max_subscriptions", "max_subs":
@@ -1685,7 +1693,7 @@ func (o *Options) processConfigFileLine(k string, v any, errors *[]error, warnin
case "reconnect_error_reports":
o.ReconnectErrorReports = int(v.(int64))
case "websocket", "ws":
if err := parseWebsocket(tk, o, errors); err != nil {
if err := parseWebsocket(tk, o, errors, warnings); err != nil {
*errors = append(*errors, err)
return
}
@@ -2321,7 +2329,7 @@ func parseJetStreamForAccount(v any, acc *Account, errors *[]error) error {
case "cluster_traffic":
vv, ok := mv.(string)
if !ok {
return &configErr{tk, fmt.Sprintf("Expected either 'system' or 'account' string value for %q, got %v", mk, mv)}
return &configErr{tk, fmt.Sprintf("Expected either 'system' or 'owner' string value for %q, got %v", mk, mv)}
}
switch vv {
case "system", _EMPTY_:
@@ -2648,6 +2656,8 @@ func parseJetStream(v any, opts *Options, errors *[]error, warnings *[]error) er
return &configErr{tk, fmt.Sprintf("Expected an absolute size for %q, got %v", mk, mv)}
}
opts.JetStreamMetaCompactSize = uint64(s)
case "meta_compact_sync":
opts.JetStreamMetaCompactSync = mv.(bool)
default:
if !tk.IsUsedVariable() {
err := &unknownConfigFieldErr{
@@ -5313,7 +5323,7 @@ func parseStringArray(fieldName string, tk token, lt *token, mv any, errors *[]e
}
}
func parseWebsocket(v any, o *Options, errors *[]error) error {
func parseWebsocket(v any, o *Options, errors *[]error, warnings *[]error) error {
var lt token
defer convertPanicToErrorList(&lt, errors)
@@ -5414,6 +5424,8 @@ func parseWebsocket(v any, o *Options, errors *[]error) error {
o.Websocket.Headers[key] = headerValue
}
}
case "ping_interval":
o.Websocket.PingInterval = parseDuration("ping_interval", tk, mv, errors, warnings)
default:
if !tk.IsUsedVariable() {
err := &unknownConfigFieldErr{
@@ -6423,3 +6435,18 @@ func expandPath(p string) (string, error) {
return filepath.Join(home, p[1:]), nil
}
// RedactArgs redacts sensitive arguments from the command line.
// For example, turns '--pass=secret' into '--pass=[REDACTED]'.
func RedactArgs(args []string) {
secret := regexp.MustCompile("^-{1,2}(user|pass|auth)(=.*)?$")
for i, arg := range args {
if secret.MatchString(arg) {
if idx := strings.Index(arg, "="); idx != -1 {
args[i] = arg[:idx] + "=[REDACTED]"
} else if i+1 < len(args) {
args[i+1] = "[REDACTED]"
}
}
}
}
+1
View File
@@ -32,6 +32,7 @@ type parseState struct {
msgBuf []byte
header http.Header // access via getHeader
scratch [MAX_CONTROL_LINE_SIZE]byte
argsa [MAX_HMSG_ARGS + 1][]byte // pre-allocated args array to avoid per-call heap escape
}
type pubArg struct {
+6
View File
@@ -63,8 +63,14 @@ func protoScanFieldValue(typ int, b []byte) (size int, err error) {
case 0:
_, size, err = protoScanVarint(b)
case 5: // fixed32
if len(b) < 4 {
return 0, errProtoInsufficient
}
size = 4
case 1: // fixed64
if len(b) < 8 {
return 0, errProtoInsufficient
}
size = 8
case 2: // length-delimited
size, err = protoScanBytes(b)
+614 -205
View File
File diff suppressed because it is too large. Load diff
+12 -12
View File
@@ -564,21 +564,20 @@ type maxConnOption struct {
// below the limit if necessary.
func (m *maxConnOption) Apply(server *Server) {
server.mu.Lock()
var (
clients = make([]*client, len(server.clients))
i = 0
)
clients := make([]*client, 0, len(server.clients))
// Map iteration is random, which allows us to close random connections.
for _, client := range server.clients {
clients[i] = client
i++
if isInternalClient(client.kind) {
continue
}
clients = append(clients, client)
}
server.mu.Unlock()
if m.newValue > 0 && len(clients) > m.newValue {
if newc := max(0, m.newValue); len(clients) > newc {
// Close connections til we are within the limit.
var (
numClose = len(clients) - m.newValue
numClose = len(clients) - newc
closed = 0
)
for _, client := range clients {
@@ -1659,7 +1658,7 @@ func (s *Server) diffOptions(newOpts *Options) ([]option, error) {
return nil, fmt.Errorf("config reload not supported for jetstream max memory and store")
}
}
case "jetstreammetacompact", "jetstreammetacompactsize":
case "jetstreammetacompact", "jetstreammetacompactsize", "jetstreammetacompactsync":
// Allowed at runtime but monitorCluster looks at s.opts directly, so no further work needed here.
case "websocket":
// Similar to gateways
@@ -2126,9 +2125,6 @@ func (s *Server) reloadAuthorization() {
resetCh <- struct{}{}
}
// Check that publish retained messages sources are still allowed to publish.
s.mqttCheckPubRetainedPerms()
// Close clients that have moved accounts
for _, client := range cclients {
client.closeConnection(ClientClosed)
@@ -2168,6 +2164,10 @@ func (s *Server) reloadAuthorization() {
s.Errorf(err.Error())
}
}
// Check that publish retained messages sources are still allowed to publish.
// Do this after dealing with JetStream.
s.mqttCheckPubRetainedPerms()
}
// Returns true if given client current account has changed (or user
+4 -7
View File
@@ -88,7 +88,7 @@ type route struct {
// an implicit route and sending to the remote.
gossipMode byte
// This will be set in case of pooling so that a route can trigger
// the creation of the next after receiving the first PONG, ensuring
// the creation of the next after receiving a PONG, ensuring
// that authentication did not fail.
startNewRoute *routeInfo
}
@@ -181,8 +181,7 @@ func (c *client) processAccountUnsub(arg []byte) {
// we have an origin cluster and we force header semantics.
func (c *client) processRoutedOriginClusterMsgArgs(arg []byte) error {
// Unroll splitArgs to avoid runtime/heap issues
a := [MAX_HMSG_ARGS + 1][]byte{}
args := a[:0]
args := c.argsa[:0]
start := -1
for i, b := range arg {
switch b {
@@ -280,8 +279,7 @@ func (c *client) processRoutedOriginClusterMsgArgs(arg []byte) error {
// Process an inbound HMSG specification from the remote route.
func (c *client) processRoutedHeaderMsgArgs(arg []byte) error {
// Unroll splitArgs to avoid runtime/heap issues
a := [MAX_HMSG_ARGS][]byte{}
args := a[:0]
args := c.argsa[:0]
var an []byte
if c.kind == ROUTER {
if an = c.route.accName; len(an) > 0 {
@@ -377,8 +375,7 @@ func (c *client) processRoutedHeaderMsgArgs(arg []byte) error {
// Process an inbound RMSG or LMSG specification from the remote route.
func (c *client) processRoutedMsgArgs(arg []byte) error {
// Unroll splitArgs to avoid runtime/heap issues
a := [MAX_RMSG_ARGS][]byte{}
args := a[:0]
args := c.argsa[:0]
var an []byte
if c.kind == ROUTER {
if an = c.route.accName; len(an) > 0 {
+3 -3
View File
@@ -3377,7 +3377,7 @@ func (s *Server) createClientEx(conn net.Conn, inProcess bool) *client {
// If there is a max connections specified, check that adding
// this new client would not push us over the max
if opts.MaxConn > 0 && len(s.clients) >= opts.MaxConn {
if opts.MaxConn < 0 || (opts.MaxConn > 0 && len(s.clients) >= opts.MaxConn) {
s.mu.Unlock()
c.maxConnExceeded()
return nil
@@ -3452,7 +3452,7 @@ func (s *Server) createClientEx(conn net.Conn, inProcess bool) *client {
pre = pre[:n]
}
conn = &tlsMixConn{conn, bytes.NewBuffer(pre)}
addr, err := readProxyProtoHeader(conn)
addr, proxyPre, err := readProxyProtoHeader(conn)
if err != nil && err != errProxyProtoUnrecognized {
// err != errProxyProtoUnrecognized implies that we detected a proxy
// protocol header but we failed to parse it, so don't continue.
@@ -3480,7 +3480,7 @@ func (s *Server) createClientEx(conn net.Conn, inProcess bool) *client {
// that it's a non-proxied connection and we want the pre-read to remain
// for the next step.
if err == nil {
pre = nil
pre = proxyPre
}
// Because we have ProxyProtocol enabled, our earlier INFO message didn't
// include the client_ip. If we need to send it again then we will include
+26 -18
View File
@@ -42,7 +42,6 @@ type winServiceWrapper struct {
}
var dockerized = false
var startupDelay = 10 * time.Second
func init() {
if v, exists := os.LookupEnv("NATS_DOCKERIZED"); exists && v == "1" {
@@ -67,6 +66,7 @@ func (w *winServiceWrapper) Execute(args []string, changes <-chan svc.ChangeRequ
status <- svc.Status{State: svc.StartPending}
go w.server.Start()
var startupDelay = 10 * time.Second
if v, exists := os.LookupEnv("NATS_STARTUP_DELAY"); exists {
if delay, err := time.ParseDuration(v); err == nil {
startupDelay = delay
@@ -86,24 +86,32 @@ func (w *winServiceWrapper) Execute(args []string, changes <-chan svc.ChangeRequ
}
loop:
for change := range changes {
switch change.Cmd {
case svc.Interrogate:
status <- change.CurrentStatus
case svc.Stop, svc.Shutdown:
w.server.Shutdown()
break loop
case reopenLogCmd:
// File log re-open for rotating file logs.
w.server.ReOpenLogFile()
case ldmCmd:
go w.server.lameDuckMode()
case svc.ParamChange:
if err := w.server.Reload(); err != nil {
w.server.Errorf("Failed to reload server configuration: %s", err)
for {
select {
case change, ok := <-changes:
if !ok {
break loop
}
default:
w.server.Debugf("Unexpected control request: %v", change.Cmd)
switch change.Cmd {
case svc.Interrogate:
status <- change.CurrentStatus
case svc.Stop, svc.Shutdown:
w.server.Shutdown()
break loop
case reopenLogCmd:
// File log re-open for rotating file logs.
w.server.ReOpenLogFile()
case ldmCmd:
go w.server.lameDuckMode()
case svc.ParamChange:
if err := w.server.Reload(); err != nil {
w.server.Errorf("Failed to reload server configuration: %s", err)
}
default:
w.server.Debugf("Unexpected control request: %v", change.Cmd)
}
case <-w.server.quitCh:
break loop
}
}
+6 -3
View File
@@ -65,6 +65,8 @@ var (
ErrCorruptStreamState = errors.New("stream state snapshot is corrupt")
// ErrTooManyResults
ErrTooManyResults = errors.New("too many matching results for request")
// ErrStoreOldUpdate is returned when a consumer update is older than the current state.
ErrStoreOldUpdate = errors.New("old update ignored")
)
// StoreMsg is the stored message format for messages that are retained by the Store layer.
@@ -99,6 +101,7 @@ type StreamStore interface {
LoadNextMsgMulti(sl *gsl.SimpleSublist, start uint64, smp *StoreMsg) (sm *StoreMsg, skip uint64, err error)
LoadLastMsg(subject string, sm *StoreMsg) (*StoreMsg, error)
LoadPrevMsg(start uint64, smp *StoreMsg) (sm *StoreMsg, err error)
LoadPrevMsgMulti(sl *gsl.SimpleSublist, start uint64, smp *StoreMsg) (sm *StoreMsg, skip uint64, err error)
RemoveMsg(seq uint64) (bool, error)
EraseMsg(seq uint64) (bool, error)
Purge() (uint64, error)
@@ -112,8 +115,8 @@ type StreamStore interface {
AllLastSeqs() ([]uint64, error)
MultiLastSeqs(filters []string, maxSeq uint64, maxAllowed int) ([]uint64, error)
SubjectForSeq(seq uint64) (string, error)
NumPending(sseq uint64, filter string, lastPerSubject bool) (total, validThrough uint64)
NumPendingMulti(sseq uint64, sl *gsl.SimpleSublist, lastPerSubject bool) (total, validThrough uint64)
NumPending(sseq uint64, filter string, lastPerSubject bool) (total, validThrough uint64, err error)
NumPendingMulti(sseq uint64, sl *gsl.SimpleSublist, lastPerSubject bool) (total, validThrough uint64, err error)
State() StreamState
FastState(*StreamState)
EncodedStreamState(failed uint64) (enc []byte, err error)
@@ -125,7 +128,7 @@ type StreamStore interface {
UpdateConfig(cfg *StreamConfig) error
Delete(inline bool) error
Stop() error
ConsumerStore(name string, cfg *ConsumerConfig) (ConsumerStore, error)
ConsumerStore(name string, created time.Time, cfg *ConsumerConfig) (ConsumerStore, error)
AddConsumer(o ConsumerStore) error
RemoveConsumer(o ConsumerStore) error
Snapshot(deadline time.Duration, includeConsumers, checkMsgs bool) (*SnapshotResult, error)
+274 -172
View File
@@ -1,4 +1,4 @@
// Copyright 2019-2025 The NATS Authors
// Copyright 2019-2026 The NATS Authors
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
@@ -425,9 +425,11 @@ type stream struct {
active bool // Indicates that there are active internal subscriptions (for the subject filters)
// and/or mirror/sources consumers are scheduled to be established or already started.
closed atomic.Bool // Set to true when stop() is called on the stream.
cisrun atomic.Bool // Indicates one checkInterestState is already running.
// Mirror
mirror *sourceInfo
mirror *sourceInfo
mirrorConsumerSetup *time.Timer
// Sources
sources map[string]*sourceInfo
@@ -618,19 +620,24 @@ const StreamMaxReplicas = 5
// AddStream adds a stream for the given account.
func (a *Account) addStream(config *StreamConfig) (*stream, error) {
return a.addStreamWithAssignment(config, nil, nil, false)
return a.addStreamWithAssignment(config, nil, nil, false, false)
}
// recoverStream recovers a stream from disk for the given account.
func (a *Account) recoverStream(config *StreamConfig) (*stream, error) {
return a.addStreamWithAssignment(config, nil, nil, false, true)
}
// AddStreamWithStore adds a stream for the given account with custome store config options.
func (a *Account) addStreamWithStore(config *StreamConfig, fsConfig *FileStoreConfig) (*stream, error) {
return a.addStreamWithAssignment(config, fsConfig, nil, false)
return a.addStreamWithAssignment(config, fsConfig, nil, false, false)
}
func (a *Account) addStreamPedantic(config *StreamConfig, pedantic bool) (*stream, error) {
return a.addStreamWithAssignment(config, nil, nil, pedantic)
return a.addStreamWithAssignment(config, nil, nil, pedantic, false)
}
func (a *Account) addStreamWithAssignment(config *StreamConfig, fsConfig *FileStoreConfig, sa *streamAssignment, pedantic bool) (*stream, error) {
func (a *Account) addStreamWithAssignment(config *StreamConfig, fsConfig *FileStoreConfig, sa *streamAssignment, pedantic, recovering bool) (*stream, error) {
s, jsa, err := a.checkForJetStream()
if err != nil {
return nil, err
@@ -677,6 +684,7 @@ func (a *Account) addStreamWithAssignment(config *StreamConfig, fsConfig *FileSt
}()
}
// Note that isClustered will be false during recovery, even if we're part of a cluster. It shouldn't be used then.
js, isClustered := jsa.jetStreamAndClustered()
jsa.mu.Lock()
if mset, ok := jsa.streams[cfg.Name]; ok {
@@ -706,25 +714,30 @@ func (a *Account) addStreamWithAssignment(config *StreamConfig, fsConfig *FileSt
jsa.usageMu.RLock()
selected, tier, hasTier := jsa.selectLimits(cfg.Replicas)
jsa.usageMu.RUnlock()
reserved := int64(0)
if !isClustered {
reserved = jsa.tieredReservation(tier, cfg)
}
jsa.mu.Unlock()
if !hasTier {
jsa.mu.Unlock()
return nil, NewJSNoLimitsError()
}
js.mu.RLock()
if isClustered {
_, reserved = tieredStreamAndReservationCount(js.cluster.streams[a.Name], tier, cfg)
}
if err := js.checkAllLimits(&selected, cfg, reserved, 0); err != nil {
// Skip if we're recovering.
if !recovering {
reserved := int64(0)
if !isClustered {
reserved = jsa.tieredReservation(tier, cfg)
}
jsa.mu.Unlock()
js.mu.RLock()
if isClustered {
_, reserved = js.tieredStreamAndReservationCount(a.Name, tier, cfg)
}
if err := js.checkAllLimits(&selected, cfg, reserved, 0); err != nil {
js.mu.RUnlock()
return nil, err
}
js.mu.RUnlock()
return nil, err
jsa.mu.Lock()
}
js.mu.RUnlock()
jsa.mu.Lock()
// Check for template ownership if present.
if cfg.Template != _EMPTY_ && jsa.account != nil {
if !jsa.checkTemplateOwnership(cfg.Template, cfg.Name) {
@@ -789,11 +802,6 @@ func (a *Account) addStreamWithAssignment(config *StreamConfig, fsConfig *FileSt
return nil, NewJSStreamSubjectOverlapError()
}
if !hasTier {
jsa.mu.Unlock()
return nil, fmt.Errorf("no applicable tier found")
}
// Setup the internal clients.
c := s.createInternalJetStreamClient()
ic := s.createInternalJetStreamClient()
@@ -827,11 +835,19 @@ func (a *Account) addStreamWithAssignment(config *StreamConfig, fsConfig *FileSt
ipqLimitByLen[*inMsg](mlen),
ipqLimitBySize[*inMsg](msz),
),
gets: newIPQueue[*directGetReq](s, qpfx+"direct gets"),
qch: make(chan struct{}),
mqch: make(chan struct{}),
uch: make(chan struct{}, 4),
sch: make(chan struct{}, 1),
gets: newIPQueue[*directGetReq](s, qpfx+"direct gets"),
qch: make(chan struct{}),
mqch: make(chan struct{}),
uch: make(chan struct{}, 4),
sch: make(chan struct{}, 1),
created: time.Now().UTC(),
}
// Add created timestamp used for the store, must match that of the stream assignment if it exists.
if sa != nil {
// The following assignment does not require mutex
// protection: sa.Created is immutable.
mset.created = sa.Created
}
// Start our signaling routine to process consumers.
@@ -895,7 +911,6 @@ func (a *Account) addStreamWithAssignment(config *StreamConfig, fsConfig *FileSt
fsCfg.SyncAlways = false
fsCfg.AsyncFlush = true
}
if err := mset.setupStore(fsCfg); err != nil {
mset.stop(true, false)
return nil, NewJSStreamStoreFailedError(err)
@@ -1080,8 +1095,12 @@ func (mset *stream) monitorQuitC() <-chan struct{} {
if mset == nil {
return nil
}
mset.mu.RLock()
defer mset.mu.RUnlock()
mset.mu.Lock()
defer mset.mu.Unlock()
// Recreate if a prior monitor routine was stopped.
if mset.mqch == nil {
mset.mqch = make(chan struct{})
}
return mset.mqch
}
@@ -1811,10 +1830,10 @@ func (s *Server) checkStreamCfg(config *StreamConfig, acc *Account, pedantic boo
}
}
// check for duplicates
// check sources for duplicates
var iNames = make(map[string]struct{})
for _, src := range cfg.Sources {
if !isValidName(src.Name) {
if src == nil || !isValidName(src.Name) {
return StreamConfig{}, NewJSSourceInvalidStreamNameError()
}
if _, ok := iNames[src.composeIName()]; !ok {
@@ -1822,6 +1841,30 @@ func (s *Server) checkStreamCfg(config *StreamConfig, acc *Account, pedantic boo
} else {
return StreamConfig{}, NewJSSourceDuplicateDetectedError()
}
if src.FilterSubject != _EMPTY_ && len(src.SubjectTransforms) != 0 {
return StreamConfig{}, NewJSSourceMultipleFiltersNotAllowedError()
}
for _, tr := range src.SubjectTransforms {
if tr.Source != _EMPTY_ && !IsValidSubject(tr.Source) {
return StreamConfig{}, NewJSSourceInvalidSubjectFilterError(fmt.Errorf("%w %s", ErrBadSubject, tr.Source))
}
err := ValidateMapping(tr.Source, tr.Destination)
if err != nil {
return StreamConfig{}, NewJSSourceInvalidTransformDestinationError(err)
}
}
// Check subject filters overlap.
for outer, tr := range src.SubjectTransforms {
for inner, innertr := range src.SubjectTransforms {
if inner != outer && subjectIsSubsetMatch(tr.Source, innertr.Source) {
return StreamConfig{}, NewJSSourceOverlappingSubjectFiltersError()
}
}
}
// Do not perform checks if External is provided, as it could lead to
// checking against itself (if sourced stream name is the same on different JetStream)
if src.External == nil {
@@ -1834,30 +1877,6 @@ func (s *Server) checkStreamCfg(config *StreamConfig, acc *Account, pedantic boo
return StreamConfig{}, NewJSSourceMaxMessageSizeTooBigError()
}
}
if src.FilterSubject != _EMPTY_ && len(src.SubjectTransforms) != 0 {
return StreamConfig{}, NewJSSourceMultipleFiltersNotAllowedError()
}
for _, tr := range src.SubjectTransforms {
if tr.Source != _EMPTY_ && !IsValidSubject(tr.Source) {
return StreamConfig{}, NewJSSourceInvalidSubjectFilterError(fmt.Errorf("%w %s", ErrBadSubject, tr.Source))
}
err := ValidateMapping(tr.Source, tr.Destination)
if err != nil {
return StreamConfig{}, NewJSSourceInvalidTransformDestinationError(err)
}
}
// Check subject filters overlap.
for outer, tr := range src.SubjectTransforms {
for inner, innertr := range src.SubjectTransforms {
if inner != outer && subjectIsSubsetMatch(tr.Source, innertr.Source) {
return StreamConfig{}, NewJSSourceOverlappingSubjectFiltersError()
}
}
}
continue
} else {
if src.External.DeliverPrefix != _EMPTY_ {
@@ -1949,7 +1968,7 @@ func (s *Server) checkStreamCfg(config *StreamConfig, acc *Account, pedantic boo
// Check for literal duplication of subject interest in config
// and no overlap with any JS or SYS API subject space.
dset := make(map[string]struct{}, len(cfg.Subjects))
for _, subj := range cfg.Subjects {
for i, subj := range cfg.Subjects {
// Make sure the subject is valid. Check this first.
if !IsValidSubject(subj) {
return StreamConfig{}, NewJSStreamInvalidConfigError(fmt.Errorf("invalid subject"))
@@ -1983,6 +2002,13 @@ func (s *Server) checkStreamCfg(config *StreamConfig, acc *Account, pedantic boo
}
}
}
// Now check if we have multiple subjects that we do not overlap ourselves
// which would cause duplicate entries (assuming no MsgID).
for _, tsubj := range cfg.Subjects[i+1:] {
if SubjectsCollide(tsubj, subj) {
return StreamConfig{}, NewJSStreamInvalidConfigError(fmt.Errorf("subject %q overlaps with %q", subj, tsubj))
}
}
// Mark for duplicate check.
dset[subj] = struct{}{}
}
@@ -2000,18 +2026,6 @@ func (s *Server) checkStreamCfg(config *StreamConfig, acc *Account, pedantic boo
return StreamConfig{}, NewJSStreamMaxStreamBytesExceededError()
}
// Now check if we have multiple subjects they we do not overlap ourselves
// which would cause duplicate entries (assuming no MsgID).
if len(cfg.Subjects) > 1 {
for _, subj := range cfg.Subjects {
for _, tsubj := range cfg.Subjects {
if tsubj != subj && SubjectsCollide(tsubj, subj) {
return StreamConfig{}, NewJSStreamInvalidConfigError(fmt.Errorf("subject %q overlaps with %q", subj, tsubj))
}
}
}
}
// Check the subject transform if any
if cfg.SubjectTransform != nil {
if cfg.SubjectTransform.Source != _EMPTY_ && !IsValidSubject(cfg.SubjectTransform.Source) {
@@ -2102,10 +2116,6 @@ func (jsa *jsAccount) configUpdateCheck(old, new *StreamConfig, s *Server, pedan
if cfg.Name != old.Name {
return nil, NewJSStreamInvalidConfigError(fmt.Errorf("stream configuration name must match original"))
}
// Can't change MaxConsumers for now.
if cfg.MaxConsumers != old.MaxConsumers {
return nil, NewJSStreamInvalidConfigError(fmt.Errorf("stream configuration update can not change MaxConsumers"))
}
// Can't change storage types.
if cfg.Storage != old.Storage {
return nil, NewJSStreamInvalidConfigError(fmt.Errorf("stream configuration update can not change storage type"))
@@ -2222,13 +2232,15 @@ func (jsa *jsAccount) configUpdateCheck(old, new *StreamConfig, s *Server, pedan
js.mu.RLock()
defer js.mu.RUnlock()
if isClustered {
_, reserved = tieredStreamAndReservationCount(js.cluster.streams[acc.Name], tier, &cfg)
_, reserved = js.tieredStreamAndReservationCount(acc.Name, tier, &cfg)
}
// reservation does not account for this stream, hence add the old value
if tier == _EMPTY_ && old.Replicas > 1 {
reserved += old.MaxBytes * int64(old.Replicas)
} else {
reserved += old.MaxBytes
if old.MaxBytes > 0 {
if tier == _EMPTY_ && old.Replicas > 1 {
reserved = addSaturate(reserved, mulSaturate(int64(old.Replicas), old.MaxBytes))
} else {
reserved = addSaturate(reserved, old.MaxBytes)
}
}
if err := js.checkAllLimits(&selected, &cfg, reserved, maxBytesOffset); err != nil {
return nil, err
@@ -2797,6 +2809,12 @@ func (mset *stream) processMirrorMsgs(mirror *sourceInfo, ready *sync.WaitGroup)
// Grab stream quit channel.
mset.mu.Lock()
msgs, qch, siqch := mirror.msgs, mset.qch, mirror.qch
// If the mirror was already canceled before we got here, exit early.
if siqch == nil {
mset.mu.Unlock()
ready.Done()
return
}
// Set the last seen as now so that we don't fail at the first check.
mirror.last.Store(time.Now().UnixNano())
mset.mu.Unlock()
@@ -3111,7 +3129,8 @@ func (mset *stream) scheduleSetupMirrorConsumerRetry() {
// Add some jitter.
next += time.Duration(rand.Intn(int(100*time.Millisecond))) + 100*time.Millisecond
time.AfterFunc(next, func() {
stopAndClearTimer(&mset.mirrorConsumerSetup)
mset.mirrorConsumerSetup = time.AfterFunc(next, func() {
mset.mu.Lock()
mset.setupMirrorConsumer()
mset.mu.Unlock()
@@ -3155,7 +3174,6 @@ func (mset *stream) setupMirrorConsumer() error {
}
mirror := mset.mirror
mirrorWg := &mirror.wg
// We want to throttle here in terms of how fast we request new consumers,
// or if the previous is still in progress.
@@ -3314,7 +3332,16 @@ func (mset *stream) setupMirrorConsumer() error {
// Wait for previous processMirrorMsgs go routine to be completely done.
// If none is running, this will not block.
mirrorWg.Wait()
mset.mu.Lock()
if mset.mirror == nil {
// Mirror config has been removed.
mset.mu.Unlock()
return
} else {
wg := &mset.mirror.wg
mset.mu.Unlock()
wg.Wait()
}
select {
case ccr := <-respCh:
@@ -3393,6 +3420,7 @@ func (mset *stream) setupMirrorConsumer() error {
"consumer": mirror.cname,
},
) {
mirror.wg.Done()
ready.Done()
}
}
@@ -3955,7 +3983,6 @@ func (mset *stream) processInboundSourceMsg(si *sourceInfo, m *inMsg) bool {
} else {
err = mset.processJetStreamMsg(m.subj, _EMPTY_, hdr, msg, 0, 0, nil, true, true)
}
if err != nil {
s := mset.srv
if strings.Contains(err.Error(), "no space left") {
@@ -3965,31 +3992,35 @@ func (mset *stream) processInboundSourceMsg(si *sourceInfo, m *inMsg) bool {
mset.mu.RLock()
accName, sname, iName := mset.acc.Name, mset.cfg.Name, si.iname
mset.mu.RUnlock()
// Can happen temporarily all the time during normal operations when the sourcing stream
// is working queue/interest with a limit and discard new.
// TODO - Improve sourcing to WQ with limit and new to use flow control rather than re-creating the consumer.
if errors.Is(err, ErrMaxMsgs) || errors.Is(err, ErrMaxBytes) {
// Can happen temporarily all the time during normal operations when the sourcing stream is discard new
// (example use case is for sourcing into a work queue)
// TODO - Maybe improve sourcing to WQ with limit and new to use flow control rather than re-creating the consumer.
if errors.Is(err, ErrMaxMsgs) || errors.Is(err, ErrMaxBytes) || errors.Is(err, ErrMaxMsgsPerSubject) {
// Do not need to do a full retry that includes finding the last sequence in the stream
// for that source. Just re-create starting with the seq we couldn't store instead.
mset.mu.Lock()
mset.retrySourceConsumerAtSeq(iName, si.sseq)
mset.mu.Unlock()
} else {
// Log some warning for errors other than errLastSeqMismatch or errMaxMsgs.
if !errors.Is(err, errLastSeqMismatch) {
// Log some warning for errors other than errLastSeqMismatch.
if !errors.Is(err, errLastSeqMismatch) && !errors.Is(err, errMsgIdDuplicate) {
s.RateLimitWarnf("Error processing inbound source %q for '%s' > '%s': %v",
iName, accName, sname, err)
}
// Retry in all type of errors if we are still leader.
// Retry in all type of errors we do not want to skip if we are still leader.
if mset.isLeader() {
// This will make sure the source is still in mset.sources map,
// find the last sequence and then call setupSourceConsumer.
iNameMap := map[string]struct{}{iName: {}}
mset.setStartingSequenceForSources(iNameMap)
mset.mu.Lock()
mset.retrySourceConsumerAtSeq(iName, si.sseq+1)
mset.mu.Unlock()
if !errors.Is(err, errMsgIdDuplicate) {
// This will make sure the source is still in mset.sources map,
// find the last sequence and then call setupSourceConsumer.
iNameMap := map[string]struct{}{iName: {}}
mset.setStartingSequenceForSources(iNameMap)
mset.mu.Lock()
mset.retrySourceConsumerAtSeq(iName, si.sseq+1)
mset.mu.Unlock()
} else {
// skipping the message but keep processing the rest of the batch
return true
}
}
}
}
@@ -4086,28 +4117,61 @@ func (mset *stream) setStartingSequenceForSources(iNames map[string]struct{}) {
return
}
// From the provided list of sources, we build a sublist that contains
// the interested filters (including transforms). As we figure out the
// starting sequence for each source, we will eliminate the source from
// the map and then refresh the sublist, which in turn makes the sublist
// ideally more specific. This allows LoadPrevMsgsMulti to work most
// effectively.
// Because this is a SimpleSublist we can't just remove the entries per
// source so we have no other option but to rebuild it from scratch, but
// this is cheap enough to do so not the end of the world.
var sl *gsl.SimpleSublist
refreshSublist := func() {
sl = gsl.NewSimpleSublist()
for iName := range iNames {
si := mset.sources[iName]
if si == nil {
continue
}
if si.sf == _EMPTY_ {
sl.Insert(fwcs, struct{}{})
} else {
sl.Insert(si.sf, struct{}{})
}
for _, sf := range si.sfs {
if sf == _EMPTY_ {
sl.Insert(fwcs, struct{}{})
} else {
sl.Insert(sf, struct{}{})
}
}
}
}
refreshSublist()
var smv StoreMsg
for seq := state.LastSeq; seq >= state.FirstSeq; {
sm, err := mset.store.LoadPrevMsg(seq, &smv)
for last := state.LastSeq; ; {
sm, seq, err := mset.store.LoadPrevMsgMulti(sl, last, &smv)
if err == ErrStoreEOF || err != nil {
break
}
seq = sm.seq - 1
last = seq - 1
if len(sm.hdr) == 0 {
continue
}
ss := getHeader(JSStreamSource, sm.hdr)
ss := sliceHeader(JSStreamSource, sm.hdr)
if len(ss) == 0 {
continue
}
streamName, indexName, sseq := streamAndSeq(bytesToString(ss))
streamName, indexName, sseq := streamAndSeq(bytesToString(ss))
if _, ok := iNames[indexName]; ok {
si := mset.sources[indexName]
si.sseq = sseq
si.dseq = 0
delete(iNames, indexName)
refreshSublist()
} else if indexName == _EMPTY_ && streamName != _EMPTY_ {
for iName := range iNames {
// TODO streamSource is a linear walk, to optimize later
@@ -4116,6 +4180,7 @@ func (mset *stream) setStartingSequenceForSources(iNames map[string]struct{}) {
si.sseq = sseq
si.dseq = 0
delete(iNames, iName)
refreshSublist()
break
}
}
@@ -4197,26 +4262,61 @@ func (mset *stream) startingSequenceForSources() {
}
}()
// Generate a list of sources and, from that, a sublist that contains
// the interested filters (including transforms). As we figure out the
// starting sequence for each source, we will eliminate the source from
// the map and then refresh the sublist, which in turn makes the sublist
// ideally more specific. This allows LoadPrevMsgsMulti to work most
// effectively.
// Because this is a SimpleSublist we can't just remove the entries per
// source so we have no other option but to rebuild it from scratch, but
// this is cheap enough to do so not the end of the world.
sources := map[string]*StreamSource{}
for _, src := range mset.cfg.Sources {
sources[src.composeIName()] = src
}
var sl *gsl.SimpleSublist
refreshSublist := func() {
sl = gsl.NewSimpleSublist()
for _, src := range sources {
if src.FilterSubject == _EMPTY_ {
sl.Insert(fwcs, struct{}{})
} else {
sl.Insert(src.FilterSubject, struct{}{})
}
for _, tr := range src.SubjectTransforms {
if tr.Destination == _EMPTY_ {
sl.Insert(fwcs, struct{}{})
} else {
sl.Insert(tr.Destination, struct{}{})
}
}
}
}
refreshSublist()
update := func(iName string, seq uint64) {
// Only update active in case we have older ones in here that got configured out.
if si := mset.sources[iName]; si != nil {
if _, ok := seqs[iName]; !ok {
seqs[iName] = seq
delete(sources, iName)
refreshSublist()
}
}
}
var smv StoreMsg
for seq := state.LastSeq; ; {
sm, err := mset.store.LoadPrevMsg(seq, &smv)
for last := state.LastSeq; ; {
sm, seq, err := mset.store.LoadPrevMsgMulti(sl, last, &smv)
if err == ErrStoreEOF || err != nil {
break
}
seq = sm.seq - 1
last = seq - 1
if len(sm.hdr) == 0 {
continue
}
ss := getHeader(JSStreamSource, sm.hdr)
ss := sliceHeader(JSStreamSource, sm.hdr)
if len(ss) == 0 {
continue
}
@@ -4546,8 +4646,6 @@ func (mset *stream) unsubscribe(sub *subscription) {
func (mset *stream) setupStore(fsCfg *FileStoreConfig) error {
mset.mu.Lock()
mset.created = time.Now().UTC()
switch mset.cfg.Storage {
case MemoryStorage:
ms, err := newMemStore(&mset.cfg)
@@ -5208,7 +5306,10 @@ func (mset *stream) getDirectRequest(req *JSApiMsgGetRequest, reply string) {
} else {
// This is a batch request, capture initial numPending.
isBatchRequest = true
np, validThrough = store.NumPending(seq, req.NextFor, false)
var err error
if np, validThrough, err = store.NumPending(seq, req.NextFor, false); err != nil {
return
}
}
// Grab MaxBytes
@@ -5301,7 +5402,10 @@ func (mset *stream) getDirectRequest(req *JSApiMsgGetRequest, reply string) {
if isBatchRequest {
// Update if the stream's last sequence has moved past our validThrough.
if mset.lseq > validThrough {
np, _ = store.NumPending(seq, req.NextFor, false)
var err error
if np, _, err = store.NumPending(seq, req.NextFor, false); err != nil {
return
}
}
hdr := fmt.Appendf(nil, eob, np, lseq)
mset.outq.send(newJSPubMsg(reply, _EMPTY_, _EMPTY_, hdr, nil, nil, 0))
@@ -5311,6 +5415,7 @@ func (mset *stream) getDirectRequest(req *JSApiMsgGetRequest, reply string) {
// processInboundJetStreamMsg handles processing messages bound for a stream.
func (mset *stream) processInboundJetStreamMsg(_ *subscription, c *client, _ *Account, subject, reply string, rmsg []byte) {
hdr, msg := c.msgParts(copyBytes(rmsg)) // Need to copy.
hdr = removeHeaderStatusIfPresent(hdr)
if mt, traceOnly := c.isMsgTraceEnabled(); mt != nil {
// If message is delivered, we need to disable the message trace headers
// to prevent a trace event to be generated when a stored message
@@ -5706,7 +5811,7 @@ func (mset *stream) processJetStreamMsg(subject, reply string, hdr, msg []byte,
mset.ddMu.Unlock()
if seq > 0 {
if canRespond {
response := append(pubAck, strconv.FormatUint(dde.seq, 10)...)
response := append(pubAck, strconv.FormatUint(seq, 10)...)
response = append(response, ",\"duplicate\": true}"...)
outq.sendMsg(reply, response)
}
@@ -5965,13 +6070,6 @@ func (mset *stream) processJetStreamMsg(subject, reply string, hdr, msg []byte,
return nil
}
// If here we will attempt to store the message.
// Assume this will succeed.
olmsgId := mset.lmsgId
mset.lmsgId = msgId
mset.lseq++
tierName := mset.tier
// Republish state if needed.
var tsubj string
var tlseq uint64
@@ -5995,7 +6093,7 @@ func (mset *stream) processJetStreamMsg(subject, reply string, hdr, msg []byte,
// If clustered this was already checked and we do not want to check here and possibly introduce skew.
// Don't error and log if we're tracing when clustered.
if !isClustered {
if exceeded, err := jsa.wouldExceedLimits(stype, tierName, mset.cfg.Replicas, subject, hdr, msg); exceeded {
if exceeded, err := jsa.wouldExceedLimits(stype, mset.tier, mset.cfg.Replicas, subject, hdr, msg); exceeded {
if err == nil {
err = NewJSAccountResourcesExceededError()
}
@@ -6054,11 +6152,7 @@ func (mset *stream) processJetStreamMsg(subject, reply string, hdr, msg []byte,
mset.srv.Warnf("Filesystem permission denied while writing msg, disabling JetStream: %v", err)
return err
}
// If we did not succeed put those values back and increment clfs in case we are clustered.
var state StreamState
mset.store.FastState(&state)
mset.lseq = state.LastSeq
mset.lmsgId = olmsgId
// If we did not succeed increment clfs in case we are clustered.
bumpCLFS()
switch err {
@@ -6079,6 +6173,8 @@ func (mset *stream) processJetStreamMsg(subject, reply string, hdr, msg []byte,
}
// If here we succeeded in storing the message.
mset.lmsgId = msgId
mset.lseq = seq
// If we have a msgId make sure to save.
// This will replace our estimate from the cluster layer if we are clustered.
@@ -6673,36 +6769,31 @@ type jsPubMsg struct {
o *consumer
}
var jsPubMsgPool sync.Pool
var jsPubMsgPool = sync.Pool{
New: func() any {
return &jsPubMsg{}
},
}
func newJSPubMsg(dsubj, subj, reply string, hdr, msg []byte, o *consumer, seq uint64) *jsPubMsg {
var m *jsPubMsg
var buf []byte
pm := jsPubMsgPool.Get()
if pm != nil {
m = pm.(*jsPubMsg)
buf = m.buf[:0]
if hdr != nil {
hdr = append(m.hdr[:0], hdr...)
}
} else {
m = new(jsPubMsg)
m := getJSPubMsgFromPool()
if m.buf == nil {
m.buf = make([]byte, 0, len(hdr)+len(msg))
}
buf := append(m.buf[:0], hdr...)
buf = append(buf, msg...)
hdr = buf[:len(hdr):len(hdr)]
msg = buf[len(hdr):]
// When getting something from a pool it is critical that all fields are
// initialized. Doing this way guarantees that if someone adds a field to
// the structure, the compiler will fail the build if this line is not updated.
(*m) = jsPubMsg{dsubj, reply, StoreMsg{subj, hdr, msg, buf, seq, 0}, o}
return m
}
// Gets a jsPubMsg from the pool.
func getJSPubMsgFromPool() *jsPubMsg {
pm := jsPubMsgPool.Get()
if pm != nil {
return pm.(*jsPubMsg)
}
return new(jsPubMsg)
return jsPubMsgPool.Get().(*jsPubMsg)
}
func (pm *jsPubMsg) returnToPool() {
@@ -6713,9 +6804,6 @@ func (pm *jsPubMsg) returnToPool() {
if len(pm.buf) > 0 {
pm.buf = pm.buf[:0]
}
if len(pm.hdr) > 0 {
pm.hdr = pm.hdr[:0]
}
jsPubMsgPool.Put(pm)
}
@@ -7218,11 +7306,30 @@ func (mset *stream) checkInterestState() {
return
}
// Ensure only one of these runs at the same time.
if !mset.cisrun.CompareAndSwap(false, true) {
return
}
defer mset.cisrun.Store(false)
var ss StreamState
mset.store.FastState(&ss)
asflr := uint64(math.MaxUint64)
for _, o := range mset.getConsumers() {
o.checkStateForInterestStream(&ss)
o.mu.RLock()
chkflr := o.chkflr
o.mu.RUnlock()
asflr = min(asflr, chkflr)
}
mset.cfgMu.RLock()
rp := mset.cfg.Retention
mset.cfgMu.RUnlock()
// Remove as many messages from the "head" of the stream if there's no interest anymore.
if rp == InterestPolicy && asflr != math.MaxUint64 {
mset.store.Compact(asflr)
}
}
@@ -7309,20 +7416,18 @@ func (mset *stream) swapSigSubs(o *consumer, newFilters []string) {
o.sigSubs = nil
}
if o.isLeader() {
if mset.csl == nil {
mset.csl = gsl.NewSublist[*consumer]()
}
// If no filters are preset, add fwcs to sublist for that consumer.
if newFilters == nil {
mset.csl.Insert(fwcs, o)
o.sigSubs = append(o.sigSubs, fwcs)
// If there are filters, add their subjects to sublist.
} else {
for _, filter := range newFilters {
mset.csl.Insert(filter, o)
o.sigSubs = append(o.sigSubs, filter)
}
if mset.csl == nil {
mset.csl = gsl.NewSublist[*consumer]()
}
// If no filters are present, add fwcs to sublist for that consumer.
if newFilters == nil {
mset.csl.Insert(fwcs, o)
o.sigSubs = append(o.sigSubs, fwcs)
} else {
// If there are filters, add their subjects to sublist.
for _, filter := range newFilters {
mset.csl.Insert(filter, o)
o.sigSubs = append(o.sigSubs, filter)
}
}
o.mu.Unlock()
@@ -7399,14 +7504,18 @@ func (mset *stream) partitionUnique(name string, partitions []string) bool {
if n == name {
continue
}
o.mu.RLock()
if o.subjf == nil {
o.mu.RUnlock()
return false
}
for _, filter := range o.subjf {
if SubjectsCollide(partition, filter.subject) {
o.mu.RUnlock()
return false
}
}
o.mu.RUnlock()
}
}
return true
@@ -7590,15 +7699,8 @@ func (mset *stream) ackMsg(o *consumer, seq uint64) bool {
return false
}
var shouldRemove bool
switch mset.cfg.Retention {
case WorkQueuePolicy:
// Normally we just remove a message when its ack'd here but if we have direct consumers
// from sources and/or mirrors we need to make sure they have delivered the msg.
shouldRemove = mset.directs <= 0 || mset.noInterest(seq, o)
case InterestPolicy:
shouldRemove = mset.noInterest(seq, o)
}
// If there's no interest left on this message for all consumers, we can remove it.
shouldRemove := mset.noInterest(seq, nil)
// If nothing else to do.
if !shouldRemove {
@@ -7709,7 +7811,7 @@ func (a *Account) RestoreStream(ncfg *StreamConfig, r io.Reader) (*stream, error
if hasTier {
if isClustered {
js.mu.RLock()
_, reserved = tieredStreamAndReservationCount(js.cluster.streams[a.Name], tier, &cfg)
_, reserved = js.tieredStreamAndReservationCount(a.Name, tier, &cfg)
js.mu.RUnlock()
} else {
reserved = jsa.tieredReservation(tier, &cfg)
+4 -1
View File
@@ -36,7 +36,6 @@ func genParts(filter []byte, parts [][]byte) [][]byte {
}
start = i + 1
} else if i < e && filter[i+1] == fwc && i+1 == e {
// We have a fwc
if i > start {
parts = append(parts, filter[start:i+1])
}
@@ -53,6 +52,10 @@ func genParts(filter []byte, parts [][]byte) [][]byte {
if next := i + 1; next == e || next < e && filter[next] != tsep {
continue
}
// Full wildcard must be terminal.
if filter[i] == fwc && i < e {
break
}
// We start with a pwc or fwc.
parts = append(parts, filter[i:i+1])
if i+1 <= e {
+102 -12
View File
@@ -16,6 +16,9 @@ package stree
import (
"bytes"
"slices"
"unsafe"
"github.com/nats-io/nats-server/v2/server/gsl"
)
// SubjectTree is an adaptive radix trie (ART) for storing subject information on literal subjects.
@@ -121,7 +124,25 @@ func (t *SubjectTree[T]) Match(filter []byte, cb func(subject []byte, val *T)) {
var raw [16][]byte
parts := genParts(filter, raw[:0])
var _pre [256]byte
t.match(t.root, parts, _pre[:0], cb)
t.match(t.root, parts, _pre[:0], func(subject []byte, val *T) bool {
cb(subject, val)
return true
})
}
// MatchUntil will match against a subject that can have wildcards and invoke
// the callback func for each matched value.
// Returning false from the callback will stop matching immediately.
// Returns true if matching ran to completion, false if callback stopped it early.
func (t *SubjectTree[T]) MatchUntil(filter []byte, cb func(subject []byte, val *T) bool) bool {
if t == nil || t.root == nil || len(filter) == 0 || cb == nil {
return true
}
// We need to break this up into chunks based on wildcards, either pwc '*' or fwc '>'.
var raw [16][]byte
parts := genParts(filter, raw[:0])
var _pre [256]byte
return t.match(t.root, parts, _pre[:0], cb)
}
// IterOrdered will walk all entries in the SubjectTree lexicographically. The callback can return false to terminate the walk.
@@ -293,7 +314,8 @@ func (t *SubjectTree[T]) delete(np *node, subject []byte, si int) (*T, bool) {
// Internal function which can be called recursively to match all leaf nodes to a given filter subject which
// once here has been decomposed to parts. These parts only care about wildcards, both pwc and fwc.
func (t *SubjectTree[T]) match(n node, parts [][]byte, pre []byte, cb func(subject []byte, val *T)) {
// Returns false if the callback requested to stop matching.
func (t *SubjectTree[T]) match(n node, parts [][]byte, pre []byte, cb func(subject []byte, val *T) bool) bool {
// Capture if we are sitting on a terminal fwc.
var hasFWC bool
if lp := len(parts); lp > 0 && len(parts[lp-1]) > 0 && parts[lp-1][0] == fwc {
@@ -304,15 +326,17 @@ func (t *SubjectTree[T]) match(n node, parts [][]byte, pre []byte, cb func(subje
nparts, matched := n.matchParts(parts)
// Check if we did not match.
if !matched {
return
return true
}
// We have matched here. If we are a leaf and have exhausted all parts or he have a FWC fire callback.
if n.isLeaf() {
if len(nparts) == 0 || (hasFWC && len(nparts) == 1) {
ln := n.(*leaf[T])
cb(append(pre, ln.suffix...), &ln.value)
if !cb(append(pre, ln.suffix...), &ln.value) {
return false
}
}
return
return true
}
// We have normal nodes here.
// We need to append our prefix
@@ -340,17 +364,23 @@ func (t *SubjectTree[T]) match(n node, parts [][]byte, pre []byte, cb func(subje
if cn.isLeaf() {
ln := cn.(*leaf[T])
if len(ln.suffix) == 0 {
cb(append(pre, ln.suffix...), &ln.value)
if !cb(append(pre, ln.suffix...), &ln.value) {
return false
}
} else if hasTermPWC && bytes.IndexByte(ln.suffix, tsep) < 0 {
cb(append(pre, ln.suffix...), &ln.value)
if !cb(append(pre, ln.suffix...), &ln.value) {
return false
}
}
} else if hasTermPWC {
// We have terminal pwc so call into match again with the child node.
t.match(cn, nparts, pre, cb)
if !t.match(cn, nparts, pre, cb) {
return false
}
}
}
// Return regardless.
return
return true
}
// If we are sitting on a terminal fwc, put back and continue.
if hasFWC && len(nparts) == 0 {
@@ -367,18 +397,21 @@ func (t *SubjectTree[T]) match(n node, parts [][]byte, pre []byte, cb func(subje
// to see if we match further down.
for _, cn := range n.children() {
if cn != nil {
t.match(cn, nparts, pre, cb)
if !t.match(cn, nparts, pre, cb) {
return false
}
}
}
return
return true
}
// Here we have normal traversal, so find the next child.
nn := n.findChild(p)
if nn == nil {
return
return true
}
n, parts = *nn, nparts
}
return true
}
// Internal iter function to walk nodes in lexicographical order.
@@ -448,3 +481,60 @@ func LazyIntersect[TL, TR any](tl *SubjectTree[TL], tr *SubjectTree[TR], cb func
})
}
}
// IntersectGSL will match all items in the given subject tree that
// have interest expressed in the given sublist. The callback will only be called
// once for each subject, regardless of overlapping subscriptions in the sublist.
func IntersectGSL[T any, SL comparable](t *SubjectTree[T], sl *gsl.GenericSublist[SL], cb func(subject []byte, val *T)) {
if t == nil || t.root == nil || sl == nil {
return
}
var _pre [256]byte
_intersectGSL(t.root, _pre[:0], sl, cb)
}
func _intersectGSL[T any, SL comparable](n node, pre []byte, sl *gsl.GenericSublist[SL], cb func(subject []byte, val *T)) {
if n.isLeaf() {
ln := n.(*leaf[T])
subj := append(pre, ln.suffix...)
if sl.HasInterest(bytesToString(subj)) {
cb(subj, &ln.value)
}
return
}
bn := n.base()
pre = append(pre, bn.prefix...)
for _, cn := range n.children() {
if cn == nil {
continue
}
subj := append(pre, cn.path()...)
if !hasInterestForTokens(sl, subj, len(pre)) {
continue
}
_intersectGSL(cn, pre, sl, cb)
}
}
// The subject tree can return partial tokens so we need to check starting interest
// only from whole tokens when we encounter a tsep.
func hasInterestForTokens[SL comparable](sl *gsl.GenericSublist[SL], subj []byte, since int) bool {
for i := since; i < len(subj); i++ {
if subj[i] == tsep {
if !sl.HasInterestStartingIn(bytesToString(subj[:i])) {
return false
}
}
}
return true
}
// Note this will avoid a copy of the data used for the string, but it will also reference the existing slice's data pointer.
// So this should be used sparingly when we know the encompassing byte slice's lifetime is the same.
func bytesToString(b []byte) string {
if len(b) == 0 {
return ""
}
p := unsafe.SliceData(b)
return unsafe.String(p, len(b))
}
+8 -67
View File
@@ -21,8 +21,6 @@ import (
"sync"
"sync/atomic"
"unicode/utf8"
"github.com/nats-io/nats-server/v2/server/stree"
)
// Sublist is a routing mechanism to handle subject distribution and
@@ -818,7 +816,9 @@ func matchLevelForAny(l *level, toks []string, np, nq *int) bool {
*nq += len(qsub)
}
}
return len(n.plist) > 0 || len(n.psubs) > 0 || len(n.qsubs) > 0
if len(n.plist) > 0 || len(n.psubs) > 0 || len(n.qsubs) > 0 {
return true
}
}
if pwc != nil {
if np != nil && nq != nil {
@@ -1327,8 +1327,9 @@ func SubjectsCollide(subj1, subj2 string) bool {
if subj1 == subj2 {
return true
}
toks1 := strings.Split(subj1, tsep)
toks2 := strings.Split(subj2, tsep)
tsa, tsb := [32]string{}, [32]string{}
toks1 := tokenizeSubjectIntoSlice(tsa[:0], subj1)
toks2 := tokenizeSubjectIntoSlice(tsb[:0], subj2)
pwc1, fwc1 := analyzeTokens(toks1)
pwc2, fwc2 := analyzeTokens(toks2)
// if both literal just string compare.
@@ -1338,9 +1339,9 @@ func SubjectsCollide(subj1, subj2 string) bool {
}
// So one or both have wildcards. If one is literal than we can do subset matching.
if l1 && !l2 {
return isSubsetMatch(toks1, subj2)
return isSubsetMatchTokenized(toks1, toks2)
} else if l2 && !l1 {
return isSubsetMatch(toks2, subj1)
return isSubsetMatchTokenized(toks2, toks1)
}
// Both have wildcards.
// If they only have partials then the lengths must match.
@@ -1726,63 +1727,3 @@ func getAllNodes(l *level, results *SublistResult) {
getAllNodes(n.next, results)
}
}
// IntersectStree will match all items in the given subject tree that
// have interest expressed in the given sublist. The callback will only be called
// once for each subject, regardless of overlapping subscriptions in the sublist.
func IntersectStree[T any](st *stree.SubjectTree[T], sl *Sublist, cb func(subj []byte, entry *T)) {
var _subj [255]byte
intersectStree(st, sl.root, _subj[:0], cb)
}
func intersectStree[T any](st *stree.SubjectTree[T], r *level, subj []byte, cb func(subj []byte, entry *T)) {
nsubj := subj
if len(nsubj) > 0 {
nsubj = append(subj, '.')
}
if r.fwc != nil {
// We've reached a full wildcard, do a FWC match on the stree at this point
// and don't keep iterating downward.
nsubj := append(nsubj, '>')
st.Match(nsubj, cb)
return
}
if r.pwc != nil {
// We've found a partial wildcard. We'll keep iterating downwards, but first
// check whether there's interest at this level (without triggering dupes) and
// match if so.
var done bool
nsubj := append(nsubj, '*')
if len(r.pwc.psubs)+len(r.pwc.qsubs) > 0 {
st.Match(nsubj, cb)
done = true
}
if r.pwc.next.numNodes() > 0 {
intersectStree(st, r.pwc.next, nsubj, cb)
}
if done {
return
}
}
// Normal node with subject literals, keep iterating.
for t, n := range r.nodes {
if r.pwc != nil && r.pwc.next.numNodes() > 0 && n.next.numNodes() > 0 {
// A wildcard at the next level will already visit these descendents
// so skip so we don't callback the same subject more than once.
continue
}
nsubj := append(nsubj, t...)
if len(n.psubs)+len(n.qsubs) > 0 {
if subjectHasWildcard(bytesToString(nsubj)) {
st.Match(nsubj, cb)
} else {
if e, ok := st.Find(nsubj); ok {
cb(nsubj, e)
}
}
}
if n.next.numNodes() > 0 {
intersectStree(st, n.next, nsubj, cb)
}
}
}
+23 -2
View File
@@ -1,4 +1,4 @@
// Copyright 2012-2024 The NATS Authors
// Copyright 2012-2025 The NATS Authors
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
@@ -20,6 +20,7 @@ import (
"errors"
"fmt"
"math"
"math/bits"
"net"
"net/url"
"reflect"
@@ -165,7 +166,7 @@ func urlsAreEqual(u1, u2 *url.URL) bool {
// e.g. comma(834142) -> 834,142
//
// This function was copied from the github.com/dustin/go-humanize
// package and is Copyright Dustin Sallings <dustin@spy.net>
// package (MIT License) and is Copyright Dustin Sallings <dustin@spy.net>
func comma(v int64) string {
sign := ""
@@ -363,3 +364,23 @@ func parallelTaskQueue(mp int) chan<- func() {
}
return tq
}
// addSaturate returns a + b, saturating at math.MaxInt64.
// Both a and b must be non-negative.
func addSaturate(a, b int64) int64 {
sum, carry := bits.Add64(uint64(a), uint64(b), 0)
if carry != 0 || sum > uint64(math.MaxInt64) {
return math.MaxInt64
}
return int64(sum)
}
// mulSaturate returns a * b, saturating at math.MaxInt64.
// Both a and b must be non-negative.
func mulSaturate(a, b int64) int64 {
hi, lo := bits.Mul64(uint64(a), uint64(b))
if hi != 0 || lo > uint64(math.MaxInt64) {
return math.MaxInt64
}
return int64(lo)
}
+222 -83
View File
@@ -31,6 +31,7 @@ import (
"strconv"
"strings"
"sync"
"sync/atomic"
"time"
"unicode/utf8"
@@ -59,6 +60,8 @@ const (
wsMaxControlPayloadSize = 125
wsFrameSizeForBrowsers = 4096 // From experiment, webrowsers behave better with limited frame size
wsCompressThreshold = 64 // Don't compress for small buffer(s)
wsMaxMsgPayloadMultiple = 8
wsMaxMsgPayloadLimit = 64 * 1024 * 1024
wsCloseSatusSize = 2
// From https://tools.ietf.org/html/rfc6455#section-11.7
@@ -127,7 +130,7 @@ type srvWebsocket struct {
server *http.Server
listener net.Listener
listenerErr error
allowedOrigins map[string]*allowedOrigin // host will be the key
allowedOrigins map[string][]*allowedOrigin // host will be the key
sameOrigin bool
connectURLs []string
connectURLsMap refCountedUrlSet
@@ -153,7 +156,7 @@ type wsUpgradeResult struct {
}
type wsReadInfo struct {
rem int
rem uint64
fs bool
ff bool
fc bool
@@ -162,31 +165,57 @@ type wsReadInfo struct {
mkey [4]byte
cbufs [][]byte
coff int
csz uint64
}
func (r *wsReadInfo) init() {
r.fs, r.ff = true, true
}
func (r *wsReadInfo) resetCompressedState() {
r.fs = true
r.ff = true
r.fc = false
r.rem = 0
r.cbufs = nil
r.coff = 0
r.csz = 0
}
// Compressed WebSocket messages have to be accumulated before they can be
// decompressed and handed to the parser, so this transport limit needs to
// allow batching several max_payload-sized NATS operations while still
// capping resource usage on the buffered compressed path.
func wsMaxMessageSize(mpay int) uint64 {
if mpay <= 0 {
mpay = MAX_PAYLOAD_SIZE
}
limit := uint64(mpay) * wsMaxMsgPayloadMultiple
if limit > wsMaxMsgPayloadLimit {
limit = wsMaxMsgPayloadLimit
}
return limit
}
// Returns a slice containing `needed` bytes from the given buffer `buf`
// starting at position `pos`, and possibly read from the given reader `r`.
// When bytes are present in `buf`, the `pos` is incremented by the number
// of bytes found up to `needed` and the new position is returned. If not
// enough bytes are found, the bytes found in `buf` are copied to the returned
// slice and the remaning bytes are read from `r`.
func wsGet(r io.Reader, buf []byte, pos, needed int) ([]byte, int, error) {
avail := len(buf) - pos
func wsGet(r io.Reader, buf []byte, pos, needed uint64) ([]byte, uint64, error) {
avail := uint64(len(buf)) - pos
if avail >= needed {
return buf[pos : pos+needed], pos + needed, nil
}
b := make([]byte, needed)
start := copy(b, buf[pos:])
start := uint64(copy(b, buf[pos:]))
for start != needed {
n, err := r.Read(b[start:cap(b)])
if err != nil {
return nil, 0, err
}
start += n
start += uint64(n)
}
return b, pos + avail, nil
}
@@ -205,12 +234,43 @@ func (c *client) isWebsocket() bool {
//
// Client lock MUST NOT be held on entry.
func (c *client) wsRead(r *wsReadInfo, ior io.Reader, buf []byte) ([][]byte, error) {
var bufs [][]byte
err := c.wsReadLoop(r, ior, buf, func(b []byte, compressed, final bool) error {
if compressed {
return errors.New("compressed websocket frames require wsReadAndParse")
}
bufs = append(bufs, b)
return nil
})
return bufs, err
}
func (c *client) wsReadAndParse(r *wsReadInfo, ior io.Reader, buf []byte) error {
mpay := int(atomic.LoadInt32(&c.mpay))
if mpay <= 0 {
mpay = MAX_PAYLOAD_SIZE
}
return c.wsReadLoop(r, ior, buf, func(b []byte, compressed, final bool) error {
if compressed {
if err := c.wsDecompressAndParse(r, b, final, mpay); err != nil {
r.resetCompressedState()
return err
}
if final {
r.fc = false
}
return nil
}
return c.parse(b)
})
}
func (c *client) wsReadLoop(r *wsReadInfo, ior io.Reader, buf []byte, handle func([]byte, bool, bool) error) error {
var (
bufs [][]byte
tmpBuf []byte
err error
pos int
max = len(buf)
pos uint64
max = uint64(len(buf))
)
for pos != max {
if r.fs {
@@ -218,69 +278,80 @@ func (c *client) wsRead(r *wsReadInfo, ior io.Reader, buf []byte) ([][]byte, err
frameType := wsOpCode(b0 & 0xF)
final := b0&wsFinalBit != 0
compressed := b0&wsRsv1Bit != 0
if b0&(wsRsv2Bit|wsRsv3Bit) != 0 {
return c.wsHandleProtocolError("RSV2 and RSV3 must be clear")
}
if compressed && !c.ws.compress {
return c.wsHandleProtocolError("compressed frame received without negotiated permessage-deflate")
}
pos++
tmpBuf, pos, err = wsGet(ior, buf, pos, 1)
if err != nil {
return bufs, err
return err
}
b1 := tmpBuf[0]
// Clients MUST set the mask bit. If not set, reject.
// However, LEAF by default will not have masking, unless they are forced to, by configuration.
if r.mask && b1&wsMaskBit == 0 {
return bufs, c.wsHandleProtocolError("mask bit missing")
return c.wsHandleProtocolError("mask bit missing")
}
// Store size in case it is < 125
r.rem = int(b1 & 0x7F)
r.rem = uint64(b1 & 0x7F)
switch frameType {
case wsPingMessage, wsPongMessage, wsCloseMessage:
if r.rem > wsMaxControlPayloadSize {
return bufs, c.wsHandleProtocolError(
return c.wsHandleProtocolError(
fmt.Sprintf("control frame length bigger than maximum allowed of %v bytes",
wsMaxControlPayloadSize))
}
if !final {
return bufs, c.wsHandleProtocolError("control frame does not have final bit set")
return c.wsHandleProtocolError("control frame does not have final bit set")
}
if compressed {
return c.wsHandleProtocolError("control frame must not be compressed")
}
case wsTextMessage, wsBinaryMessage:
if !r.ff {
return bufs, c.wsHandleProtocolError("new message started before final frame for previous message was received")
return c.wsHandleProtocolError("new message started before final frame for previous message was received")
}
r.ff = final
r.fc = compressed
case wsContinuationFrame:
// Compressed bit must be only set in the first frame
if r.ff || compressed {
return bufs, c.wsHandleProtocolError("invalid continuation frame")
return c.wsHandleProtocolError("invalid continuation frame")
}
r.ff = final
default:
return bufs, c.wsHandleProtocolError(fmt.Sprintf("unknown opcode %v", frameType))
return c.wsHandleProtocolError(fmt.Sprintf("unknown opcode %v", frameType))
}
switch r.rem {
case 126:
tmpBuf, pos, err = wsGet(ior, buf, pos, 2)
if err != nil {
return bufs, err
return err
}
r.rem = int(binary.BigEndian.Uint16(tmpBuf))
r.rem = uint64(binary.BigEndian.Uint16(tmpBuf))
case 127:
tmpBuf, pos, err = wsGet(ior, buf, pos, 8)
if err != nil {
return bufs, err
return err
}
if r.rem = binary.BigEndian.Uint64(tmpBuf); r.rem&(uint64(1)<<63) != 0 {
return c.wsHandleProtocolError("invalid 64-bit payload length")
}
r.rem = int(binary.BigEndian.Uint64(tmpBuf))
}
if r.mask {
// Read masking key
tmpBuf, pos, err = wsGet(ior, buf, pos, 4)
if err != nil {
return bufs, err
return err
}
copy(r.mkey[:], tmpBuf)
r.mkpos = 0
@@ -290,7 +361,7 @@ func (c *client) wsRead(r *wsReadInfo, ior io.Reader, buf []byte) ([][]byte, err
if wsIsControlFrame(frameType) {
pos, err = c.wsHandleControlFrame(r, frameType, ior, buf, pos)
if err != nil {
return bufs, err
return err
}
continue
}
@@ -299,53 +370,26 @@ func (c *client) wsRead(r *wsReadInfo, ior io.Reader, buf []byte) ([][]byte, err
r.fs = false
}
if pos < max {
var b []byte
var n int
n = r.rem
n := r.rem
if pos+n > max {
n = max - pos
}
b = buf[pos : pos+n]
b := buf[pos : pos+n]
pos += n
r.rem -= n
// If needed, unmask the buffer
if r.mask {
r.unmask(b)
}
addToBufs := true
// Handle compressed message
if r.fc {
// Assume that we may have continuation frames or not the full payload.
addToBufs = false
// Make a copy of the buffer before adding it to the list
// of compressed fragments.
r.cbufs = append(r.cbufs, append([]byte(nil), b...))
// When we have the final frame and we have read the full payload,
// we can decompress it.
if r.ff && r.rem == 0 {
b, err = r.decompress()
if err != nil {
return bufs, err
}
r.fc = false
// Now we can add to `bufs`
addToBufs = true
}
if err := handle(b, r.fc, r.ff && r.rem == 0); err != nil {
return err
}
// For non compressed frames, or when we have decompressed the
// whole message.
if addToBufs {
bufs = append(bufs, b)
}
// If payload has been fully read, then indicate that next
// is the start of a frame.
if r.rem == 0 {
r.fs = true
}
}
}
return bufs, nil
return nil
}
func (r *wsReadInfo) Read(dst []byte) (int, error) {
@@ -389,6 +433,9 @@ func (r *wsReadInfo) nextCBuf() []byte {
}
func (r *wsReadInfo) ReadByte() (byte, error) {
for len(r.cbufs) > 0 && len(r.cbufs[0]) == 0 {
r.nextCBuf()
}
if len(r.cbufs) == 0 {
return 0, io.EOF
}
@@ -398,33 +445,71 @@ func (r *wsReadInfo) ReadByte() (byte, error) {
return b, nil
}
func (r *wsReadInfo) decompress() ([]byte, error) {
r.coff = 0
// As per https://tools.ietf.org/html/rfc7692#section-7.2.2
// add 0x00, 0x00, 0xff, 0xff and then a final block so that flate reader
// does not report unexpected EOF.
func (c *client) wsDecompressAndParse(r *wsReadInfo, b []byte, final bool, mpay int) error {
limit := wsMaxMessageSize(mpay)
if len(b) > 0 {
if r.csz+uint64(len(b)) > limit {
return ErrMaxPayload
}
r.cbufs = append(r.cbufs, append([]byte(nil), b...))
r.csz += uint64(len(b))
}
if !final {
return nil
}
if r.csz+uint64(len(compressLastBlock)) > limit {
return ErrMaxPayload
}
r.cbufs = append(r.cbufs, compressLastBlock)
// Get a decompressor from the pool and bind it to this object (wsReadInfo)
// that provides Read() and ReadByte() APIs that will consume the compressed
// buffers (r.cbufs).
r.csz += uint64(len(compressLastBlock))
r.coff = 0
d, _ := decompressorPool.Get().(io.ReadCloser)
if d == nil {
d = flate.NewReader(r)
} else {
d.(flate.Resetter).Reset(r, nil)
}
// This will do the decompression.
b, err := io.ReadAll(d)
decompressorPool.Put(d)
// Now reset the compressed buffers list.
r.cbufs = nil
return b, err
defer func() {
d.Close()
decompressorPool.Put(d)
r.cbufs = nil
r.coff = 0
r.csz = 0
}()
lr := io.LimitedReader{R: d, N: int64(mpay + 1)}
buf := make([]byte, 32*1024)
total := 0
for {
n, err := lr.Read(buf)
if n > 0 {
pn := n
if total+n > mpay {
pn = mpay - total
}
if pn > 0 {
if err := c.parse(buf[:pn]); err != nil {
return err
}
}
total += n
if total > mpay {
return ErrMaxPayload
}
}
if err == nil {
continue
}
if err == io.EOF {
return nil
}
return err
}
}
// Handles the PING, PONG and CLOSE websocket control frames.
//
// Client lock MUST NOT be held on entry.
func (c *client) wsHandleControlFrame(r *wsReadInfo, frameType wsOpCode, nc io.Reader, buf []byte, pos int) (int, error) {
func (c *client) wsHandleControlFrame(r *wsReadInfo, frameType wsOpCode, nc io.Reader, buf []byte, pos uint64) (uint64, error) {
var payload []byte
var err error
@@ -443,6 +528,9 @@ func (c *client) wsHandleControlFrame(r *wsReadInfo, frameType wsOpCode, nc io.R
status := wsCloseStatusNoStatusReceived
var body string
lp := len(payload)
if lp == 1 {
return pos, c.wsHandleProtocolError("close frame payload cannot be 1 byte")
}
// If there is a payload, the status is represented as a 2-byte
// unsigned integer (in network byte order). Then, there may be an
// optional body.
@@ -450,6 +538,9 @@ func (c *client) wsHandleControlFrame(r *wsReadInfo, frameType wsOpCode, nc io.R
if hasStatus {
// Decode the status
status = int(binary.BigEndian.Uint16(payload[:wsCloseSatusSize]))
if !wsIsValidCloseStatus(status) {
return pos, c.wsHandleProtocolError(fmt.Sprintf("invalid close status code %v", status))
}
// Now if there is a body, capture it and make sure this is a valid UTF-8.
if hasBody {
body = string(payload[wsCloseSatusSize:])
@@ -686,6 +777,21 @@ func (c *client) wsHandleProtocolError(message string) error {
return errors.New(message)
}
func wsIsValidCloseStatus(code int) bool {
switch code {
case wsCloseStatusNoStatusReceived, 1004, 1006, wsCloseStatusTLSHandshake:
return false
}
if code < 1000 || code >= 5000 {
return false
}
// 1016-2999 are currently reserved.
if code >= 1016 && code <= 2999 {
return false
}
return true
}
// Create a close message with the given `status` and `body`.
// If the `body` is more than the maximum allows control frame payload size,
// it is truncated and "..." is added at the end (as a hint that message
@@ -745,6 +851,10 @@ func (s *Server) wsUpgrade(w http.ResponseWriter, r *http.Request) (*wsUpgradeRe
if key == _EMPTY_ {
return nil, wsReturnHTTPError(w, r, http.StatusBadRequest, "key missing")
}
decoded, err := base64.StdEncoding.DecodeString(key)
if err != nil || len(decoded) != 16 {
return nil, wsReturnHTTPError(w, r, http.StatusBadRequest, "invalid websocket key")
}
// Point 6.
if !wsHeaderContains(r.Header, "Sec-Websocket-Version", "13") {
return nil, wsReturnHTTPError(w, r, http.StatusBadRequest, "invalid version")
@@ -766,7 +876,10 @@ func (s *Server) wsUpgrade(w http.ResponseWriter, r *http.Request) (*wsUpgradeRe
// We will do masking if asked (unless we reject for tests)
noMasking := r.Header.Get(wsNoMaskingHeader) == wsNoMaskingValue && !wsTestRejectNoMasking
h := w.(http.Hijacker)
h, ok := w.(http.Hijacker)
if !ok {
return nil, wsReturnHTTPError(w, r, http.StatusBadRequest, "websocket upgrade not supported")
}
conn, brw, err := h.Hijack()
if err != nil {
if conn != nil {
@@ -814,9 +927,11 @@ func (s *Server) wsUpgrade(w http.ResponseWriter, r *http.Request) (*wsUpgradeRe
// Check for X-Forwarded-For header
if cips, ok := r.Header[wsXForwardedForHeader]; ok {
cip := cips[0]
if net.ParseIP(cip) != nil {
ws.clientIP = cip
if len(cips) > 0 {
cip := cips[0]
if net.ParseIP(cip) != nil {
ws.clientIP = cip
}
}
}
@@ -946,7 +1061,11 @@ func (w *srvWebsocket) checkOrigin(r *http.Request) error {
if err != nil {
return err
}
if oh != rh || op != rp {
rs := "http"
if r.TLS != nil {
rs = "https"
}
if oh != rh || op != rp || !strings.EqualFold(u.Scheme, rs) {
return errors.New("not same origin")
}
// I guess it is possible to have cases where one wants to check
@@ -955,9 +1074,16 @@ func (w *srvWebsocket) checkOrigin(r *http.Request) error {
}
if !listEmpty {
w.mu.RLock()
ao := w.allowedOrigins[oh]
origins := w.allowedOrigins[oh]
w.mu.RUnlock()
if ao == nil || u.Scheme != ao.scheme || op != ao.port {
var allowed bool
for _, ao := range origins {
if u.Scheme == ao.scheme && op == ao.port {
allowed = true
break
}
}
if !allowed {
return errors.New("not in the allowed list")
}
}
@@ -1011,7 +1137,17 @@ func validateWebsocketOptions(o *Options) error {
}
// Make sure that allowed origins, if specified, can be parsed.
for _, ao := range wo.AllowedOrigins {
if _, err := url.Parse(ao); err != nil {
u, err := url.ParseRequestURI(ao)
if err != nil {
return fmt.Errorf("unable to parse allowed origin: %v", err)
}
if u.Scheme != "http" && u.Scheme != "https" {
return fmt.Errorf("unable to parse allowed origin %q: allowed origins must be absolute URLs with http or https scheme", ao)
}
if u.Host == _EMPTY_ {
return fmt.Errorf("unable to parse allowed origin %q: host is required", ao)
}
if _, _, err := wsGetHostAndPort(u.Scheme == "https", u.Host); err != nil {
return fmt.Errorf("unable to parse allowed origin: %v", err)
}
}
@@ -1083,9 +1219,9 @@ func (s *Server) wsSetOriginOptions(o *WebsocketOpts) {
}
h, p, _ := wsGetHostAndPort(u.Scheme == "https", u.Host)
if ws.allowedOrigins == nil {
ws.allowedOrigins = make(map[string]*allowedOrigin, len(o.AllowedOrigins))
ws.allowedOrigins = make(map[string][]*allowedOrigin, len(o.AllowedOrigins))
}
ws.allowedOrigins[h] = &allowedOrigin{scheme: u.Scheme, port: p}
ws.allowedOrigins[h] = append(ws.allowedOrigins[h], &allowedOrigin{scheme: u.Scheme, port: p})
}
}
@@ -1306,7 +1442,7 @@ func (s *Server) createWSClient(conn net.Conn, ws *websocket) *client {
return c
}
if opts.MaxConn > 0 && len(s.clients) >= opts.MaxConn {
if opts.MaxConn < 0 || (opts.MaxConn > 0 && len(s.clients) >= opts.MaxConn) {
s.mu.Unlock()
c.maxConnExceeded()
return nil
@@ -1384,7 +1520,7 @@ func (c *client) wsCollapsePtoNB() (net.Buffers, int64) {
cp.Reset(buf)
}
var csz int
for _, b := range nb {
for i, b := range nb {
for len(b) > 0 {
n, err := cp.Write(b)
if err != nil {
@@ -1396,7 +1532,10 @@ func (c *client) wsCollapsePtoNB() (net.Buffers, int64) {
}
b = b[n:]
}
nbPoolPut(b) // No longer needed as contents written to compressor.
// Use original slice since capacity will change to zero
// in the loop after consuming the buffer, which will make
// nbPoolPut discard it.
nbPoolPut(nb[i])
}
if err := cp.Flush(); err != nil {
c.Errorf("Error during compression: %v", err)
+12 -12
View File
@@ -23,7 +23,7 @@ A [Go](http://golang.org) client for the [NATS messaging system](https://nats.io
go get github.com/nats-io/nats.go@latest
# To get a specific version:
go get github.com/nats-io/nats.go@v1.47.0
go get github.com/nats-io/nats.go@v1.49.0
# Note that the latest major version for NATS Server is v2:
go get github.com/nats-io/nats-server/v2@latest
@@ -134,7 +134,7 @@ The simplest form is to use the helper method UserCredentials(credsFilepath).
nc, err := nats.Connect(url, nats.UserCredentials("user.creds"))
```
The helper methods creates two callback handlers to present the user JWT and sign the nonce challenge from the server.
The helper method creates two callback handlers to present the user JWT and sign the nonce challenge from the server.
The core client library never has direct access to your private key and simply performs the callback for signing the server challenge.
The helper will load and wipe and erase memory it uses for each connect or reconnect.
@@ -177,7 +177,7 @@ nc, err := nats.Connect("tls://nats.demo.io:4443")
// We provide a helper method to make this case easier.
nc, err = nats.Connect("tls://localhost:4443", nats.RootCAs("./configs/certs/ca.pem"))
// If the server requires client certificate, there is an helper function for that too:
// If the server requires client certificate, there is a helper function for that too:
cert := nats.ClientCert("./configs/certs/client-cert.pem", "./configs/certs/client-key.pem")
nc, err = nats.Connect("tls://localhost:4443", cert)
@@ -210,17 +210,17 @@ if err != nil {
// "*" matches any token, at any level of the subject.
nc.Subscribe("foo.*.baz", func(m *Msg) {
fmt.Printf("Msg received on [%s] : %s\n", m.Subject, string(m.Data));
fmt.Printf("Msg received on [%s] : %s\n", m.Subject, string(m.Data))
})
nc.Subscribe("foo.bar.*", func(m *Msg) {
fmt.Printf("Msg received on [%s] : %s\n", m.Subject, string(m.Data));
fmt.Printf("Msg received on [%s] : %s\n", m.Subject, string(m.Data))
})
// ">" matches any length of the tail of a subject, and can only be the last token
// E.g. 'foo.>' will match 'foo.bar', 'foo.bar.baz', 'foo.foo.bar.bax.22'
nc.Subscribe("foo.>", func(m *Msg) {
fmt.Printf("Msg received on [%s] : %s\n", m.Subject, string(m.Data));
fmt.Printf("Msg received on [%s] : %s\n", m.Subject, string(m.Data))
})
// Matches all of the above
@@ -237,7 +237,7 @@ nc.Publish("foo.bar.baz", []byte("Hello World"))
// Normal subscribers will continue to work as expected.
nc.QueueSubscribe("foo", "job_workers", func(_ *Msg) {
received += 1;
received += 1
})
```
@@ -267,9 +267,9 @@ fmt.Println("All clear!")
// FlushTimeout specifies a timeout value as well.
err := nc.FlushTimeout(1*time.Second)
if err != nil {
fmt.Println("All clear!")
} else {
fmt.Println("Flushed timed out!")
} else {
fmt.Println("All clear!")
}
// Auto-unsubscribe after MAX_WANTED messages received
@@ -285,7 +285,7 @@ nc1.Subscribe("foo", func(m *Msg) {
fmt.Printf("Received a message: %s\n", string(m.Data))
})
nc2.Publish("foo", []byte("Hello World!"));
nc2.Publish("foo", []byte("Hello World!"))
```
@@ -339,7 +339,7 @@ nc, err = nats.Connect("nats://localhost:4222", nats.UserInfo("foo", "bar"))
// For token based authentication:
nc, err = nats.Connect("nats://localhost:4222", nats.Token("S3cretT0ken"))
// You can even pass the two at the same time in case one of the server
// You can even pass the two at the same time in case one of the servers
// in the mesh requires token instead of user name and password.
nc, err = nats.Connect("nats://localhost:4222",
nats.UserInfo("foo", "bar"),
@@ -372,7 +372,7 @@ msg, err := sub.NextMsgWithContext(ctx)
```
## Backwards compatibility
## Backward compatibility
In the development of nats.go, we are committed to maintaining backward compatibility and ensuring a stable and reliable experience for all users. In general, we follow the standard go compatibility guidelines.
However, it's important to clarify our stance on certain types of changes:
+1 -1
View File
@@ -95,7 +95,7 @@ func (nc *Conn) oldRequestWithContext(ctx context.Context, subj string, hdr, dat
s.AutoUnsubscribe(1)
defer s.Unsubscribe()
err = nc.publish(subj, inbox, hdr, data)
err = nc.publish(subj, inbox, false, hdr, data)
if err != nil {
return nil, err
}
+2 -2
View File
@@ -107,7 +107,7 @@ func (c *EncodedConn) Publish(subject string, v any) error {
if err != nil {
return err
}
return c.Conn.publish(subject, _EMPTY_, nil, b)
return c.Conn.publish(subject, _EMPTY_, false, nil, b)
}
// PublishRequest will perform a Publish() expecting a response on the
@@ -120,7 +120,7 @@ func (c *EncodedConn) PublishRequest(subject, reply string, v any) error {
if err != nil {
return err
}
return c.Conn.publish(subject, reply, nil, b)
return c.Conn.publish(subject, reply, true, nil, b)
}
// Request will create an Inbox and perform a Request() call
+9 -9
View File
@@ -4,19 +4,19 @@ go 1.24.0
require (
github.com/golang/protobuf v1.4.2
github.com/klauspost/compress v1.18.0
github.com/klauspost/compress v1.18.2
github.com/nats-io/jwt/v2 v2.8.0
github.com/nats-io/nats-server/v2 v2.12.0
github.com/nats-io/nkeys v0.4.11
github.com/nats-io/nats-server/v2 v2.12.3
github.com/nats-io/nkeys v0.4.12
github.com/nats-io/nuid v1.0.1
google.golang.org/protobuf v1.23.0
)
require (
github.com/antithesishq/antithesis-sdk-go v0.4.3-default-no-op // indirect
github.com/google/go-tpm v0.9.5 // indirect
github.com/minio/highwayhash v1.0.3 // indirect
golang.org/x/crypto v0.42.0 // indirect
golang.org/x/sys v0.36.0 // indirect
golang.org/x/time v0.13.0 // indirect
github.com/antithesishq/antithesis-sdk-go v0.5.0-default-no-op // indirect
github.com/google/go-tpm v0.9.7 // indirect
github.com/minio/highwayhash v1.0.4-0.20251030100505-070ab1a87a76 // indirect
golang.org/x/crypto v0.46.0 // indirect
golang.org/x/sys v0.39.0 // indirect
golang.org/x/time v0.14.0 // indirect
)
+18 -18
View File
@@ -1,5 +1,5 @@
github.com/antithesishq/antithesis-sdk-go v0.4.3-default-no-op h1:+OSa/t11TFhqfrX0EOSqQBDJ0YlpmK0rDSiB19dg9M0=
github.com/antithesishq/antithesis-sdk-go v0.4.3-default-no-op/go.mod h1:IUpT2DPAKh6i/YhSbt6Gl3v2yvUZjmKncl7U91fup7E=
github.com/antithesishq/antithesis-sdk-go v0.5.0-default-no-op h1:Ucf+QxEKMbPogRO5guBNe5cgd9uZgfoJLOYs8WWhtjM=
github.com/antithesishq/antithesis-sdk-go v0.5.0-default-no-op/go.mod h1:IUpT2DPAKh6i/YhSbt6Gl3v2yvUZjmKncl7U91fup7E=
github.com/golang/protobuf v1.4.0-rc.1/go.mod h1:ceaxUfeHdC40wWswd/P6IGgMaK3YpKi5j83Wpe3EHw8=
github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:xKAWHe0F5eneWXFV3EuXVDTCmh+JuBKY0li0aMyXATA=
github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrUpVNzEA03Pprs=
@@ -12,27 +12,27 @@ github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMyw
github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
github.com/google/go-cmp v0.5.9 h1:O2Tfq5qg4qc4AmwVlvv0oLiVAGB7enBSJ2x2DqQFi38=
github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
github.com/google/go-tpm v0.9.5 h1:ocUmnDebX54dnW+MQWGQRbdaAcJELsa6PqZhJ48KwVU=
github.com/google/go-tpm v0.9.5/go.mod h1:h9jEsEECg7gtLis0upRBQU+GhYVH6jMjrFxI8u6bVUY=
github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo=
github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ=
github.com/minio/highwayhash v1.0.3 h1:kbnuUMoHYyVl7szWjSxJnxw11k2U709jqFPPmIUyD6Q=
github.com/minio/highwayhash v1.0.3/go.mod h1:GGYsuwP/fPD6Y9hMiXuapVvlIUEhFhMTh0rxU3ik1LQ=
github.com/google/go-tpm v0.9.7 h1:u89J4tUUeDTlH8xxC3CTW7OHZjbjKoHdQ9W7gCUhtxA=
github.com/google/go-tpm v0.9.7/go.mod h1:h9jEsEECg7gtLis0upRBQU+GhYVH6jMjrFxI8u6bVUY=
github.com/klauspost/compress v1.18.2 h1:iiPHWW0YrcFgpBYhsA6D1+fqHssJscY/Tm/y2Uqnapk=
github.com/klauspost/compress v1.18.2/go.mod h1:R0h/fSBs8DE4ENlcrlib3PsXS61voFxhIs2DeRhCvJ4=
github.com/minio/highwayhash v1.0.4-0.20251030100505-070ab1a87a76 h1:KGuD/pM2JpL9FAYvBrnBBeENKZNh6eNtjqytV6TYjnk=
github.com/minio/highwayhash v1.0.4-0.20251030100505-070ab1a87a76/go.mod h1:GGYsuwP/fPD6Y9hMiXuapVvlIUEhFhMTh0rxU3ik1LQ=
github.com/nats-io/jwt/v2 v2.8.0 h1:K7uzyz50+yGZDO5o772eRE7atlcSEENpL7P+b74JV1g=
github.com/nats-io/jwt/v2 v2.8.0/go.mod h1:me11pOkwObtcBNR8AiMrUbtVOUGkqYjMQZ6jnSdVUIA=
github.com/nats-io/nats-server/v2 v2.12.0 h1:OIwe8jZUqJFrh+hhiyKu8snNib66qsx806OslqJuo74=
github.com/nats-io/nats-server/v2 v2.12.0/go.mod h1:nr8dhzqkP5E/lDwmn+A2CvQPMd1yDKXQI7iGg3lAvww=
github.com/nats-io/nkeys v0.4.11 h1:q44qGV008kYd9W1b1nEBkNzvnWxtRSQ7A8BoqRrcfa0=
github.com/nats-io/nkeys v0.4.11/go.mod h1:szDimtgmfOi9n25JpfIdGw12tZFYXqhGxjhVxsatHVE=
github.com/nats-io/nats-server/v2 v2.12.3 h1:KRv+1n7lddMVgkJPQer+pt36TcO0ENxjilBmeWdjcHs=
github.com/nats-io/nats-server/v2 v2.12.3/go.mod h1:MQXjG9WjyXKz9koWzUc3jYUMKD8x3CLmTNy91IQQz3Y=
github.com/nats-io/nkeys v0.4.12 h1:nssm7JKOG9/x4J8II47VWCL1Ds29avyiQDRn0ckMvDc=
github.com/nats-io/nkeys v0.4.12/go.mod h1:MT59A1HYcjIcyQDJStTfaOY6vhy9XTUjOFo+SVsvpBg=
github.com/nats-io/nuid v1.0.1 h1:5iA8DT8V7q8WK2EScv2padNa/rTESc1KdnPw4TC2paw=
github.com/nats-io/nuid v1.0.1/go.mod h1:19wcPz3Ph3q0Jbyiqsd0kePYG7A95tJPxeL+1OSON2c=
golang.org/x/crypto v0.42.0 h1:chiH31gIWm57EkTXpwnqf8qeuMUi0yekh6mT2AvFlqI=
golang.org/x/crypto v0.42.0/go.mod h1:4+rDnOTJhQCx2q7/j6rAN5XDw8kPjeaXEUR2eL94ix8=
golang.org/x/crypto v0.46.0 h1:cKRW/pmt1pKAfetfu+RCEvjvZkA9RimPbh7bhFjGVBU=
golang.org/x/crypto v0.46.0/go.mod h1:Evb/oLKmMraqjZ2iQTwDwvCtJkczlDuTmdJXoZVzqU0=
golang.org/x/sys v0.21.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.36.0 h1:KVRy2GtZBrk1cBYA7MKu5bEZFxQk4NIDV6RLVcC8o0k=
golang.org/x/sys v0.36.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
golang.org/x/time v0.13.0 h1:eUlYslOIt32DgYD6utsuUeHs4d7AsEYLuIAdg7FlYgI=
golang.org/x/time v0.13.0/go.mod h1:eL/Oa2bBBK0TkX57Fyni+NgnyQQN4LitPmob2Hjnqw4=
golang.org/x/sys v0.39.0 h1:CvCKL8MeisomCi6qNZ+wbb0DN9E5AATixKsvNtMoMFk=
golang.org/x/sys v0.39.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
golang.org/x/time v0.14.0 h1:MRx4UaLrDotUKUdCIqzPC48t1Y9hANFKIRpNx+Te8PI=
golang.org/x/time v0.14.0/go.mod h1:eL/Oa2bBBK0TkX57Fyni+NgnyQQN4LitPmob2Hjnqw4=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8=
google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0=
+749
View File
@@ -0,0 +1,749 @@
# Migrating from Legacy JetStream API to `jetstream` Package
This guide helps you migrate from the legacy JetStream API in the `nats` package
(`nats.JetStreamContext`) to the new `jetstream` package
(`github.com/nats-io/nats.go/jetstream`).
- [Why Migrate?](#why-migrate)
- [Getting Started](#getting-started)
- [Stream Management](#stream-management)
- [Consumer Management](#consumer-management)
- [Publishing](#publishing)
- [Consuming Messages](#consuming-messages)
- [Replacing js.Subscribe()](#replacing-jssubscribe)
- [Replacing js.PullSubscribe()](#replacing-jspullsubscribe)
- [Ordered Consumers](#ordered-consumers)
- [Push Consumers](#push-consumers)
- [Subscription Options Mapping](#subscription-options-mapping)
- [Error Handling in Consume/Messages](#error-handling-in-consumemessages)
- [Message Acknowledgement](#message-acknowledgement)
- [KeyValue Store](#keyvalue-store)
- [Object Store](#object-store)
## Why Migrate?
The legacy JetStream API (`nats.JetStreamContext`) is deprecated. The `jetstream`
package provides a cleaner, more predictable API with several key improvements:
- **Explicit resource management.** Streams and consumers are created and managed
explicitly. The legacy `js.Subscribe()` implicitly created consumers behind
the scenes, leading to surprising behavior.
- **Pull consumers as the default.** Pull consumers with `Consume()` and
`Messages()` provide the same continuous message delivery as the legacy push-based
`Subscribe()`, but with better flow control and no slow consumer issues.
- **`context.Context` throughout.** All API calls accept `context.Context` for
timeout and cancellation, replacing the mix of `MaxWait`, `AckWait`, and
`Context()` options.
- **Clear interface separation.** Instead of one large `JetStreamContext` interface,
functionality is split across focused interfaces: `JetStream`, `Stream` and
`Consumer`.
## Getting Started
The core NATS connection remains unchanged. Only the JetStream initialization
differs:
```go
import (
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
)
nc, _ := nats.Connect(nats.DefaultURL)
```
**Legacy:**
```go
js, _ := nc.JetStream()
// With domain
js, _ := nc.JetStream(nats.Domain("hub"))
// With custom API prefix
js, _ := nc.JetStream(nats.APIPrefix("myprefix"))
```
**New:**
```go
js, _ := jetstream.New(nc)
// With domain
js, _ := jetstream.NewWithDomain(nc, "hub")
// With custom API prefix
js, _ := jetstream.NewWithAPIPrefix(nc, "myprefix")
```
### Initialization Options
| Legacy | New |
|-----------------------------------|--------------------------------------------|
| `nats.Domain(domain)` | `jetstream.NewWithDomain(nc, domain)` |
| `nats.APIPrefix(prefix)` | `jetstream.NewWithAPIPrefix(nc, prefix)` |
| `nats.PublishAsyncMaxPending(n)` | `jetstream.WithPublishAsyncMaxPending(n)` |
| `nats.PublishAsyncErrHandler(cb)` | `jetstream.WithPublishAsyncErrHandler(cb)` |
## Stream Management
`StreamConfig` is essentially the same struct — it just lives in the `jetstream`
package now. The new API takes `StreamConfig` by value (not pointer) and
management methods return a `Stream` handle instead of `*StreamInfo`.
| Legacy | New | Notes |
|---------------------------------|-------------------------------------|-------------------------------------------------------------------|
| `js.AddStream(cfg)` | `js.CreateStream(ctx, cfg)` | Also: `CreateOrUpdateStream()` |
| `js.UpdateStream(cfg)` | `js.UpdateStream(ctx, cfg)` | |
| `js.DeleteStream(name)` | `js.DeleteStream(ctx, name)` | |
| `js.StreamInfo(name)` | `s.Info(ctx)` / `s.CachedInfo()` | Get stream handle first via `js.Stream(ctx, name)` |
| `js.PurgeStream(name, opts...)` | `s.Purge(ctx, opts...)` | Options: `WithPurgeSubject`, `WithPurgeSequence`, `WithPurgeKeep` |
| `js.GetMsg(name, seq)` | `s.GetMsg(ctx, seq)` | |
| `js.GetLastMsg(name, subj)` | `s.GetLastMsgForSubject(ctx, subj)` | |
| `js.DeleteMsg(name, seq)` | `s.DeleteMsg(ctx, seq)` | Also: `s.SecureDeleteMsg()` |
| `js.Streams()` | `js.ListStreams(ctx)` | Returns lister with `.Info()` channel and `.Err()` |
| `js.StreamNames()` | `js.StreamNames(ctx)` | Returns lister with `.Name()` channel and `.Err()` |
The key architectural difference is that stream-specific operations (purge, get/delete
messages) now live on the `Stream` interface instead of the top-level context. Get
a stream handle first, then operate on it:
```go
s, _ := js.Stream(ctx, "ORDERS")
s.Purge(ctx)
msg, _ := s.GetMsg(ctx, 100)
```
## Consumer Management
The biggest conceptual change: in the legacy API, `js.Subscribe()` would
implicitly create consumers. In the new API, consumer creation is always explicit
and separate from message consumption.
| Legacy | New | Notes |
|------------------------------------------|----------------------------------------|----------------------------------------------------------------|
| `js.AddConsumer(stream, cfg)` | `js.CreateConsumer(ctx, stream, cfg)` | Also: `CreateOrUpdateConsumer()`, `UpdateConsumer()` |
| `js.Subscribe(subj, handler)` (implicit) | No equivalent | Must create consumer explicitly first |
| `js.ConsumerInfo(stream, name)` | `cons.Info(ctx)` / `cons.CachedInfo()` | Get consumer handle first via `js.Consumer(ctx, stream, name)` |
| `js.DeleteConsumer(stream, name)` | `js.DeleteConsumer(ctx, stream, name)` | |
| `js.Consumers(stream)` | `s.ListConsumers(ctx)` | Returns lister with `.Info()` channel and `.Err()` |
| `js.ConsumerNames(stream)` | `s.ConsumerNames(ctx)` | Returns lister with `.Name()` channel and `.Err()` |
Consumer management is available at two levels:
- On `JetStream` — requires stream name as parameter (e.g. `js.CreateConsumer(ctx, "ORDERS", cfg)`), bypassing the need to fetch a stream
- On `Stream` — no stream name needed (e.g. `s.CreateConsumer(ctx, cfg)`)
The new API provides three creation methods:
- `CreateConsumer` — fails if the consumer already exists with different config
- `UpdateConsumer` - fails if the consumer does not exist
- `CreateOrUpdateConsumer` — creates or updates as needed
**Additional notes on consumer behavior:**
- The default ack policy changed between the APIs. In the legacy API,
`AddConsumer()` defaulted to `AckNone`. In the new API, the default is
`AckExplicit`.
- In the legacy API, `sub.Unsubscribe()` on an implicitly created
consumer would automatically delete that consumer on the server. The new API
does not perform any automatic cleanup - consumers must be deleted explicitly
via `DeleteConsumer()`, or via `InactiveThreshold` on the consumer
config to let the server remove it automatically after a period of inactivity.
Push consumers use separate methods: `CreatePushConsumer`, `CreateOrUpdatePushConsumer`,
`UpdatePushConsumer`, and `PushConsumer` (for getting a handle).
```go
s, _ := js.Stream(ctx, "ORDERS")
cons, _ := s.CreateOrUpdateConsumer(ctx, jetstream.ConsumerConfig{
Durable: "processor",
})
```
## Publishing
Publishing is largely the same, with the addition of `context.Context` for
synchronous operations.
### Synchronous Publish
**Legacy:**
```go
ack, _ := js.Publish("ORDERS.new", []byte("hello"))
ack, _ = js.PublishMsg(&nats.Msg{
Subject: "ORDERS.new",
Data: []byte("hello"),
})
```
**New:**
```go
ack, _ := js.Publish(ctx, "ORDERS.new", []byte("hello"))
ack, _ = js.PublishMsg(ctx, &nats.Msg{
Subject: "ORDERS.new",
Data: []byte("hello"),
})
```
### Async Publish
**Legacy:**
```go
ackF, _ := js.PublishAsync("ORDERS.new", []byte("hello"))
select {
case ack := <-ackF.Ok():
fmt.Println(ack.Sequence)
case err := <-ackF.Err():
fmt.Println(err)
}
// Wait for all pending acks
<-js.PublishAsyncComplete()
```
**New:**
```go
// Async publish does not take context (returns immediately)
ackF, _ := js.PublishAsync("ORDERS.new", []byte("hello"))
select {
case ack := <-ackF.Ok():
fmt.Println(ack.Sequence)
case err := <-ackF.Err():
fmt.Println(err)
}
<-js.PublishAsyncComplete()
```
### Publish Options
| Legacy | New |
|------------------------------------------|---------------------------------------------------|
| `nats.MsgId(id)` | `jetstream.WithMsgID(id)` |
| `nats.ExpectStream(name)` | `jetstream.WithExpectStream(name)` |
| `nats.ExpectLastSequence(seq)` | `jetstream.WithExpectLastSequence(seq)` |
| `nats.ExpectLastSequencePerSubject(seq)` | `jetstream.WithExpectLastSequencePerSubject(seq)` |
| `nats.ExpectLastMsgId(id)` | `jetstream.WithExpectLastMsgID(id)` |
| `nats.RetryWait(dur)` | `jetstream.WithRetryWait(dur)` |
| `nats.RetryAttempts(n)` | `jetstream.WithRetryAttempts(n)` |
| `nats.StallWait(dur)` | `jetstream.WithStallWait(dur)` |
## Consuming Messages
This is the most significant area of change. The legacy API offered many
subscription flavors (`Subscribe`, `SubscribeSync`, `QueueSubscribe`,
`ChanSubscribe`, `PullSubscribe`) that blurred the line between consumer
creation, stream lookup and message consumption. The new API separates these
concerns: first create a consumer, then choose how to receive messages.
With the exception of PullSubscribe, all legacy subscription flavors utilized push consumers under the hood. The new API recommends pull consumers for all use cases, as they provide better flow control and no risk of slow consumer issues. Pull-based consumption is available via `Consume()` and `Messages()`, which maintain persistent pull subscriptions with pre-buffering for efficient continuous delivery. Push consumers are still supported for users who prefer that model, but pull consumers are the recommended default.
### Replacing `js.Subscribe()`
The legacy `js.Subscribe()` created a push consumer behind the scenes (unless
explicitly specified otherwise via `nats.Bind()` or `nats.Durable()`) and
delivered messages either via a callback. In the new API, the recommended
replacement is a **pull consumer** with `Consume()` or `Messages()`. These
provide the same continuous delivery with better flow control.
#### Legacy: callback subscription
```go
sub, _ := js.Subscribe("ORDERS.*", func(msg *nats.Msg) {
fmt.Printf("Received: %s\n", string(msg.Data))
msg.Ack()
}, nats.Durable("processor"), nats.ManualAck)
defer sub.Unsubscribe()
```
#### New: callback with `Consume()`
`Consume()` is the closest equivalent to `js.Subscribe()` — it delivers messages
to a callback function continuously.
```go
s, _ := js.Stream(ctx, "ORDERS")
cons, _ := s.CreateOrUpdateConsumer(ctx, jetstream.ConsumerConfig{
Durable: "processor",
FilterSubject: "ORDERS.*",
})
cc, _ := cons.Consume(func(msg jetstream.Msg) {
fmt.Printf("Received: %s\n", string(msg.Data()))
msg.Ack()
})
defer cc.Stop()
```
> Note: `ManualAck()` is not needed — messages are never auto-acknowledged in
> the new API.
#### New: iterator with `Messages()`
`Messages()` provides an iterator-based approach, useful when you want explicit
control over when the next message is fetched.
```go
cons, _ := s.CreateOrUpdateConsumer(ctx, jetstream.ConsumerConfig{
Durable: "processor",
FilterSubject: "ORDERS.*",
})
iter, _ := cons.Messages()
for {
msg, err := iter.Next()
if err != nil {
// handle error
}
fmt.Printf("Received: %s\n", string(msg.Data()))
msg.Ack()
}
// Call iter.Stop() when done
```
Both `Consume()` and `Messages()` maintain overlapping pull requests to the
server, providing efficient continuous delivery without gaps.
#### Legacy: synchronous subscription
```go
sub, _ := js.SubscribeSync("ORDERS.*", nats.Durable("processor"))
msg, _ := sub.NextMsg(time.Second)
```
**New:** Use `Messages()` and call `Next()`:
```go
cons, _ := s.CreateOrUpdateConsumer(ctx, jetstream.ConsumerConfig{
Durable: "processor",
FilterSubject: "ORDERS.*",
})
iter, _ := cons.Messages()
msg, _ := iter.Next()
```
#### Legacy: queue subscription
```go
// Multiple instances share work via a queue group
sub, _ := js.QueueSubscribe("ORDERS.*", "workers", handler,
nats.Durable("processor"))
```
**New with pull consumers:** With pull consumers, there is no need for an
explicit queue group. Multiple application instances (or goroutines) calling
`Consume()` or `Messages()` on the same durable consumer will naturally
distribute messages among themselves — the server tracks pending acknowledgements
and avoids delivering the same message to multiple consumers:
```go
cons, _ := s.CreateOrUpdateConsumer(ctx, jetstream.ConsumerConfig{
Durable: "processor",
})
cc, _ := cons.Consume(handler)
defer cc.Stop()
```
**New with push consumers:** If you need push-based queue semantics, set
`DeliverGroup` on a push consumer — this is the direct equivalent of the legacy
queue group:
```go
cons, _ := s.CreateOrUpdatePushConsumer(ctx, jetstream.ConsumerConfig{
Durable: "processor",
DeliverSubject: "deliver.orders",
DeliverGroup: "workers",
})
cc, _ := cons.Consume(handler)
defer cc.Stop()
```
> **Note:** Push consumers with `DeliverGroup` cannot be flow controlled. If you
> experience slow consumer issues, consider using pull-based consumers instead —
> multiple instances on the same durable consumer achieve the same work
> distribution without the slow consumer risk.
#### Legacy: channel subscription
```go
ch := make(chan *nats.Msg, 64)
sub, _ := js.ChanSubscribe("ORDERS.*", ch, nats.Durable("processor"))
for msg := range ch {
msg.Ack()
}
```
**New:** There is no direct channel-based equivalent. Use `Consume()` or
`Messages()` instead.
### Replacing `js.PullSubscribe()`
The legacy pull subscription required creating a subscription and then calling
`Fetch()` in a loop.
#### Legacy: pull subscribe + fetch loop
```go
sub, _ := js.PullSubscribe("ORDERS.*", "processor")
for {
msgs, _ := sub.Fetch(10, nats.MaxWait(5*time.Second))
for _, msg := range msgs {
fmt.Printf("Received: %s\n", string(msg.Data))
msg.Ack()
}
}
```
**New with `Fetch()`/`FetchNoWait()` (one-off batch):**
If you specifically need one-off batch fetching, `Fetch()` is available directly
on the consumer — no separate subscription step:
```go
cons, _ := s.CreateOrUpdateConsumer(ctx, jetstream.ConsumerConfig{
Durable: "processor",
FilterSubject: "ORDERS.*",
})
// non-blocking, returns a `FetchResult` that provides messages and error
msgs, _ := cons.Fetch(10, jetstream.FetchMaxWait(5*time.Second))
for msg := range msgs.Messages() {
fmt.Printf("Received: %s\n", string(msg.Data()))
msg.Ack()
}
if msgs.Error() != nil {
// handle error
}
```
> **Warning:** `Fetch()`, `FetchNoWait()`, and `FetchBytes()` are one-off,
> single pull requests. They do not perform pre-buffering optimizations. For
> continuous message processing, always prefer `Consume()` or `Messages()`.
> When using `FetchBytes()`, the requested byte size must stay under the
> client's max pending bytes limit (64MB by default), otherwise it will trigger
> slow consumer errors on the underlying subscription.
### Ordered Consumers
Ordered consumers provide strictly ordered, gap-free message delivery. The library
automatically recreates the underlying consumer on sequence gaps or heartbeat
failures.
**Legacy:**
```go
sub, _ := js.Subscribe("ORDERS.*", handler, nats.OrderedConsumer())
```
**New:**
```go
cons, _ := js.OrderedConsumer(ctx, "ORDERS", jetstream.OrderedConsumerConfig{
FilterSubjects: []string{"ORDERS.*"},
})
// Use the same consumption methods as regular consumers
cc, _ := cons.Consume(func(msg jetstream.Msg) {
fmt.Printf("Received: %s\n", string(msg.Data()))
})
defer cc.Stop()
```
### Push Consumers
Pull consumers are recommended for most use cases, but push consumers are also
supported. Push consumers require `DeliverSubject` in their config and only
support `Consume()` (not `Fetch()` or `Messages()`).
**Legacy:**
```go
sub, _ := js.Subscribe("ORDERS.*", handler,
nats.Durable("processor"),
nats.DeliverSubject("deliver.orders"),
nats.IdleHeartbeat(30*time.Second),
)
```
**New:**
```go
cons, _ := s.CreateOrUpdatePushConsumer(ctx, jetstream.ConsumerConfig{
Durable: "processor",
FilterSubject: "ORDERS.*",
DeliverSubject: "deliver.orders",
IdleHeartbeat: 30 * time.Second,
})
cc, _ := cons.Consume(func(msg jetstream.Msg) {
fmt.Printf("Received: %s\n", string(msg.Data()))
msg.Ack()
})
defer cc.Stop()
```
### Subscription Options Mapping
Most legacy `SubOpt` options map directly to `ConsumerConfig` fields. Since
consumer creation is explicit, these are set at creation time rather than passed
as subscription options.
| Legacy SubOpt | New ConsumerConfig field |
|-------------------------------------|---------------------------------------------------------------------------|
| `nats.Durable("name")` | `Durable: "name"` |
| `nats.ConsumerName("name")` | `Name: "name"` |
| `nats.Description("desc")` | `Description: "desc"` |
| `nats.DeliverAll()` | `DeliverPolicy: jetstream.DeliverAllPolicy` |
| `nats.DeliverLast()` | `DeliverPolicy: jetstream.DeliverLastPolicy` |
| `nats.DeliverLastPerSubject()` | `DeliverPolicy: jetstream.DeliverLastPerSubjectPolicy` |
| `nats.DeliverNew()` | `DeliverPolicy: jetstream.DeliverNewPolicy` |
| `nats.StartSequence(seq)` | `DeliverPolicy: jetstream.DeliverByStartSequencePolicy, OptStartSeq: seq` |
| `nats.StartTime(t)` | `DeliverPolicy: jetstream.DeliverByStartTimePolicy, OptStartTime: &t` |
| `nats.AckExplicit()` | `AckPolicy: jetstream.AckExplicitPolicy` |
| `nats.AckAll()` | `AckPolicy: jetstream.AckAllPolicy` |
| `nats.AckNone()` | `AckPolicy: jetstream.AckNonePolicy` |
| `nats.ManualAck()` | Not needed (messages are never auto-acked) |
| `nats.MaxDeliver(n)` | `MaxDeliver: n` |
| `nats.MaxAckPending(n)` | `MaxAckPending: n` |
| `nats.BackOff(durations)` | `BackOff: durations` |
| `nats.ReplayOriginal()` | `ReplayPolicy: jetstream.ReplayOriginalPolicy` |
| `nats.ReplayInstant()` | `ReplayPolicy: jetstream.ReplayInstantPolicy` |
| `nats.RateLimit(bps)` | `RateLimit: bps` |
| `nats.HeadersOnly()` | `HeadersOnly: true` |
| `nats.InactiveThreshold(dur)` | `InactiveThreshold: dur` |
| `nats.ConsumerFilterSubjects(s...)` | `FilterSubjects: s` |
| `nats.ConsumerReplicas(n)` | `Replicas: n` |
| `nats.ConsumerMemoryStorage()` | `MemoryStorage: true` |
The following options have no direct equivalent — use the consumer handle
directly instead:
| Legacy SubOpt | New equivalent |
|-------------------------------|---------------------------------------------------------------------|
| `nats.Bind(stream, consumer)` | `js.Consumer(ctx, stream, consumer)` or `s.Consumer(ctx, consumer)` |
| `nats.BindStream(stream)` | Use `js.Stream(ctx, stream)` to get a stream handle |
| `nats.OrderedConsumer()` | `js.OrderedConsumer(ctx, stream, cfg)` |
### Consume/Messages Options
`Consume()` and `Messages()` accept options that control pull request behavior:
| Option | Description |
|----------------------------|--------------------------------------------------------------|
| `PullMaxMessages(n)` | Max messages buffered (default: 500) |
| `PullMaxBytes(n)` | Max bytes buffered (mutually exclusive with PullMaxMessages) |
| `PullExpiry(dur)` | Pull request timeout (default: 30s) |
| `PullHeartbeat(dur)` | Idle heartbeat interval |
| `PullThresholdMessages(n)` | Refill threshold (default: 50% of max) |
| `PullThresholdBytes(n)` | Byte-based refill threshold |
| `StopAfter(n)` | Auto-stop after N messages |
| `ConsumeErrHandler(fn)` | Custom error handler |
### Error Handling in Consume/Messages
Both `Consume()` and `Messages()` handle server-sent status messages internally.
Some errors are terminal (stop consumption), while others are recoverable
(consumption continues).
**Terminal errors** — consumption stops automatically:
- `ErrConsumerDeleted` — the consumer was deleted on the server
- `ErrBadRequest` — invalid request (e.g. misconfigured consumer)
- Connection closed — for `Consume()` this surfaces as `ErrConnectionClosed`;
for `Messages()`, `Next()` returns `ErrMsgIteratorClosed`
**Recoverable errors** — reported via error handler, consumption continues:
- `ErrNoHeartbeat` — missed idle heartbeats from server; a new pull request
is issued automatically
- `ErrConsumerLeadershipChanged` — consumer moved to a different server in the
cluster; pending counts are reset
- `nats.ErrNoResponders` — no JetStream service available (temporary)
#### Error handling with `Consume()`
Use `ConsumeErrHandler` to be notified about both terminal and recoverable errors:
```go
cc, _ := cons.Consume(func(msg jetstream.Msg) {
msg.Ack()
}, jetstream.ConsumeErrHandler(func(cc jetstream.ConsumeContext, err error) {
if errors.Is(err, jetstream.ErrConsumerDeleted) ||
errors.Is(err, jetstream.ErrBadRequest) {
log.Fatalf("terminal consumer error: %v", err)
}
log.Printf("recoverable consumer error: %v", err)
}))
defer cc.Stop()
```
#### Error handling with `Messages()`
With `Messages()`, terminal errors are returned directly by `Next()`. By default,
`ErrNoHeartbeat` is also returned by `Next()` (controlled by
`WithMessagesErrOnMissingHeartbeat`), but it is not terminal — you can continue
calling `Next()`:
```go
iter, _ := cons.Messages()
for {
msg, err := iter.Next()
if err != nil {
if errors.Is(err, jetstream.ErrMsgIteratorClosed) {
// iterator was stopped (either explicitly or due to connection close)
break
}
if errors.Is(err, jetstream.ErrNoHeartbeat) {
// recoverable — new pull request is issued, keep going
log.Println("missed heartbeat, re-pulling")
continue
}
// ErrConsumerDeleted, ErrBadRequest are terminal
log.Fatalf("terminal error: %v", err)
}
msg.Ack()
}
```
## Message Acknowledgement
Ack methods are similar, with minor naming changes. The main difference is that
message fields are accessed via methods instead of struct fields.
| Legacy | New |
|-------------------------|------------------------------|
| `msg.Ack()` | Unchanged |
| `msg.AckSync()` | `msg.DoubleAck(ctx)` |
| `msg.Nak()` | Unchanged |
| `msg.NakWithDelay(dur)` | Unchanged |
| `msg.InProgress()` | Unchanged |
| `msg.Term()` | Unchanged |
| N/A | `msg.TermWithReason(reason)` |
| `msg.Metadata()` | Unchanged |
### Accessing Message Data
**Legacy:** Direct struct fields on `*nats.Msg`:
```go
fmt.Println(string(msg.Data))
fmt.Println(msg.Subject)
fmt.Println(msg.Header.Get("key"))
```
**New:** Methods on `jetstream.Msg` interface:
```go
fmt.Println(string(msg.Data()))
fmt.Println(msg.Subject())
fmt.Println(msg.Headers().Get("key"))
```
## KeyValue Store
The KV API is nearly identical. The main changes are:
1. All methods take `context.Context` as the first parameter
2. New `CreateOrUpdateKeyValue()` and `UpdateKeyValue()` methods
3. Types live in the `jetstream` package
**Legacy:**
```go
js, _ := nc.JetStream()
kv, _ := js.CreateKeyValue(&nats.KeyValueConfig{
Bucket: "profiles",
})
kv.Put("sue.color", []byte("blue"))
entry, _ := kv.Get("sue.color")
fmt.Println(string(entry.Value()))
watcher, _ := kv.Watch("sue.*")
defer watcher.Stop()
```
**New:**
```go
js, _ := jetstream.New(nc)
kv, _ := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{
Bucket: "profiles",
})
kv.Put(ctx, "sue.color", []byte("blue"))
entry, _ := kv.Get(ctx, "sue.color")
fmt.Println(string(entry.Value()))
watcher, _ := kv.Watch(ctx, "sue.*")
defer watcher.Stop()
```
### KV Management Methods
| Legacy | New |
|-----------------------------|---------------------------------------|
| `js.KeyValue(bucket)` | `js.KeyValue(ctx, bucket)` |
| `js.CreateKeyValue(cfg)` | `js.CreateKeyValue(ctx, cfg)` |
| N/A | `js.UpdateKeyValue(ctx, cfg)` |
| N/A | `js.CreateOrUpdateKeyValue(ctx, cfg)` |
| `js.DeleteKeyValue(bucket)` | `js.DeleteKeyValue(ctx, bucket)` |
| `js.KeyValueStoreNames()` | `js.KeyValueStoreNames(ctx)` |
| `js.KeyValueStores()` | `js.KeyValueStores(ctx)` |
## Object Store
Same pattern as KV — all methods gain `context.Context`, types move to `jetstream`
package.
**Legacy:**
```go
js, _ := nc.JetStream()
os, _ := js.CreateObjectStore(&nats.ObjectStoreConfig{
Bucket: "configs",
})
os.PutString("config-1", "data")
result, _ := os.Get("config-1")
data, _ := io.ReadAll(result)
```
**New:**
```go
js, _ := jetstream.New(nc)
os, _ := js.CreateObjectStore(ctx, jetstream.ObjectStoreConfig{
Bucket: "configs",
})
os.PutString(ctx, "config-1", "data")
result, _ := os.Get(ctx, "config-1")
data, _ := io.ReadAll(result)
```
### Object Store Management Methods
| Legacy | New |
|--------------------------------|------------------------------------------|
| `js.ObjectStore(bucket)` | `js.ObjectStore(ctx, bucket)` |
| `js.CreateObjectStore(cfg)` | `js.CreateObjectStore(ctx, cfg)` |
| N/A | `js.UpdateObjectStore(ctx, cfg)` |
| N/A | `js.CreateOrUpdateObjectStore(ctx, cfg)` |
| `js.DeleteObjectStore(bucket)` | `js.DeleteObjectStore(ctx, bucket)` |
+22 -25
View File
@@ -121,7 +121,7 @@ func main() {
messageCounter++
}
fmt.Printf("received %d messages\n", messageCounter)
fmt.Printf("Received %d messages\n", messageCounter)
if msgs.Error() != nil {
fmt.Println("Error during Fetch(): ", msgs.Error())
@@ -224,7 +224,7 @@ _ = s.Purge(ctx, jetstream.WithPurgeSequence(100))
_ = s.Purge(ctx, jetstream.WithPurgeKeep(10))
```
- Get and messages from stream
- Get and delete messages from a stream
```go
// get message from stream with sequence number == 100
@@ -240,7 +240,7 @@ _ = s.DeleteMsg(ctx, 100)
- Get information about a stream
```go
// Fetches latest stream info from server
// Fetches the latest stream info from server
info, _ := s.Info(ctx)
fmt.Println(info.Config.Name)
@@ -310,7 +310,7 @@ cons2 := js.CreateOrUpdateConsumer(ctx, "ORDERS", jetstream.ConsumerConfig{
// or an illegal property is to be updated (e.g. AckPolicy)
updated, _ := js.UpdateConsumer(ctx, "ORDERS", jetstream.ConsumerConfig{
AckPolicy: jetstream.AckExplicitPolicy,
Description: "updated consumer"
Description: "updated consumer",
})
// get consumer handle
@@ -336,7 +336,7 @@ cons, _ := stream.CreateConsumer(ctx, jetstream.ConsumerConfig{
})
// get consumer handle
cons, _ = stream.Consumer(ctx, "ORDERS", "foo")
cons, _ = stream.Consumer(ctx, "foo")
// delete a consumer
stream.DeleteConsumer(ctx, "foo")
@@ -395,20 +395,19 @@ js, _ := jetstream.New(nc)
// create a consumer (this is an idempotent operation)
cons, _ := js.OrderedConsumer(ctx, "ORDERS", jetstream.OrderedConsumerConfig{
// Filter results from "ORDERS" stream by specific subject
FilterSubjects: []{"ORDERS.A"},
FilterSubjects: []string{"ORDERS.A"},
})
```
### Receiving messages from pull consumers
The `Consumer` interface covers allows fetching messages on demand, with
pre-defined batch size on bytes limit, or continuous push-like receiving of
The `Consumer` interface allows fetching messages on demand, with a
pre-defined batch size or byte limit, or continuous push-like receiving of
messages.
#### __Single fetch__
This pattern pattern allows fetching a defined number of messages in a single
RPC.
This pattern allows fetching a defined number of messages in a single RPC.
- Using `Fetch` or `FetchBytes`, consumer will return up to the provided number
of messages/bytes. By default, `Fetch()` will wait 30 seconds before timing out
@@ -481,10 +480,10 @@ single messages on demand.
Subject filtering is achieved by configuring a consumer with a `FilterSubject`
value.
##### Using `Consume()` receive messages in a callback
##### Using `Consume()` to receive messages in a callback
```go
cons, _ := js.CreateOrUpdateConsumer("ORDERS", jetstream.ConsumerConfig{
cons, _ := js.CreateOrUpdateConsumer(ctx, "ORDERS", jetstream.ConsumerConfig{
AckPolicy: jetstream.AckExplicitPolicy,
// receive messages from ORDERS.A subject only
FilterSubject: "ORDERS.A"
@@ -498,7 +497,7 @@ consContext, _ := c.Consume(func(msg jetstream.Msg) {
defer consContext.Stop()
```
Similarly to `Messages()`, `Consume()` can be supplied with options to modify
Similar to `Messages()`, `Consume()` can be supplied with options to modify
the behavior of a single pull request:
- `PullMaxMessages(int)` - up to provided number of messages will be buffered
@@ -511,7 +510,6 @@ the behavior of a single pull request:
request. If the value is set too low, the consumer will stall and not be able
to consume messages.
- `PullExpiry(time.Duration)` - timeout on a single pull request to the server
type PullThresholdMessages int
- `PullThresholdMessages(int)` - amount of messages which triggers refilling the
buffer
- `PullThresholdBytes(int)` - amount of bytes which triggers refilling the
@@ -521,10 +519,10 @@ request. An error will be triggered if at least 2 heartbeats are missed
- `ConsumeErrHandler(func (ConsumeContext, error))` - when used, sets a
custom error handler on `Consume()`, allowing e.g. tracking missing
heartbeats.
- `PullMaxMessagesWithBytesLimit` - up to the provided number of messages will
be buffered and a single fetch size will be limited to the provided value.
- `PullMaxMessagesWithBytesLimit(int, int)` - up to the provided number of messages
will be buffered and a single fetch size will be limited to the provided value.
This is an advanced option and should be used with caution. Most of the time,
`PullMaxMessages` or `PullMaxBytes` should be used instead. Note that he byte
`PullMaxMessages` or `PullMaxBytes` should be used instead. Note that the byte
limit should never be set to a value lower than the maximum message size that
can be expected from the server. If the byte limit is lower than the maximum
message size, the consumer will stall and not be able to consume messages.
@@ -568,7 +566,6 @@ iter, _ := cons.Messages(jetstream.PullMaxMessages(10), jetstream.PullMaxBytes(1
request. If the value is set too low, the consumer will stall and not be able
to consume messages.
- `PullExpiry(time.Duration)` - timeout on a single pull request to the server
type PullThresholdMessages int
- `PullThresholdMessages(int)` - amount of messages which triggers refilling the
buffer
- `PullThresholdBytes(int)` - amount of bytes which triggers refilling the
@@ -576,10 +573,10 @@ type PullThresholdMessages int
- `PullHeartbeat(time.Duration)` - idle heartbeat duration for a single pull
request. An error will be triggered if at least 2 heartbeats are missed (unless
`WithMessagesErrOnMissingHeartbeat(false)` is used)
- `PullMaxMessagesWithBytesLimit` - up to the provided number of messages will
be buffered and a single fetch size will be limited to the provided value.
- `PullMaxMessagesWithBytesLimit(int, int)` - up to the provided number of messages
will be buffered and a single fetch size will be limited to the provided value.
This is an advanced option and should be used with caution. Most of the time,
`PullMaxMessages` or `PullMaxBytes` should be used instead. Note that he byte
`PullMaxMessages` or `PullMaxBytes` should be used instead. Note that the byte
limit should never be set to a value lower than the maximum message size that
can be expected from the server. If the byte limit is lower than the maximum
message size, the consumer will stall and not be able to consume messages.
@@ -623,7 +620,7 @@ can be set to prevent the consumer from receiving more messages than it can
handle.
```go
cons, _ := js.CreateOrUpdatePushConsumer("ORDERS", jetstream.ConsumerConfig{
cons, _ := js.CreateOrUpdatePushConsumer(ctx, "ORDERS", jetstream.ConsumerConfig{
DeliverSubject: nats.NewInbox()
AckPolicy: jetstream.AckExplicitPolicy,
// receive messages from ORDERS.A subject only
@@ -672,7 +669,7 @@ setting various headers. Additionally, for `PublishMsg()` headers can be set
directly on `nats.Msg`.
```go
// All 3 implementations are work identically
// All 3 implementations work identically
ack, err := js.PublishMsg(ctx, &nats.Msg{
Data: []byte("hello"),
Subject: "ORDERS.new",
@@ -973,14 +970,14 @@ js.DeleteObjectStore(ctx, "configs")
Object Stores support Watchers, which can be used to watch for changes on
objects in a given bucket. Watcher will receive a notification on a channel when
a change occurs. By default, watcher will return latest information for all
a change occurs. By default, watcher will return the latest information for all
objects in a bucket. After sending all initial values, watcher will send nil on
the channel to signal that all initial values have been sent and it will start
sending updates when changes occur.
>__NOTE:__ Watchers do not retrieve values for objects, only metadata (containing
>information such as object name, bucket name, object size etc.). If object data
>is required, `Get` method should be used.
>is required, the `Get` method should be used.
Watcher supports several configuration options:
Loaded 100 of 292 files, more files were not shown because too many files have changed in this diff. Show more