* reverseproxy: propagate TCP half-close on upgraded streams
handleUpgradeResponse starts one copy goroutine per direction but returns as
soon as the first one reports a result, and returning runs two unconditional
deferred closes (the client connection and the backend connection). A client
that ends its send direction with a TCP half-close therefore causes the whole
tunnel to be torn down: the backend connection is reset mid-conversation and
whatever it was about to send is never delivered.
Treat a clean EOF as the end of one direction rather than the end of the
tunnel. When a copy finishes without error, propagate the half-close to the
destination with CloseWrite where the connection supports it, and keep waiting
for the other direction so pending bytes can still drain. Real copy errors, the
stream timeout and request cancellation/shutdown still tear down immediately.
If the destination does not support CloseWrite, the peer cannot be told that
the direction ended, so the previous behavior is kept and the tunnel is torn
down rather than held open until the stream timeout.
This is the same class that was fixed upstream in net/http/httputil
(https://go.dev/issue/35892); Caddy has its own upgraded-stream copier, so the
standard library fix does not apply here.
Fixes#8026
* reverseproxy: follow net/http/httputil shape for the half-close
Replace the copyResult struct and the closeWriter/closeWrite helpers with
upstream's plain error channel and errCopyDone sentinel, propagating
CloseWrite inside the copier, as requested in review.
Behaviour is unchanged. The wait is repeated twice inside the existing
select because handleUpgradeResponse also selects on the stream timeout
and sizes the channel at 2 so both goroutines can exit when it fires
(#7418), which net/http/httputil has no equivalent of.
* reverseproxy: explain the repeated wait next to it
Move the reasoning for why the wait is repeated inside the select, rather
than written as upstream's two-line form, into a comment beside the loop
so a future reader does not have to find the review discussion.
* reverseproxy: add an upgraded-stream half-close integration test
Drive a real upgraded stream through the handler and check that closing one
direction is propagated to the other side instead of tearing the tunnel down.
Mirrors net/http/httputil's TestReverseProxyWebSocketHalfTCP, which covers the
same class upstream (https://go.dev/issue/35892): a backend that hijacks and
hands back a *net.TCPConn, the handler in front of it, and a client that dials
it directly, then the four close-read / close-write combinations.
Against the unfixed handler the two half-close cases fail - the surviving
direction never delivers its pending bytes - while the two close-read cases
pass, so the test isolates exactly the reported behaviour.
* reverseproxy: give the half-close test reads a deadline
Without one, a regression blocks in ReadFull until the package timeout
instead of failing the test, which is the flakiness an end-to-end test of
this kind is usually accused of. Ten seconds is far above any real latency
here and turns a hang into a clear failure.
* reverseproxy: hijack the test backend via http.NewResponseController
Matches how the rest of the tree reaches Hijack (responsewriter.go,
caddyauth.go, streaming.go itself) and keeps working if the writer is
wrapped, where a direct http.Hijacker assertion would not.
* reverse proxy: rewrite requests and responses for websocket over http2
* delete protocol pseudo-header
* modify cloned requests
* set request variable to track if it's a h2 websocket
* use request bodu
* rewrite request body
* use WebSocket instead of Websocket in the headers
* use logger check for zap loggers
* fix lint
* reverseproxy: Add more debug logs
This makes debug logging very noisy when reverse proxying, but I guess
that's the point.
This has shown to be useful in troubleshooting infrastructure issues.
* Update modules/caddyhttp/reverseproxy/streaming.go
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
* Update modules/caddyhttp/reverseproxy/streaming.go
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
* Add opt-in `trace_logs` option
* Rename to VerboseLogs
---------
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
* caddyhttp: Make use of http.ResponseController
Also syncs the reverseproxy implementation with stdlib's which now uses ResponseController as well https://github.com/golang/go/commit/2449bbb5e614954ce9e99c8a481ea2ee73d72d61
* Enable full-duplex for HTTP/1.1
* Appease linter
* Add warning for builds with Go 1.20, so it's less surprising to users
* Improved godoc for EnableFullDuplex, copied text from stdlib
* Only wrap in encode if not already wrapped
* reverseproxy: Mask the WS close message when we're the client
* weakrand
* Bump golangci-lint version so path ignores work on Windows
* gofmt
* ugh, gofmt everything, I guess
* reverseproxy: Close hijacked conns on reload/quit
We also send a Close control message to both ends of
WebSocket connections. I have tested this many times in
my dev environment with consistent success, although
the variety of scenarios was limited.
* Oops... actually call Close() this time
* CloseMessage --> closeMessage
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
* Use httpguts, duh
* Use map instead of sync.Map
Co-authored-by: Francis Lavoie <lavofr@gmail.com>
* reverseproxy: Sync up `handleUpgradeResponse` with stdlib
I had left this as a TODO for when we bump to minimum 1.17, but I should've realized it was under `internal` so it couldn't be used directly.
Copied the functions we needed for parity. Hopefully this is ok!
* Add tests and fix godoc comments
Co-authored-by: Matthew Holt <mholt@users.noreply.github.com>
* ci: Use golangci's github action for linting
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Fix most of the staticcheck lint errors
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Fix the prealloc lint errors
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Fix the misspell lint errors
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Fix the varcheck lint errors
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Fix the errcheck lint errors
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Fix the bodyclose lint errors
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Fix the deadcode lint errors
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Fix the unused lint errors
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Fix the gosec lint errors
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Fix the gosimple lint errors
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Fix the ineffassign lint errors
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Fix the staticcheck lint errors
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Revert the misspell change, use a neutral English
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Remove broken golangci-lint CI job
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Re-add errantly-removed weakrand initialization
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* don't break the loop and return
* Removing extra handling for null rootKey
* unignore RegisterModule/RegisterAdapter
Co-authored-by: Mohammed Al Sahaf <msaa1990@gmail.com>
* single-line log message
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* Fix lint after a1808b0dbf209c615e438a496d257ce5e3acdce2 was merged
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Revert ticker change, ignore it instead
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Ignore some of the write errors
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Remove blank line
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Use lifetime
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* close immediately
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* Preallocate configVals
Signed-off-by: Dave Henderson <dhenderson@gmail.com>
* Update modules/caddytls/distributedstek/distributedstek.go
Co-authored-by: Mohammed Al Sahaf <msaa1990@gmail.com>
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
* reverseproxy: Enable error logging for connection upgrades
* reverseproxy: Change some of the error levels, unsugar
* Use unsugared log in one spot
Co-authored-by: Matthew Holt <mholt@users.noreply.github.com>
* reverse proxy: Support more h2 stream scenarios (#3556)
* reverse proxy: add integration test for better h2 stream (#3556)
* reverse proxy: adjust comments as francislavoie suggests
* link to issue #3556 in the comments
Turhan ACAR