The active health check body is user-supplied and commonly JSON, but it
was expanded with ReplaceAll, which blanks any `{...}` the replacer does
not recognize. Only globals are available here, so a JSON
health_request_body was sent as an empty body.
This is not limited to bodies without placeholders. A body that does use
one is mangled too: `{"token":"{env.SOME_VAR}"}` is sent as `"}`, since
the opening `{"token":"` is consumed as an unrecognized placeholder
before the real one is reached.
Use ReplaceKnown, matching the header values a few lines below, so
unrecognized braces are left intact and real placeholders still expand.
Only the body is changed. The Host header on the same path keeps
ReplaceAll deliberately: Go's HTTP server blanks a Host containing
braces, so leaving `{...}` intact there would send a Host the upstream
discards, where blanking it lets Go fall back to the URL host.
* Implement HTTPoxy mitigation in FastCGI
Added HTTPoxy mitigation to prevent trusting client-supplied Proxy header for HTTP_PROXY environment variable.
* Implement test for HTTPoxy vulnerability protection
Add test to ensure HTTPoxy vulnerability is mitigated by dropping client-supplied Proxy headers.
* gofmt: format code
* revert unrelated gofmt change to replacer_test.go
* reverseproxy: isolate active health-check state per distinct check config
Multiple reverse_proxy handlers configured with different active health
checks (health_uri, health_headers, ...) against the same upstream dial
address currently share a single Host in the global pool, so one
handler's failing probes mark the address unhealthy for every other
handler. Key the pool by dial address plus a stable fingerprint of the
active health-check config, so distinct checks get independent health
state.
The fingerprint is strictly internal to pool identity: the Prometheus
upstreams_healthy label and the /reverse_proxy/upstreams admin endpoint
continue to report the plain dial address, unchanged.
Dynamic upstreams are intentionally out of scope here: they resolve
through a separate per-lookup path (dynamicHosts) and collapsing there
has different lifetime semantics; noted for a follow-up.
Fixes#7870
* reverseproxy: use strings.Cut in hostKeyAddress
Satisfies the modernize linter; behaviour is unchanged, since Cut returns
the whole string when the separator is absent.
* reverseproxy: expose the health-check fingerprint as a public discriminator
Health state is now kept per (dial address, active health check config),
but both user-visible surfaces still reported address alone:
- caddy_reverse_proxy_upstreams_healthy was labeled only by upstream, so
every handler sharing an address wrote the same series concurrently and
the reported value was whichever updater ran last. The metric gains a
health_check label carrying the config fingerprint ("" when no active
checks), so each health target owns its series; aggregate across checks
with sum/min by (upstream).
- /reverse_proxy/upstreams reported one entry per pool key but with only
the plain address, so consumers indexing by address silently discarded
all but one entry. Entries now carry health_check (omitted when empty),
and the endpoint documents that (address, health_check) is the entry's
identity — one entry per health target, deliberately not aggregated,
since any aggregation here would be lossy and undocumented.
Tests: two handlers on one address with different checks must produce two
metric series reflecting their own state (fails if the fingerprint is
dropped from the label), and two admin entries distinguished by non-empty
fingerprints.
* reverseproxy: narrow the fix to per-Upstream active health counters
Move the consecutive active pass/fail counters from Host onto Upstream,
alongside the active unhealthy state that already lives there, instead of
re-keying the global host pool.
Host is keyed by dial address alone, but an active health check is
configured per handler, so two handlers dialing the same address with
different health_uri or health_headers share those counters and can push
each other over their own thresholds. Upstream is already per-handler and
already carries the active unhealthy flag, so the counters belong next to
it and the pool keeps its plain dial-address keys.
This drops the host key fingerprint and its exposure in the metric label
and the admin upstreams endpoint; the metric series identity is left for
separate consideration.
---------
Co-authored-by: SillyZir <269283839+SillyZir@users.noreply.github.com>
Co-authored-by: Zen Dodd <mail@steadytao.com>
Caddy's underscore header filter (GHSA-f59h-q822-g45g) only checked
for `_`. PHP folds `.` to `_` when registering $_SERVER keys the same
way CGI/FastCGI folds `-` to `_`, so a dotted alias (e.g. Remote.User)
survived the filter and collided with the legitimate hyphenated
header once it reached a PHP/FastCGI backend, bypassing forward_auth
copy_headers the same way the underscore alias did.
Extends the filter to drop `.` symmetrically, adds an
`expected_dot_headers` allowlist mirroring `expected_underscore_headers`,
and handles header names containing both separators (only an exact
allowlist entry is honored there, since a prefix glob's free-form
suffix can't be vetted for an embedded second separator).
Root cause identified by @iliaal in the FrankenPHP advisory
GHSA-49wc-4hcv-v58q.
Preallocate the sources slice in MultiUpstreams.Provision and the
netAddrs slice in UpstreamResolver.ParseAddresses now that their final
lengths are known, avoiding the intermediate append regrowth
allocations. For a 4-address resolver this reduces ParseAddresses from
11 to 9 allocs/op and 528 to 384 B/op; the benefit scales with the
number of sources/addresses.
| before | after |
| sec/op | sec/op vs base |
UpstreamResolverParseAddresses-8 | 2.289µ |1.946µ ~ (p=0.165) noisy |
B/op: 528 → 384 -27.27% (p=0.000)
allocs/op: 11 → 9 -18.18% (p=0.000)
Adds BenchmarkUpstreamResolverParseAddresses as per policy.
Build the Via request and response headers with strconv-based
concatenation instead of fmt.Sprintf, avoiding fmt's reflection and
formatting overhead on every proxied request/response. Same single
allocation, ~40% faster for that construction in isolation.
| before (fmt) | after (strconv) |
| sec/op | sec/op vs base |
PrepareRequest-8 | 6.136µ ± 11% | 4.491µ ± 33% -26.80% (p=0.007 n=10)|
B/op: 1008 -> 1008 (unchanged)
allocs/op: 12 -> 12 (unchanged)
No behavior change. Adds BenchmarkPrepareRequest, as per policy, exercising the
request preparation path.
* feat(fastcgi): populate SERVER_ADDR by default
Populate the SERVER_ADDR FastCGI environment variable using the local socket address of the incoming connection from the request context. This improves out-of-the-box compatibility for PHP applications that rely on $_SERVER['SERVER_ADDR'] (such as legacy frameworks and custom telemetry packages) when migrating from Nginx or Apache.
Co-developed-by: Gemini AI <renich+gemini@woralelandia.com>
Signed-off-by: Rénich Bon Ćirić <renich@woralelandia.com>
* fix(fastcgi): validate SERVER_ADDR IP format and add test coverage
Validate that SERVER_ADDR is only populated when the local address parses as a valid IP address, preventing Unix socket paths from being assigned if SplitHostPort fails. Add test cases covering IPv4, IPv6, missing context, Unix sockets, and explicit overrides.
Co-developed-by: Gemini AI <renich+gemini@woralelandia.com>
Signed-off-by: Rénich Bon Ćirić <renich@woralelandia.com>
The random_choose selection policy is meant to implement
power-of-d-choices: sample d available upstreams uniformly, then pick
the least-loaded of that sample. The sampling loop, however, was not a
correct reservoir sample (Algorithm R): it never filled the first k
reservoir slots unconditionally, instead writing every candidate to a
random slot j = rand(i+1), which can evict an earlier candidate while
leaving another slot nil. It also derived j from the upstream's index
in the pool rather than from the number of available upstreams seen,
skewing the sample whenever unavailable upstreams precede available
ones.
As a result the reservoir frequently held fewer than min(k, available)
upstreams, so the least-load comparison often never happened. Most
notably, with two upstreams and 'random_choose 2' (the canonical
power-of-two-choices setup), half of all requests were routed to the
more-loaded upstream even when it was saturated and the other idle --
identical behavior to plain 'random'. The nil slots this leaves behind
were also the cause of the panic reported in #3810, which was patched
by skipping nils in leastRequests rather than by fixing the sampling.
Replace the loop with a standard Algorithm R reservoir sample over the
available upstreams: fill the first k slots, then replace a random slot
with probability k/n. Every available upstream is now sampled uniformly
and the reservoir always holds min(k, available) candidates.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* reverseproxy: validate on weighted_round_robin policy
Validate that weighted_round_robin has a non-zero total weight.
This prevents configurations such as:
weighted_round_robin 0 0
from being accepted and causing a divide-by-zero panic during request handling.
* test: validation test on zero weight upstreams.
* test: provision called instead of totalweight setting
* reverseproxy: validate on negative upstream weights
* test: regression test on weighted_round_robin selection policy
* reverseproxy: replace placeholders specified for sni while using http3
* add test for placeholder
* reverseproxy: replace placeholders specified for sni while using http3
* add test for placeholder
* reverseproxy: test HTTP/3 SNI host placeholder
---------
Co-authored-by: Zen Dodd <mail@steadytao.com>
* feat: drop headers with underscore in their names
* feat: Caddyfile binding and tests for underscore-in-header drop
Add the `allow_underscore_in_headers` global server option, refine the
doc comment, and cover the filter end-to-end: server-level unit tests
(drop, opt-out, debug log, RFC-7230 space rejection), a fastcgi unit
test for the trimmed header name replacer, and forward_auth integration
tests for both the default-drop and opt-out paths.
* remove allow_underscore_in_headers option for now
Both fallbacks in splitPos relied on golang.org/x/text/search with
search.IgnoreCase, which performs Unicode equivalence matching far beyond
ASCII case folding. Combined with the validated-ASCII guarantee on every
SplitPath entry, that fallback turned non-PHP filenames into PHP scripts:
- when the inner loop hit a non-ASCII byte and the IndexString fallback
returned -1, the loop broke without resetting match=false, so a stale
match=true caused a non-existent .php to be reported (PoC:
"/name.<U+00A1>.txt").
- search.IgnoreCase folded fullwidth, mathematical and circled letters
onto ASCII, so "/shell.<math sans-serif php>",
"/shell.<fullwidth p>hp", "/shell.<circled php>" were all detected as
".php" files.
Replace the fallback with strict byte-level ASCII case-insensitive
matching: any byte >= utf8.RuneSelf in the path can never be part of a
match, since SplitPath entries are validated ASCII-only and lower-cased
in Provision(). This keeps the hot path branch-light and removes the
x/text/search dependency from the main module.
Reported against FrankenPHP as GHSA-3g8v-8r37-cgjm and
GHSA-v4h7-cj44-8fc8. The vulnerable function in this module was adapted
from the same FrankenPHP code.
* reverseproxy: Add ability to clear dynamic upstreams cache during retries
This is an optional interface for dynamic upstream modules to implement if they cache results.
TODO: More documentation; this is an experiment.
* Add some godoc
* Export interface; update godoc
* admin: Redact sensitive request headers in API logs
* Fix govulncheck and typed atomic lint failures
* Sync Go module metadata after dependency downgrade
* add 'root' key to Helper.State for access in frankenphp's `php_server` directive
* clone state before passing it to child directives, but keep sharing it among sibling directives
* propagate named route state from children to parent
* use BlockState to set "root" instead
* gofmt -w .
* go fmt ./...
* here we go
When using copy_headers in a forward_auth block, client-supplied headers with
the same names were not being removed before being forwarded to the backend.
This happens because PR #6608 added a MatchNot guard that skips the Set
operation when the auth service does not return a given header. That guard
prevents setting headers to empty strings, which is the correct behavior,
but it also means a client can send X-User-Id: admin in their request and
if the auth service validates the token without returning X-User-Id, Caddy
skips the Set and the client value passes through unchanged to the backend.
The fix adds an unconditional delete route for each copy_headers entry,
placed just before the existing conditional set route. The delete always runs
regardless of what the auth service returns. The conditional set still only
runs when the auth service provides that header.
The end result is:
- Client-supplied headers are always removed
- When the auth service returns the header, the backend gets that value
- When the auth service does not return the header, the backend sees nothing
Existing behavior is unchanged for any deployment where the auth service
returns all of the configured copy_headers entries.
Fixes GHSA-7r4p-vjf4-gxv4
This refactors the initial approach in PR #7281, replacing the UsagePool
with a dedicated package-level sync.Map and atomic.Int64 to track
in-flight requests without global lock contention.
It also introduces a lookup map in the admin API to fix a potential
O(n^2) iteration over upstreams, ensuring that draining upstreams
are correctly exposed across config reloads without leaking memory.
Co-authored-by: Y.Horie <u5.horie@gmail.com>
reverseproxy: optimize in-flight tracking and admin API
- Replaced sync.RWMutex with sync.Map and atomic.Int64 to avoid lock contention under high RPS.
- Introduced a lookup map in the admin API to fix a potential O(n^2) iteration over upstreams.
When a request arrives via a Unix domain socket (RemoteAddr == "@"),
net.SplitHostPort fails, causing addForwardedHeaders to strip all
X-Forwarded-* headers even when the connection is trusted via
trusted_proxies_unix.
Handle Unix socket connections before parsing RemoteAddr: if untrusted,
strip headers for security; if trusted, let clientIP remain empty (no
peer IP for a Unix socket hop) and fall through to the shared header
logic, preserving the existing XFF chain without appending a spurious
entry.
Amp-Thread-ID: https://ampcode.com/threads/T-019c4225-a0ad-7283-ac56-e2c01eae1103
Co-authored-by: Amp <amp@ampcode.com>
Islam Elsayed