* httpcaddyfile: new tls_automate_names global option
Provisioning a certificate for a name that is not served requires giving
it a site block of its own:
*.example.com {
}
foo.example.com {
respond "Real site"
}
That asks the tls app for the wildcard, but it also adds a route to the
http app, so every name pointed at the server that has no site block of
its own -- bar.example.com here -- gets an empty but valid response
rather than no match at all. Wanting a certificate and wanting to serve
a name are separate things, and the Caddyfile had no way to say only the
first.
Name them in the new global option instead:
{
tls_automate_names *.example.com
}
foo.example.com {
respond "Real site"
}
The names are added to the automate certificate loader and to an
automation policy built from the global options, so a name listed here
is managed exactly as it would be from a site block, with the same
issuers; the only difference in the adapted config is that no route is
added for it. Repeating the option appends rather than replaces, so a
long list can be split over several lines.
Names that cannot get a public certificate are given the internal
issuer, the same treatment a site block gives them. Names that already
appear in the automate list are skipped, but a name that also has a site
block is left listed here as well: a site block for http:// only is not
managed by auto-HTTPS, so dropping the name because a block exists could
silently leave it without a certificate.
The option needs no http app at all, so a config consisting only of
global options now adapts to a tls app on its own -- enough to keep
certificates renewed for a mail or XMPP server, or for names served by a
layer 4 app.
Closes#7122
* httpcaddyfile: reuse an existing policy for an automated name
A name given to tls_automate_names may already have an automation policy
from its own site block. Adding a second policy for the same subject is
not just redundant: the adapter rejects overlapping subjects, so
{
email nobody@example.com
tls_automate_names foo.example.com
}
foo.example.com {
tls {
ca https://acme.example.test/directory
}
}
failed to adapt at all, with "hostname appears in more than one
automation policy, making certificate management ambiguous".
Keep such a name in the automate loader but leave its policy alone. The
site block's policy is the more specific of the two, and the loader entry
is still wanted, since a site block served only over HTTP is not managed
by auto-HTTPS.
* Pin that tls_automate_names overrides auto_https off
The names given to the option are managed whether or not auto-HTTPS is
disabled: unlike the hostless-key block above it, that code is not gated
on auto_https, because naming a subject explicitly is a stronger signal
than the general switch. Nothing enforced it, so add the adaptation
fixture @steadytao asked for.
Provision previously only checked for a single opening brace at the
start of a destination, so values like {result}suffix, {result, and
{result}} passed validation. strings.Trim(dest, "{}") then silently
stored a wrong placeholder name (e.g. result}suffix), making the
intended value unavailable.
Require the destination to be exactly one placeholder: a single
opening brace at the start, a single closing brace at the end, and a
non-empty name.
Fixes#8073
The basic auth provider expanded account usernames and passwords with
repl.ReplaceAll, which blanks any {...} the replacer does not recognize
Credentials are config data, so a username like alice-{bogus} was stored
as alice- and a password like ab{cd}ef was quietly rewritten into
different valid base64, turning a config mistake into a working
password the admin never set
Use ReplaceKnown, matching the respond headers fix in #8014 and the
header handler fix in #4880, so unknown braces survive and real
placeholders like {env.*} still expand
* internal: fix MaxSizeSubjectsListForLog off-by-one when maxToDisplay is 0
The loop appended before checking the length, so with maxToDisplay=0
one domain leaked into the output before the break check. Reorder so
the length check comes first.
Signed-off-by: Mohammed Al Sahaf <msaa1990@gmail.com>
* Update internal/logs.go
Co-authored-by: Zen Dodd <mail@steadytao.com>
Signed-off-by: Mohammed Al Sahaf <mohammed@caffeinatedwonders.com>
* fix doc
Signed-off-by: Mohammed Al Sahaf <msaa1990@gmail.com>
---------
Signed-off-by: Mohammed Al Sahaf <msaa1990@gmail.com>
Signed-off-by: Mohammed Al Sahaf <mohammed@caffeinatedwonders.com>
Co-authored-by: Zen Dodd <mail@steadytao.com>
The set body is user-supplied and commonly JSON, but it was expanded with
ReplaceAll, which blanks any {...} the replacer does not recognize, so a
plain JSON body was sent empty and a body mixing JSON with a real
placeholder was mangled
Use ReplaceKnown, matching the respond body, so unrecognized braces are
left intact and real placeholders still expand. Regression test pins both
directions
Co-authored-by: hktitof <hktitof@users.noreply.github.com>
* caddyhttp: fix randString sameCase dictionary to match its docs
The doc comment says randString excludes confusing characters like
I, l, 1, 0, O. When sameCase is true, uppercase letters and the
characters l and o should be excluded. But the sameCase dictionary
still contained '0'. Drop it.
Signed-off-by: Mohammed Al Sahaf <msaa1990@gmail.com>
* Update modules/caddyhttp/errors.go
Signed-off-by: Matt Holt <mholt@users.noreply.github.com>
* elaborate the doc, and expand tests
Signed-off-by: Mohammed Al Sahaf <msaa1990@gmail.com>
* Update modules/caddyhttp/errors_randstring_test.go
Co-authored-by: Zen Dodd <mail@steadytao.com>
Signed-off-by: Mohammed Al Sahaf <mohammed@caffeinatedwonders.com>
---------
Signed-off-by: Mohammed Al Sahaf <msaa1990@gmail.com>
Signed-off-by: Matt Holt <mholt@users.noreply.github.com>
Signed-off-by: Mohammed Al Sahaf <mohammed@caffeinatedwonders.com>
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
Co-authored-by: Zen Dodd <mail@steadytao.com>
when a request exceeds the request_body max_size limit, the request_body handler wraps the http.MaxBytesError into a caddyhttp.HandlerError carrying status 413, but the {http.request.body} and {http.request.body_base64} placeholders ran io.Copy with the error ignored, so they silently returned the truncated prefix as though it were the complete body. docs promise a 413 for reads past max_size, and silently truncating is a bad failure mode in templates and vars_regexp where the body value drives decisions. see #7691 and the narrow follow-up prescribed when #7692 was closed
reading the body now returns a dedicated RequestBodyLimitError marker instead of a generic HandlerError, and only when the read failure is actually the max_size limit. the consumers (template placeholder function, vars and vars_regexp matchers) recognize exactly that marker and wrap it in a status carrying handler error so the oversized request fails with 413, while every unrelated error value keeps its old behavior: templates still render it as text and the vars matchers still match on its error text. the surfaced error is stripped of its generated id and stack trace so a placeholder stringified into a response body cannot leak the call stack
negative regressions prove an unrelated HandlerError in templates, vars and vars_regexp does not start controlling request handling, plus integration tests for the template and vars_regexp 413 paths
ai assisted (GLM agent) under Abdel's direction and local verification
Signed-off-by: Abdel <hktitof@gmail.com>
* Move websocket header normalization later
A regression since #6621, these headers can be rewritten by cloneHeaders
and by .Add from user and transport ops. Moving this closer to where
it's round-tripped feels like the nicest solution.
I've put the harness I used to reproduce the issue at
https://codeberg.org/nfreya/caddy-websocket-headers-repro
* Integration test for normalized websocket headers
---------
Co-authored-by: Zen Dodd <mail@steadytao.com>
Co-authored-by: Matt Holt <mholt@users.noreply.github.com>
A request whose body has no length of its own — Transfer-Encoding: chunked, or
HTTP/2 and HTTP/3 requests sent without the header — cannot be forwarded over
FastCGI until it has been buffered in full, because CGI/1.1 requires
CONTENT_LENGTH and php-fpm hangs when it is absent or wrong and the body is not
empty. When request_buffers is too small to hold the whole body, the length
stays unknown and the request is refused with 411.
The refusal said none of that. RoundTrip passes r.ContentLength to Post, which
is -1 for such a request, so the operator got a bare "411 Length Required" with
either no error at all or strconv's "invalid syntax" on a value they never
wrote. On #7386 that sent two people looking for the fault in their backend.
The 411 now carries the reason and the remedy, and the unknown-length case is
told apart from a genuinely malformed value. ParseUint becomes ParseInt plus an
explicit negative check, which is strictly tighter: values above MaxInt64 used
to be accepted and are unreachable anyway, since CONTENT_LENGTH is always
FormatInt of an int64 by the time Do sees it.
No status code changes. Every request that was refused before is still refused;
raising request_buffers is still what makes a large chunked body work.
Tests: the reachable path through Post, the unusable CONTENT_LENGTH values Do
itself guards against, an integration test driving a chunked body through a
fastcgi reverse_proxy over a unix socket at each side of the buffer boundary
(including request_buffers -1, which succeeds at any size and is the remedy),
and a boundary test recording that a body exactly the size of the buffer counts
as partial.
That last one is deliberately not changed here. Telling "exactly the limit"
apart from "more to come" costs a read that a paused stream may never answer,
and bufferedBody also backs response_buffers: measured against a body that
delivers exactly the limit and stops, the current code hands the buffered
prefix on immediately, while peeking one byte first delivers nothing at all.
Co-authored-by: Aditya <205600203+Rohilalala@users.noreply.github.com>
Co-authored-by: Zen Dodd <mail@steadytao.com>
* reverseproxy: build TLS client config once so HTTP/3 keeps tls_trust_pool
reuse the config built for the HTTP/1.1+2 transport and give the HTTP/3
transport a clone of it to honour the tls_trust_pool
Signed-off-by: Timo Stark <tippexs91@googlemail.com>
Co-authored-by: Faiyaz Rahman <faiyazrahman03@gmail.com>
* Adding issue reference
---------
Signed-off-by: Timo Stark <tippexs91@googlemail.com>
Co-authored-by: Faiyaz Rahman <faiyazrahman03@gmail.com>
* reverseproxy: propagate TCP half-close on upgraded streams
handleUpgradeResponse starts one copy goroutine per direction but returns as
soon as the first one reports a result, and returning runs two unconditional
deferred closes (the client connection and the backend connection). A client
that ends its send direction with a TCP half-close therefore causes the whole
tunnel to be torn down: the backend connection is reset mid-conversation and
whatever it was about to send is never delivered.
Treat a clean EOF as the end of one direction rather than the end of the
tunnel. When a copy finishes without error, propagate the half-close to the
destination with CloseWrite where the connection supports it, and keep waiting
for the other direction so pending bytes can still drain. Real copy errors, the
stream timeout and request cancellation/shutdown still tear down immediately.
If the destination does not support CloseWrite, the peer cannot be told that
the direction ended, so the previous behavior is kept and the tunnel is torn
down rather than held open until the stream timeout.
This is the same class that was fixed upstream in net/http/httputil
(https://go.dev/issue/35892); Caddy has its own upgraded-stream copier, so the
standard library fix does not apply here.
Fixes#8026
* reverseproxy: follow net/http/httputil shape for the half-close
Replace the copyResult struct and the closeWriter/closeWrite helpers with
upstream's plain error channel and errCopyDone sentinel, propagating
CloseWrite inside the copier, as requested in review.
Behaviour is unchanged. The wait is repeated twice inside the existing
select because handleUpgradeResponse also selects on the stream timeout
and sizes the channel at 2 so both goroutines can exit when it fires
(#7418), which net/http/httputil has no equivalent of.
* reverseproxy: explain the repeated wait next to it
Move the reasoning for why the wait is repeated inside the select, rather
than written as upstream's two-line form, into a comment beside the loop
so a future reader does not have to find the review discussion.
* reverseproxy: add an upgraded-stream half-close integration test
Drive a real upgraded stream through the handler and check that closing one
direction is propagated to the other side instead of tearing the tunnel down.
Mirrors net/http/httputil's TestReverseProxyWebSocketHalfTCP, which covers the
same class upstream (https://go.dev/issue/35892): a backend that hijacks and
hands back a *net.TCPConn, the handler in front of it, and a client that dials
it directly, then the four close-read / close-write combinations.
Against the unfixed handler the two half-close cases fail - the surviving
direction never delivers its pending bytes - while the two close-read cases
pass, so the test isolates exactly the reported behaviour.
* reverseproxy: give the half-close test reads a deadline
Without one, a regression blocks in ReadFull until the package timeout
instead of failing the test, which is the flakiness an end-to-end test of
this kind is usually accused of. Ten seconds is far above any real latency
here and turns a hang into a clear failure.
* reverseproxy: hijack the test backend via http.NewResponseController
Matches how the rest of the tree reaches Hijack (responsewriter.go,
caddyauth.go, streaming.go itself) and keeps working if the writer is
wrapped, where a direct http.Hijacker assertion would not.
The respond directive expanded header field names and values with
repl.ReplaceAll, which blanks any {...} the replacer does not recognize
Header values are config data, often JSON or carrying literal braces, so a
value like {key:value} was sent empty and a-{b}-c came out a--c
Use ReplaceKnown, matching the header handler fix in #4880 (same class as
#4418) and the respond body expansion a few lines below, so unknown braces
survive and real placeholders still expand
Opening the bbolt database only reported timeout, which sounds like a
broken config. Catch bolt.ErrTimeout with errors.Is and say another
process holds the lock, and that caddy reload is the right command when
Caddy is already running. The error stays fatal.
The active health check body is user-supplied and commonly JSON, but it
was expanded with ReplaceAll, which blanks any `{...}` the replacer does
not recognize. Only globals are available here, so a JSON
health_request_body was sent as an empty body.
This is not limited to bodies without placeholders. A body that does use
one is mangled too: `{"token":"{env.SOME_VAR}"}` is sent as `"}`, since
the opening `{"token":"` is consumed as an unrecognized placeholder
before the real one is reached.
Use ReplaceKnown, matching the header values a few lines below, so
unrecognized braces are left intact and real placeholders still expand.
Only the body is changed. The Host header on the same path keeps
ReplaceAll deliberately: Go's HTTP server blanks a Host containing
braces, so leaving `{...}` intact there would send a Host the upstream
discards, where blanking it lets Go fall back to the URL host.
App.Emit does a fair amount of work before it can discover that no handler
is bound: it derives three loggers, one of which formats the event's UUID
even when debug logging is off, and registers a replacer callback. Only
then does it reach "shortcut if event not bound at all".
Some events are emitted on every TLS handshake -- CertMagic emits
tls_get_certificate as the first statement of GetCertificateWithContext --
so on a server with no events configuration that work runs per handshake
and is discarded every time.
Return early when neither the event's name nor the catch-all is bound and
debug logging is off, which are exactly the conditions under which nothing
can observe the event. caddy.NewEvent still runs, so the returned Event is
unchanged for callers.
Benchmarks are included; measurements are in the pull request.
* caddytls: synchronize storage cleaner with TLS.Stop via context and WaitGroup
* Unify the storage cleanup synchronization to both tls and ech.
* Cannot embed the sync.WaitGroup directly as the TLS struct is copied.
* caddytls: propagate cancellable context to ECH rotation and add sync tests
Pass cancellable context to ECH key rotation so in-flight storage locks
and operations unblock when TLS.Stop is invoked. Add comprehensive tests
verifying TLS.Stop cleanly unblocks and waits for storage cleaner and
ECH workers.
---------
Co-authored-by: Zen Dodd <mail@steadytao.com>
The admin host check compared r.Host against the allowed URL's Host
with byte-for-byte equality. url.Parse does not normalize host case,
so an allowed 'http://Example.com:2019' rejected a client Host of
'example.com:2019' with 'host not allowed', and the same happened for
an uppercase variant of a lowercase entry like localhost.
The Origin check got this treatment in 7973 already, the DNS rebinding
Host check right next to it did not. Both read the same allowedOrigins
list, so this applies the same strings.EqualFold treatment there.
Regression test covers both fold directions, the default localhost
entry and the negative case that must keep failing
- willCycle now reports a cycle when from == to, so addEdge rejects
self-loops (a self-importing file was previously accepted).
- removeNode now drops the removed node's outgoing edges and any
incoming edges pointing at it, keeping the adjacency map consistent.
Signed-off-by: Mohammed Al Sahaf <msaa1990@gmail.com>
The admin origin allow-list compared origin.Host against the allowed
URL's Host with byte-for-byte equality. url.Parse does not normalize
host case, so an allowed 'http://Example.com:8080' rejected a client
origin of 'http://example.com:8080' even though RFC 3986 §3.2.2 says
host names are case-insensitive. Use strings.EqualFold.
Signed-off-by: Mohammed Al Sahaf <msaa1990@gmail.com>
Per https://go.dev/ref/mod#go-mod-file-ident, '@' is not a valid module
path character. The old implementation used strings.LastIndex to allow
inputs like github.com/@user/module@v1.0.0 to parse, but such inputs
are not valid module paths per the spec. Reject them explicitly.
Signed-off-by: Mohammed Al Sahaf <msaa1990@gmail.com>
A port range is inclusive on both ends, so a range from start to end
spans (end - start + 1) ports. The old (end - start) > maxPortSpan
check let 0-65535 through even though that spans 65536 ports, one
above the 65535 cap. Include the trailing port in the count.
Signed-off-by: Mohammed Al Sahaf <msaa1990@gmail.com>
v1.0.0 renamed SetGoMemLimitWithOpts to Set and dropped the other Set*
helpers. The options this call passes are unchanged, as are the 0.9
default ratio and the AUTOMEMLIMIT handling, so behavior is the same.
Only the already-deprecated AUTOMEMLIMIT_DEBUG and AUTOMEMLIMIT_EXPERIMENT
environment variables are gone, and neither is used here.
StripPathSuffix is documented to behave like StripPathPrefix: the suffix
is matched in normalized (unescaped) space except where the pattern uses
an escape sequence. But suffix stripping was implemented as
reverse(trimPathPrefix(reverse(escapedPath), reverse(suffix)))
Reversing the strings moves the '%' to the *end* of each "%xx" escape,
which defeats trimPathPrefix's escape detection (it expects '%' to
precede the two hex digits). As a result the escape-aware, normalized
comparison never happened for suffixes: a decoded pattern failed to
match a percent-encoded path.
Concretely, StripPathPrefix "/a/b/c" strips "/a%2Fb/c/d" to "/d", but the
mirror StripPathSuffix "/b/c" left "/a/b%2Fc" untouched instead of
producing "/a"; likewise StripPathSuffix "bc" did not strip "/a%62c".
This has been the behavior since #4948, which introduced both the
escape-aware trimPathPrefix and the reverse-based suffix trimming.
Replace the reverse trick with a dedicated trimPathSuffix that iterates
from the ends of both strings and applies the same escape-aware,
case-insensitive comparison as trimPathPrefix. An escape in the pattern
itself is still compared literally, so "%2fsuffix" continues to require
the path to contain that exact escape.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* admin: normalize request path in remote admin access-control check
Co-authored-by: atlarix-agent <agent@atlarix.dev>
* admin: fix empty allowedPath regression and dead code in path normalization
path.Clean("") returns ".", so cleaning allowedPath unconditionally
silently broke the allow-all behavior when Paths: [""] is configured.
Short-circuit the empty case before cleaning to preserve that behavior.
Also remove the dead strings.HasSuffix(allowedPath, "/") branch —
after path.Clean the path never has a trailing slash, so the unified
reqPath == allowedPath || HasPrefix(reqPath, allowedPath+"/") form
covers exact match, subpath boundary, and trailing-slash requests.
Co-authored-by: atlarix-agent <agent@atlarix.dev>
Co-authored-by: iabdullah215 <muhammadabdullah8040@gmail.com>
* admin: validate non-canonical configured paths at provisioning
path.Clean(allowedPath) silently broadens misconfigured values like
// or /.. into /, which grants unintended access to all endpoints.
Reject non-canonical paths during provisioning in
replaceRemoteAdminServer so misconfigurations fail fast with a
clear error. The path.Clean in adminPathAllowed remains as
defense-in-depth but is now a safe no-op on validated inputs.
Co-authored-by: atlarix-agent <agent@atlarix.dev>
* admin: validate non-canonical configured paths at provisioning
path.Clean(allowedPath) silently broadens misconfigured values like
// or /.. into /, which grants unintended access to all endpoints.
Reject non-canonical paths during provisioning in
replaceRemoteAdminServer so misconfigurations fail fast with a
clear error. The path.Clean in adminPathAllowed remains as
defense-in-depth but is now a safe no-op on validated inputs.
Co-authored-by: atlarix-agent <agent@atlarix.dev>
* admin: fix TrimRight → TrimSuffix in provisioning path validation
strings.TrimRight strips all trailing slashes, so a configured path
like /foo// passed validation (both slashes trimmed to /foo matching
path.Clean output) but was silently broadened to /foo at runtime.
Use strings.TrimSuffix instead, which removes exactly one trailing
slash — the only form the exemption was meant to allow (users write
/pki/ca/prod/ meaning the /pki/ca/prod scope).
Also update the // test case: with TrimSuffix, // is just / + one
trailing slash, which is valid under the exemption. Add a new test
for /foo// (double trailing slashes → wantErr: true).
Co-authored-by: atlarix-agent <agent@atlarix.dev>
* admin: reject non-canonical root permission path
* admin: preserve trailing-slash permission semantics
---------
Co-authored-by: atlarix-agent <agent@atlarix.dev>
Co-authored-by: iabdullah215 <muhammadabdullah8040@gmail.com>
Co-authored-by: Zen Dodd <mail@steadytao.com>
* rewrite: don't drop a trailing '=' from the query string
buildQueryString scanned for '=' unconditionally when looking for the
end of a component, but '=' only delimits a key from its value once;
any further '=' bytes are literal data. When the query ended with '=',
that byte was consumed as a delimiter with nothing following it to
re-emit, so it was silently lost:
?x=1&sig=YWJjZA== => ?x=1&sig=YWJjZA=
This corrupts base64 padding in the last query parameter, which is the
shape of an S3 presigned URL (X-Amz-Signature), turning a valid
signature into a 403. Only the final '=' of the query was affected;
?sig=YWJjZA==&x=1 came through intact.
Disable the '=' search while consuming a value so that only '&' ends it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* rewrite: honor a query string injected by a replacement value
Which URI components get written back was decided from the literal
config string, before placeholders were expanded, but an injected query
is only detected after expansion. The two were never reconciled: for
`rewrite * {rp.header.X-Accel-Redirect}` there is no literal '?', so
qsStart stayed -1, and the correctly-built query string was computed and
then discarded by the `if qsStart >= 0` guard.
Only half the split was applied. The path was still truncated at the
injected '?', so the query was not preserved either -- it was dropped,
and any query already on the request survived in its place:
GET /orig?keep=me, X-Accel-Redirect: /hello?some=param
=> /hello?keep=me
Track whether a query was actually injected and include that in the
write-back condition. Appending a literal '?' to the rewrite value was
the known workaround precisely because it set qsStart; that keeps
working and is now unnecessary.
The flag is only set where the injected query is adopted, so an
explicitly configured query still wins, and a value with no '?' still
leaves the query untouched -- which is what the implicit rewrites of
try_files and php_fastcgi rely on. Those stay safe regardless, since
escapePathPlaceholders already escapes the two placeholders they use, so
a client-supplied %3F cannot split the URI.
Fixes#5208
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* rewrite: drop a fragment injected by a replacement value
The scan that separates path, query and fragment runs on the literal
config string, so a '#' arriving later via a replacement value was never
treated as a delimiter. It leaked into whichever component it landed in:
X-Accel-Redirect: /hello?p=x#frag => RawQuery = "p=x#frag"
Everything after '#' is fragment (RFC 3986 section 4.2) and a fragment is
never sent to the server, so drop it before the path is split, mirroring
how the scan already handles a literal '#'. An escaped %23 is unaffected,
so a real '#' in a path or query is still expressible, and a configured
fragment still wins over an injected one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Zen Dodd