mirror of
https://github.com/caddyserver/caddy.git
synced 2026-10-31 08:21:39 -04:00
4f45ded755
* docs: add Caddyfile lexer/parser/formatter unification design spec Design for unifying the Caddyfile lexer/parser and the standalone formatter by making the token pipeline drive formatting: raw-source-span capture, comment tokens, separator-kind metadata, follow-imports and (default-off) braced-wrap modes. Includes adversarial-review hardening and empirically-found intentional improvements over the legacy formatter. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: add phased implementation plan for Caddyfile formatter unification Task-by-task TDD plan: Phase 1 (token model + formatter rewrite + fuzz invariants + legacy oracle), Phase 2 (follow-imports mode + caddy fmt --imports), Phase 3 (default-off braced-wrap). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: vendor legacy Caddyfile formatter as differential oracle Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * caddyfile: add format-mode Token fields and accessors Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * caddyfile: add Lex and LexOptions; Tokenize delegates to Lex Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * caddyfile: capture verbatim raw source spans in format mode Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * caddyfile: emit comment tokens in format mode Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * caddyfile: record token separator kind in format mode Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * caddyfile: split structural braces in format mode Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * caddyfile: rewrite Format as a token-based renderer (core layout) Replace the rune-by-rune state machine with a renderer that consumes the format-mode token stream (Lex with Comments+Raw). Adds FormatWithOptions and FormatOptions{WrapUnbracedSite}; Format keeps its signature. Comment placement, continuations, empty-block expansion, and the import/}-glue rules land in Tasks 9-11; the six dependent TestFormatter cases are temporarily skipped with annotations. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * caddyfile: token renderer comments (I1/I3) and continuations (D1) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * caddyfile: token renderer empty blocks (I2) and post-} break (I4) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * caddyfile: import-brace exception, drop nesting cap, drop < quirk Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * caddyfile: fix formatter idempotency/semantic bugs, retire legacy oracle Fix two fuzz-found bugs plus several related idempotency issues in the token-based Caddyfile formatter, and retire the legacy byte-scanner formatter used as a test oracle. Bug B (formatter.go): a stray "}" at nesting 0 was glued to an unquoted predecessor because precededBySpace is unreliable after an unquoted token. Only suppress the separating space for an inline close brace when the previous token is quoted (same guard as comment gluing), so "0 }" no longer merges into "0}". Bug A (lexer.go): an empty-bodied heredoc hitting EOF with no closing marker was silently dropped, breaking Format idempotency. Surface it as an "incomplete heredoc" error ONLY in format mode (opts.Comments || opts.Raw); Tokenize/Parse keep the exact prior silent-drop behavior. Format then falls back to preserving the trimmed input. Additional format-mode-only fixes surfaced by property fuzzing: correct line-break accounting for empty-bodied heredocs and continuation-framed quoted tokens; normalize the raw slice / clear continuation on split and standalone structural braces; guard the address/comment/brace fold so a next-line "{" is not folded onto a comment trailing an opening brace; and a fallback + idempotency backstop in FormatWithOptions for degenerate tokens (ambiguous heredoc openers, unterminated/escaped quotes, dangling escapes, messy/leading line continuations). The parse path is unchanged. Property fuzz targets (formatter_test.go): keep FuzzFormatIdempotent, FuzzFormatNoPanic, and a redesigned FuzzFormatSemanticPreserve that asserts format-mode token-text preservation (robust to the parser's lenient grouping of ambiguous brace layouts). Add regression tests TestFormatStrayCloseBraceKeepsTokens and TestFormatIncompleteHeredocIdempotent. Remove FuzzFormatParity, containsParityExclusion, maxNesting and delete formatter_legacy_test.go (legacyFormat oracle + TestLegacyOracleMatchesCurrent). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * caddyfile: seed corpus and FormattingDifference stability tests Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * caddyfile: extract pure import-glob resolution helper Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * caddyfile: import discovery for follow-imports formatting Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * caddyfile: FormatImports formats root plus imported files Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * cmd: add 'caddy fmt --imports' to format imported files Adds a --imports flag to 'caddy fmt' that calls caddyfile.FormatImports to format the root Caddyfile and all files it references via import directives. Rejects --imports with stdin (no source directory to resolve against). With --overwrite each file is written back; otherwise each file's output is printed to stdout with a '# <path>' header, with optional per-file diff via --diff. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * caddyfile: detect single unbraced site block Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * caddyfile: implement default-off unbraced->braced site wrapping Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * caddyfile: make WrapUnbracedSite decision on normalized stream (idempotency fix) FormatWithOptions wrapped before formatting, deciding on the raw token stream. isSingleUnbracedSite uses source-line gaps, but formatTokens removes blank lines and folds dangling braces, so a shape read as "not a single site" on pass 1 could read as a single unbraced site on pass 2 and get wrapped, breaking Format(Format(x)) == Format(x). Decide the wrap on the already-normalized stream instead: normalize, re-lex, and wrap only if isSingleUnbracedSite holds there. tokens becomes the wrapped stream so the backstop still compares like-for-like; once wrapped the output is braced, so no double-wrap on re-runs. Format (zero options) still never wraps. Adds a regression reproducer and FuzzWrapUnbracedSiteIdempotent. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * caddyfile: satisfy golangci-lint (rangeint + De Morgan) Modernize three index loops to range-over-int and simplify a negated boolean per staticcheck QF1001. No behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * caddyfile: modernize remaining index loops (rangeint) Convert the remaining for-i index loops to range-over-int per golangci-lint modernize. No behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * caddyfile: fix formatter and import correctness issues (#7881) * caddyfile: fix formatter and import correctness issues * noot * noot * caddyfile: document parser-driven import discovery * test: make formatter import tests portable (#7882) * caddyfile: propagate import observer into named route blocks Named route blocks expand their imports through a separate parser built in expandImportsInBlock, which copied only definedSnippets and importGraph. The importObserver hook was left nil, so FormatImports never saw files imported from inside a &(name) { ... } block and "caddy fmt --imports" silently skipped them. Carry the observer into that parser and cover it with a discovery test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * caddyfile: detect end-of-input-swallowing tokens from token shape hasUnformattableToken documented an unterminated quote/backtick as a shape that cannot be rendered idempotently, but never implemented the check. The condition was only caught dynamically by trailingNewlineChangesTokens, which returns early for input that already ends in a newline. That early return is wrong for a token that swallowed the newline: once Format's mandatory trailing newline sits inside an unterminated quote, appending another one no longer changes the stream, so the second pass rendered what the first pass had preserved verbatim. Format(`"""`) => `"""` (fallback, preserved) Format(`"""` + newline) => `"" "` (rendered, space inserted) Decide it from the token instead: a non-quoted token whose verbatim source ends in a newline swallowed it, since a newline otherwise terminates a token. This also covers an escaped quote ("\"") running to end-of-input, which has the same shape. A lone "\r" is kept in raw but does not terminate a token, so it is deliberately excluded; heredoc tokens carry wasQuoted == '<'. Master's rune-based formatter was idempotent on these inputs, so this restores parity. Found by FuzzFormatIdempotent; the inputs are added as portable seeds and table cases rather than testdata/fuzz files. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * caddyfile: stop Format from mutating its input buffer The three fallback paths returned append(bytes.TrimSpace(input), '\n'). TrimSpace returns a subslice sharing the caller's backing array, so the append wrote the newline into the caller's buffer. os.ReadFile returns a slice with one byte of spare capacity, which is exactly what the append needs, so this reached real callers: $ printf 'foo "unterminated \n\n' > Caddyfile $ caddy fmt --diff foo "unterminated <- reported unchanged; its trailing - whitespace was in fact stripped cmd/commandfuncs.go compares input against output after calling Format, and by then input had been overwritten to match. The same aliasing corrupted FuzzFormatIdempotent, which reported failures whose recorded inputs did not reproduce. Copy into a fresh buffer instead, and assert both non-mutation and non-aliasing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * caddyfile: preserve repeated import expansions during formatting Deduplicate formatted files without skipping imports with different arguments, block substitutions, or lexical aliases. Track active physical-file ancestry to prevent symlink cycles, including snippet and block expansion contexts. Add regression coverage and align the upstream glued-brace expectation with the token-based formatter's literal-token behavior. Validation: go test -race -short ./..., go build ./..., and golangci-lint run --timeout 10m all pass. AI assistance: OpenAI Codex (GPT-6) assisted with implementation and testing. * docs: remove formatter implementation plan and design spec * caddyfile: require whitespace before line continuations Preserve attached trailing backslashes as literal text, including Windows root paths. Require a space or tab before an unquoted continuation backslash, and preserve significant leading whitespace in formatter fallback output. Add lexer, formatter, CRLF, adapter, and fuzz regression coverage. Validated with race tests, build, lint, formatter fuzzing, and the 165-example forum corpus. AI assistance: OpenAI Codex (GPT-6) assisted with implementation, regression tests, and validation. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: a <a@tuxpa.in>