# Hot-code orderfile

`lightpanda.ld` is an LLD linker script that gathers the functions and
read-only data touched by a typical session into two contiguous output
sections, `.text.hot` and `.rodata.hot`, placed in front of the cold `.text`
and `.rodata`. It only affects layout; no code changes.

## Why

The kernel maps file-backed pages in 64KB fault-around windows
(`/sys/kernel/debug/fault_around_bytes`). With the default link order the hot
code is scattered across the whole 40MB of text, so every window that holds a
single executed byte becomes resident: ~17MB of `.text` and ~4MB of `.rodata`
for a hot set that is really 7.5MB + 2MB. Packing the hot set contiguously
takes the CDP bench (`demo/puppeteer/cdp.js`, 100 runs) from VmHWM ~28.2MB to
~25.3MB with the stock V8 archive and ~21.2MB once V8 is addressable too (see
below), with no change in run duration.

## How it is applied

- Opt-in via `-Dorderfile=orderfile/lightpanda.ld`; CI passes it for the Linux
  release artifact and the e2e bench build. Local release builds don't, so
  they keep the sub-second link. Linux/LLD only (it is an ELF linker script).
- The exe and every C library it links are built with
  `-ffunction-sections -fdata-sections` so each function/datum has its own
  input section for the script to address.
- Patterns that match nothing are ignored by LLD, so a stale script degrades
  gracefully: renamed or removed functions simply fall back into cold `.text`.
  Zig's `__anon_NNN` names (~7% of the patterns) renumber on unrelated
  changes, so the profile decays a little with every commit; it is
  regenerated weekly (below).
- Link-time cost: LLD name-checks every input section against every pattern
  of every input-section description, so the script scopes patterns to their
  object file (`*api.o(...)`). The shipped script covers Zig/Rust/C only
  (~4k patterns, ~2s of link). Once V8's functions are addressable (below) the
  script grows to ~26k patterns and the release link to ~26s; one description
  per pattern instead of per file would make that 85s (the line breaks inside
  a description are free, they exist for the diff). Debug builds pay nothing.
- The prebuilt V8 archive is compiled with `-fno-unique-section-names`
  (Chromium hardcodes it), so all of V8's function sections are named `.text`
  and the script cannot address them. `mark_hot_sections.zig` (run
  automatically by `build.zig` when an orderfile is set) rewrites the archive,
  renaming just the *hot* sections to `.text.hot.<symbol>` /
  `.rodata.hot.<symbol>` so the script gathers all of V8 with one
  `.text.hot.*` glob (which keeps the link at ~2s instead of ~26s). The
  browser's own hot Zig/Rust/C sections are matched by explicit per-file
  patterns. The rewritten archive is a build cache artifact (~132MB -> ~158MB;
  LLVM objects share .shstrtab with .strtab, so each touched member grows);
  the linked binary is byte-for-byte unaffected.
- V8's embedded builtins blob (the `Builtins_*` symbols, one ~2MB `.text`
  section) is deliberately left cold. Pulling it into `.text.hot` shifts the
  layout so that V8's runtime code range (a separate mmap) no longer reaches
  the blob with a pc-relative call, and V8 then copies the whole 2MB blob into
  an executable anonymous mapping at startup (+~1.8MB RSS, deterministic).
  `mark_hot_sections.zig` and `gen_order.py` both skip `Builtins_*`.

## Regenerating the profile

```bash
# root for /sys/kernel/debug/fault_around_bytes; ../demo checked out with
# `npm install` done; node, go, python3 and binutils on the PATH.
orderfile/tools/regen.sh -Doptimize=ReleaseFast -Dsnapshot_path=../../snapshot.bin -Dcpu=x86_64
```

The script

1. builds with an empty linker script (the unordered binary; the
   `--verbose-link` line of that build is where `gen_order.py` gets the
   objects and archives it maps symbols to sections with);
2. runs the CDP bench (`demo/puppeteer/cdp.js`, 100 runs) against it with
   fault-around set to 4KB, dumps the pages that are resident
   (`pagemap.py`), turns them into hot symbol lists in address order
   (`hotlist.py`) and into the new `lightpanda.ld` / `v8.txt`
   (`gen_order.py`);
3. re-runs that build's `ld.lld` line with the new script to check that it
   links (the script is a cache input of the whole exe compilation, so
   building with it would be a second full compile; the relink takes
   seconds and is the same link, bar V8's hot-marked archive coming from
   the previous `v8.txt`, which does not affect whether the script parses).

Whether the new profile is any good is not measured here; the e2e bench's
`MAX_VmHWM` gate is, on the next push to `main`.

The profile is a list of resident pages, so every function sharing a 4KB page
with a hot one is counted as hot. Profiling the *unordered* binary keeps that
over-inclusion random: a profile of the ordered binary would re-include every
function that went cold but still sits in `.text.hot` next to a hot one, and
the hot set would only ever grow. Over-inclusion is the cheap side of the
trade anyway (a hot function left out costs a whole 64KB cold window; one
included needlessly costs its own size), and `.text.hot` is packed in
address order of the profile, so functions that run together stay together.

The binary is benched from tmpfs (`RAMDIR`, default `/dev/shm`): on ext4 with
a recent kernel, large page-cache folios are mapped whole and hide the
64KB-window behaviour that CI sees. The `x86_64` profile is applied to the
`aarch64` release too; it carries over as far as the section and symbol
names coincide, nothing is measured on `aarch64`.

To look at what is still resident outside the hot sections, run the bench by
hand and use `pagemap.py` + `nm` on the live process.