https://github.com/lightpanda-io/browser/pull/3271 introduce an orderfile so
that hot sections of code were grouped together in the binary, resulting in more
efficient loading. But, it excluded v8 functions because v8 is compiled with
`-fno-unique-section-names` so each section gets the same name. Because of this
the orderfile can't target specific "hot" or "cold" functions.
As a follow up to 3271, I thought we'd be able to re-compile v8 without that
flag, but:
1 - The flag isn't directly exposed by v8, so we'd need to change v8's own build
BUILD.gn in our build process
2 - It would make the .a file ~ 40MB larger (though the final lightpanda binary
would stay the same size
3 - If we wanted to created two .a files (one with -fno-unique-section-names and
one without), it would require a full rebuild (for both x86-64 and arm).
I didn't love those compromises, so I asked Claude if the sections names could
be generated in a separate file and then that could be used when generating out
lightpanda.ld. What claude came up with was a small Zig script
(mark_hot_sections.zig) which is now run as part of the build. It takes the
v8.a file (which still has `-fno-unique-section-names`), it takes a text list
of hot functions, and it re-generates v8.a with a special .text.hot and
.rodata.hot sections. lightpanda.ld can include these.
This PR depends on https://github.com/lightpanda-io/zig-v8-fork/pull/202 but
202 doesn't require a v8 rebuild. It merely allows prebuilt_v8_path to be
a lazypath, which we need because the prebuilt_v8_path that we pass is now
generated from this build script.
So we're trying to limit glibc to 2.43.0 do to an issue in Zig's self-hosted
compiler. I'm on 2.44.0 and I hard-coded 2.43.0. But for anyone on a version
prior to 2.43.0, that causes issues. So this takes the host version but limits
it to 2.43.0
The explicit version only exists to force Zig's bundled CRT (zig#31272),
but it also selects symbol versions. PR #3231 added tiny-skia, which
calls acosf, and glibc 2.43 re-versioned that symbol, so the binary now
fails to load on hosts with an older glibc:
libm.so.6: version `GLIBC_2.43' not found
Use the detected host glibc instead: still an explicit version, so the
bundled CRT is still used, but never newer than what the binary runs on.
CI build, for e2e-test and nightly now build with:
-Dorderfile/lightpanda.ld
This file informs the build on how to organize the code in the binary, grouping
hot code together so that we have to load less of the binary into memory.
lightpanda.ld will drift: we'll refactor our code, add new features, update
dependencies, update Zig, ... So it has to be re-generated. But we can do that
automatically in the CI (say, before the nightly build). That's for a follow up
PR.
This does not currently cover V8. V8 is being build with
`-no-unique-section-names`, so we don't get names that we can correctly
organize. The real win comes from doing this in V8, since a lot of V8 is cold.
This PR can land as-is, a zig-v8-fork PR will remove that flag, and then we can
have a follow up PR with an lightpanda.ld that includes the v8 symbols.
This is opt-in (via the -Dorderfile flag) because it adds ~20 seconds of
linking time.
Cargo prints its progress to stderr. Zig's build sees the stderr content and
prints it as a "failed command: ...". Capture cargo's stderr to silence this
fake-error. A status != 0 still causes the error to be printed.
Deduplicate the three executable blocks and the vendored C library
module setup, drop vestigial error unions, and replace hand-rolled
formatting with b.fmt/b.pathJoin/std.mem.concat.
The snapshot-creator and skills executables now get the same
sanitize_c/sanitize_thread flags as the browser executable, which
they were silently missing.
Prebuilt V8 discovery now reports every cache miss with the probed
path. Previously a miss on the static archive path fell back to the
source build without a message, and the dev_fast message claimed a
source build was starting when the graph might not compile anything.
Assuming the conditions are met for -dev_fast to work (linux, debug, x86-64
without sanitizers), the -dev_fast now defaults to true.
download-v8 make target always downloads the .a (as before) and on linux x86-64
it also downloads the .so.
Supports prebuild-v8 shared library and adds a `download-v8-shared` make target.
Depends on: https://github.com/lightpanda-io/zig-v8-fork/pull/198
The auto-discovery of the .so is done in the build.zig, which is different than
the auto-discovery of the .a which is done in the Makefile. This is intentional
and if we inverse -Ddev_fast so that it's on by default, the .a's auto discovery
will be moved to build.zig for consistency and explicitness. As-is, we have no
way to tell whether -Dprebuilt_v8_path=v8.a is being injected by Makefile or
is explicit (which isn't a problem so long as we're not doing dev_fast by
default).
Zig 0.17 removes @cImport in favor of translate-c steps in the build
system. Each library's link function now creates an addTranslateC step
exposed as a module import: curl and isocline translate their headers
straight from the dependency tree, while sqlite3 uses the artifact's
emitted include tree since the amalgamation lives in a sub-dependency.
Terminal.zig and prompt_assist.zig previously instantiated two
independent @cImport namespaces for isocline; they now share one
translated module, so its types are identical across both files.
Make sure jemallocator is disabled in Release builds (we only use it in Debug
builds to collect metrics. Supposedly something changed in our 0.16 build that
causes it to be initialized even though it isn't used. Not sure, for this, I'm
just doing what Claude tell me, but disabling something in Release that I know
we aren't using sounds ok).
Built against https://github.com/lightpanda-io/zig-v8-fork/tree/zig-0.16 but
it doesn't require a new v8 build.
Built against https://github.com/lightpanda-io/boringssl-zig/tree/zig-0.16
since the current fork we point to isn't updated.
A global std.Io instance, lp.io. Way easier this way and requires 0 changes to
our libcurl integration / event loop.
Network code uses a new layer that does what Zig 0.15's posix package used to
do. Again, quicker migration that way. But, as long as we have the global IO,
and given the half-baked nature of networking in std.Io 0.16, this just makes
sense. Things can be migrated as needed.
The std.time.* -> std.Io.Timestamp/Clock/Duration resulted in _a lot_ of
changes. ArrayList = .{} -> ArrayList -> .empty also resulted in a lot of
changes, but that's obviously superficial. As is the trimLeft/trimRight ->
trimStart/trimEnd rename.
Locking adopt the `Uncancelable` variants, e.g. mutex.lockUncancelable() to
preserve the error-free signature (and, because cancellation would be something
we'd have to put more thought into).
std.json.ObjectMap is now unmanaged, so the allocator had to be passed along.
However, there's still a deprecated managed variant of MemoryPool, so I switched
to it (we can do a small follow up PR to move to the unmanaged after).
I tried use_llvm = false, but it locks my computer, consuming RAM until MacOS
gives me a popup I've never seen before, begging me to start killing processes.
Agent and the networking stuff saw the most significant changes.
The /save script documentation lived as a hand-maintained string in
Agent.zig whose primitives table drifted whenever a tool changed. Move
it to src/script/skill.zig and render the reference (signatures, option
lists, enums, defaults, per-parameter descriptions) from Schema.all()
at first use, keeping the curated per-tool notes behind an exhaustive
switch on Tool so a new or renamed tool is a compile error until its
doc entry exists.
The rendered skill is shared by three consumers:
- the /save and revision system prompts (built lazily in Agent.zig)
- a new mcp://skill/pandascript resource
- `zig build skills`, which writes zig-out/skills/<name>/SKILL.md with
Claude Code frontmatter via a registry-based generator exe, so
future Lightpanda skills are one registry entry each
Schema.FieldEntry now retains per-parameter schema descriptions, which
previously existed only in the raw JSON.
This is a bit all over the place.
1 - Replace libidn2 with rust-idna. It looks like there are different idna
profiles, and rust-idna (from the servo project) implements the whatwg
one. libidn2 would be too strict in some cases and not strict enough in
others. (Gemini says I could use libidn2 for this, but what it suggested
didn't work, and I couldn't figure it out myself, and claude insisted it
_did not_ have the correct implementation for what we want).
2 - We previously only ran a URL through idna if it wasn't ascii. Turns out
we also need to run it if there's a "xn--" (aka, an IDNA ACE prefix) in
there. This helps us pass hundreds of WPT cases, and it's pretty cheap.
3 - Implement more of the Area WebAPI. Mostly copied from Anchor.
4 - Add username/password accessor to Anchor/Area
5 - window.open validates the URL (i.e. tries to resolve it and handles the
error)
6 - Invalid idna conversion maps to a TypeError
7 - Cleanup closed popups on the next tick (like destroyed pages), rather than
at an interval or on shutdown. This one seems unrelated, but some of these
tests are opening hundreds (thousands?) of popups and then closing them.
AFAIK, these aren't being used and I've personally not have reason to reference
/ look at them for months.
I have no issues though if we want to keep them in.
The build script wrote the version line to stdout, polluting any pipeline
that captures program output via 'zig build run' or similar. Banners
belong on stderr.
Replaces the linenoise library with isocline for improved REPL
functionality. Updates build scripts and moves completion state from
global variables into the Terminal struct.
config.h already has a hand-added _LIBIDN2_LP_DECLS guarded block at
the end declaring extern strverscmp for the same gnulib-overlay reason.
Move the strchrnul prototype into that block (still #ifdef __APPLE__
since glibc Linux already declares it) so all project-added externs
live in one place, the autoconf-generated section stays untouched, and
the precedent for "how this codebase handles missing glibc extensions"
is one consistent pattern instead of two.
Refresh the build.zig and strchrnul.c comments to point at the
relocated declaration.
No functional change — verified across -target x86_64-macos.{13.0,
14.0,15.4} (lookup.c pattern + the three TUs that include both
<config.h> and <string.h>) plus x86_64-linux-gnu, all under
-Wall -Wextra -Werror.
Move the macOS strchrnul declaration from a side-channel header injected
via -include into vendor/libidn2/config.h itself, gated on __APPLE__.
config.h is the canonical place where libidn2 documents its
HAVE_STRCHRNUL detection — co-locating the platform fallback prototype
there mirrors gnulib's own approach (which declares missing symbols in
its substituted <string.h>) and removes the per-platform build-flag
asymmetry plus the dedicated include path.
The Darwin-only strchrnul.c implementation is unchanged; build.zig now
just adds it on Darwin without touching the lib/ flags array.
Verified that on both `-target x86_64-macos.14.0` (no libc strchrnul)
and `-target x86_64-macos.15.4` (libc has it) a TU mimicking lookup.c
(`#include "config.h"` then call strchrnul) compiles cleanly with no
implicit-declaration error and no redeclaration warning.
libidn2's lib/lookup.c calls strchrnul() unconditionally — a glibc
extension that macOS libc neither defines nor declares. Upstream's
portable build relies on gnulib substituting <string.h> with a
declaration and linking gl/strchrnul.c, but that overlay is not wired
up here. The result was an "implicit function declaration" hard error
on every macOS nightly build.
Add a Darwin-only shim under vendor/libidn2/darwin/ and inject its
prototype into the libidn2 lib/ sources via -include. The shim mirrors
strchrnul's semantics without falling through to rawmemchr (also
glibc-only) since libidn2 only ever searches for '.'.
Links to libidn2 and builds libcurl with it. This makes libcurl work, and by
extension browser, work on international domain names, e.g.
zig build run -- fetch "https://räksmörgås.se/"
With it available, we can use it in our WebAPIs which should also support these
domains, e.g:
testing.expectEqual('xn--rksmrgs-5wao1o.se', new URL('https://räksmörgås.se').hostname);
There is more integration to be done here, but this is a first step.
claude wrote all of the build.zig code.
I don't have a strong opinion about this feature, I just dislike that our WPT
/url/* tests are at 1704 / 9095 and, this is the biggest chunk (although, this
specific commit just does the basic integration and probably won't fix too many
WPT cases directly).
Profiling `zig build --release=fast` after a one-file edit showed the
cost was three parallel exe compiles of the same module — each re-runs
LLVM codegen when anything in `lightpanda_module` changes. All C/C++
deps (v8, sqlite, curl, brotli, nghttp2, boringssl) already cache across
builds and are not the bottleneck.
Default install now builds only the main `lightpanda` exe.
`lightpanda-snapshot-creator` and `legacy_test` move to a new named step
and can be built explicitly via `zig build extras`. CI is unaffected:
nightly.yml already invokes `snapshot_creator` as an explicit step, and
no job consumes the legacy_test artifact.
Measured (release=fast, one source file modified, 36/36 steps):
before: real 3m32s, compile exe lightpanda ~3m, others ~2m each
after: real 2m43s, only compile exe lightpanda
Also tried and rejected:
- self-hosted Zig backend for Debug: fatal linker error, unhandled
relocation type R_X86_64_PC64 in V8 objects.
- mold via `mold -run`: ~4s delta, within noise — Zig invokes LLD
as a library, so no external `ld` process exists to intercept.
This adds an app.storage which is a union around configurable storage engine
(currently, only sqlite).
It is _not_ being used anywhere right now. The goal is to get feedback on
the implementation and then move cache to it.
This doesn't expose a generic query API. The goal is that the storage will
expose high level methods, e.g. `cacheGet(req: CacheGetRequest)` and every
storage engine will translate the `storage.CacheGetRequest` as needed.
A thin wrapper around the Sqlite C api is included, e.g. exec(SQL, .{args}) a
`rows` and `row` fetcher. A connection pool is included. By default, an
in-memory DB is currently created. And a `migrations` table with an id of `1`
is created/inserted. I don't imagine needing fancy migratations.
Some WPT tests need to interact with the browser in a way that isn't possible
with web apis. Browsers need to expose a way for tests to do this and then use
the testdriver-vendor.js to hook into these special WPT actions.
This commit sets up the infrastructure for supporting this and includes
the delete_all_cookies functionality needed by various cookie tests (e.g.
/cookies/attributes/attributes-ctl.sub.html).
A new compilation flag, `-Dwpt_extensions`, can be specified. When specified
a `window.webdriver` accessor is defined and a `WebDriver` type is exposed.
Note that, while I only implemented delete_all_cookies for now, I've seen other
tests fail because of missing vendor-specific implementation.
Scott Taylor