browser/docs/agent-tutorial.md

Agent tutorial — Hacker News, end-to-end

This walks you from "I just built ./lightpanda" to a recorded, replayable, self-healing browser script — and then drives the same script from an external MCP client. Every section ends with a command you can run; nothing references later sections.

For the flag/command/tool tables, see agent.md. This document is the tutorial; that one is the reference.

What you'll build

One session against Hacker News:

1. Log in with your account.
2. Confirm the login by reading the username out of the header.
3. Record the whole flow to a .lp file.
4. Replay it offline, with no LLM.
5. Break a selector on purpose; watch --self-heal repair the file.
6. Drive the same script from an external agent over MCP.

The finished artifact already exists in the repo as hn_login.lp. Diff your recording against it at the end as a sanity check.

Prerequisites

• ./lightpanda on your PATH (build with zig build).
• A Hacker News account.
• One LLM API key for sections that need natural language and self-healing — Anthropic, OpenAI, Gemini, or a local Ollama. Sections 4–7 work with no key at all.

Export your HN credentials as LP_* env vars. The convention is LP_<SITE>_<FIELD> — a short site identifier (HN for Hacker News, GH for GitHub, …) lets you keep credentials for multiple sites in your environment without collisions. The unprefixed LP_USERNAME / LP_PASSWORD form is the generic fallback when you only have one site.

In bash or zsh:

export LP_HN_USERNAME="your-hn-handle"
export LP_HN_PASSWORD="your-hn-password"

In fish:

set -gx LP_HN_USERNAME "your-hn-handle"
set -gx LP_HN_PASSWORD "your-hn-password"

The LP_ prefix matters. The agent resolves $LP_* references inside the Lightpanda subprocess, so the literal secret never enters the LLM context; and the getEnv tool refuses to read anything that doesn't start with LP_, so the model can't probe your other env vars.

Verify they're set before continuing — substitution fails silently if a variable is missing (the literal $LP_HN_USERNAME ends up typed into the form), and the TYPE confirmation message intentionally echoes the placeholder name rather than the resolved value, so the response text won't tell you. Confirm directly:

./lightpanda agent --no-llm
> /getEnv LP_HN_USERNAME

/getEnv returns the literal value if set, or "not set" if missing. Only $LP_* references in fill values are substituted; other $ characters in your password (my$ecret, $5.99) are passed through verbatim.

1. First contact: the REPL

./lightpanda agent

On startup the agent prints a one-line notice telling you which mode it landed in — which provider (and which env var won), or "basic REPL (no LLM)" if no key is set. The REPL writes its history to .lp-history in the working directory, so up-arrow works across runs.

Try the meta commands:

> /help
> /help goto
> /quit

/help lists every browser tool. /help <tool> prints its JSON schema. /quit exits cleanly. If you have no API key yet and want to poke around without an LLM, ./lightpanda agent --no-llm forces the basic REPL.

2. The shortest possible win: --task

Before doing anything complicated, prove the LLM + browser stack works end-to-end:

./lightpanda agent --task "what is the top story on news.ycombinator.com?"

--task runs a single user turn, prints the final answer on stdout, and exits. Tool calls, progress, and errors all go to stderr, so redirecting stdout gives you a clean answer:

./lightpanda agent --task "top story on news.ycombinator.com?" > out.txt

If you need to feed the model a local file, repeat --task-attachment <path> for each one.

3. Driving the browser by hand

Now back to the REPL. We'll write the HN login flow one command at a time so you can see how each step depends on what the previous one showed.

> GOTO https://news.ycombinator.com/login

GOTO takes an unquoted URL. The page is now loaded.

Commands must be uppercase. click '#foo' is forwarded to the LLM as natural language; only CLICK '#foo' runs as a command. TAB completion in the REPL fills in the caps for you — typing cli<TAB> rewrites the line to CLICK.

Inspect it before clicking anything:

> TREE

TREE prints the semantic tree to stdout. Two forms are visible — the login form and the create-account form below it — and each one contains two unlabeled textboxes:

8 form
 13 'username:'
 15 [i] textbox
 18 'password:'
 20 [i]
 22 [i] button 'login' value='login'
30 form
 35 'username:'
 37 [i] textbox
 …

Notice the textboxes have no accessible name — "username:" is a sibling text node, not a <label for="…">. This is typical of older pages. The ARIA-name tool reflects that:

> /findElement role=textbox name=username
[]

Empty result. Slash commands accept a single positional argument (for tools with one required field), key=value pairs, or a raw {json} blob; findElement filters by accessible name, which we don't have here.

For unlabeled login forms, jump to detectForms, which reads the HTML directly and surfaces each form's action plus each input's name attribute:

> /detectForms

You'll see two forms, the first with action: "login" and fields named acct and pw. That's enough to synthesize the CSS selector yourself: scope by form action to avoid colliding with the create-account form, then key on the input's name attribute.

Selector rule, load-bearing: the click-family tools (CLICK, TYPE, HOVER, SELECT, CHECK) accept CSS selectors only. The backend node IDs findElement and detectForms return are invalidated by any DOM mutation, and they cannot be serialized into PandaScript — a session that uses them is not replayable. Always synthesize a CSS selector from the attributes (id, class, name, action, tag_name) and use that.

Now fill the form:

> TYPE 'form[action="login"] input[name="acct"]' '$LP_HN_USERNAME'
> TYPE 'form[action="login"] input[name="pw"]' '$LP_HN_PASSWORD'
> CLICK 'form[action="login"] input[type="submit"][value="login"]'
> WAIT '#logout'

$LP_* references in TYPE values are resolved at execution time inside the subprocess. The LLM never sees the literal credential.

The WAIT '#logout' line is doing two jobs at once, and it's worth unpacking because the pattern recurs in every recorded script:

• It's a synchronization point. CLICK on the submit button returns as soon as the click dispatches, not after the server responds. Without a wait, the next command races the login redirect and may run against the pre-login DOM. WAIT '<selector>' blocks until that selector appears, so the script resumes only after HN's logged-in page has rendered.
• It's an implicit assertion. HN renders the #logout link only when the session is authenticated. If the credentials are wrong (or HN throws up a captcha, or rate-limits, or the form layout changed), #logout never appears and WAIT times out — the script fails loudly at the line where the failure actually happened, instead of silently succeeding and producing garbage downstream.

You generally want a WAIT like this after every state-changing action that triggers async work: pick a selector that only exists in the post-action state, and you get free regression protection. Waiting on the URL (location.pathname === "/news") or a generic element that exists on both pages is weaker — both can be true before the navigation finishes.

Confirm by pulling structured data off the page. EXTRACT takes a JSON schema object — each value describes what to lift out, and the whole result is printed to stdout as one JSON object. The simplest form is a flat selector lookup:

> EXTRACT '{"karma": "#karma"}'
{"karma":"42"}

The schema grammar is small but covers the cases you'd reach for:

• "<sel>" — textContent.trim() of the first match (string, or null if no match).
• "" — the matched element's own text (only meaningful inside a fields block, where there's an outer element to refer to).
• ["<sel>"] — text of every match (string array).
• {"selector": "<sel>", "attr": "<name>"} — the first match's attribute value.
• [{"selector": "<sel>", "fields": {…}}] — array of objects, where each fields entry is resolved relative to the matched element.

After a page-changing action (click, navigation, form submit) the previous TREE snapshot is stale; re-inspect before the next interaction. Hop back to the front page and pull the story list to exercise the structured form:

> GOTO https://news.ycombinator.com
> EXTRACT '''
{
  "topStories": [{
    "selector": ".athing",
    "fields": {
      "rank": ".rank",
      "title": ".titleline > a",
      "url": {"selector": ".titleline > a", "attr": "href"}
    }
  }]
}
'''

Triple-quoted (''' or """) values let a schema span multiple lines — the REPL keeps reading until it sees the matching closing quote. The result is a single JSON object printed to stdout: {"topStories":[{"rank":"1","title":"…","url":"…"}, …]}.

The schema is parsed in Zig before the page-side walker runs, so a typo like a stray comma surfaces here as Error: invalid EXTRACT schema JSON instead of a confusing V8 stack trace.

4. Recording the session

The same flow, but recorded to a file. Quit the REPL, then:

./lightpanda agent -i hn.lp

-i <path> opens an interactive REPL that appends state-mutating commands to <path>. Retype the same sequence — login (GOTO, two TYPEs, CLICK, WAIT), then the front-page hop and structured pull (GOTO, multi-line EXTRACT) — then /quit.

Inspect the result:

cat hn.lp

You should see the seven mutating commands and nothing else — no TREE, no MARKDOWN, no slash-command lookups. The recorder filters on a per-command flag (Command.isRecorded()) so read-only inspection never pollutes the script; EXTRACT is recorded (it changes what the script will output on replay even though it doesn't mutate the page).

Diff it against the checked-in fixture:

diff hn.lp hn_login.lp

Modulo trailing newlines, they should match. That fixture is what the rest of this tutorial uses.

5. Replaying deterministically

./lightpanda agent hn_login.lp

No --provider, no LLM, no token spend. The recorded script runs top to bottom against a fresh browser. This is the form you want for regression tests and CI: it's pure replay.

LOGIN, ACCEPT_COOKIES, and natural-language lines are the only script entries that require an LLM. A pure recording from -i never contains them, so it always replays without --provider.

6. Selector drift and --self-heal

Real pages change. Simulate selector drift by editing your copy:

cp hn_login.lp hn_broken.lp
sed -i 's/input\[name="acct"\]/input[name="user"]/' hn_broken.lp

input[name="user"] doesn't exist on HN's login form, so a plain replay fails:

./lightpanda agent hn_broken.lp

TYPE, CHECK, and SELECT go a step further than just "did the selector resolve" — a post-exec verifier checks that the DOM actually reflects the intent (the input ended up with the value you typed, the checkbox flipped, the option got selected). That's what catches silent drift before it propagates.

Now re-run with self-heal:

./lightpanda agent --self-heal --provider anthropic hn_broken.lp

(Substitute your provider.) On failure, the agent runs a short, budget-capped LLM turn against the current page state, gets a replacement command, runs it, and atomically rewrites hn_broken.lp in place. A hn_broken.lp.bak is written before any mutation, and the rewritten line is prefixed with a header:

# [Auto-healed] Original: TYPE 'form[action="login"] input[name="user"]' '$LP_USERNAME'
TYPE 'form[action="login"] input[name="acct"]' '$LP_USERNAME'

Self-heal is intentionally narrow: one replacement per failure, no navigation, capped budget. It's there to recover from selector drift, not to redesign the script.

Re-run without --self-heal to prove replay is back to deterministic:

./lightpanda agent hn_broken.lp

7. Same script, external agent (MCP)

Everything above used Lightpanda's built-in agent. If you're driving Lightpanda from a different agent (Claude Code, a custom MCP client, your own harness), use lightpanda mcp instead — same browser tools, no API key on the Lightpanda side, the calling agent supplies the LLM.

Register the server with your MCP client:

{
  "mcpServers": {
    "lightpanda": {
      "command": "/path/to/lightpanda",
      "args": ["mcp"]
    }
  }
}

Record a session over MCP

From the external agent, call:

1. record_start { "path": "hn.lp" } — begins appending state-mutating tool calls to hn.lp. The path must be relative and free of ...
2. The same browser tools you'd call anyway: goto, fill, click, waitForSelector. Each one that succeeds is appended verbatim; query-only tools (tree, markdown, findElement, consoleLogs) are never recorded.
3. record_comment { "text": "logged in" } — drop a breadcrumb above the next recorded line. Useful for marking the boundary between LLM-driven phases.
4. record_stop {} — closes the recording and returns {path, line_count}.

The output file is byte-equivalent to what -i hn.lp produced in section 4. It replays via the agent CLI without modification:

./lightpanda agent hn.lp

Replay with self-heal over MCP

MCP doesn't carry a --self-heal flag — self-heal is a two-tool roundtrip the calling agent orchestrates:

1. Read the script. For each non-blank, non-comment line, call script_step { "line": "<line>" }. Comments and blanks are no-ops on the Lightpanda side.
2. On isError: true, the structured error message tells you what failed. Hand the current page state and the failing line to your own LLM; have it return a replacement PandaScript line (or several).
3. Call script_heal { "path": "...", "replacements": [{ "original_line": "...", "replacement_lines": ["..."] }] }. Each original_line must match verbatim. Lightpanda writes <path>.bak first, then atomically rewrites the file with the # [Auto-healed] Original: … header prepended to the first replacement — same format as section 6.
4. Continue from the next line.

script_step deliberately does not auto-record: the script is already the source of truth during replay, so double-recording would diverge the file from itself. LOGIN, ACCEPT_COOKIES, and natural-language lines are rejected — those need an LLM, which is the caller's responsibility.

Where to go next

• agent.md — full reference: every flag, every PandaScript command, every browser tool, plus the security model and auto-detection rules.
• hn_login.lp — the fixture this tutorial builds.
• lightpanda mcp --help and lightpanda agent --help — current flag listings straight from the binary.