mirror of
https://github.com/twentyhq/twenty.git
synced 2026-09-23 22:16:33 -04:00
Félix Malfait
0bcce772d2
## What A post-merge browser QA agent, riding the environment `ci-e2e-main` already boots on every push to main. After the deterministic Playwright suite, the QA Scout: 1. Resolves the merged PR for the pushed commit and collects its metadata, file list, and diff. Skips i18n translation PRs and titles carrying `[no-qa]` / `[noqa]`, mirroring `mustBeQa` in the eng app. 2. Runs Claude Code (`claude-code-action`, same pin as `claude.yml`) with the Playwright MCP against the already-running app at `localhost:3000`, following the new skill at `.claude/skills/qa-scout/SKILL.md`: derive 2 to 5 user-visible scenarios from the diff, execute them in the browser, and diff its own window of the server and worker logs to catch swallowed backend exceptions behind a green-looking UI (the 2.35 timelineActivity failure mode). 3. Writes `verdict.json` (`PASS | INVESTIGATE | FAIL`) and `report.md`, uploaded as the `qa-scout-report` artifact and rendered in the job summary. 4. On `FAIL` / `INVESTIGATE`, a separate job posts (or updates) a single marked comment on the merged PR, opening with a `[!CAUTION]` admonition. `PASS` stays quiet. The skill also works locally: engineers can run `/qa-scout` in Claude Code against a dev stack. ## Why Post-merge QA of merged PRs is today a fully manual Build Companion queue, and the deterministic e2e suite covers 7 flows. The Scout pre-chews that QA: it picks the risky flows for each specific diff, and it reads the logs while it clicks, which is exactly where silently-swallowed regressions hide. `hasQaBeenDoneOnMain` stays human-owned; the Scout's report lands where the human QA starts. ## Safety and rollout - **Shadow-safe**: every Scout step is `continue-on-error`, so it can never redden `ci-e2e-main` or affect `ci-e2e-main-status-check`. The suite's own behavior is unchanged (server/worker logs are now teed to files; console output is identical). - **Privilege separation**: the agent step runs under the workflow's read-only token and only gets `Read/Grep/Glob/Write`, a short Bash allowlist, and the Playwright MCP (headless Chrome, isolated profile, origin-restricted to localhost). `pull-requests: write` exists only in the `qa-scout-comment` job, which reads the structured artifact and never the agent's raw context, so untrusted page/log content the agent reads can never reach a privileged credential. - **Never on fork or PR events**: the Scout only runs on push to main and manual `workflow_dispatch`; merge-queue PR runs of this workflow are untouched. - **Kill switch**: set repo variable `QA_SCOUT_DISABLE_COMMENTS=true` to keep verdicts in artifacts/summaries only (shadow mode). - **Re-run**: Actions, CI E2E Main, Run workflow with `pr_number` re-runs the Scout for any merged PR. ## Validation - Workflow YAML parses; both embedded scripts pass `bash -n`; the heredoc MCP config is valid JSON. - The `mustBeQa` mirror was table-tested against real title shapes (`[no-qa]`, `[NOQA]`, `i18n - translations`, mid-title i18n mentions). - `@playwright/mcp@0.0.79` (current latest) flags used (`--headless`, `--browser chrome`, `--isolated`, `--output-dir`, `--allowed-origins`) verified against its CLI help. - `.gitignore` gains `!.claude/skills/` so the skill can live next to the committed `.claude/settings.json`. ## Follow-ups (separate PRs) - Wire the verdict into the eng app: a webhook route (like `/s/main-ci-failing`) writing the verdict onto the `pullRequest` record so Build Companion shows it in the QA queue. - Point the same skill at staging after deploy/upgrade, where aged data lives. - Have the nightly/deep variant draft Playwright specs for stable scenarios so the deterministic suite grows. --- _Generated by [Claude Code](https://claude.ai/code/session_01B1ML7TTsrqPbPnCtox9rBX)_ <!-- This is an auto-generated description by cubic. --> <a href="https://cubic.dev/pr/twentyhq/twenty/pull/24856?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. -->
