Compare commits

..
1 Commits
Author SHA1 Message Date
Andrey Antukh 04d7b623aa Add account lockout after failed login attempts
Implement per-account brute-force protection using a Redis-backed
failed-login counter. After 5 failed attempts within 15 minutes, the
account is temporarily locked out and all login attempts (including
with the correct password) are rejected with a 429 response.

Closes #11397

AI-assisted-by: longcat-2.0
2026-08-28 11:30:00 +02:00
646 changed files with 11030 additions and 36140 deletions

No files matched your search

-91
View File
@@ -1,91 +0,0 @@
# Agent skills
This folder is the single home for the skills our coding agents use.
Each skill is a folder with a `SKILL.md` inside — a short instruction
manual that an agent loads only when it needs it.
One copy serves every tool:
- **opencode** reads this folder directly.
- **Claude Code** reads it through the `.claude/skills` symlink.
- **Codex** reads it directly.
To change how the agents behave, edit the `SKILL.md` here. There is no
second copy to keep in sync.
## How the skills are organized
**Flows** are the six skills you invoke by name. Each one covers one step
in the life of a change: plan it, review the plan, implement it, review
the code, open the pull request.
**References** hold the quality standards. A flow's reviewer loads them;
you rarely touch them directly.
**Procedures** define how one concrete step is done — a plan document, an
issue, a commit. Flows call them, but they also work on their own.
**Utilities** are small helpers for everyday work: search, file lookup,
JSON, REPL access, and so on.
## Flows
| Skill | What it does | When you would say |
|---|---|---|
| [`make-a-plan`](skills/make-a-plan/SKILL.md) | Researches the task, writes an implementation plan, asks you the open questions in plain language, and saves the plan to `.agents/plans/`. | "make a plan for the token refresh bug" |
| [`review-plan`](skills/review-plan/SKILL.md) | Evaluates a plan before anyone writes code: completeness, ordering, risks. Approves it or asks for changes. | "review this plan before we start" |
| [`implement-plan`](skills/implement-plan/SKILL.md) | Shows you the full flow first — the issue and branch it will create (or the branch it continues on), the execution style, and the task checklist — and, after your go-ahead, executes a ready plan. Default: every task, one commit. On request ("step by step"): one task, one commit, your confirmation between tasks. On request ("direct"): no issue and no branch, commits on the current branch. | "implement the plan" · "step by step, one commit per task" · "direct, no branch" |
| [`review-code`](skills/review-code/SKILL.md) | Reviews a diff, branch, or PR and returns findings ranked by impact. | "review my changes before I push" |
| [`create-pr`](skills/create-pr/SKILL.md) | Opens a pull request for the current branch — with checks on base branch, commits, issue, and push state — or updates an existing PR's title and description. | "open a PR for this branch" |
| [`resolve-git-conflicts`](skills/resolve-git-conflicts/SKILL.md) | Untangles merge or rebase conflicts: explains both sides, proposes a resolution, applies it after you approve. Never runs `git rebase --continue`. | "resolve these conflicts" |
## References
| Skill | What it holds |
|---|---|
| [`plan-review-criteria`](skills/plan-review-criteria/SKILL.md) | The plan review rubric: six axes, severity levels, approval standard, output format. The `review-plan` reviewer loads it. |
| [`code-review-criteria`](skills/code-review-criteria/SKILL.md) | The code review rubric: five axes, core principles (DRY, KISS, YAGNI), severity format, verdict. The `review-code` reviewer loads it. |
## Procedures
| Skill | What it does |
|---|---|
| [`planner`](skills/planner/SKILL.md) | The spec of a good plan: context, architecture decisions, tasks with acceptance criteria, checkpoints. Used by `make-a-plan`. |
| [`create-issue`](skills/create-issue/SKILL.md) | Creates a GitHub issue that follows Penpot conventions. Used by `implement-plan`; also works on its own. |
| [`create-commit`](skills/create-commit/SKILL.md) | Makes a commit the Penpot way: emoji subject, clear body, `AI-assisted-by` trailer. Used by `implement-plan`; also works alone when you say "commit this". |
## Utilities
| Skill | What it does |
|---|---|
| [`bat-cat`](skills/bat-cat/SKILL.md) | Read files in the terminal with syntax highlighting and line numbers. |
| [`fd-find`](skills/fd-find/SKILL.md) | Find files by name or pattern, respecting `.gitignore`. |
| [`ripgrep`](skills/ripgrep/SKILL.md) | Fast content search with regular expressions. |
| [`jq-json-processor`](skills/jq-json-processor/SKILL.md) | Slice, filter, and reshape JSON output. |
| [`nrepl-eval`](skills/nrepl-eval/SKILL.md) | Run Clojure or ClojureScript code in the live REPL sessions (backend and frontend). |
| [`taiga`](skills/taiga/SKILL.md) | Look up Penpot issues, user stories, and tasks in Taiga. |
| [`testing`](skills/testing/SKILL.md) | The repo's testing rules and TDD workflow, loaded before writing tests. |
| [`local-ci`](skills/local-ci/SKILL.md) | Run CI-style lint, test, and format checks for the modules you touched with `scripts/ci`, and read the logs when they fail. |
| [`security-and-hardening`](skills/security-and-hardening/SKILL.md) | Security checks for code that handles user input, auth, or external services. |
| [`ste`](skills/ste/SKILL.md) | Rewrites prose in Simplified Technical English. Loads only when you name it. |
| [`refine-prompt`](skills/refine-prompt/SKILL.md) | Rewrites a rough prompt into a clearer one. Never runs the prompt. |
| [`update-changelog`](skills/update-changelog/SKILL.md) | Regenerates `CHANGES.md` from a GitHub milestone. |
## A typical round
1. `/make-a-plan` — you get a plan and a saved file in `.agents/plans/`.
2. `/review-plan` — a second opinion; approve or request changes.
3. `/implement-plan` — the code gets written and committed. Starting from a base branch, it also opens the GitHub issue and the `issue-NNNN` branch; the plans that follow continue on that same branch.
4. `/review-code` — a reviewer checks the commit.
5. `/create-pr` — the branch goes up as a pull request.
Every step also works on its own, and you can always say what you want
in plain words — the agents pick the right skill from what you say.
## Adding or changing a skill
Create a folder here with a `SKILL.md` inside. The file needs `name` and
`description` in its frontmatter, and a clear "When to use" section so
agents know when to reach for it. Keep one job per skill, and keep the
two families apart: flows are named with a verb first; reference skills
end in `-criteria`.
-105
View File
@@ -1,105 +0,0 @@
---
name: create-pr
description: PR flow — open a new PR for the current task branch (validates base branch, commits, issue and push state) or update an existing PR's title or description to match Penpot conventions. Use it when the user asks to open or create a PR, in any phrasing.
---
# Create PR
Two modes. **Open mode** takes the current task branch to a new, validated
PR. **Update mode** rewrites an existing PR's title or description. Gather
information, validate, and act in one pass. If validation fails, STOP with a
single coherent message that lists every problem and states exactly what
information is missing — never fix or work around problems silently.
Both modes require an authenticated `gh` CLI (`gh auth status`) and never
push — the user pushes from their own shell.
## When to use
- The user asks to open or create a NEW PR for the current task branch, in
any phrasing ("open a PR", "create the pull request", "put this up for
review") — or runs `/create-pr`. → **Open mode**.
- The user asks to fix or update an EXISTING PR's title or description to
match conventions. → **Update mode**.
If the running agent cannot write (for example, the plan agent), say so and
stop — this skill needs the build agent.
## Open mode
### 1. Gather context (read-only)
- Current branch: `git rev-parse --abbrev-ref HEAD`.
- Target base branch: run `./scripts/detect-target-branch` from the repo root.
It prints the nearest ancestor branch of HEAD (exit 0) or fails (exit 1).
- Commits: `git log --oneline <base>..HEAD`.
- Push state (local): `git rev-parse --verify origin/<branch>` and compare
with HEAD. It reads the local remote-tracking ref — no network, no SSH. It
reflects the last push or fetch this clone knows about.
- Issue: from the session context, or from the branch name — `issue-NNNN`
maps to issue NNNN; recover its title and body with `gh issue view NNNN`.
### 2. Validate — stop with one message if anything fails
Run all checks before reporting, then report every failure together:
1. **Base branch not usable.** If the script fails (exit 1), or its output —
after stripping an optional `remotes/origin/` prefix — is not one of the
canonical branches (`develop`, `staging`, `main`), stop and ask the user
to re-run with more context — for example, passing the base branch
explicitly in their invocation. An explicit base given by the user
overrides the script's output.
2. **On a base branch.** There is no task branch to merge — say so and stop.
3. **No commits.** The branch has no commits ahead of the base — say so and
stop.
4. **No clear issue.** There is no issue in the session context, and the
branch name has no `issue-NNNN` pattern (or `gh issue view` finds nothing)
— say so and stop. Exception: the user's invocation says `no issue` /
`without issue` — then continue without an issue reference.
5. **Branch not pushed.** The remote-tracking ref `origin/<branch>` is
missing, or `git rev-parse origin/<branch>` differs from HEAD — the
branch was never pushed, or has commits the remote does not have. Never
push yourself; ask the user to push and to run `/create-pr` again
afterwards, then stop.
### 3. Already-open PR
Check whether a PR already exists for this branch (`gh pr list --head
<branch>`). If one exists, report its URL and stop — do not create a second
one. Title or description fixes belong to Update mode.
### 4. Write and create the PR
Write the title and body following `mem:workflow/creating-prs` (title format,
description structure, writing principles) and `mem:workflow/creating-commits`
(commit type emojis). Derive the title and body from the commits and, when
there is one, from the issue body. Reference the issue with `Closes #NNNN`.
```bash
gh pr create --repo penpot/penpot --title "<TITLE>" --body-file /tmp/pr-body.md
```
### 5. Report
Report the PR URL and stop.
## Update mode
1. Identify the PR: the number given by the user, or `gh pr list --head
<branch>`.
2. Write the new title and/or body following `mem:workflow/creating-prs`.
3. Apply and verify:
```bash
gh pr edit <NUMBER> --repo penpot/penpot --title "<TITLE>" --body-file /tmp/pr-body.md
gh pr view <NUMBER> --repo penpot/penpot --json title,body
```
4. Report and stop.
## User context
Extra context in the user's invocation (the message that triggered this skill)
plays the role command arguments play elsewhere: overrides such as `no issue` /
`without issue`, an explicit base branch (`from origin/staging`), a PR number
for Update mode, and so on.
-144
View File
@@ -1,144 +0,0 @@
---
name: implement-plan
description: Implementation flow — execute a ready plan from the session context: read the plan, detect the flow, then present the full picture (issue and branch to create or the branch to continue on, execution style, task checklist) and wait for confirmation. Default is every task with one final commit; on request ("step by step"), one task and one commit at a time with a pause after each; on request ("direct"), no issue and no branch — the commit lands on the current branch. Use it when the user asks to implement or execute a plan, in any phrasing.
---
# Implement Plan
This flow is run once a plan is ready (for example, from plan mode). Execute
the plan already prepared in the current session context. It never pushes —
the user pushes.
By default it ends with exactly one commit. When the user asks for it
("step by step"), it commits once per task instead and waits for the
user's confirmation after each one (see *Execution modes*).
## When to use
- The user asks to implement or execute a plan, in any phrasing:
"implement the plan", "execute it", "go build it" — or runs
`/implement-plan`.
- A ready, reviewed plan is in the session context or a plan file path
was given (typically after `/make-a-plan` or `/review-plan`).
Do not use it to produce plans — that is the `make-a-plan` flow.
## 1. Read the plan first
Identify the plan to execute — from the file path the user gave, the
arguments, or the session context. Read it completely. Read the required
memories before writing any code: `mem:critical-info` and the core memory
of every module the plan touches, plus the deeper memories they reference
(AGENTS.md governs this).
## 2. Detect the flow (no questions)
Inspect the current branch with `git rev-parse --abbrev-ref HEAD`, pick the
mode, and announce it in one line before presenting anything. Detection is
read-only: nothing is created until the user confirms (step 3).
- **On a base branch** (`main`, `develop`, `staging`) → **standalone mode**:
a new GitHub issue and a branch `issue-NNNN` will be created after the
user's confirmation.
- **On any other branch** (a feature branch, typically `issue-NNNN`) →
**continue mode**: the implementation continues on the current branch.
No issue or branch is created. The branch name provides the issue
reference when it follows the `issue-NNNN` pattern.
Arguments override detection: `standalone`, `continue`, `direct`
(`no branch` / `direct commit`), `no issue` / `without issue`, or an
explicit base such as `from origin/develop`.
**Direct mode** (`direct`, `no branch`, `direct commit`): no issue and
no branch — the implementation and the commit land on the current branch
as it is, even when it is a base branch. Best for small or tooling-only
changes the user wants committed in place.
**Standalone while already on a feature branch:** stop and explain that this
would stack branches. Ask the user to re-run with an explicit base, for
example `from origin/develop` — then branch from that base instead of HEAD.
## 3. Present the checklist and wait
Before touching the repository, show the user the full picture:
- **The flow**: whether the GitHub issue and the branch will be created
(standalone mode — give the planned branch name, `issue-NNNN` or
`plan-<slug>`), whether you continue on the current branch
(continue mode — name it), or whether everything lands on the current
branch as it is (direct mode — name it, and say so when it is a base
branch).
- **The execution style**: batch or step-by-step (see *Execution modes*).
- A checklist (todolist) of the plan's tasks, in order.
Then WAIT for the user's explicit confirmation. Do not start until you
have it. If the plan has no discrete tasks, ask the user how to split
it, or propose running it as a single change.
## 4. Execute the plan
**Standalone setup, after the confirmation:** create the issue with the
**`create-issue`** skill, following the *Creating Issues from Draft Body*
flow in `mem:workflow/creating-issues`. Derive the issue title and body
from the plan, capture the new issue's number — call it **NNNN** — and
create the branch from the current HEAD:
```
git checkout -b issue-NNNN
```
If the arguments say `no issue` / `without issue`, skip the issue and
create a branch named `plan-<slug>` instead, where `<slug>` is the plan
title, lowercase and hyphen-separated.
If the arguments say `direct` / `no branch` / `direct commit`, skip the
issue and the branch: implement and commit on the current branch as it
is. If it is a base branch, the checklist presentation already said so —
no further confirmation is needed.
### Batch mode (default)
Implement every task in one go. Work methodically, keeping changes
focused on what the issue requires. Respect the plan's proposed
parallelization when it applies.
When the implementation is complete, load the **`create-commit`** skill
and follow its workflow to commit the changes. Provide a brief summary
of what was implemented and why, the issue reference (`issue-NNNN`) when
there is one, and the model name you are running as so the
`AI-assisted-by` trailer is set correctly.
### Step-by-step mode (on request)
When the user asks for it — "step by step", "task by task", "one commit
per task" — loop one task at a time:
- Execute exactly ONE task.
- Commit it now: load the **`create-commit`** skill and follow it —
one commit per task, never two tasks in one commit. Same inputs as
always: what and why, the issue reference, your model name.
- Show the user the result (what changed, files touched, how it was
verified).
- WAIT for the user's confirmation before starting the next task.
Never batch in this mode: no two tasks in one commit, and no new task
before the user confirms. If a task turns out much bigger than planned,
stop and ask the user before splitting it.
## When you are done
End by suggesting the next steps (suggestions, not a required pipeline — any
instruction from me overrides them):
- `/review-code` — to review the changes just committed; it routes to
`/make-a-plan` by itself if the findings need one.
- `/create-pr` — when the task is done and the branch is ready to merge.
## User context
Extra context in the user's invocation (the message that triggered this
skill) plays the role command arguments play elsewhere: `standalone`,
`continue`, `direct` (`no branch` / `direct commit`), `no issue` /
`without issue`, an explicit base such as `from origin/develop`, or
`step by step` / `one commit per task` for the step-by-step execution
mode. Modes combine freely, for example "standalone step by step".
-95
View File
@@ -1,95 +0,0 @@
---
name: local-ci
description: Run local CI-style checks with ./scripts/ci (lint, tests, format) per monorepo module. Use when verifying changes before declaring work done, running lint or tests locally, fixing formatting, or repairing Clojure delimiter errors.
---
# Local CI
Run the same checks CI runs, locally, for the modules you touched, with
`scripts/ci`. Each task writes a log file; the final summary says what
passed and what failed.
Full details: `mem:scripts/ci` (file: `.serena/memories/scripts/ci.md`)
## When to use
- After implementing or fixing code — verify every module you touched
before declaring the work done.
- When the user asks to run CI, lint, tests, or format checks locally.
- When you changed `common/` — validate its consumers too.
**Skip:** while exploring, planning, or reading code.
## Command reference
Run from the repo root:
```bash
./scripts/ci [OPTIONS] [MODULES...]
```
Modules: `frontend` `backend` `common` `render-wasm` `exporter` `mcp`
`plugins` `library`, or `--all` for every module.
With no task flags it runs three tasks per module, in order: **lint**,
**test**, **fmt** (format check; `--fix` formats files instead).
| Flag | Effect |
|------|--------|
| `--all` | Run every module |
| `--exclude MOD` | Skip one module (repeatable) |
| `--lint` / `--no-lint` | Run only lint / drop lint |
| `--test` / `--no-test` | Run only tests / drop tests |
| `--fmt` / `--no-fmt` | Run only format check / drop it |
| `--fix` | Format files instead of checking (other tasks unaffected) |
| `--paren-repair` | Fix delimiter errors in Clojure/CLJS files |
| `--fail-fast` | Stop at the first failure |
| `--quiet` | Suppress failure output |
| `--dry-run` | Show what would run, execute nothing |
| `--clean` | Delete the `.ci-logs/` directory |
## Reading failures
Every task writes its full output to `.ci-logs/<module>-<task>.log`. On
failure the script prints only the last 30 lines. To diagnose a failure,
**read the log file** — never re-run the command piped through filters
(repo rule: redirect to a file first, then read it). The exit code is 1
when any task failed; the summary lists each failed `module:task` and its
log path.
## Typical workflows
```bash
# Verify a module you changed: lint + tests + format check
./scripts/ci frontend
# Fast pass while iterating: lint only
./scripts/ci --lint frontend
# Lint + format check, skip the long test suite
./scripts/ci --no-test frontend
# Format the module without running the test suite
./scripts/ci --fix --no-test frontend
# Broke delimiters in Clojure/CLJS files: repair first, then lint
./scripts/ci --paren-repair frontend
./scripts/ci --lint frontend
# Changed common/ — validate its consumers too
./scripts/ci frontend backend exporter
# Preview what would run, without running it
./scripts/ci --dry-run --all
```
## Gotchas
- Run from the repo root.
- Test tasks are long-running (backend runs `clojure -M:dev:test`); give
the bash call a generous timeout (1020 minutes) instead of letting it
time out mid-run.
- `mcp` has no lint task — it shows as skipped, not failed.
- `--paren-repair` only fixes delimiters; run lint afterwards to catch
what remains. See `mem:scripts/paren-repair`.
- What to run and how to read test results: `mem:testing`.
-100
View File
@@ -1,100 +0,0 @@
---
name: make-a-plan
description: Planning flow — research the subject of this session, produce an implementation plan with the planner skill, resolve open questions with the user in plain language, and save the final plan to .agents/plans/. Use it when the user asks to plan, design, or break down a task, in any phrasing.
---
# Make a Plan
Act as a senior software engineer: research the subject of this session in depth and
produce a well-grounded, actionable implementation plan.
If the running agent cannot write (for example, the plan agent), say so and
stop — this skill needs the build agent to save the plan.
## When to use
- The user asks to plan, design, or break down a task, in any phrasing:
"make a plan", "how would we build X", "design an approach for Y" —
or runs `/make-a-plan`.
- The user asks to rework or extend an existing plan (for example, after
review findings) — revise the saved plan file in place.
Do not use it to execute a plan — that is the `implement-plan` flow.
## Instructions
1. **Produce the plan** with the `planner` skill. By default, research the
subject of this session and draft the plan yourself. If I ask for it (for
example, `delegated` in the user context), delegate to the `general` subagent
instead — the delegate must also follow the `planner` skill and receive all
the relevant session context (a review, user feedback, and so on).
2. Before asking me to decide anything, explain the plan and every open question in
plain language. Assume I know only the high-level project goal, not the codebase,
architecture, implementation terms, or the problem this task solves.
3. Once all decisions are answered and the plan is final, save it verbatim to the
announced path under `.agents/plans/` (create the directory if it does not
exist). This step is the flow's explicit authorization to write the plan
file — the only write allowed here. If I later ask for changes, update the
saved file directly.
4. Present me with a clear, self-contained summary of the plan's most relevant points
only after all required decisions have been answered. Write it for someone who knows
only the project's high-level goal and may not know the plan's low-level context.
Explain necessary technical language in plain terms, include the problem being
solved and the proposed outcome, and do not assume that listing technical task names
is enough.
### Hard rule — read-only while planning
While this flow runs, act read-only: research with read-only tools only.
Never edit source files, never run builds, tests, linters, or any command that
modifies state, and never commit. The single allowed write is the plan file in
step 3. This rule expires when I approve the plan or move on to another task;
then you act as a normal build agent again.
When the plan contains open questions, do not show them as bare technical questions or
assume that I understand the technical language or technical words used in the plan.
For each question, first explain:
- What part of the user problem the decision affects.
- The relevant concept from the beginning, with a small concrete example.
- What each available option would make the system do.
- The practical benefits, costs, risks, and user-visible consequences of each option.
- Which option the planner recommends and why.
Only after that explanation, use the `question` tool to ask the decision with clear,
non-technical option labels. Put the recommended option first and mark it as
`(Recommended)`. Group related questions when their context is shared, but do not ask a
question whose meaning has not already been explained.
If I say that I do not understand a question or its choices, do not treat my previous
answer as valid. Explain the concepts again from the high-level project goal, use a more
concrete example, explain the implications, and ask the question again with the
`question` tool. Repeat this until I can make an informed choice. If one answer creates
new design consequences or additional decisions, explain those consequences before
asking any new question.
Distinguish clearly between requirements already fixed by the roadmap or existing
architecture and choices that actually require my input. Do not ask me to choose an
implementation detail when the plan can resolve it safely without changing the public
behavior. If there are no decisions that require my input, say so and present the
summary.
IMPORTANT: **Under no circumstances execute the plan. Wait for the user to review it
after all possible questions have been answered.** The final summary must explain the
problem being solved, the proposed behavior, the main user-visible workflow, important
constraints and risks, what is deliberately out of scope, and the path where the plan
is saved. Never assume that a short list of task names is enough context. End
the final response by suggesting the next steps, in this order:
1. `/review-plan` — to get a second opinion on the plan before executing it.
2. `/implement-plan` — to execute the plan from the current session context.
These are suggestions, not a required pipeline — any instruction from me
overrides them (for example, asking you to implement the plan directly).
## User context
Extra context in the user's invocation (the message that triggered this skill)
plays the role command arguments play elsewhere: for example, `delegated` to
hand the research and drafting to the `general` subagent, or corrections and
feedback about a previous plan.
-205
View File
@@ -1,205 +0,0 @@
---
name: planner
description: Read-only planning and architecture analysis — produce a structured implementation plan with task breakdown, acceptance criteria, sizing, and checkpoints. Always output to the user with the plan, suggested save path and the next steps.
---
# Planner
Produce a plan that another engineer or agent can execute without guessing.
## When to Use
- The user asks for a plan, design, or analysis of a feature or bug.
- The user wants to understand which parts of the codebase a task will touch.
- The user needs a step-by-step implementation plan with file paths, function
names, and test strategy.
- The user asks "how would I implement X?" or "what's involved in fixing Y?".
- The user is about to start non-trivial work and wants a bite-sized task
breakdown.
- A task feels too large or vague to start.
- Work needs to be parallelized across multiple agents or sessions.
Do not use for a small change with obvious scope or an existing executable plan.
## CRITICAL: Required Reading Before Planning
Before drafting any plan, work through the project's own guidance:
1. Read `critical-info` (`.serena/memories/critical-info.md`) — the entry point
that describes the monorepo structure and module dependency graph.
2. From `critical-info`, identify which modules your task affects.
3. Read each affected module's core memory, e.g. `mem:frontend/core`,
`mem:backend/core`, `mem:common/core`, `mem:exporter/core`,
`mem:render-wasm/core`. Follow `mem:` references deeper as needed.
4. For each affected module, note its lint, format, and test commands so the
plan can include concrete verification steps.
Skipping this step is the #1 cause of incorrect or incomplete plans.
## Constraints
- You are **analysis-only** — never create, edit, or delete source code. The
only file you may write is the plan itself, and only when the command or
user explicitly instructs you to save it.
- You do **not** run builds, tests, linters, or any commands that modify state.
- You do **not** create git commits or interact with version control.
- You do **not** execute shell commands beyond read-only searches (`rg`, `ls`,
`find`, `cat`, `bat`).
- Your output is a structured plan or analysis, ready for handoff to an
engineer agent or developer.
## Planning Process
1. Define the problem, desired outcome, constraints, and exclusions.
2. Trace the current behavior through the affected modules.
3. Map dependencies and choose an implementation order that builds foundations
before their consumers.
4. Identify open product or architecture decisions. Resolve implementation
details from existing conventions when they do not affect public behavior.
5. Identify edge cases, security and data risks, performance bounds, breaking
changes, and external dependencies.
6. Split the work into small, ordered tasks. Prefer complete testable slices
over unrelated layer-wide batches. Apply DRY and KISS to the proposed
implementation.
7. Define exact acceptance criteria and verification for every task.
8. Add a checkpoint after every two or three tasks in a longer plan.
9. State which tasks can run in parallel and which must remain sequential.
## Task Format
Each task follows this structure:
```markdown
## Task [N]: [Short descriptive title]
**Description:** One or two paragraphs explaining what this task accomplishes.
Should be clear and concise.
**Rationale:** Why this task exists and why this approach over the obvious
alternatives — design decisions, trade-offs, constraints discovered during
analysis. One or two sentences; skip only if genuinely trivial.
**Code sketch (optional):** Signature-, type-, or shape-level example when the
intended interface is non-obvious. Keep it short — a skeleton that fixes the
contract (function signature, model fields, error shape), never a full
implementation. Omit when the task is mechanical.
**Acceptance criteria:**
- [ ] [Specific, testable condition]
- [ ] [Specific, testable condition]
**Verification:**
- [ ] Relevant tests pass (module-specific test command).
- [ ] Lint/formatter passes (module-specific check command), if applicable.
- [ ] The core flow works end-to-end, if applicable.
**Dependencies:** [Task numbers this depends on, or "None"]
**Files likely touched:**
- `path/to/file.clj`
- `path/to/file_test.clj`
**Estimated scope:** [XS: 1 file | S: 1-2 files | M: 3-5 files | L: 5+ files]
```
Use commands from `mem:testing` and affected module memories. Never substitute
generic text such as "run the tests" when the project documents an exact
command.
When possible, design each task with TDD in mind: acceptance criteria double as a test
list, and the natural first step of the task is writing those tests before the
implementation. Some tasks resist this (config, migrations, pure wiring) — for those, keep
the usual verification steps.
## Task Sizing
| Size | Files | Scope | Example |
|------|-------|-------|---------|
| **XS** | 1 | Single function, config change, or schema tweak | Add a validation rule |
| **S** | 1-2 | One handler or component method | Add a new RPC endpoint |
| **M** | 3-5 | One vertical feature slice | Bookmark CRUD with tests |
| **L** | 5-8 | Multi-component feature | Search with filtering and pagination |
| **XL** | 8+ | **Too large — break it down further** | — |
Split a task when it contains independent outcomes, spans unrelated systems, or cannot be
completed and verified in one focused session (if a task is XL, it should be broken into
smaller tasks; agents perform best on S and M tasks).
## Task order and checkpoints
Arrange tasks so that:
1. Dependencies are satisfied (build foundation first)
2. Each task leaves the system in a working state
3. Verification checkpoints occur after every 2-3 tasks
4. High-risk tasks are early (fail fast)
Add explicit checkpoints with the relevant module commands:
```markdown
### Checkpoint: After Tasks 1-3
- [ ] Relevant tests pass (module-specific command).
- [ ] The relevant build or compilation passes, if applicable.
- [ ] The core flow works end-to-end.
```
## Output Format
The plan is always delivered in the response so the user sees it regardless
of which agent is running the skill. File writes follow `Constraints`
by default announce the path instead of writing.
Announce the save path `.agents/plans/YYYY-MM-DD-<slug>.md` (today's date,
lowercase hyphen-separated slug, e.g. `2026-09-10-add-batch-get-profiles`;
an explicit user path wins).
End the response by suggesting the next steps: `/review-plan` to get a second
opinion on the plan and `/implement-plan` to execute it.
### Plan Structure
Use this document shape:
```markdown
# Plan: Title
## Context
## Affected Modules
## Architecture Decisions
## Risks and Considerations
## Approach
## Task List
## Verification and Testing
## Parallelization
## Open Questions
```
Omit empty sections only when they do not apply. Every implementation task
still requires acceptance criteria, verification, dependencies, likely files,
and scope.
When the plan is purely analytical (e.g. a code review or feasibility study
with no implementation), skip the **Approach** and **Task List** sections and
lead with **Findings** instead, keeping the rest of the structure.
## Common Rationalizations
| Rationalization | Reality |
|---|---|
| "I'll figure it out as I go" | That's how you end up with a tangled mess and rework. 10 minutes of planning saves hours. |
| "The tasks are obvious" | Write them down anyway. Explicit tasks surface hidden dependencies and forgotten edge cases. |
| "Planning is overhead" | Planning is the task. Implementation without a plan is just typing. |
| "I can hold it all in my head" | Context windows are finite. Written plans survive session boundaries and compaction. |
## Verification Checklist
Before delivering the plan, confirm:
- [ ] Every task has acceptance criteria
- [ ] Every task has a verification step
- [ ] Task dependencies are identified and ordered correctly
- [ ] No task is XL or larger — break it down instead
- [ ] Checkpoints exist after every 2-3 tasks
- [ ] The response states the plan's path (saved or suggested) and suggests
`/review-plan` and `/implement-plan`
- [ ] The plan is ready for human review
@@ -1,47 +0,0 @@
---
name: resolve-git-conflicts
description: Conflict resolution flow — understand the local git conflicts, present a resolution plan, and resolve them after the user approves it. Never continues the rebase. Use it when the repo has unresolved conflicts (rebase, merge, cherry-pick) or the user asks to resolve them.
---
# Resolve Git Conflicts
Resolve conflicts in the local repository. The user handles finishing the
rebase themselves — you must **never** run `git rebase --continue`,
`git rebase --skip`, `git merge --continue`, or anything similar.
## When to use
- The repository has unresolved conflicts — during a rebase, merge, or
cherry-pick — whether the user asks about them or not.
- The user asks to resolve conflicts, in any phrasing: "fix the merge
conflicts", "resolve these", "what's conflicting here?".
## Phase 1 — Understand the problem (read-only)
1. Run `git status` to detect the conflict state (rebase, merge, cherry-pick, etc.) and list conflicted files.
2. For each conflicted (unmerged) file, understand the situation **without modifying anything**:
- Read the file and identify the conflict markers (`<<<<<<<`, `=======`, `>>>>>>>`).
- Inspect both sides — `git show <ours>:<file>` and `git show <theirs>:<file>` — plus `git log`/`git show` on the commits involved to understand intent.
- Identify what each side changed and why, and how they should be combined.
## Phase 2 — Present the resolution plan
3. **Present a clear plan to the user before touching any file.** For each conflicted file, state:
- What each side changed and why.
- Your proposed resolution and the reasoning behind it.
- How the two sides are combined (both additive → merge; both modify the same code → keep the semantically correct version, merging intent from both sides when clear from code and context).
4. **Ask the user only when genuinely unclear.** Do not ask about anything you can determine yourself from the code, commit messages, or context. Only decisions that are not determinable and change the outcome (e.g. conflicting product decisions, which side to discard) warrant a question. **Collect all such questions together in an "Open Questions" section at the end of the plan**, so the user has full context to answer them properly.
5. **Wait for the user to accept the plan** (and answer any open questions) before editing, staging, or otherwise modifying anything.
## Phase 3 — Execute
6. Resolve each conflicted file by editing the file to the agreed merged content and removing all conflict markers.
## Phase 4 — Stage and verify
7. **Stage every resolved file** with `git add <file>`. Do not stage unrelated untracked files unless clearly part of the resolution.
8. Verify no conflict markers remain (search for `<<<<<<<` / `>>>>>>>` in resolved files) and that `git status` shows no unmerged paths.
## Phase 5 — Report
9. Briefly report the conflict state, how each conflicted file was resolved (and any answers received to open questions), and stop — do **not** run `git rebase --continue` or any other continuation command.
-73
View File
@@ -1,73 +0,0 @@
---
name: review-code
description: Code review flow — review a diff, PR, or code change, delegating the review to a subagent that follows the code-review-criteria skill. Use it when the user asks to review code or a PR, in any phrasing.
---
# Review Code
Act as a senior software engineer and perform a thorough code review.
## When to use
- The user asks to review code, in any phrasing: "review this diff",
"review the PR", "check my changes", "code review" — or runs
`/review-code`.
- A commit, branch, PR, or diff is ready and the user wants it assessed
before merge.
## Instructions
1. **Determine what is being reviewed** from the user context: a working-tree
diff, a commit range, a branch, a PR (number or URL), or specific files. If
the target is ambiguous, ask before reviewing.
2. Delegate the review to the `general` subagent (via the task tool), unless the
user specifies another agent. Include in the prompt the
**`code-review-criteria`** skill name and all user context.
3. When the subagent returns, output the review to the user verbatim. Do not
summarize it and do not act on its findings.
4. Right after the review, suggest how to proceed based on the findings. These
are suggestions — the user decides:
- **Approve (no required changes):** say so — there is nothing to address.
- **Minor findings (nits):** applying them directly as-is is fine once the
review is done — no plan needed.
- **Substantive findings:** suggest `/make-a-plan` to make a plan to address
them.
### Hard rule — read-only while reviewing
This flow is read-only **for the duration of the review**: from the moment it
starts until the user considers the review finished (including any feedback,
questions, or clarifications about it). During that period, never fix,
implement, edit files or create commits — not even "obvious" fixes derived from
the findings. Once the user explicitly states the review is done (or moves on to
a different task), this rule no longer applies and you act as a normal build
agent again.
## Instructions for the subagent
1. Load the **`code-review-criteria`** skill and follow its process and output
format.
2. Read `AGENTS.md` (if present) and follow its instructions for finding and
reading all related testing documentation from memories before reviewing.
3. Return in your final message the COMPLETE review, verbatim, exactly as the
skill instructs it to be produced. Do not summarize it — include the full
structured review.
### Strong rules for the subagent
1. Do not invent problems. Every finding must be real and actionable.
2. Read-only: do not modify any file and do not create a commit — reviewing
never writes.
3. Be specific and constructive. "This could be better" is not helpful — explain
why and how.
4. Prioritize by impact. One structural issue outweighs ten nits.
5. Missing tests are an issue, not a suggestion. Report as a severity-tagged
finding — never as a recommendation.
6. Skip generated files, lockfile-only changes, and unrelated modifications
unless they introduce security risks.
## User context
Extra context in the user's invocation (the message that triggered this skill)
plays the role command arguments play elsewhere: for example, a PR number or
URL, a commit range, specific files, or a different agent to run the review.
-71
View File
@@ -1,71 +0,0 @@
---
name: review-plan
description: Plan review flow — evaluate an implementation plan before it is executed, delegating the review to a subagent that follows the plan-review-criteria skill. Use it when the user asks to review a plan, in any phrasing.
---
# Review Plan
Act as a senior software engineer and perform a thorough review of an
implementation plan.
## When to use
- The user asks to review a plan, in any phrasing: "review this plan",
"does this plan look right?", "second opinion on the plan" — or runs
`/review-plan`.
- A plan was just produced (typically by `/make-a-plan`) and the user
wants it evaluated before executing it.
## Instructions
1. **Determine the plan under review** from the session context (for example, a
plan just produced by `/make-a-plan`) or from a plan file path given by the
user (typically under `.agents/plans/`). If a file path is given, read the
file first so the complete plan is in context.
2. Delegate the review to the `general` subagent (via the task tool), unless the
user specifies another agent. Include in the prompt the
**`plan-review-criteria`** skill name and all user context.
3. When the subagent returns, output the review to the user verbatim. Do not
summarize it and do not act on its findings.
4. Right after the review, suggest the next step based on the verdict. These
are suggestions — the user decides, and any instruction overrides them:
- **Approve** → suggest `/implement-plan` to execute it.
- **Request changes** → suggest `/make-a-plan` to make a plan to address the
findings.
### Hard rule — read-only while reviewing
This flow is read-only **for the duration of the review**: from the moment it
starts until the user considers the review finished (including any feedback,
questions, or clarifications about it). During that period, never fix,
implement, edit files or create commits — not even "obvious" fixes derived from
the findings. Once the user explicitly states the review is done (or moves on to
a different task), this rule no longer applies and you act as a normal build
agent again.
## Instructions for the subagent
1. Load the **`plan-review-criteria`** skill and follow its process and output
format.
2. Read `AGENTS.md` (if present) and follow its instructions for finding and
reading all related documentation and testing memories before reviewing.
3. Return in your final message the COMPLETE review, verbatim, exactly as the
skill instructs it to be produced. Do not summarize it — include the full
structured review.
### Strong rules for the subagent
1. Do not invent problems. Every finding must be real and actionable.
2. Read-only: do not modify any file and do not create a commit — reviewing
never writes.
3. Be specific and constructive. "This could be better" is not helpful — explain
why and how.
4. Prioritize by impact. One structural issue outweighs ten nits.
5. Judge the plan as the implementer would: every task executable without
guessing, ordering follows the dependency graph, risks named.
## User context
Extra context in the user's invocation (the message that triggered this skill)
plays the role command arguments play elsewhere: for example, a plan file path
to review, or a different agent to run the review.
-1
View File
@@ -1 +0,0 @@
../.agents/skills
-44
View File
@@ -1,44 +0,0 @@
name: _ADHOC
run-name: >-
_ADHOC (${{ inputs.gh_ref }}${{ inputs.nitrate_ref != '' && format(' / nitrate:{0}', inputs.nitrate_ref) || '' }})
on:
workflow_dispatch:
inputs:
gh_ref:
description: 'Branch/ref to build in penpot/penpot'
type: string
required: true
nitrate_ref:
description: 'Branch/ref to build admin-console in penpot/penpot-nitrate (defaults to gh_ref)'
type: string
required: false
force:
description: 'Rebuild and overwrite even if already built/promoted'
type: boolean
required: false
default: false
jobs:
build-bundle:
uses: ./.github/workflows/build-bundle.yml
secrets: inherit
with:
gh_ref: ${{ inputs.gh_ref }}
force: ${{ inputs.force }}
build-docker:
needs: build-bundle
uses: ./.github/workflows/build-docker.yml
secrets: inherit
with:
gh_ref: ${{ inputs.gh_ref }}
force: ${{ inputs.force }}
build-docker-admin-console:
uses: ./.github/workflows/build-docker-admin-console.yml
secrets: inherit
with:
gh_ref: ${{ inputs.nitrate_ref || inputs.gh_ref }}
force: ${{ inputs.force }}
+1 -35
View File
@@ -9,11 +9,6 @@ on:
type: string
required: true
default: 'develop'
force:
description: 'Rebuild and overwrite even if this version already exists in S3'
type: boolean
required: false
default: false
workflow_call:
inputs:
gh_ref:
@@ -21,11 +16,6 @@ on:
type: string
required: true
default: 'develop'
force:
description: 'Rebuild and overwrite even if this version already exists in S3'
type: boolean
required: false
default: false
# Literal group name: under `workflow_call`, `github.workflow` resolves to the
# caller's workflow, which put this workflow and the other reusable one called
@@ -44,8 +34,6 @@ jobs:
outputs:
gh_ref: ${{ steps.vars.outputs.gh_ref }}
bundle_version: ${{ steps.vars.outputs.bundle_version }}
sha: ${{ steps.vars.outputs.sha }}
commit_title: ${{ steps.vars.outputs.commit_title }}
exists: ${{ steps.check.outputs.exists }}
steps:
@@ -60,12 +48,10 @@ jobs:
run: |
echo "gh_ref=${{ inputs.gh_ref || github.ref_name }}" >> $GITHUB_OUTPUT
echo "bundle_version=$(git describe --tags --always)" >> $GITHUB_OUTPUT
echo "sha=$(git rev-parse --short=12 HEAD)" >> $GITHUB_OUTPUT
echo "commit_title=$(git log -1 --pretty=%s)" >> $GITHUB_OUTPUT
# The uploaded zip carries its version as S3 metadata. If the
# existing object was already built from this same commit, the
# whole build job is skipped. `force` bypasses this check entirely.
# whole build job is skipped.
- name: Check if this bundle is already built
id: check
env:
@@ -73,16 +59,6 @@ jobs:
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
AWS_DEFAULT_REGION: ${{ secrets.AWS_REGION }}
run: |
if [ "${{ inputs.force }}" = "true" ]; then
echo "exists=false" >> $GITHUB_OUTPUT
{
echo "### 🔁 Bundle build forced"
echo ""
echo "\`force: true\` — skipping the S3 version check."
} >> "$GITHUB_STEP_SUMMARY"
exit 0
fi
EXISTING_VERSION=$(aws s3api head-object \
--bucket ${{ secrets.S3_BUCKET }} \
--key "penpot-${{ steps.vars.outputs.gh_ref }}.zip" \
@@ -141,16 +117,6 @@ jobs:
s3://${{ secrets.S3_BUCKET }}/penpot-${{ needs.check.outputs.gh_ref }}.zip \
--metadata bundle-version=${{ needs.check.outputs.bundle_version }}
- name: Write step summary
run: |
{
echo "### ✅ Bundle built"
echo ""
echo "- Version: \`${{ needs.check.outputs.bundle_version }}\` (\`git describe --tags --always\`)"
echo "- Commit: [\`${{ needs.check.outputs.sha }}\`](https://github.com/${{ github.repository }}/commit/${{ needs.check.outputs.sha }}) — ${{ needs.check.outputs.commit_title }}"
echo "- Built at: $(date -u +'%Y-%m-%d %H:%M:%S UTC')"
} >> "$GITHUB_STEP_SUMMARY"
# ── 3. Single failure notification for the whole workflow ─────────────
notify:
name: Notify failure
-12
View File
@@ -1,16 +1,7 @@
name: _DEVELOP
run-name: >-
_DEVELOP (develop @ ${{ github.sha }})
on:
workflow_dispatch:
inputs:
force:
description: 'Rebuild and overwrite even if already built/promoted'
type: boolean
required: false
default: false
schedule:
- cron: '16 5-20 * * 1-5'
@@ -24,7 +15,6 @@ jobs:
secrets: inherit
with:
gh_ref: "develop"
force: ${{ inputs.force || false }}
build-docker:
needs: build-bundle
@@ -32,11 +22,9 @@ jobs:
secrets: inherit
with:
gh_ref: "develop"
force: ${{ inputs.force || false }}
build-docker-admin-console:
uses: ./.github/workflows/build-docker-admin-console.yml
secrets: inherit
with:
gh_ref: "develop"
force: ${{ inputs.force || false }}
@@ -13,11 +13,6 @@ on:
type: string
required: false
default: 'develop'
force:
description: 'Rebuild and overwrite even if already built'
type: boolean
required: false
default: false
workflow_call:
inputs:
gh_ref:
@@ -29,11 +24,6 @@ on:
type: string
required: false
default: 'develop'
force:
description: 'Rebuild and overwrite even if already built'
type: boolean
required: false
default: false
secrets:
ORG_WORKFLOW_TOKEN:
description: 'Token with Actions write access on penpot-nitrate'
@@ -57,7 +47,6 @@ jobs:
gh workflow run "$WORKFLOW" --repo "$REPO" --ref "$DISPATCH_REF" \
-f gh_ref="$GH_REF" \
-f force="${{ inputs.force }}" \
-f caller_run_id="$DISTINCT_ID" \
-f caller_run_url="$CALLER_URL"
+1 -1
View File
@@ -6,7 +6,7 @@ on:
jobs:
build-and-push:
name: Build and push DevEnv Docker image
runs-on: penpot-standar-runner
runs-on: penpot-extended-runner
steps:
- name: Set common environment variables
+20 -51
View File
@@ -8,11 +8,6 @@ on:
type: string
required: true
default: 'develop'
force:
description: 'Rebuild and overwrite even if this sha is already promoted'
type: boolean
required: false
default: false
workflow_call:
inputs:
gh_ref:
@@ -20,11 +15,6 @@ on:
type: string
required: true
default: 'develop'
force:
description: 'Rebuild and overwrite even if this sha is already promoted'
type: boolean
required: false
default: false
# Literal group name: under `workflow_call`, `github.workflow` resolves to the
# caller's workflow, which put this workflow and the other reusable one called
@@ -46,13 +36,12 @@ jobs:
# ── 1. Resolve the build key and check the whole set at once ───────────
prepare:
name: Prepare
runs-on: penpot-standar-runner
runs-on: penpot-extended-runner
timeout-minutes: 15
outputs:
gh_ref: ${{ steps.vars.outputs.gh_ref }}
bundle_version: ${{ steps.vars.outputs.bundle_version }}
sha: ${{ steps.vars.outputs.sha }}
commit_title: ${{ steps.vars.outputs.commit_title }}
build_key: ${{ steps.vars.outputs.build_key }}
exists: ${{ steps.check.outputs.exists }}
steps:
@@ -70,8 +59,6 @@ jobs:
run: |
GH_REF="${{ inputs.gh_ref || github.ref_name }}"
echo "gh_ref=$GH_REF" >> $GITHUB_OUTPUT
echo "sha=$(git rev-parse --short=12 HEAD)" >> $GITHUB_OUTPUT
echo "commit_title=$(git log -1 --pretty=%s)" >> $GITHUB_OUTPUT
BUNDLE_VERSION=$(aws s3api head-object \
--bucket ${{ secrets.S3_BUCKET }} \
@@ -80,11 +67,15 @@ jobs:
--output text)
echo "bundle_version=$BUNDLE_VERSION" >> $GITHUB_OUTPUT
# Image content = bundle + docker build context, so the build key
# combines both.
CTX_HASH=$(git rev-parse "HEAD:docker/images" | cut -c1-12)
echo "build_key=${BUNDLE_VERSION}-${CTX_HASH}" >> $GITHUB_OUTPUT
# The image set is a single block, so a single set-level check is
# enough: `promote` drops a marker object in S3 only after every
# image was built AND every branch tag was moved. Marker present
# means there is nothing at all to do for this commit. `force`
# bypasses this check entirely.
# means there is nothing at all to do for this build key.
- name: Check if this image set is already built
id: check
env:
@@ -92,30 +83,15 @@ jobs:
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
AWS_DEFAULT_REGION: ${{ secrets.AWS_REGION }}
run: |
if [ "${{ inputs.force }}" = "true" ]; then
echo "exists=false" >> $GITHUB_OUTPUT
mkdir -p "$BUNDLE_CACHE"
find "$BUNDLE_CACHE" -type f -mtime +1 -delete || true
ZIP="$BUNDLE_CACHE/penpot-${{ steps.vars.outputs.bundle_version }}.zip"
aws s3 cp "s3://${{ secrets.S3_BUCKET }}/penpot-${{ steps.vars.outputs.gh_ref }}.zip" "$ZIP.$$.tmp"
mv "$ZIP.$$.tmp" "$ZIP"
{
echo "### 🔁 Image set build forced"
echo ""
echo "\`force: true\` — skipping the S3 marker check."
} >> "$GITHUB_STEP_SUMMARY"
exit 0
fi
if aws s3api head-object \
--bucket ${{ secrets.S3_BUCKET }} \
--key "markers/images-sha-${{ steps.vars.outputs.sha }}" \
--key "markers/images-${{ steps.vars.outputs.build_key }}" \
> /dev/null 2>&1; then
echo "exists=true" >> $GITHUB_OUTPUT
{
echo "### ⏭️ Image set build skipped"
echo ""
echo "The whole set was already built and promoted for \`sha-${{ steps.vars.outputs.sha }}\`."
echo "The whole set was already built and promoted for \`${{ steps.vars.outputs.build_key }}\`."
} >> "$GITHUB_STEP_SUMMARY"
else
echo "exists=false" >> $GITHUB_OUTPUT
@@ -125,7 +101,7 @@ jobs:
# prune stale bundles while at it.
mkdir -p "$BUNDLE_CACHE"
find "$BUNDLE_CACHE" -type f -mtime +1 -delete || true
ZIP="$BUNDLE_CACHE/penpot-${{ steps.vars.outputs.bundle_version }}.zip"
ZIP="$BUNDLE_CACHE/penpot-${{ steps.vars.outputs.build_key }}.zip"
if [ ! -f "$ZIP" ]; then
aws s3 cp "s3://${{ secrets.S3_BUCKET }}/penpot-${{ steps.vars.outputs.gh_ref }}.zip" "$ZIP.$$.tmp"
mv "$ZIP.$$.tmp" "$ZIP"
@@ -135,7 +111,7 @@ jobs:
# ── 2. One build per image, in parallel, only when needed ──────────────
build:
name: Build ${{ matrix.image }}
runs-on: penpot-standar-runner
runs-on: penpot-extended-runner
timeout-minutes: 60
needs: prepare
if: needs.prepare.outputs.exists == 'false'
@@ -166,7 +142,7 @@ jobs:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
# To avoid the "429 Too Many Requests" error when downloading
# To avoid the 429 Too Many Requests error when downloading
# images from DockerHub for unregistered users.
# https://docs.docker.com/docker-hub/usage/
- name: Login to DockerHub Registry
@@ -197,7 +173,7 @@ jobs:
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
AWS_DEFAULT_REGION: ${{ secrets.AWS_REGION }}
run: |
ZIP="$BUNDLE_CACHE/penpot-${{ needs.prepare.outputs.bundle_version }}.zip"
ZIP="$BUNDLE_CACHE/penpot-${{ needs.prepare.outputs.build_key }}.zip"
if [ ! -f "$ZIP" ]; then
echo "Bundle not found in host cache; falling back to S3."
mkdir -p "$BUNDLE_CACHE"
@@ -237,7 +213,7 @@ jobs:
sbom: true
# Immutable tag only; branch tags are moved atomically for the
# whole image set by the `promote` job.
tags: ${{ secrets.DOCKER_REGISTRY }}/${{ matrix.image }}:sha-${{ needs.prepare.outputs.sha }}
tags: ${{ secrets.DOCKER_REGISTRY }}/${{ matrix.image }}:build-${{ needs.prepare.outputs.build_key }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=registry,ref=${{ secrets.DOCKER_REGISTRY }}/${{ matrix.image }}:buildcache
cache-to: type=registry,ref=${{ secrets.DOCKER_REGISTRY }}/${{ matrix.image }}:buildcache,mode=max
@@ -248,7 +224,7 @@ jobs:
# the S3 marker guarantees the branch tags were already moved.
promote:
name: Promote image set
runs-on: penpot-standar-runner
runs-on: penpot-extended-runner
timeout-minutes: 10
needs: [prepare, build]
@@ -273,7 +249,7 @@ jobs:
for image in $ALL_IMAGES; do
docker buildx imagetools create \
-t "${{ secrets.DOCKER_REGISTRY }}/$image:${{ needs.prepare.outputs.gh_ref }}" \
"${{ secrets.DOCKER_REGISTRY }}/$image:sha-${{ needs.prepare.outputs.sha }}"
"${{ secrets.DOCKER_REGISTRY }}/$image:build-${{ needs.prepare.outputs.build_key }}"
done
# The marker is written LAST: its presence certifies that all five
@@ -285,24 +261,17 @@ jobs:
AWS_DEFAULT_REGION: ${{ secrets.AWS_REGION }}
run: |
echo "${{ github.run_id }}" | aws s3 cp - \
"s3://${{ secrets.S3_BUCKET }}/markers/images-sha-${{ needs.prepare.outputs.sha }}"
- name: Write step summary
run: |
"s3://${{ secrets.S3_BUCKET }}/markers/images-${{ needs.prepare.outputs.build_key }}"
{
echo "### ✅ Image set promoted"
echo ""
echo "- Version: \`${{ needs.prepare.outputs.bundle_version }}\` (\`git describe --tags --always\`)"
echo "- Commit: [\`${{ needs.prepare.outputs.sha }}\`](https://github.com/${{ github.repository }}/commit/${{ needs.prepare.outputs.sha }}) — ${{ needs.prepare.outputs.commit_title }}"
echo "- Built at: $(date -u +'%Y-%m-%d %H:%M:%S UTC')"
echo ""
echo "All \`:${{ needs.prepare.outputs.gh_ref }}\` tags now point to \`sha-${{ needs.prepare.outputs.sha }}\`."
echo "All \`:${{ needs.prepare.outputs.gh_ref }}\` tags now point to \`build-${{ needs.prepare.outputs.build_key }}\`."
} >> "$GITHUB_STEP_SUMMARY"
# ── 4. Single failure notification for the whole workflow ─────────────
notify:
name: Notify failure
runs-on: penpot-standar-runner
runs-on: penpot-extended-runner
timeout-minutes: 5
needs: [prepare, build, promote]
if: failure()
-12
View File
@@ -1,16 +1,7 @@
name: _STAGING
run-name: >-
_STAGING (staging)
on:
workflow_dispatch:
inputs:
force:
description: 'Rebuild and overwrite even if already built/promoted'
type: boolean
required: false
default: false
schedule:
- cron: '36 5-20 * * 1-5'
@@ -24,7 +15,6 @@ jobs:
secrets: inherit
with:
gh_ref: "staging"
force: ${{ inputs.force || false }}
build-docker:
needs: build-bundle
@@ -32,11 +22,9 @@ jobs:
secrets: inherit
with:
gh_ref: "staging"
force: ${{ inputs.force || false }}
build-docker-admin-console:
uses: ./.github/workflows/build-docker-admin-console.yml
secrets: inherit
with:
gh_ref: "staging"
force: ${{ inputs.force || false }}
+1 -13
View File
@@ -1,16 +1,7 @@
name: _TAG
run-name: >-
_TAG (${{ github.ref_name }} @ ${{ github.sha }})
on:
workflow_dispatch:
inputs:
force:
description: 'Rebuild and overwrite even if already built/promoted (manual re-releases only)'
type: boolean
required: false
default: false
push:
tags:
- '*'
@@ -27,7 +18,6 @@ jobs:
secrets: inherit
with:
gh_ref: ${{ github.ref_name }}
force: ${{ inputs.force || false }}
build-docker:
needs: build-bundle
@@ -35,18 +25,16 @@ jobs:
secrets: inherit
with:
gh_ref: ${{ github.ref_name }}
force: ${{ inputs.force || false }}
build-docker-admin-console:
uses: ./.github/workflows/build-docker-admin-console.yml
secrets: inherit
with:
gh_ref: ${{ github.ref_name }}
force: ${{ inputs.force || false }}
notify:
name: Notifications
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
needs:
- build-docker
- build-docker-admin-console
+1 -1
View File
@@ -19,7 +19,7 @@ permissions:
jobs:
release:
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
outputs:
version: ${{ steps.vars.outputs.gh_ref }}
release_notes: ${{ steps.extract_release_notes.outputs.release_notes }}
+1 -1
View File
@@ -32,7 +32,7 @@ jobs:
test-exporter:
if: ${{ !github.event.pull_request.draft }}
name: "Exporter Tests"
runs-on: penpot-extended-runner
runs-on: penpot-runner-02
container:
image: penpotapp/devenv:latest
volumes:
+10 -15
View File
@@ -102,14 +102,17 @@ jobs:
if: ${{ !github.event.pull_request.draft }}
name: "Integration Tests (${{ matrix.shard }})"
runs-on: penpot-extended-runner
timeout-minutes: 40
timeout-minutes: ${{ github.base_ref == 'staging' && 60 || 25 }}
needs: build-integration
# TEMPORARY (release stabilization): PRs targeting `staging` run on a
# single serial shard, so new flakes cannot block the release work.
# Remove the `github.base_ref` branch below to restore full parallelism.
strategy:
fail-fast: false
matrix:
shard: ${{ fromJSON(inputs.shards || '[1, 2, 3, 4]') }}
shard: ${{ fromJSON(inputs.shards || (github.base_ref == 'staging' && '[1]' || '[1, 2, 3, 4]')) }}
container:
image: penpotapp/devenv:latest
@@ -150,8 +153,12 @@ jobs:
working-directory: ./frontend
env:
WORKERS: ${{ inputs.workers }}
BASE_REF: ${{ github.base_ref }}
run: |
WORKERS=${WORKERS:-2}
# TEMPORARY (release stabilization): see the note on the matrix above.
if [ -z "$WORKERS" ]; then
if [ "$BASE_REF" = "staging" ]; then WORKERS=1; else WORKERS=2; fi
fi
echo "Running shard ${{ matrix.shard }}/${{ strategy.job-total }} with $WORKERS workers"
pnpm exec playwright test --project default \
--workers="$WORKERS" \
@@ -229,18 +236,6 @@ jobs:
fi
jq -r -f ../.github/scripts/playwright-summary.jq report.json >> "$GITHUB_STEP_SUMMARY"
# Kept for 30 days so flakiness rates can be aggregated across runs
# without scraping job logs.
- name: Upload JSON report
uses: actions/upload-artifact@v7
if: always()
with:
name: integration-json-report
path: frontend/report.json
overwrite: true
if-no-files-found: ignore
retention-days: 30
- name: Upload HTML report
uses: actions/upload-artifact@v7
with:
+1 -2
View File
@@ -98,11 +98,10 @@ opencode.json
/.idea
*.iml
/.claude
/CLAUDE.md
/.playwright-mcp
/.devenv/mcp/
/opencode.json
/.agents/plans
/.opencode/plans
/.opencode/reports
/.opencode/prompts
/.ci-logs
-10
View File
@@ -1,10 +0,0 @@
---
description: Create a PR for the current task branch or update an existing one — loads and follows the create-pr skill
agent: build
---
Load the **`create-pr`** skill and follow it as your only instruction.
## User input, overrides and additional context
$ARGUMENTS
+36 -4
View File
@@ -1,10 +1,42 @@
---
description: Execute a ready plan — task checklist, your confirmation, then all tasks with one commit (default) or step by step with a commit and a pause per task; creates issue + branch when on a base branch, or commits on the current branch with "direct" — loads and follows the implement-plan skill
description: Execute a ready plan end-to-end — create a GitHub issue, branch issue-NNNN, implement the plan, then commit via the create-commit skill
agent: build
---
Load the **`implement-plan`** skill and follow it as your only instruction.
# Implement Plan
## User input, overrides and additional context
This command is run once a plan is ready (for example, from plan mode). Execute
the plan already prepared in the current session context — it does not take
extra arguments. Follow these steps in order.
$ARGUMENTS
## 1. Create the issue
Use the **`create-issue`** skill, following the *Creating Issues from Draft Body*
flow in `mem:workflow/creating-issues`. Derive the issue title and body from the
plan. Capture the new issue's number — call it **NNNN** (needed for the branch
name and the commit reference).
## 2. Create the branch
Create and switch to a branch named after the issue:
```
git checkout -b issue-NNNN
```
(Replace NNNN with the issue number from step 1.)
## 3. Execute the plan
Implement the prepared plan from the session context. Work methodically, keeping
changes focused on what the issue requires. Do not commit — the commit happens in
step 4.
## 4. Commit with the create-commit skill
After the implementation is complete, load the **`create-commit`** skill and
follow its workflow to commit the changes. Provide a brief summary of what was
implemented and why, the issue reference (`issue-NNNN`), and the model name you
are running as so the `AI-assisted-by` trailer is set correctly.
Do not push. Pushing is handled separately by the user.
-10
View File
@@ -1,10 +0,0 @@
---
description: Investigate the chosen task, produce an implementation plan, and save it — loads and follows the make-a-plan skill
agent: build
---
Load the **`make-a-plan`** skill and follow it as your only instruction.
## User input, overrides and additional context
$ARGUMENTS
+36 -2
View File
@@ -1,6 +1,40 @@
---
description: Resolve local git conflicts and stage the resolved files; never continues the rebase — loads and follows the resolve-git-conflicts skill
description: Resolve local git conflicts and stage the resolved files with git add — never continues the rebase
agent: build
---
Load the **`resolve-git-conflicts`** skill and follow it as your only instruction.
# Fix Git Conflicts
Resolve conflicts in the local repository. The user handles finishing the
rebase themselves — you must **never** run `git rebase --continue`,
`git rebase --skip`, `git merge --continue`, or anything similar.
## Phase 1 — Understand the problem (read-only)
1. Run `git status` to detect the conflict state (rebase, merge, cherry-pick, etc.) and list conflicted files.
2. For each conflicted (unmerged) file, understand the situation **without modifying anything**:
- Read the file and identify the conflict markers (`<<<<<<<`, `=======`, `>>>>>>>`).
- Inspect both sides — `git show <ours>:<file>` and `git show <theirs>:<file>` — plus `git log`/`git show` on the commits involved to understand intent.
- Identify what each side changed and why, and how they should be combined.
## Phase 2 — Present the resolution plan
3. **Present a clear plan to the user before touching any file.** For each conflicted file, state:
- What each side changed and why.
- Your proposed resolution and the reasoning behind it.
- How the two sides are combined (both additive → merge; both modify the same code → keep the semantically correct version, merging intent from both sides when clear from code and context).
4. **Ask the user only when genuinely unclear.** Do not ask about anything you can determine yourself from the code, commit messages, or context. Only decisions that are not determinable and change the outcome (e.g. conflicting product decisions, which side to discard) warrant a question. **Collect all such questions together in an "Open Questions" section at the end of the plan**, so the user has full context to answer them properly.
5. **Wait for the user to accept the plan** (and answer any open questions) before editing, staging, or otherwise modifying anything.
## Phase 3 — Execute
6. Resolve each conflicted file by editing the file to the agreed merged content and removing all conflict markers.
## Phase 4 — Stage and verify
7. **Stage every resolved file** with `git add <file>`. Do not stage unrelated untracked files unless clearly part of the resolution.
8. Verify no conflict markers remain (search for `<<<<<<<` / `>>>>>>>` in resolved files) and that `git status` shows no unmerged paths.
## Phase 5 — Report
9. Briefly report the conflict state, how each conflicted file was resolved (and any answers received to open questions), and stop — do **not** run `git rebase --continue` or any other continuation command.
-10
View File
@@ -1,10 +0,0 @@
---
description: Code review — review a diff, PR, or code change — loads and follows the review-code skill
agent: build
---
Load the **`review-code`** skill and follow it as your only instruction.
## User input, overrides and additional context
$ARGUMENTS
-10
View File
@@ -1,10 +0,0 @@
---
description: Plan review — evaluate an implementation plan before executing it — loads and follows the review-plan skill
agent: build
---
Load the **`review-plan`** skill and follow it as your only instruction.
## User input, overrides and additional context
$ARGUMENTS
+25
View File
@@ -0,0 +1,25 @@
Act as a senior software engineer and perform a thorough review.
## Instructions
1. **Determine what is being reviewed** from the provided context:
- **If it is a plan** (implementation plan, design document, task breakdown) → load the **`plan-review`** skill.
- **If it is code** (diff, PR, code change) → load the **`code-review`** skill.
2. Read `AGENTS.md` and follow its instructions for finding and reading all related testing documentation from memories before reviewing.
3. **Skip generated files, lockfile-only changes, and unrelated modifications** unless they introduce security risks.
4. Follow the loaded skill's process and produce its output format.
## Strong Rules
1. Do not invent problems. Every finding must be real and actionable.
2. Do not modify any code and do not create a commit — this command only reviews.
3. Be specific and constructive. "This could be better" is not helpful — explain why and how.
4. Prioritize by impact. One structural issue outweighs ten nits.
5. Missing tests are an issue, not a suggestion. Report as a severity-tagged finding — never as a recommendation.
## Context
$ARGUMENTS
@@ -9,11 +9,6 @@ metadata: {"clawdbot":{"emoji":"🦇","requires":{"bins":["bat"]},"install":[{"i
`cat` with syntax highlighting, line numbers, and Git integration.
## When to use
- Reading or displaying a file in the terminal — prefer it over plain
`cat`: syntax highlighting, line numbers, git-side indicators.
## Quick Start
### Basic usage
@@ -1,9 +1,9 @@
---
name: code-review-criteria
description: Code review criteria — the five review axes, core principles, severity format, and verdict for reviewing code changes. Loaded by the reviewer subagent of the review-code flow. Not a user-facing flow — to review code, use the review-code flow.
name: code-review
description: Conducts multi-axis code review. Use before merging any change. Use when reviewing code written by yourself, another agent, or a human. Use when you need to assess code quality across multiple dimensions before it enters the main branch.
---
# Code Review Criteria and Quality
# Code Review and Quality
## Overview
@@ -13,10 +13,11 @@ Multi-dimensional code review with quality gates. Every change gets reviewed bef
## When to Use
- The reviewer subagent of the `review-code` flow loads this skill to perform
the review of a code change.
- To review code, always go through the `review-code` flow — never load this
skill directly for that. This is the criteria reference, not the flow.
- Before merging any PR or change
- After completing a feature implementation
- When another agent or model produced code you need to evaluate
- When refactoring existing code
- After any bug fix (review both the fix and the regression test)
## Core Principles
@@ -105,8 +106,6 @@ For detailed security guidance, see `security-and-hardening`.
| **Low:** | Minor, optional | Author may ignore — formatting, style preferences |
| **Suggestion:** | Worth considering | Not required, but improves the code |
**Unique finding IDs.** Assign every finding a stable identifier: `F1`, `F2`, `F3`, … numbered in order of severity (Critical first, then High, Medium, Low, Suggestion). Use the ID everywhere the finding is mentioned — in section headers, in the verdict, in follow-up discussion. Never renumber within a review. Example: `**F3 (High)** — `app/validate.cljs:42` — duplicate branch logic…`.
For each finding, describe the circumstances under which it could fail: specific inputs, load conditions, timing, or user actions that trigger the problem. "This crashes when input is null" is actionable; "this might crash" is not.
Lead with what matters: correctness and security first, then structural issues, then everything else. A few high-conviction comments beat a long list.
@@ -123,11 +122,11 @@ Briefly explain what the code does and give an overall assessment.
### Critical and High-Priority Issues
List problems that could cause security incidents, data loss, crashes, incorrect behavior, or major performance degradation. Each finding gets its unique ID (`F1`, `F2`, …). For each: state the severity, identify the file/function/code section, explain why it's a problem, describe failure circumstances, and provide a concrete improvement with corrected code when useful.
List problems that could cause security incidents, data loss, crashes, incorrect behavior, or major performance degradation. For each: state the severity, identify the file/function/code section, explain why it's a problem, describe failure circumstances, and provide a concrete improvement with corrected code when useful.
### Other Findings
List medium- and low-priority issues, including maintainability and design concerns. Continue the ID sequence started above (`F3`, `F4`, …).
List medium- and low-priority issues, including maintainability and design concerns.
### Suggested Refactoring
@@ -149,8 +148,6 @@ Choose one:
- **Approve with minor changes** — Good to merge after addressing low/medium issues
- **Request changes** — Critical or high issues must be resolved before merge
List the finding IDs the verdict depends on (e.g. "Request changes: F1, F4").
## Change Sizing
Small, focused changes are easier to review, faster to merge, and safer to deploy.
@@ -234,13 +231,25 @@ For supply-chain risk triage, follow the `security-and-hardening` skill.
## Verification
Before emitting the verdict, verify the change as it stands. This is the reviewer's own due diligence — it covers the state of the code at review time, not the later resolution of findings (fixing findings is the author's job; confirming them is a new review):
After review is complete:
- [ ] Tests pass — run them yourself, don't trust the claim
- [ ] All Critical issues are resolved
- [ ] All Required (no-prefix) changes are resolved or explicitly deferred with justification
- [ ] Tests pass
- [ ] Build succeeds
- [ ] The verification story is documented (what changed, how it was verified)
- [ ] Dependency upgrades reviewed against changelog, isolated per package, verified by green suite
## Multi-Model Review Pattern
Use different models for different review perspectives:
```
Model A writes the code → Model B reviews → Model A addresses feedback → Human makes the final call
```
Different models have different blind spots.
## See Also
- For detailed security review guidance, see `security-and-hardening`
File renamed without changes.
+39
View File
@@ -0,0 +1,39 @@
---
name: create-pr
description: Create or update a GitHub PR following Penpot conventions.
---
# Skill: create-pr
Create or update a GitHub PR. Read and follow:
- `mem:workflow/creating-prs` — title format, description structure, writing principles
- `mem:workflow/creating-commits` — commit type emojis
## When to Use
- Creating a new PR from a feature branch
- Updating an existing PR's title or description to match conventions
## Prerequisites
- `gh` CLI authenticated (`gh auth status`)
## Commands
**Create:**
```bash
gh pr create --repo penpot/penpot --title "<TITLE>" --body-file /tmp/pr-body.md
```
**Update:**
```bash
gh pr edit <NUMBER> --repo penpot/penpot --title "<TITLE>" --body-file /tmp/pr-body.md
```
**Verify:**
```bash
gh pr view <NUMBER> --repo penpot/penpot --json title,body
```
@@ -9,11 +9,6 @@ metadata: {"clawdbot":{"emoji":"📂","requires":{"bins":["fd"]},"install":[{"id
User-friendly alternative to `find` with smart defaults.
## When to use
- Locating files or directories by name or pattern — prefer it over
plain `find`: simpler syntax, smart defaults, respects `.gitignore`.
## Quick Start
### Basic search
@@ -9,11 +9,6 @@ metadata: {"clawdbot":{"emoji":"🔍","requires":{"bins":["jq"]},"install":[{"id
Process, filter, and transform JSON data with jq.
## When to use
- Parsing, filtering, or transforming JSON from commands, files, or API
responses — slicing, reshaping, or validating JSON output.
## Quick Examples
### Basic filtering
@@ -10,12 +10,6 @@ Evaluate Clojure (or ClojureScript) code via a running nREPL server using
Full documentation: `mem:scripts/nrepl-eval` (file: `.serena/memories/scripts/nrepl-eval.md`)
## When to use
- Evaluating Clojure or ClojureScript code against the running nREPL
sessions (backend 6064, frontend 3447) — live inspection, patching, or
debugging.
## Quick Reference
```bash
@@ -1,9 +1,9 @@
---
name: plan-review-criteria
description: Plan review criteria — the six review axes, severity rubric, approval standard, and output format for reviewing implementation plans. Loaded by the reviewer subagent of the review-plan flow. Not a user-facing flow — to review a plan, use the review-plan flow.
name: plan-review
description: Reviews implementation plans for quality, completeness, and actionability. Use after a plan is produced by the planner skill, before starting implementation. Use when evaluating a plan written by yourself, another agent, or a human.
---
# Plan Review Criteria
# Plan Review
## Overview
@@ -13,10 +13,10 @@ Multi-dimensional plan review with quality gates. Every plan gets reviewed befor
## When to Use
- The reviewer subagent of the `review-plan` flow loads this skill to perform
the review of a plan.
- To review a plan, always go through the `review-plan` flow — never load this
skill directly for that. This is the criteria reference, not the flow.
- After the planner skill produces a plan
- Before starting implementation on any non-trivial task
- When reviewing a plan written by another agent or a human
- When a plan feels too large, vague, or risky to start
**Do NOT use for:** Single-file changes with obvious scope, or when the task is trivial enough to just do.
@@ -87,7 +87,7 @@ Can an implementer actually execute this?
### 6. Proposed Code Quality *(when the plan includes implementation details)*
If the plan proposes code shapes, function signatures, data structures, or API designs, evaluate those proposals against `code-review-criteria`:
If the plan proposes code shapes, function signatures, data structures, or API designs, evaluate those proposals against `code-review` criteria:
- **Correctness:** Do the proposed types/signatures handle edge cases (null, empty, boundaries)?
- **Readability:** Are proposed names descriptive and consistent with project conventions?
@@ -215,7 +215,7 @@ Check that the plan can actually confirm it worked:
If the plan includes code snippets, types, or API designs:
```
- Load code-review-criteria skill for criteria
- Load code-review skill for criteria
- Check proposed signatures for edge cases
- Verify naming follows project conventions
- Confirm abstractions follow existing patterns
@@ -310,6 +310,6 @@ If the plan includes code snippets, types, or API designs:
## See Also
- For producing plans, use the `planner` skill
- For reviewing implemented code, use `code-review-criteria` — also the criteria source for axis 6
- For reviewing implemented code, use `code-review` — also the criteria source for axis 6
- For security-specific concerns, see `security-and-hardening`
- For testing strategy guidance, see `testing`
+271
View File
@@ -0,0 +1,271 @@
---
name: planner
description: Read-only planning and architecture analysis for Penpot — produce a structured implementation plan (Context, Affected modules, Approach, Risks, Testing). Always output to the user; additionally save to .opencode/plans/YYYY-MM-DD-<title>.md.
---
# Planner
Read-only senior software architect role for Penpot. Produces structured
implementation plans that engineers or other agents can execute. Never writes
or modifies code.
## When to Use
- The user asks for a plan, design, or analysis of a feature or bug.
- The user wants to understand which parts of the codebase a task will touch.
- The user needs a step-by-step implementation plan with file paths, function
names, and test strategy.
- The user asks "how would I implement X?" or "what's involved in fixing Y?".
- The user is about to start non-trivial work and wants a bite-sized task
breakdown.
Do **not** use this skill to actually implement anything — it is read-only.
## Role
You are a Senior Software Architect working on Penpot, an open-source design
tool. Your sole responsibility is planning and analysis — you do NOT write or
modify code.
You help users understand the codebase, design solutions, and create detailed
implementation plans that other agents or developers can execute. Document
everything they need to know: which files to touch for each task, code patterns,
tests, and how to verify correctness. Apply DRY and KISS principles.
Do **not** suggest commit messages or commit names anywhere in your plans or
responses — committing is the developer's responsibility.
## Required Reading Before Planning
Before drafting any plan, work through the project's own guidance:
1. Read `critical-info` (`.serena/memories/critical-info.md`) — the entry point
that describes the monorepo structure and module dependency graph.
2. From `critical-info`, identify which modules your task affects.
3. Read each affected module's core memory, e.g. `mem:frontend/core`,
`mem:backend/core`, `mem:common/core`, `mem:exporter/core`,
`mem:render-wasm/core`. Follow `mem:` references deeper as needed.
4. For each affected module, note its lint, format, and test commands so the
plan can include concrete verification steps.
Skipping this step is the #1 cause of incorrect or incomplete plans.
## The Planning Process
### Phase 1: Architecture Analysis
1. Read the spec, requirements, or feature request.
2. Analyze the codebase architecture and identify affected modules.
3. Read project conventions (starting with `critical-info` and module core
memories) before drafting.
4. Map dependencies between components (see the dependency graph in
`critical-info`).
5. Identify risks, edge cases, performance implications, and breaking changes.
### Phase 2: Task Breakdown
Implementation order follows the monorepo's dependency graph:
`frontend -> common`, `backend -> common`, `exporter -> common`,
`frontend -> render-wasm`. Build shared foundations first, then layer
consumers on top.
#### Slice Vertically
Instead of building all of common, then all of backend, then all of frontend —
build one complete feature path at a time:
```
Task 1: common data types + schema ← foundation
Task 2: backend RPC handler + persistence
Task 3: frontend UI component + API integration
```
Each vertical slice delivers working, testable functionality.
#### Write Tasks
Each task follows this structure:
```markdown
## Task [N]: [Short descriptive title]
**Description:** One paragraph explaining what this task accomplishes.
**Acceptance criteria:**
- [ ] [Specific, testable condition]
- [ ] [Specific, testable condition]
**Verification:**
- [ ] Tests pass (module-specific test command)
- [ ] Lint/formatter passes (module-specific check command)
**Dependencies:** [Task numbers this depends on, or "None"]
**Files likely touched:**
- `path/to/file.clj`
- `path/to/file_test.clj`
```
Replace "module-specific test command" with the actual commands for the module
(e.g. `clojure -M:dev:test` for backend/common, `npx shadow-cljs compile test && npx karma start` for frontend,
or the commands noted in the module's core memory).
#### Estimate Scope
| Size | Files | Scope |
|------|-------|-------|
| **XS** | 1 | Single function, config change, or schema tweak |
| **S** | 1-2 | One handler or component method |
| **M** | 3-5 | One vertical feature slice |
| **L** | 5-8 | Multi-component feature |
| **XL** | 8+ | **Too large — break it down further** |
If a task is L or larger, break it into smaller tasks. Agents perform best on
S and M tasks.
**When to break a task down further:**
- It would take more than one focused session
- You cannot describe the acceptance criteria in 3 or fewer bullet points
- It touches two or more independent subsystems
- You find yourself writing "and" in the task title (a sign it is two tasks)
#### Order and Checkpoints
Arrange tasks so that:
1. Dependencies are satisfied (build foundation first)
2. Each task leaves the system in a working state
3. Verification checkpoints occur after every 2-3 tasks
4. High-risk tasks are early (fail fast)
Add explicit checkpoints with the relevant module commands:
```markdown
## Checkpoint: After Tasks 1-3
- [ ] All tests pass (module-specific command)
- [ ] Lint/format passes (module-specific command)
- [ ] Core flow works end-to-end
- [ ] Review with human before proceeding
```
## Requirements
- Analyze the codebase architecture and identify affected modules.
- Read project conventions before drafting (start with `critical-info` and
affected module core memories).
- Break down complex features or bugs into atomic, actionable steps.
- Propose solutions with clear rationale, trade-offs, and sequencing.
- Identify risks, edge cases, performance implications, and breaking changes.
- Apply DRY and KISS principles to the proposed implementation.
- Define a testing strategy aligned with each affected module's tooling.
- Every task must have acceptance criteria and verification steps.
- Checkpoints must exist between major phases.
## Constraints
- You are **analysis-only** — never create, edit, or delete source code.
- The only file write you may attempt is the plan itself, saved to
`.opencode/plans/`.
- You do **not** run builds, tests, linters, or any commands that modify state.
- You do **not** create git commits or interact with version control.
- You do **not** execute shell commands beyond read-only searches.
- Your output is a structured plan or analysis, ready for handoff to an
engineer agent or developer.
## Output Format
The plan is always delivered in the response so the user sees it regardless
of which agent is running the skill.
Additionally, save the plan to:
```
.opencode/plans/YYYY-MM-DD-<plan-one-line-title>.md
```
Use today's date in the user's local timezone. The `<plan-one-line-title>`
slug is lowercase, hyphen-separated, and a short summary of the task
(e.g. `add-batch-get-profiles-for-file-comments`). Create the
`.opencode/plans/` directory if it does not exist.
Always attempt the write. If the user explicitly provides a target file path,
use that path instead of the default.
### Plan Document Template
```markdown
# Plan: [Feature/Project Name]
## Context
[One paragraph: what is the problem or feature request? Why is it needed?]
## Affected Modules
[Which modules of the monorepo are involved? Reference module paths and any
`mem:` memories that were consulted.]
## Architecture Decisions
- [Key decision 1 and rationale]
- [Key decision 2 and rationale]
## Risks & Considerations
[Edge cases, performance implications, breaking changes, migration concerns,
security implications.]
## Approach
[Step-by-step implementation plan with file paths, function names, and code
shape where applicable. Group steps into atomic, ordered tasks.]
## Task List
### Phase 1: Foundation
- [ ] Task 1: ...
- [ ] Task 2: ...
### Checkpoint: Phase 1
- [ ] Tests pass, lint/formatter clean (module-specific commands)
### Phase 2: Core Features
- [ ] Task 3: ...
- [ ] Task 4: ...
### Checkpoint: Phase 2
- [ ] End-to-end flow works
### Phase 3: Polish
- [ ] Task 5: ...
- [ ] Task 6: ...
### Checkpoint: Complete
- [ ] All acceptance criteria met
- [ ] Ready for review
## Testing Strategy
[How to verify: which test commands to run per module, what cases to cover,
manual verification steps, lint/format checks. Consult each module's core
memory for the exact commands.]
## Parallelization Opportunities
- **Safe to parallelize:** Independent feature slices across separate
modules, tests for already-implemented features
- **Must be sequential:** Shared common schema changes, database migrations
- **Needs coordination:** Features that share a contract (define the contract
first, then parallelize)
## Open Questions
- [Question needing human input]
```
When the plan is purely analytical (e.g. a code review or feasibility study
with no implementation), skip the **Approach** and **Task List** sections and
lead with **Findings** instead, keeping the rest of the structure.
## Verification Checklist
Before starting implementation, confirm:
- [ ] Every task has acceptance criteria
- [ ] Every task has a verification step
- [ ] Task dependencies are identified and ordered correctly
- [ ] No task touches more than ~5 files
- [ ] Checkpoints exist between major phases
- [ ] The human has reviewed and approved the plan
@@ -9,11 +9,6 @@ metadata: {"clawdbot":{"emoji":"🔎","requires":{"bins":["rg"]},"install":[{"id
Fast, smart recursive search. Respects `.gitignore` by default.
## When to use
- Searching file contents across the repo for regex patterns — the
default code search, respects `.gitignore`.
## Quick Start
### Basic search
@@ -9,13 +9,6 @@ Apply the ASD-STE100 standard to all prose you produce in this task. Do not anno
Compliance note (for you, not for output): the official specification and its dictionary are copyright ASD. This skill encodes paraphrased rules and a publicly sourced word list. For certified aerospace/defense deliverables, tell the user that full compliance requires the free official specification (asd-ste100.org) and a human sign-off. Never claim certified compliance.
## When to use
Only when the user explicitly invokes it: they type `/ste`, or say "use
the ste skill" / "apply ASD-STE100". Requests like "simplify this",
"make it clearer", or "shorter sentences" do NOT invoke it — respond
normally unless it is named.
## Step 0 — Classify the text
Before writing a single sentence, decide: is this **procedural** text (instructions someone follows) or **descriptive** text (explanation, background, description)? Every limit below depends on this. Mixed documents get classified section by section.
@@ -11,12 +11,6 @@ Fetch information from Taiga public API for the **Penpot** project
**No authentication required** — only public project data is accessed.
## When to use
- The user asks about Penpot issues, user stories, or tasks tracked in
Taiga — fetch them via the public API (project id 345963), no
authentication needed.
## Prerequisites
- `python3` — the `scripts/taiga.py` CLI script is self-contained (stdlib only)
File renamed without changes.
@@ -357,39 +357,6 @@ Insert the new version section right after the `# CHANGELOG` header (before
the previous version entry). Use the `edit` tool with enough context to make
a unique match.
### 8b. Propose and populate the `:rocket: Epics and highlights` subsection
After inserting the version section, proactively create or populate the
`### :rocket: Epics and highlights` subsection. This section surfaces the
most impactful changes for self-hosted users checking for updates.
**When to create:** If the version section does not already have a
`### :rocket: Epics and highlights` subsection, create one. Place it before
`### :sparkles:` (matching existing order in CHANGES.md).
**How to identify highlights:** Review the `:sparkles:` entries for the
version and select 25 of the most impactful/user-visible ones. Criteria:
- New user-visible features (not internal refactors)
- Significant capability additions
- Items that create "FOMO" for self-hosted users on older versions
**Use release notes as hints:** Check
`frontend/src/app/main/ui/releases/v2_<MINOR>.cljs` for the corresponding
version. The slide titles and feature descriptions there are curated
marketing content indicating what the team considers highlight-worthy. Match
those themes to changelog entries. Treat these files as optional hints — they
may not exist for every version.
**Format requirement:** Every `:rocket:` entry MUST follow the standard
changelog format with issue/PR references:
```
- <description> [#<ISSUE>](https://github.com/penpot/penpot/issues/<ISSUE>) (PR: [#<PR>](https://github.com/penpot/penpot/pull/<PR>))
```
An entry without issue AND PR references is a highlight gap (warning, not an anomaly).
**Preserve existing entries:** If the `:rocket:` section already exists from
a prior run, preserve its entries. Do not remove or rewrite them.
### 9. Verify
Read the top of `CHANGES.md` and confirm:
@@ -501,30 +468,18 @@ Markdown viewer.
## What is an anomaly
**An anomaly is a milestone-mismatch between an issue and its referenced
PR.** There are two anomaly types, plus two highlight gaps (warnings that
do not count toward the anomaly total):
PR.** It indicates that the changelog claim "this issue is fixed by this PR,
all in milestone M" is inconsistent with the actual milestone assignments.
There are exactly two types:
1. **Issue is in the milestone, but its referenced PR is in a different
milestone (or has no milestone).** The changelog claims a fix in this
release, but the PR is being released elsewhere — the fix may not
actually ship here.
2. **PR is in the milestone, but the issue it closes is in a different
milestone.** The PR is being released here, but the issue it fixes is
being released in a different version — the changelog pairing is
misleading.
**Exception — issue with no milestone is NOT an anomaly.** Milestones
are only required for issues tracked in the "Main" project. A milestone
PR that closes an issue with no milestone references an issue from
another (probably private) project; that is expected and the issue is
not part of this changelog. Do not report it.
3. **missing-highlights (gap):** A released X.Y.0 version section has no
`### :rocket: Epics and highlights` subsection. Patches (X.Y.Z) never
carry highlights, so only minors/majors are checked.
4. **missing-highlight-reference (gap):** A `:rocket:` entry lacks the
required issue AND PR references. Every highlight entry must follow the
standard changelog format with `[#ISSUE]` and `(PR: [#PR])` links
(multi-PR `(PR: [#A](...), [#B](...))` accepted).
milestone (or has no milestone).** The PR is being released here, but
the issue it fixes is being released in a different version (or never
tracked in a milestone) — the changelog pairing is misleading.
**Anything else is not an anomaly.** Other discrepancies (exclusion
labels on in-changelog issues, missing valid issues, unmerged PR
@@ -680,10 +635,6 @@ for pr_num in sorted(changelog_prs):
if get_pr_milestone(pr_num) != MILESTONE: continue
for issue_num in pr.get('closing_issues', []):
issue_ms = get_issue_milestone(issue_num)
# No milestone = issue from another (probably private) project —
# milestones are only required for the "Main" project. Not an
# anomaly, and the issue never belongs in this changelog.
if issue_ms is None: continue
if issue_ms != MILESTONE:
anomalies_b.append({
'pr': pr_num,
@@ -692,40 +643,6 @@ for pr_num in sorted(changelog_prs):
'issue_milestone': issue_ms, # may be None
})
# --- Type C: released X.Y.0 version sections without :rocket: subsection ---
# Patches (X.Y.Z with Z != 0) never carry :rocket: by design — only minors/majors (X.Y.0).
anomalies_c = [] # list of version strings
rocket_heading_re = re.compile(r'^### :rocket:', re.MULTILINE)
version_sections = re.split(r'(?=^## \d+\.\d+\.\d+)', content, flags=re.MULTILINE)
for vs in version_sections:
m = re.match(r'^## (\d+\.\d+\.\d+)(.*)', vs)
if not m: continue
ver, suffix = m.group(1), m.group(2)
if 'unreleased' in suffix.lower(): continue
if ver.split('.')[2] != '0': continue
if not rocket_heading_re.search(vs):
anomalies_c.append(ver)
# --- Type D: :rocket: entries without issue AND PR references ---
# Both are required: `[#ISSUE](.../issues/N)` and `(PR: [#PR](.../pull/M))`.
# Multi-PR entries `(PR: [#A](...), [#B](...))` are accepted.
anomalies_d = [] # list of dicts: {version, line}
issue_ref_re = re.compile(r'\[#\d+\]\(https://github\.com/penpot/penpot/issues/\d+\)')
pr_ref_re = re.compile(r'\(PR:\s*\[#\d+\]\(https://github\.com/penpot/penpot/pull/\d+\)(\s*,\s*\[#\d+\]\(https://github\.com/penpot/penpot/pull/\d+\))*\)')
for vs in version_sections:
m = re.match(r'^## (\d+\.\d+\.\d+)(.*)', vs)
if not m: continue
ver = m.group(1)
rocket_match = rocket_heading_re.search(vs)
if not rocket_match: continue
# Extract the :rocket: subsection body (up to next ### or ##)
rocket_body = vs[rocket_match.end():]
rocket_body = re.split(r'(?m)^#{2,3}\s', rocket_body)[0]
for line in rocket_body.splitlines():
line = line.strip()
if line.startswith('- ') and not (issue_ref_re.search(line) and pr_ref_re.search(line)):
anomalies_d.append({'version': ver, 'line': line[:100]})
# --- Write report ---
def fmt_ms(ms):
return ms if ms else "_none_"
@@ -737,17 +654,13 @@ with open(OUTPUT, 'w') as f:
n_a = len(anomalies_a)
n_b = len(anomalies_b)
n_c = len(anomalies_c)
n_d = len(anomalies_d)
f.write('## Summary\n\n')
f.write(f'- **Issue in {MILESTONE}, referenced PR in different milestone or no milestone:** {n_a}\n')
f.write(f'- **PR in {MILESTONE}, closing issue in a different milestone:** {n_b}\n')
f.write(f'- **Total anomalies:** {n_a + n_b}\n')
f.write(f'- **Released X.Y.0 version missing :rocket: section (gap):** {n_c}\n')
f.write(f'- **:rocket: entry without issue AND PR references (gap):** {n_d}\n\n')
f.write(f'- **PR in {MILESTONE}, closing issue in different milestone or no milestone:** {n_b}\n')
f.write(f'- **Total anomalies:** {n_a + n_b}\n\n')
# --- Anomalies section (milestone mismatches only) ---
# --- Anomalies section ---
if n_a or n_b:
f.write('## Anomalies\n\n')
f.write('These are milestone mismatches between an issue in the changelog '
@@ -773,7 +686,7 @@ with open(OUTPUT, 'w') as f:
f.write('\n')
if n_b:
f.write(f'\n### PR in {MILESTONE}, closing issue in a different milestone\n\n')
f.write(f'\n### PR in {MILESTONE}, closing issue in different milestone or no milestone\n\n')
by_pr = {}
for b in anomalies_b:
by_pr.setdefault(b['pr'], []).append(b)
@@ -786,37 +699,9 @@ with open(OUTPUT, 'w') as f:
badge = '🔴' if e['issue_milestone'] is None else '⚠️'
f.write(f' - {badge} Closing {issue_link(e["issue"])} is in milestone **{ms_label}** (expected: {MILESTONE})\n')
f.write('\n')
else:
f.write('✅ No anomalies found. All (issue, PR) pairs in the changelog have aligned milestone assignments.\n\n')
# --- Highlight gaps (warnings, not anomalies) ---
if n_c or n_d:
f.write('## Highlight gaps\n\n')
f.write('These are warnings, not anomalies: they do not affect the '
'milestone-mismatch total above. They track `:rocket:` coverage '
'across all released X.Y.0 versions. Historical entries (e.g. '
'Taiga links) predate the current reference convention and are '
'expected to appear here.\n\n')
if n_c:
f.write(f'### Released X.Y.0 version missing :rocket: section\n\n')
f.write('These released minors/majors have no `### :rocket: Epics and highlights` subsection. '
'Add highlights to help self-hosted users understand what they are missing.\n\n')
for ver in anomalies_c:
f.write(f'- Version **{ver}**\n')
f.write('\n')
if n_d:
f.write(f'### :rocket: entry without issue AND PR references\n\n')
f.write('These highlight entries lack the required issue AND PR references. '
'Add `[#ISSUE](...)` and `(PR: [#PR](...))` links.\n\n')
for d in anomalies_d:
f.write(f'- **{d["version"]}**: `{d["line"]}`\n')
f.write('\n')
elif not (n_a or n_b):
f.write('✅ No highlight gaps found. All released X.Y.0 versions have properly referenced :rocket: entries.\n\n')
# --- Context ---
f.write('---\n\n')
f.write('## Context\n\n')
@@ -831,22 +716,13 @@ print(f"Anomaly report written to {OUTPUT}")
PYEOF
```
This generates `CHANGES-ISSUES.md` containing anomalies and highlight gaps:
This generates `CHANGES-ISSUES.md` containing **only the anomalies**
milestone mismatches between issues and their referenced PRs:
1. **Issue in milestone, referenced PR in different milestone or no milestone**
the changelog claims a fix here, but the PR is released elsewhere.
2. **PR in milestone, closing issue in a different milestone**
2. **PR in milestone, closing issue in different milestone or no milestone**
the PR is released here, but the issue it fixes belongs to another version.
(An issue with *no* milestone belongs to another, probably private,
project — milestones are only required on the "Main" project — so it is
neither an anomaly nor a changelog candidate.)
3. **missing-highlights (gap, warning)** — a released X.Y.0 version section
has no `### :rocket: Epics and highlights` subsection. Patches (X.Y.Z)
never carry highlights.
4. **missing-highlight-reference (gap, warning)** — a `:rocket:` entry lacks
the required issue AND PR references.
Gaps do not count toward the anomaly total.
**Rule violations are not in the report** — they are workflow errors the
LLM must fix directly in `CHANGES.md` during step 6a (pre-flight checks).
@@ -920,17 +796,11 @@ self-contained and clickable in any Markdown viewer.
issue from a different project or context. If the PR title and issue title
are clearly unrelated, or the PR predates the issue by years, treat it as a
data glitch and skip it.
- **Anomaly = milestone mismatch only; gaps are warnings.** The report's
anomaly total counts only milestone mismatches: (1) the issue is in this
milestone but the referenced PR is in a different milestone (or unassigned),
and (2) the PR is in this milestone but the issue it closes is in a
different milestone. `:rocket:` highlight gaps (missing section on a
released X.Y.0, entry without issue AND PR references) are reported in a
separate `Highlight gaps` section and never count toward the anomaly total. An
*unassigned* (milestone-less) issue closed by a milestone PR is **not**
an anomaly: milestones are required only for the "Main" project, so such
issues come from another (probably private) project and are not changelog
candidates. These anomalies are reported because the changelog pairing is
- **Anomaly = milestone mismatch only.** The report contains only milestone
mismatches: (1) the issue is in this milestone but the referenced PR is
in a different milestone (or unassigned), and (2) the PR is in this
milestone but the issue it closes is in a different milestone (or
unassigned). These are anomalies because the changelog pairing is
*misleading* — the human needs to decide whether the milestone or the
changelog is wrong. All other discrepancies (exclusion labels, missing
valid issues, unmerged PR references, duplicates, stale milestone
-1
View File
@@ -7,7 +7,6 @@ Backend: JVM Clojure; Integrant; PostgreSQL; Redis/Valkey; RPC; HTTP; storage; m
- RPC, DB helpers, workers, cron: `mem:backend/rpc-db-worker-subtleties`
- Storage abstraction, logical buckets, object lifecycle, deduplication, access, and garbage collection: `mem:backend/storage`.
- HTTP sessions, config, media processing, and file data persistence: `mem:backend/http-storage-filedata-subtleties`.
- Embedded Ladybug graph experiment, projection, incremental sync, console, and risks: `mem:backend/graph-experiment`
- Auth flows, permission model, teams, projects, invitations, comments, webhooks, audit: `mem:backend/auth-permissions-product-domains`
- Services, task-queue/Pub-Sub topology constraints -> `mem:prod-infra/core`.
@@ -1,632 +0,0 @@
# Graph Experiment
## Scope
- Purpose: project Penpot file data into an embedded Ladybug graph database.
- Purpose: keep the graph current with Penpot file changes.
- Purpose: expose a read-only graph console for backend debugging.
- This is an experiment, not a replacement for PostgreSQL file storage.
- The graph subsystem is off unless `:graph` is in the backend flags.
- The main Penpot frontend has no graph feature code for this subsystem.
- The graph console is a backend-served HTML template with JavaScript.
## Memory Links
- Read `mem:backend/core` for backend architecture, HTTP routes, DB rules, and test commands.
- Read `mem:backend/rpc-db-worker-subtleties` for RPC and message bus behavior.
- Read `mem:backend/http-storage-filedata-subtleties` for file data loading and realization.
- Read `mem:common/changes-architecture` for the change record vocabulary.
- Read `mem:frontend/routing-app-shell-subtleties` for the existing notification WebSocket.
- Read `mem:prod-infra/core` for Redis or Valkey message bus topology.
## Branch Surface
- The graph experiment adds about 6,336 lines and changes about 27 files.
- The graph implementation lives under `backend/src/app/graph/`.
- The graph console lives at `backend/resources/app/templates/graph-console.tmpl`.
- The existing debug page gains graph links in `backend/resources/app/templates/debug.tmpl`.
- The existing debug HTTP routes gain graph handlers in `backend/src/app/http/debug.clj`.
- The backend system passes the message bus to the debug route component in `backend/src/app/main.clj`.
- The backend adds Ladybug and Arrow dependencies in `backend/deps.edn`.
- The backend adds JVM options for Ladybug and Arrow native access.
- The common flag registry adds `:graph` in `common/src/app/common/flags.cljc`.
- The graph experiment adds `graph_sync_parity_test.clj` and `graph_binder_gate_test.clj`.
## System Model
### Storage layers
- PostgreSQL remains the source of truth for Penpot files.
- The graph database stores a projection of one file.
- A persistent graph uses a `.lbug` path under `PENPOT_GRAPH_DIR`.
- The default graph directory is `/tmp/penpot-graph`.
- A debug session uses a Ladybug `:memory:` database.
- A debug session database lives inside the backend JVM process.
- A debug session does not survive a backend restart.
- A debug session does not store file data back to PostgreSQL.
### Two graph update paths
- Cold projection reads the complete file and rebuilds the graph.
- Incremental sync reads file change records and updates the open graph.
- Both paths must produce the same graph for the same file state.
- The parity test treats cold projection as the reference path.
- A reload discards the session graph and uses cold projection again.
## Main Namespaces
### `app.graph.ladybug`
- Opens and closes Ladybug `Database` and `Connection` objects.
- Installs and loads the Ladybug JSON extension.
- Executes Cypher statements.
- Executes prepared statements.
- Binds scalar parameters.
- Formats UUID, string, integer, number, JSON, and timestamp values.
- Formats compound values such as arrays, maps, and structs.
- Converts Ladybug values back to Clojure values.
- Limits normal query results to 200 rows by default.
- Detects result truncation with `:truncated?`.
- Uses query timeout `0` by default.
- Query timeout `0` disables the timeout.
- Provides `validate-on-connection!` for parse, bind, and read-only checks.
- `exec-prepared-on-connection!` prepares every statement before the first execution.
- A prepare failure stops the batch before a mutation runs.
### `app.graph.schema`
- Provides the public schema facade.
- Exposes schema version `penpot-graph-slice-4`.
- Delegates node and relationship definitions to `app.graph.schema.nodes`.
### `app.graph.schema.nodes`
- Holds the single registry for graph node tables.
- Generates node DDL.
- Generates relationship DDL.
- Maps Penpot shape types to graph tables.
- Projects source attributes into graph attributes.
- Formats graph column values.
- Quotes reserved graph labels such as `Group` and `Boolean`.
- Defines container tables and shape tables.
- Defines `IsChildOf`, `IsInstanceOf`, `RefersTo`, and `FillsSwapSlot`.
### `app.graph.schema.contract`
- Records deliberate graph contract decisions.
- Renames graph columns such as `:revn` to `revision`.
- Drops attributes that do not belong in this graph slice.
- Records attributes that the graph does not project.
- Applies per-table dropped attributes.
- Defines type overrides for vectors, transforms, colors, maps, and JSON arrays.
- Maps selected map keys to the frontend JSON naming convention.
- `:background-blur` remains a declared unprojected attribute.
### `app.graph.schema.projection`
- Derives projected schemas from canonical Malli schemas.
- Builds the projected document schema.
- Builds projected shape schemas.
- Selects the schema for each shape type.
### `app.graph.schema.types`
- Maps Malli types to Ladybug types.
- Maps matrices to `DOUBLE[6]`.
- Maps points to `DOUBLE[2]`.
- Maps rectangles to `DOUBLE[4]`.
- Maps colors to `UINT32`.
- Maps collections to Ladybug arrays.
- Maps `:map-of` schemas to `MAP`.
- Maps closed scalar maps to `STRUCT`.
- Maps other complex values to `JSON`.
### `app.graph.schema.values`
- Coerces source values to graph column values.
- Writes fixed vectors with deterministic order.
- Packs colors into the graph color representation.
- Sorts set values when deterministic output is needed.
### `app.graph.arrow`
- Loads projection rows with Apache Arrow.
- Creates temporary staged node and relationship tables.
- Uses `COPY ... FROM (MATCH ...)` for bulk loading.
- Groups relationship loads by source and target table pair.
- Resolves relationship endpoints with joins.
- Does not use `createArrowRelTable` for UUID relationship endpoints.
- Keeps the Arrow `RootAllocator` alive until Ladybug releases staged buffers.
- Closes the allocator after the connection and database close sequence.
### `app.graph.ingest`
- Fetches a complete file with `bfc/get-file` and `:realize? true`.
- Rejects missing files.
- Rejects files without file data.
- Can run file data validation before projection.
- Creates the DDL.
- Loads nodes and edges through Arrow.
- Executes post-load transforms.
- Writes graph metadata last.
- Treats the final metadata write as the complete-build marker.
- Supports a persistent database path and an open connection.
### `app.graph.projection.document`
- Projects `Document`, `Page`, `Component`, and supported shape nodes.
- Skips the page root frame.
- Creates `IsChildOf` edges from shapes to parents.
- Creates page edges to the document.
- Creates component edges to the document.
- Stores page order in `Page.index` and edge `position`.
- Reverses the stored `:shapes` list for Penpot z-order.
- Adds `page-id` to every projected shape.
- Propagates an instance head `component-id` to descendants.
- Stops component inheritance at a non-Frame shape with its own component ID.
- Skips deleted components during cold projection.
- Logs unsupported shape types and missing shape records.
### `app.graph.projection.transforms`
- Runs after the base nodes and edges load.
- `link-component-instances` creates `IsInstanceOf` edges.
- A Frame needs `component-file` to qualify as an instance head.
- `link-shape-refs` creates `RefersTo` edges from `shape-ref`.
- Ladybug limits multi-label relationship `MERGE` statements.
- The transform emits one statement for each shape-table pair.
- `link-swap-slots` creates `FillsSwapSlot` edges.
- Swap slot IDs come from `swap-slot-<uuid>` entries in `touched`.
- The transform removes swap slot entries from `touched` after edge creation.
- The transform order matters because it reads and then changes `touched`.
### `app.graph.meta`
- Stores graph provenance in `GraphMeta`.
- Stores schema version, source revision, producer, and build time.
- The source revision identifies the file revision used for cold projection.
### `app.graph.stats` and `app.graph.report`
- `app.graph.stats` counts graph nodes and relationships from the live catalog.
- `app.graph.report` prints ingest information for REPL use.
## Cold Projection Flow
1. Get the file row and realized file data from PostgreSQL.
2. Read the file revision from the file row.
3. Build the node and edge projection.
4. Create all graph tables from the graph schema.
5. Load node rows with Arrow.
6. Load relationship rows with Arrow.
7. Run `CHECKPOINT;`.
8. Run the registered derived transforms.
9. Write `GraphMeta` as the final build step.
10. Return file ID, file revision, database path, projection stats, and transform stats.
### Projection node groups
- `Document` contains file-level attributes without the file data blob.
- `Document.options` receives file-level options from the data blob.
- `Page` contains page attributes without the page object map.
- `Component` contains component attributes without component object maps.
- Shape tables contain the supported shape attributes.
- The graph stores selected derived attributes such as `page-id`.
### Projection relationship groups
- Structural edges use `IsChildOf`.
- Page and component edges point to `Document`.
- Derived edges come from the post-load transform registry.
## Incremental Sync
### Change source
- `app.rpc.commands.files-update` persists the file update first.
- The same command publishes a `:file-change` message to the file topic.
- The topic key is the file UUID.
- The message contains the file ID, profile ID, session ID, revision, version, and changes.
- Library changes also publish a team-topic message.
- The graph session only consumes the file-topic `:file-change` messages.
### Session subscription
- `app.graph.debug/start-sync-loop!` creates a channel with a dropping buffer of 64.
- The session subscribes the channel to the file UUID topic.
- The loop reads one message at a time.
- The loop ignores message types other than `:file-change`.
- The loop stops when the channel closes.
- `destroy-session!` closes the channel and purges its message bus subscription.
### Session state
- Sessions are stored in a global `defonce` atom.
- The map key is the string form of `profile-id`.
- One profile has one graph session.
- Loading another file first destroys the old session.
- A session stores the Ladybug database and connection.
- A session stores a shared lock for graph access.
- A session stores file metadata.
- A session stores the incremental sync index.
- A session stores the message bus channel.
- A session stores load time and profile ID.
- The session keeps projection statistics but drops full projection rows after index creation.
### Sync index
- `build-index` starts from the complete cold projection.
- The index stores the graph file ID and document ID.
- The index stores the current graph revision.
- The index stores page IDs, names, and positions.
- The index stores component IDs, names, and deleted state.
- The index stores shape table, parent, position, frame, page, and component context.
- The index stores child IDs by parent ID.
- The index supports later change application without another PostgreSQL file read.
### Change application
- `apply-changes!` processes the change list in source order.
- Each supported change returns a new index and a list of Cypher statements.
- Unsupported changes enter the `:skipped` result.
- Supported changes enter the `:applied` result.
- The function collects all statements before it executes them.
- The function appends a document revision statement when at least one change applies.
- The index revision advances only when at least one change applies.
- A larger incoming revision than the index revision creates a warning.
- A revision gap does not trigger catch-up.
### Shape change rules
- `:add-obj` reuses `projection.document/denormalized-shape`.
- `:add-obj` creates the shape node and its parent edge.
- `:mod-obj` applies supported `:set` operations to graph columns.
- `:mod-obj` keeps false and zero values as values.
- `:del-obj` deletes shapes in deep post-order.
- `:mov-objects` detaches shapes from the old parent.
- `:mov-objects` closes the old sibling position gap.
- `:mov-objects` inserts shapes at the new position.
- `:mov-objects` updates `parent_id` and `frame_id`.
- `:mov-objects` rewrites container `shapes` values.
- The parent columns and child lists must match a cold projection.
### Page and component change rules
- Page add creates a projected page node and a document edge.
- Page delete removes the page subtree.
- Page modification updates supported page attributes.
- Component add creates a component node and document edge.
- Component modification updates supported component attributes.
- Component delete uses a soft-delete state.
- Component restore removes the soft-delete state.
- Component purge removes the component node and document edge.
- Component sync paths need more parity coverage than the current tests provide.
## Session Locking
- The sync loop and HTTP handlers share one lock per session.
- The lock protects one Ladybug connection from concurrent access.
- Queries acquire the lock before binder validation and execution.
- Graph data export acquires the lock before catalog reads.
- Session export acquires the lock before `EXPORT DATABASE`.
- A long query blocks sync for the same session.
- A sync batch blocks queries for the same session.
- Ladybug connection thread safety is not assumed.
## Graph Query Rules
- The console accepts Cypher text.
- Blank query text raises a validation error.
- The query first passes Ladybug prepare and bind checks.
- The query must pass the engine read-only analysis.
- A mutating query is rejected.
- The graph console does not provide a write path.
- A session graph is rebuilt from the file by Reload.
- Normal query results have a 200-row limit.
- Query results use string values for the HTML console representation.
- JSON requests receive a Transit JSON response with the query and result.
- HTML requests receive the rendered console with the result.
## Graph Data Export
### G6 data
- `/dbg/actions/graph-data` reads the live Ladybug database.
- It does not read the sync index for nodes and edges.
- It therefore shows database drift if a batch fails after index update.
- Node export covers all registered node tables.
- Relationship export reads the Ladybug relationship catalog.
- Relationship export includes source, target, relationship name, and position.
- Node and relationship export uses a 100,000-row limit.
- The response reports `truncated` when a limit cuts the result.
- The response reports buffer-manager memory usage.
### `.lbug` export
- `source=file` rebuilds the persistent graph from PostgreSQL file data.
- `source=file` runs a synchronous full ingest for each request.
- `source=session` exports the caller profile's live in-memory graph.
- Session export uses Ladybug `EXPORT DATABASE` to Parquet files.
- Session export creates a new `.lbug` database with `IMPORT DATABASE`.
- The temporary Parquet staging directory is deleted after import.
- The final session `.lbug` file remains in the system temporary directory.
- The HTTP response streams the database file to the caller.
## HTTP Routes and Access
- The graph routes live in `backend/src/app/http/debug.clj`.
- The graph route list is added only when `:graph` is enabled.
- `/dbg/graph` serves the graph console page.
- `/dbg/actions/graph-files` returns the profile file tree.
- `/dbg/actions/graph-load` loads a file into the profile session.
- `/dbg/actions/graph-unload` closes the profile session.
- `/dbg/actions/graph-reload` rebuilds the loaded file graph.
- `/dbg/actions/graph-query` runs a read-only Cypher query.
- `/dbg/actions/graph-sync-status` returns the sync state.
- `/dbg/actions/graph-data` returns nodes and edges for G6.
- `/dbg/actions/graph-export` streams a `.lbug` database.
- The `/dbg` session middleware remains active.
- The `/dbg` admin middleware remains active.
- A devenv host with a profile ID passes the debug authorization rule.
- Other hosts need a profile email in the configured admin set.
- `/dbg/actions/graph-files` lists reachable teams, projects, and files.
- The file tree query has a 500-file limit.
- The graph handlers resolve graph namespaces at call time.
- The backend requires `app.graph.debug` and `app.graph.ingest` when the flag is on.
- Ladybug native loading then fails during route initialization instead of first use.
## Console Frontend
### Page type
- `graph-console.tmpl` is a backend resource template.
- It is not a Rumext component.
- It is not part of the main frontend route table.
- The page uses browser `fetch` calls and a browser WebSocket.
- The page loads G6 version `5.1.1` from jsDelivr.
### File tree
- The page fetches `/dbg/actions/graph-files`.
- The response contains team, project, and file groups.
- The page creates the tree with DOM APIs.
- A file click submits the graph load form.
- The page shows a message when no file exists.
### Graph rendering
- The page fetches `/dbg/actions/graph-data`.
- The page converts graph nodes and edges to G6 data.
- The page skips repaint when the node and edge signature does not change.
- The page marks added, removed, and changed graph entities.
- The page supports tree, dagre, circular, force, and combo layouts.
- The page supports collapsed container combos.
- The page has render guards at 4,000 nodes and 8,000 edges.
- The `?safe` query option bypasses the render guard.
- The page shows graph size by node count and relationship count.
- The page shows buffer-manager memory in MiB.
- The page reports a CDN failure when G6 is undefined.
### Query result filtering
- A query can return `filter_*` columns with node IDs.
- The HTML result table hides columns with the `filter_` prefix.
- The JSON result keeps the full result.
- The graph view uses the hidden IDs to select matching nodes.
- The graph view re-runs the query after graph refresh.
- This keeps the query filter aligned with the current graph.
- A user column named `filter_*` follows the same hiding rule.
### Node inspector
- A node click creates a query for that node.
- The inspector calls `/dbg/actions/graph-query` with JSON negotiation.
- The inspector displays the full projected row.
- The inspector uses table and ID values from the graph data.
## WebSocket Data Flow
1. The page opens `/ws/notifications` with a random `session-id` query value.
2. The page sends `:subscribe-file` with a Transit UUID value.
3. The server makes sure that the file exists and that the profile has read permission.
4. The server subscribes the connection to the file topic.
5. `files_update` publishes `:file-change` to the same topic.
6. The graph session consumes the message from its message bus subscription.
7. The WebSocket server sends the message to the browser connection.
8. The browser adds the change to the changelog.
9. The browser fetches sync status after 150 milliseconds.
10. The browser fetches graph data after a 400-millisecond debounce.
11. The browser repaints the G6 graph when the graph data changes.
### WebSocket reconnect behavior
- The page reconnects after three seconds when the socket closes.
- The page resubscribes to the file after the socket opens.
- The page refreshes sync status after reconnect.
- The page refreshes graph data after reconnect.
- Reconnect does not recover dropped message-bus changes.
- The page shows the sync error or skipped-change state when the status reports it.
## Feature Flag and Runtime Dependencies
- `:graph` is defined in `common/src/app/common/flags.cljc`.
- The flag is off by default.
- `com.ladybugdb/lbug` version `0.19.1` is a backend dependency.
- `org.apache.arrow/arrow-memory-netty` version `18.2.0` supports Arrow `RootAllocator`.
- The JVM uses `--enable-native-access=ALL-UNNAMED`.
- The JVM uses `--add-opens=java.base/java.nio=ALL-UNNAMED`.
- The JVM uses `--sun-misc-unsafe-memory-access=allow`.
- The JVM options appear in the development alias and backend launch scripts.
- A Ladybug version change needs new binder and parity tests.
- A JDK version change needs a startup test with the graph flag enabled.
## Tests
### `backend-tests.graph-sync-parity-test`
- Uses two Ladybug `:memory:` databases.
- Does not use PostgreSQL or a live graph session.
- Projects initial file data into database A.
- Applies changes to database A through incremental sync.
- Applies the same changes to file data.
- Projects the changed file data into database B.
- Compares every node row and relationship row.
- Reports differences by table, row key, and column.
- Covers shape add, shape modification, shape deletion, movement, and page changes.
- Contains a test that injects a sync defect and expects a graph difference.
- Does not cover all component change variants.
- Does not cover every movement insertion mode.
### `backend-tests.graph-binder-gate-test`
- Creates the live graph DDL in a Ladybug `:memory:` database.
- Prepares each sync statement template without executing it.
- Detects parse errors and missing tables.
- Detects missing columns and bad label quoting.
- Reports the expected read-only classification.
- Covers reserved node labels across the node registry.
- Reports an error result for an invalid statement.
### Test gaps
- No automated HTTP handler tests cover graph routes.
- No automated session lifecycle tests cover load and unload.
- No automated WebSocket tests cover graph subscription.
- No automated export tests cover persistent and session sources.
- Component add, modify, delete, restore, and purge need parity tests.
- Page delete needs parity coverage.
- Movement with `:after-shape` needs parity coverage.
- Buffer overflow and revision gap behavior need tests.
- Partial batch failure and recovery need tests.
- Query timeout and long-query behavior need tests.
## Known Risks and Limits
### Dropped changes
- The sync channel uses a dropping buffer of 64.
- A burst can discard file-change messages.
- The sync loop logs a revision gap when it sees a larger revision.
- The sync loop does not fetch missing rows from `file_change`.
- Reload is the only built-in recovery path.
### Partial batch state
- `apply-changes!` does not provide Ladybug transaction atomicity.
- A statement failure can leave a partly changed graph.
- The in-memory index can advance before the database state is complete.
- `/dbg/actions/graph-data` reads the database and exposes this drift.
- Reload rebuilds the graph from PostgreSQL file data.
### Query resource use
- The default session query timeout is zero.
- A costly query can hold the session lock for a long time.
- The same lock blocks incremental sync.
- The graph export also holds the same lock during catalog reads.
- The graph schema has a high memory floor.
- The console reports about 115 MiB for the wide slice before file data.
### Session lifecycle
- Sessions have no TTL.
- Sessions remain until unload, replacement, or process shutdown.
- Each session owns native Ladybug memory.
- Many profiles can create many native databases.
- A profile load replaces its previous session.
- Two browser tabs for one profile share one graph session.
### Temporary files
- Session export leaves the final `.lbug` file in the system temporary directory.
- Long-lived servers can accumulate exported session databases.
- The staging directory is deleted after import.
### Browser dependency
- The graph view depends on a runtime CDN request.
- A network restriction can remove the G6 view.
- Queries and session status still use backend endpoints without G6.
### Data exposure
- The graph console can list many files available to the profile.
- The console can load complete projected file data.
- The console can export a graph database.
- The console can inspect all projected node attributes.
- The console is safe only when the `/dbg` access boundary is correct.
- The graph flag must remain off for deployments that do not need this tool.
### Contract drift
- The graph schema is a deliberate slice of the Penpot file model.
- New source attributes do not enter the graph automatically in all cases.
- Dropped and unprojected attributes need an explicit contract decision.
- `applied_tokens` key mapping depends on the JSON naming function.
- `filter_*` is a frontend convention, not a graph schema guarantee.
### Ladybug dialect coupling
- Cypher strings contain Ladybug-specific syntax.
- Label quoting handles reserved labels explicitly.
- Relationship transforms depend on Ladybug relationship limits.
- Arrow loading depends on Ladybug `COPY FROM (MATCH ...)` behavior.
- A dependency upgrade needs schema, binder, Arrow, and parity checks.
## REPL Helpers
- `app.srepl.main` resolves graph functions only when a helper runs.
- `graph-smoke-test!` runs a basic Ladybug operation.
- `graph-query-test!` runs a graph query test.
- `ingest-file-to-graph!` projects a file into a graph database.
- These helpers use `requiring-resolve` to keep the graph dependency lazy.
## Operational Invariants
- PostgreSQL file data remains authoritative.
- Cold projection and incremental sync must produce equal graph state.
- The graph revision must identify the last applied file revision.
- The document revision must update when a sync batch applies.
- A missing or skipped change must remain visible in sync status.
- A graph query from the console must be read-only.
- A graph session must serialize connection access.
- Graph routes must remain behind the `:graph` flag and `/dbg` access control.
- The Arrow allocator must outlive all Ladybug operations that use its buffers.
- `GraphMeta` must be written after the full ingest and transforms finish.
## Key Files
- `backend/src/app/graph/ladybug.clj`: Ladybug API and query gates.
- `backend/src/app/graph/arrow.clj`: Arrow bulk load.
- `backend/src/app/graph/ingest.clj`: Complete file ingest.
- `backend/src/app/graph/debug.clj`: Session lifecycle, sync loop, query, and export.
- `backend/src/app/graph/sync.clj`: Incremental change application.
- `backend/src/app/graph/meta.clj`: Graph provenance.
- `backend/src/app/graph/stats.clj`: Graph counts.
- `backend/src/app/graph/report.clj`: REPL ingest report.
- `backend/src/app/graph/projection/document.clj`: Base document projection.
- `backend/src/app/graph/projection/transforms.clj`: Derived relationship transforms.
- `backend/src/app/graph/schema/nodes.clj`: Node and relationship registry.
- `backend/src/app/graph/schema/contract.clj`: Projection contract decisions.
- `backend/src/app/graph/schema/projection.clj`: Malli projection schemas.
- `backend/src/app/graph/schema/types.clj`: Malli-to-Ladybug type mapping.
- `backend/src/app/graph/schema/values.clj`: Value coercion.
- `backend/src/app/http/debug.clj`: Graph route registration and handlers.
- `backend/src/app/http/websocket.clj`: File WebSocket subscription handlers.
- `backend/src/app/rpc/commands/files_update.clj`: File-change publication.
- `backend/src/app/main.clj`: Integrant message bus wiring.
- `backend/resources/app/templates/graph-console.tmpl`: Graph console browser code.
- `backend/resources/app/templates/debug.tmpl`: Debug page graph links.
- `common/src/app/common/flags.cljc`: `:graph` feature flag.
- `backend/test/backend_tests/graph_sync_parity_test.clj`: Cold versus sync parity.
- `backend/test/backend_tests/graph_binder_gate_test.clj`: Cypher binder gate.
## Development Commands
- Run backend commands from the `backend/` directory.
- Run focused parity tests with `clojure -M:dev:test --focus backend-tests.graph-sync-parity-test`.
- Run focused binder tests with `clojure -M:dev:test --focus backend-tests.graph-binder-gate-test`.
- Run the backend test suite with `clojure -M:dev:test`.
- Examine Clojure formatting with `pnpm run check-fmt:clj`.
- Run backend Clojure lint with `pnpm run lint:clj`.
- Write test output to a file before reading or filtering it.
-12
View File
@@ -14,11 +14,7 @@ You are working on the GitHub project `penpot/penpot`, a monorepo.
- Before `git commit``mem:workflow/creating-commits` (subject format, body, `AI-assisted-by: model-name` trailer)
- Before `gh issue create``mem:workflow/creating-issues` (title derivation, body template, labels, Issue Type)
- Before `gh pr create` / `gh pr edit``mem:workflow/creating-prs` (title format, body structure, "Note:" line)
- Before a repo-wide pnpm version update → `mem:workflow/updating-pnpm` (workspace
layout, `corepack use` sweep order, the stamp-missing-field and
ignored-builds gotchas, verification steps)
- **Never `git push`, force-push, or modify `git origin`** (or any other remote). The user pushes from their own shell; if a push is required, say so and wait. Never amend a commit that the user has already pushed unless explicitly asked.
- **Never edit `CHANGES.md` by hand.** The changelog is generated from GitHub milestones during the release process; update it only via the `update-changelog` skill flow or on explicit user request.
- You have access to the GitHub CLI `gh` or corresponding MCP tools.
- Issues are also managed on Taiga. Read issues using the `read_taiga_issue` tool.
- Before writing code, analyze the task in depth and describe your plan. If the task is complex, break it down into atomic steps.
@@ -74,14 +70,6 @@ module. You can read it from `mem:<MODULE>/core`
- `scripts/error-reports.mjs` — Query error reports via RPC API with token
authentication. Supports list/get operations with filtering and pagination.
See `mem:scripts/error-reports`.
- `scripts/clean-node-modules` — Remove stale `node_modules` from all pnpm
workspaces (root, modules, member packages). Keeps the shared pnpm store
at `<repo>/.pnpm-store` unless `--store`; ignores `external/` and
`.opencode/`. Usage and reinstall steps: `mem:workflow/updating-pnpm`.
- `scripts/ci` — CI orchestration script: runs lint, tests, and format
checks per module (`frontend backend common render-wasm exporter mcp
plugins library`). Logs go to `.ci-logs/`; read the log file on failure.
See `mem:scripts/ci`.
# Dependency graph
+1 -2
View File
@@ -6,7 +6,6 @@ Compose-based dev environment under `docker/devenv/`, driven by `manage.sh`. Par
- `penpotdev-infra`: shared `postgres`, `minio`, `minio-setup`, `mailer`, `ldap`. File: `docker-compose.infra.yml`.
- `penpotdev-wsN` (N=0,1,…): per-instance `main` + `redis` (Valkey). File: `docker-compose.main.yml`. ws0 (a.k.a. `main`) binds `$PWD`; ws1+ bind clones at `${PENPOT_WORKSPACES_DIR}/wsN/` (default `~/.penpot/penpot_workspaces/`), maintained by the developer.
- Optional overlay `docker-compose.opencode.yml`: added by `instance-compose` as an extra `-f` only when `PENPOT_OPENCODE_CONFIG_DIR` is set (i.e. `run-devenv --opencode-config-dir DIR` ran in this process). Bind-mounts the host dir at `/home/penpot/.config/opencode` (`:z`). Flag-only, per-call; not read from ambient env. Parser `parse-opencode-config-dir` absolutizes (`~`, realpath) because compose resolves relative bind sources against the compose file's dir. Only instances brought up with the flag get the mount.
- All projects join external network `penpot_shared`. Created idempotently by `ensure-devenv-network`, never removed by lifecycle commands.
## Source-of-truth files
@@ -66,7 +65,7 @@ No `--delete` on the working-tree pass: gitignored caches in the workspace survi
## CLI surface
- `run-devenv --agentic [--ws main|0|wsN|N] [--sync] [--serena-context CTX] [--opencode-config-dir DIR]`: bring one instance up. Agentic only — MCP and Serena windows are always created. Default target main. Errors out if the target is already running. `--sync` is rejected on main; on ws1+ it's optional (forced only when the workspace dir does not exist yet). `--opencode-config-dir DIR` bind-mounts DIR at `~/.config/opencode` in-container via the optional overlay above; mount applies at container creation, so changing it requires stop + re-run.
- `run-devenv --agentic [--ws main|0|wsN|N] [--sync] [--serena-context CTX]`: bring one instance up. Agentic only — MCP and Serena windows are always created. Default target main. Errors out if the target is already running. `--sync` is rejected on main; on ws1+ it's optional (forced only when the workspace dir does not exist yet).
- `stop-devenv [--ws main|0|wsN|N] [--all]`: stop instances. Flags mutually exclusive. `--ws N` stops just that workspace. `--ws 0` or no flag stops ws0; shared infra shuts down only if no other instances remain. `--all` stops every ws highest-first then ws0, then infra.
- `run-devenv`: legacy alias, ws0 non-agentic attached.
- `attach-devenv [--ws main|0|wsN|N]`: pure attach. Fails fast if instance/session missing.
@@ -19,20 +19,6 @@
- Raster `Fill::Image`: skip `save_layer` unless the shape has an image filter; plain
Rect/Frame (no corners) also skip the container clip (`draw_image_fill` in fills.rs).
- `can_render_directly` paints onto Current (no Fills/Strokes blit) for plain geometry and
for stroke-free text (SrcOver, no blur/shadows). Multi-style text is fine: span styles
live in Paragraph `TextStyle`s. Text skips the `nested_fills` guard (fills are on spans).
`draw_text` only `save_layer`s when stroke-group opacity is set; plain fill paint is direct.
- Plain text fill paint reuses `TextContent.layout` paragraphs when
`has_usable_paint_layout` (paragraphs present + version match; during
interactive transforms rotation/move skips width check via
`modifier_changes_text_layout`, resize falls back to `layout_width` vs
`get_width(selrect.width())`), via `text::try_paint_from_layout_cache`.
The walker computes `text_layout_cache_rotation_only` from `tree` and
passes it into `render_shape`; stroke/shadow paths pass `false`.
- `TextContentLayout` paragraphs are `Rc`-shared on `Clone` so modifier clones
(rotate/pan) keep the paint cache; `needs_update` is paragraphs-empty only.
Decorations are skipped when no span requests underline/strike.
- Zoom settle: visible tiles present via `FrameType::ViewportReady` before interest-ring
work; crop-cache rebuild is deferred to the later `Full` so the soft→sharp snap is
compose+present only.
-61
View File
@@ -1,61 +0,0 @@
# CI (scripts/ci)
`scripts/ci` runs CI-style checks — lint, tests, format — for one or more
monorepo modules and prints a per-task summary. It is the local equivalent
of CI; use it to verify changes before declaring work done.
## When to use
- After implementing or fixing code in a module: run its checks before
finishing (AGENTS.md: run the applicable lint and format checks).
- When `common/` changed: validate its consumers too (frontend, backend,
exporter; see the dependency graph in `mem:critical-info`).
- To fix formatting across a module (`--fix`) or repair delimiters
(`--paren-repair`) before linting.
## How to use (CLI)
Run from the repo root:
```bash
./scripts/ci MODULE... # lint + test + fmt per module
./scripts/ci --all --no-test # lint + fmt on all modules
./scripts/ci --lint frontend # lint only
./scripts/ci --fix --no-test frontend # format files, skip tests
./scripts/ci --paren-repair --all # fix delimiters in all Clojure modules
./scripts/ci --dry-run --all # preview what would run
```
Modules: `frontend backend common render-wasm exporter mcp plugins library`.
Flags:
- Default tasks: `lint`, `test`, `fmt` (format check; `--fix` formats
instead).
- `--lint` / `--test` / `--fmt` run one task only; `--no-lint` /
`--no-test` / `--no-fmt` drop one task from the default set.
- `--paren-repair` runs only the delimiter repair — it wraps
`scripts/paren-repair` over each module's Clojure/CLJS sources; see
`mem:scripts/paren-repair`.
- `--all` selects every module; `--exclude MOD` drops one (repeatable).
- `--fail-fast` stops at the first failure; `--quiet` suppresses failure
output; `--dry-run` prints commands without running; `--clean` removes
the log directory.
## Logs and exit codes
- Full output of every task: `.ci-logs/<module>-<task>.log`.
- On failure the script prints the last 30 lines; the final summary lists
every failed `module:task` with its log path.
- Exit code 0 when all selected tasks passed, 1 otherwise.
- Diagnose failures by reading the log file — never pipe test output
through filters (AGENTS.md hard rule).
## Notes
- `mcp` has no lint task (shows as skipped). `render-wasm` uses `./lint`,
`./test`, and `cargo fmt`.
- Test tasks are long-running (backend: `clojure -M:dev:test`); use a
generous timeout when calling it from an agent shell.
- Skill entry point: `.agents/skills/local-ci/SKILL.md`.
- Testing principles and output discipline: `mem:testing`.
+1 -1
View File
@@ -351,5 +351,5 @@ gh issue view <NUMBER> --repo penpot/penpot --json title
## See Also
- End-to-end orchestration entry point: the `create-issue` skill at
`.agents/skills/create-issue/SKILL.md`. The skill is a thin entry
`.opencode/skills/create-issue/SKILL.md`. The skill is a thin entry
point; this memory is the canonical home for all issue-creation rules.
+1 -7
View File
@@ -1,12 +1,6 @@
# Creating Pull Requests
PR only on explicit request.
## Branch Naming
- Primary: `issue-NNNN` — one branch per GitHub issue (e.g. `issue-11525`).
- No issue: free-form descriptive name, dash-separated, no slashes (e.g. `fix-ellipse-icon-typo`, `feat-auto-link-libraries`).
- If the user already created the branch, use it as-is — never rename.
PR only on explicit request. Branch: issue/feature-specific; fallback `<type>/<short-description>` (`fix/...`, `feat/...`, `refactor/...`, `docs/...`, `chore/...`, `perf/...`).
## Target Branch
@@ -1,91 +0,0 @@
# Updating pnpm Across All Workspaces
Canonical procedure. Run it from the repo root with the log redirected to a
file (never pipe tool output through filters).
## Layout facts
- The repo has 11 pnpm workspaces, each with its own `pnpm-workspace.yaml`
and `pnpm-lock.yaml`: the repo root plus `backend`, `common`, `docs`,
`exporter`, `frontend`, `library`, `mcp`, `media-processor`, `plugins`,
and `render-wasm`.
- Every package inside a module workspace (for example all `plugins/apps/*`
and `plugins/libs/*` packages) is a plain member of that module's
workspace. Members must not carry their own `pnpm-workspace.yaml` or
`pnpm-lock.yaml`; their dependencies resolve through the parent
workspace's lockfile.
- One shared pnpm store for the whole repo: `<repo>/.pnpm-store`. Every
workspace yaml sets it explicitly: `storeDir: .pnpm-store` at the root,
`storeDir: ../.pnpm-store` in each module. pnpm resolves the value
against the workspace root, so all workspaces land on the same store.
Do not remove these lines: nested workspaces do not inherit settings,
and without them each workspace may resolve a different store.
- The store survives `node_modules` cleans. It is content-addressed and
integrity-verified, so it cannot go stale; staleness lives in
node_modules. Only `scripts/clean-node-modules --store` removes it.
- Every `package.json` (about 35 of them) must carry a `packageManager` field
with the identical `pnpm@<version>+sha512.<hash>` value. Do not let them drift.
- CI pins no pnpm version; workflows rely on corepack reading
`packageManager`. Fixing the fields fixes CI.
## Procedure
1. Resolve the target tag first and note the version. Example:
`npm view pnpm dist-tags --json` for `next-12` (latest 12.x). The tag
moves over time; always re-check.
2. List every directory with a `package.json`, excluding `node_modules`
(`fd -H -t f package.json -E node_modules`). This list is the work set;
do not maintain a hand-written list.
3. Run `corepack use pnpm@<tag>` in workspace roots first, then members.
`corepack use` stamps `packageManager` in the nearest package.json and
runs an install. Member runs repeat the workspace install; after the root
run they are quick no-ops.
4. If a run fails, fix the cause (see gotchas) and re-run that directory.
## Gotchas
- `corepack use` only updates an existing `packageManager` field. If a
package.json lacks the field, corepack walks up to the nearest ancestor
that has one and stamps that file instead; the member stays unstamped.
After the sweep, assert every package.json carries the field. For a
missing one, insert the identical `pnpm@<version>+sha512.<hash>` string,
then re-run `corepack use pnpm@<tag>` in that directory.
- A workspace may fail with `ERR_PNPM_IGNORED_BUILDS`, and pnpm then writes
a placeholder scaffold into its `pnpm-workspace.yaml`:
`allowBuilds: esbuild: set this to true or false`. Current pnpm writes
only the `allowBuilds` placeholder; any legacy key still present
(`ignoredBuiltDependencies`, `onlyBuiltDependencies`,
`neverBuiltDependencies`) is ignored since pnpm 11. Repo convention is
`allowBuilds: esbuild: true`. Replace the placeholder and drop the
legacy entry, then re-run.
- `plugins/apps/composable-test-suite` once had its own
`pnpm-workspace.yaml` and acted as a nested workspace root. That state is
gone on purpose: pnpm picks the nearest `pnpm-workspace.yaml` walking up,
so a nested one silently forks install and lockfile behavior. Do not
reintroduce it.
- Expect metadata-only lockfile diffs when only the pnpm version moves:
the pnpm self-reference entries, plus a new `packageManagerDependencies`
section in lockfiles last written by older pnpm. Large diffs mean
re-resolution; inspect them before accepting.
## Verification
- Every `packageManager` field is byte-identical (same version and hash).
- `pnpm --version` in each workspace prints the target version.
- `pnpm install --frozen-lockfile` succeeds in each of the 11 workspaces.
- `git diff` on lockfiles matches the expectations above.
## Cleaning stale node_modules
- `scripts/clean-node-modules` removes every workspace `node_modules`: the
repo root, all module workspaces, and all member packages. Use it when
installs misbehave after dependency changes: clean, reinstall, done.
- Flags: `-n/--dry-run` lists without deleting; `--store` also removes the
shared pnpm store at `<repo>/.pnpm-store` (the next install re-downloads
what it held). `external/` (vendored dependency trees with their own
lifecycles) and `.opencode/` are always ignored.
- The script never touches the pnpm store by default, so the reinstall
after cleaning reuses cached packages (zero downloads).
- After cleaning, run `pnpm install` in each workspace root to restore the
development environment; `frontend` postinstall also reinstalls and
builds `plugins-runtime`.
+2 -25
View File
@@ -8,15 +8,9 @@
wait for the user to push. Do not change the remote URL, do not switch SSH↔HTTPS.
- **Never amend a commit that has been pushed** unless the user explicitly asks.
If the user pushes, treat that commit as final from the agent's side.
- **Never edit `CHANGES.md` by hand** in commits or PRs. The changelog is
generated from GitHub milestones during the release process; update it only
via the `update-changelog` skill flow or on explicit user request.
- **Never pipe test output directly to filters** (`| head`, `| tail`, `| grep`, etc.).
Always redirect to a file first: `command > /tmp/output.txt 2>&1`, then read/grep the file.
This prevents hiding test failures. See `mem:testing` for details.
- **`.claude/skills` is a symlink to `.agents/skills`.**
Edit skills only in their canonical location (`.agents/skills`); never edit
through `.claude/skills`.
- **Read the workflow memory BEFORE the corresponding action**:
- Before `git commit``mem:workflow/creating-commits` (commit format, AI-assisted-by trailer)
- Before `gh issue create``mem:workflow/creating-issues` (title derivation, body template, Issue Type)
@@ -46,27 +40,10 @@ Skipping this step is the #1 cause of incorrect or incomplete work.
`github.com/penpot/penpot/security/advisories/GHSA-*`, extract the GHSA ID
from the URL and run `python3 scripts/gh.py advisories <GHSA-ID>` to fetch
full advisory details before proceeding.
- **Issue or PR mentioned** — When the user mentions a penpot/penpot issue or
PR (URL like `github.com/penpot/penpot/issues/<n>` / `.../pull/<n>`, or a
bare `#<n>` when context clearly refers to this repo), fetch details via CLI
instead of WebFetch:
- Issue → `gh issue view <n> --repo penpot/penpot` (add `--comments` when
discussion context matters).
- Single PR → `gh pr view <n> --repo penpot/penpot`.
- Multiple PRs (list, file, or milestone) → `python3 scripts/gh.py prs ...`.
Do this before proceeding. Only use WebFetch if the CLI fails.
## Writing Rules
Writing rules, from Orwell, 1946. These govern prose: docs, PR text, messages. Never touch code or technical terms; swap in everyday words only where precision survives.
1. Never use a metaphor, simile or other figure of speech which you are used to seeing in print.
2. Never use a long word where a short one will do.
3. If it is possible to cut a word out, always cut it out.
4. Never use the passive where you can use the active.
5. Never use a foreign phrase, a scientific word or a jargon word if you can think of an everyday English equivalent.
6. Break any of these rules sooner than say anything outright barbarous.
Review every prose output against these rules before delivering.
Use the `ste` skill when the user explicitly requests STE, `/ste`, or ASD-STE100.
---
@@ -148,6 +125,6 @@ precision while maintaining a strong focus on maintainability and performance.
- `scripts/nrepl-eval.mjs` — Evaluate Clojure code via nREPL (backend + frontend).
- `scripts/check-commit` — Validate commit messages against Penpot's commit guidelines.
- `scripts/check-fmt-clj` — Check Clojure formatting without modifying files.
- `scripts/ci` — CI orchestration script for running lint, tests, and format checks across modules. See `mem:scripts/ci`.
- `scripts/ci` — CI orchestration script for running lint, tests, and format checks across modules. See `scripts/ci --help`.
- `scripts/gh.py` — Multi-purpose GitHub CLI helper. Subcommands: `issues` (list issues in a milestone), `prs` (fetch PR details), `advisories` (list/inspect security advisories). See `python3 scripts/gh.py --help`.
+5 -184
View File
@@ -1,60 +1,7 @@
# CHANGELOG
## 2.19.0 (Unreleased)
### :rocket: Epics and highlights
- Add configurable keyboard shortcuts [#9924](https://github.com/penpot/penpot/issues/9924) (PR: [#10237](https://github.com/penpot/penpot/pull/10237))
- Improve path operations and edition in the path editor [#10889](https://github.com/penpot/penpot/issues/10889) (PR: [#10807](https://github.com/penpot/penpot/pull/10807))
- Add auto-linking of libraries during import based on slugified name [#9263](https://github.com/penpot/penpot/issues/9263) (PR: [#9958](https://github.com/penpot/penpot/pull/9958))
### :bug: Bugs fixed
- Fix copying text from Penpot to the clipboard not working on MS Windows [#11303](https://github.com/penpot/penpot/issues/11303) (PR: [#11305](https://github.com/penpot/penpot/pull/11305))
- Fix performance issue with WebGL render [#11240](https://github.com/penpot/penpot/issues/11240) (PR: [#11259](https://github.com/penpot/penpot/pull/11259))
- Fix comment bubbles rendering on top of workspace dropdown menus [#10283](https://github.com/penpot/penpot/issues/10283) (PR: [#11201](https://github.com/penpot/penpot/pull/11201))
- Fix inconsistent Mixed label in blur options and numeric inputs across 24 locales (by @filipsajdak) [#11148](https://github.com/penpot/penpot/issues/11148) (PR: [#11151](https://github.com/penpot/penpot/pull/11151))
- Fix overlay shifting left when shown with top-center alignment in viewer prototype (by @filipsajdak) [#9048](https://github.com/penpot/penpot/issues/9048) (PR: [#10454](https://github.com/penpot/penpot/pull/10454))
- Fix internal error when clicking the Copy button on the Access Token page (by @0xTHAC0) [#8496](https://github.com/penpot/penpot/issues/8496) (PR: [#11156](https://github.com/penpot/penpot/pull/11156))
- Fix `disable-registration` flag not preventing non-users from creating accounts in the share prototypes page (by @0xTHAC0) [#5164](https://github.com/penpot/penpot/issues/5164) (PR: [#11199](https://github.com/penpot/penpot/pull/11199))
- Fix "Cannot assign to read only property 'toString'" error during text resize (by @makesomethingshit) [#10168](https://github.com/penpot/penpot/issues/10168) (PR: [#11521](https://github.com/penpot/penpot/pull/11521))
- Fix plugin postMessage channel broadcasting messages to all plugins without origin validation [#10968](https://github.com/penpot/penpot/issues/10968) (PR: [#10970](https://github.com/penpot/penpot/pull/10970))
- Fix MCP plugin page navigation while connected crashing the workspace (by @makesomethingshit) [#11001](https://github.com/penpot/penpot/issues/11001) (PR: [#11521](https://github.com/penpot/penpot/pull/11521))
- Fix shortcut search never matching on key combination, only on action label [#11003](https://github.com/penpot/penpot/issues/11003) (PR: [#11081](https://github.com/penpot/penpot/pull/11081))
- Fix Shift + special character key shortcut capturing the shifted character instead of the physical key [#11004](https://github.com/penpot/penpot/issues/11004) (PR: [#11081](https://github.com/penpot/penpot/pull/11081))
- Fix reassigning the "Paste" shortcut not updating the UI or taking effect in the workspace [#11005](https://github.com/penpot/penpot/issues/11005) (PR: [#11081](https://github.com/penpot/penpot/pull/11081))
- Fix font-size dropdown clipping multi-digit values in Firefox (by @0xTHAC0) [#11008](https://github.com/penpot/penpot/issues/11008) (PR: [#11162](https://github.com/penpot/penpot/pull/11162), [#11500](https://github.com/penpot/penpot/pull/11500))
- Fix exporting shortcuts producing an invalid "toggle-fullscreen" entry that breaks re-import [#11032](https://github.com/penpot/penpot/issues/11032) (PR: [#11081](https://github.com/penpot/penpot/pull/11081))
- Fix plugin API missing permission checks in tokens, shapes, variants, flows, layouts, and user identity [#11137](https://github.com/penpot/penpot/issues/11137) (PR: [#11139](https://github.com/penpot/penpot/pull/11139))
- Fix library summary Redis cache keys omitting the tenant [#11407](https://github.com/penpot/penpot/issues/11407) (PR: [#11408](https://github.com/penpot/penpot/pull/11408))
- Fix active theme name in the inspect tab displaying an id instead of the name [#11437](https://github.com/penpot/penpot/issues/11437) (PR: [#11439](https://github.com/penpot/penpot/pull/11439))
- Fix triple-click not selecting the full line in text editor v3 [#11483](https://github.com/penpot/penpot/issues/11483) (PR: [#11493](https://github.com/penpot/penpot/pull/11493))
- Fix pasted text losing formatting on last lines after resizing and adding new lines from the top [#11501](https://github.com/penpot/penpot/issues/11501) (PR: [#11503](https://github.com/penpot/penpot/pull/11503))
- Fix variant property dropdown appearing empty and throwing an internal error when the component has no sibling variants [#11524](https://github.com/penpot/penpot/issues/11524) (PR: [#11499](https://github.com/penpot/penpot/pull/11499))
### :sparkles: New features & Enhancements
- Make backend storage resilient to interrupted writes, missing files and stalled cleanup [#11344](https://github.com/penpot/penpot/issues/11344) (PR: [#11345](https://github.com/penpot/penpot/pull/11345))
- Implement RTL support in the text editor v3 [#11262](https://github.com/penpot/penpot/issues/11262)
- Improve path operations and edition in the path editor [#10889](https://github.com/penpot/penpot/issues/10889) (PR: [#10807](https://github.com/penpot/penpot/pull/10807))
- Add configurable keyboard shortcuts [#9924](https://github.com/penpot/penpot/issues/9924) (PR: [#10237](https://github.com/penpot/penpot/pull/10237))
- Add auto-linking of libraries during import based on slugified name [#9263](https://github.com/penpot/penpot/issues/9263) (PR: [#9958](https://github.com/penpot/penpot/pull/9958))
- Add support for internal libraries and file sync for Design Tokens [#9334](https://github.com/penpot/penpot/issues/9334)
- Warn self-hosted users when their Penpot version is outdated and surface what they're missing [#10497](https://github.com/penpot/penpot/issues/10497) (PR: [#11411](https://github.com/penpot/penpot/pull/11411))
- Add dedicated RPC methods for plugin registry operations with permission validation [#10952](https://github.com/penpot/penpot/issues/10952) (PR: [#10957](https://github.com/penpot/penpot/pull/10957))
- Document MCP and internal resolver environment variables (by @ShreyashAgare26) [#11318](https://github.com/penpot/penpot/issues/11318) (PR: [#11572](https://github.com/penpot/penpot/pull/11572))
- Add tokens source indicator to assets tab [#11365](https://github.com/penpot/penpot/issues/11365) (PR: [#11439](https://github.com/penpot/penpot/pull/11439))
- Export multiple fills to SVG [#11466](https://github.com/penpot/penpot/issues/11466) (PR: [#11467](https://github.com/penpot/penpot/pull/11467))
- Add Penpot-specific board size presets (file thumbnail, template cover, plugin icon/cover) [#11561](https://github.com/penpot/penpot/issues/11561) (PR: [#11565](https://github.com/penpot/penpot/pull/11565))
## 2.18.0 (Unreleased)
### :rocket: Epics and highlights
- Group toolbar drawing tools into shape and free-draw flyouts [#9316](https://github.com/penpot/penpot/issues/9316) (PR: [#9480](https://github.com/penpot/penpot/pull/9480), [#10354](https://github.com/penpot/penpot/pull/10354))
- Add dedicated Line and Arrow drawing tools (by @davidv399) [#9145](https://github.com/penpot/penpot/issues/9145) (PR: [#9146](https://github.com/penpot/penpot/pull/9146))
- Show and manage comments while designing in the workspace [#10239](https://github.com/penpot/penpot/issues/10239) (PR: [#10275](https://github.com/penpot/penpot/pull/10275))
### :bug: Bugs fixed
- Fix MCP integration hanging when the Penpot tab is backgrounded or frozen by the browser [#10323](https://github.com/penpot/penpot/issues/10323) (PR: [#10392](https://github.com/penpot/penpot/pull/10392))
@@ -64,136 +11,18 @@
- Fix plugin API addTheme calls failing with the signature shown in the high-level overview [#10074](https://github.com/penpot/penpot/issues/10074) (PR: [#10359](https://github.com/penpot/penpot/pull/10359))
- Fix empty text shape not being deleted on editor exit [#10540](https://github.com/penpot/penpot/issues/10540) (PR: [#10541](https://github.com/penpot/penpot/pull/10541))
- Fix broken token pills showing wrong default state when not selected [#10524](https://github.com/penpot/penpot/issues/10524) (PR: [#10535](https://github.com/penpot/penpot/pull/10535))
- Replace hyphens with bullets in subscription benefits list [#10547](https://github.com/penpot/penpot/issues/10547) (PR: [#10523](https://github.com/penpot/penpot/pull/10523))
- Fix Chinese (zh-CN) translation showing wrong label for Intersection in board path menu (by @sawirricardo) [#10346](https://github.com/penpot/penpot/issues/10346) (PR: [#10381](https://github.com/penpot/penpot/pull/10381))
- Fix invalid formulas being accepted in numeric inputs (by @AKnassa) [#9581](https://github.com/penpot/penpot/issues/9581) (PR: [#10659](https://github.com/penpot/penpot/pull/10659))
- Fix radial gradient handles blowing up in size when rotated on ellipses (by @AKnassa) [#10069](https://github.com/penpot/penpot/issues/10069) (PR: [#10666](https://github.com/penpot/penpot/pull/10666))
- Fix plugin API validation errors being too generic to diagnose the failure (by @AKnassa) [#10072](https://github.com/penpot/penpot/issues/10072) (PR: [#10667](https://github.com/penpot/penpot/pull/10667))
- Fix crash with referential integrity error when deleting a component inside a grid (by @Alotor) [#10101](https://github.com/penpot/penpot/issues/10101) (PR: [#10956](https://github.com/penpot/penpot/pull/10956))
- Fix component copies not preserving rotation when the main component has changes [#10109](https://github.com/penpot/penpot/issues/10109) (PR: [#10574](https://github.com/penpot/penpot/pull/10574))
- Fix text width and height staying stale after setting growType in the plugin API [#10207](https://github.com/penpot/penpot/issues/10207) (PR: [#9898](https://github.com/penpot/penpot/pull/9898))
- Fix padding not painted until expanding the 4-sides padding option [#10278](https://github.com/penpot/penpot/issues/10278) (PR: [#10602](https://github.com/penpot/penpot/pull/10602))
- Fix files with custom fonts breaking with a referential integrity error when moved between teams (by @filipsajdak) [#10496](https://github.com/penpot/penpot/issues/10496) (PR: [#10837](https://github.com/penpot/penpot/pull/10837))
- Fix clicking overlapping comment bubbles zooming to 20000% without showing the comments [#10526](https://github.com/penpot/penpot/issues/10526) (PR: [#10543](https://github.com/penpot/penpot/pull/10543))
- Fix user menu subsections in the dashboard not closing when hovering away from the parent option (by @AKnassa) [#10549](https://github.com/penpot/penpot/issues/10549) (PR: [#10639](https://github.com/penpot/penpot/pull/10639))
- Fix self-hosted env-generated config.js being cached for 7 days so PENPOT_FLAGS changes did not reach already-cached browsers (by @filipsajdak) [#10556](https://github.com/penpot/penpot/issues/10556) (PR: [#11146](https://github.com/penpot/penpot/pull/11146))
- Fix color of selected text in light theme [#10570](https://github.com/penpot/penpot/issues/10570) (PR: [#10614](https://github.com/penpot/penpot/pull/10614))
- Fix margin input order being inconsistent with padding inputs and between collapsed and expanded states [#10578](https://github.com/penpot/penpot/issues/10578) (PR: [#10797](https://github.com/penpot/penpot/pull/10797))
- Fix uncaught DOMException when writing image/svg+xml content to the clipboard (by @AKnassa) [#10596](https://github.com/penpot/penpot/issues/10596) (PR: [#10663](https://github.com/penpot/penpot/pull/10663))
- Fix tick icons not aligned in the font selector [#10597](https://github.com/penpot/penpot/issues/10597) (PR: [#10774](https://github.com/penpot/penpot/pull/10774))
- Fix incorrect padding values when multiple shapes are selected [#10598](https://github.com/penpot/penpot/issues/10598) (PR: [#10602](https://github.com/penpot/penpot/pull/10602))
- Fix integrity errors related to variants not being repaired [#10606](https://github.com/penpot/penpot/issues/10606) (PR: [#10768](https://github.com/penpot/penpot/pull/10768))
- Fix changing password showing 'Password should be at least 8 characters' error on the old password field (by @AKnassa) [#10626](https://github.com/penpot/penpot/issues/10626) (PR: [#10661](https://github.com/penpot/penpot/pull/10661))
- Fix stroke caps disappearing when dragging [#10633](https://github.com/penpot/penpot/issues/10633) (PR: [#10634](https://github.com/penpot/penpot/pull/10634))
- Fix layout padding being saved as string after invalid input in multi-selection, causing persistence errors (by @niwinz) [#10638](https://github.com/penpot/penpot/issues/10638) (PR: [#10758](https://github.com/penpot/penpot/pull/10758))
- Fix inconsistent theme handling between Penpot and plugins [#10676](https://github.com/penpot/penpot/issues/10676) (PR: [#10677](https://github.com/penpot/penpot/pull/10677))
- Fix image stroke (strokeImage) support missing in the plugin API Stroke interface [#10682](https://github.com/penpot/penpot/issues/10682) (PR: [#10683](https://github.com/penpot/penpot/pull/10683))
- Fix SVG images not working as fill in the WebGL renderer [#10705](https://github.com/penpot/penpot/issues/10705) (PR: [#10707](https://github.com/penpot/penpot/pull/10707))
- Fix background blur not working on text shapes [#10706](https://github.com/penpot/penpot/issues/10706) (PR: [#10712](https://github.com/penpot/penpot/pull/10712))
- Fix background blur not applying on strokes [#10713](https://github.com/penpot/penpot/issues/10713) (PR: [#10716](https://github.com/penpot/penpot/pull/10716))
- Fix text shape with empty content breaking workspace updates [#10725](https://github.com/penpot/penpot/issues/10725) (PR: [#10731](https://github.com/penpot/penpot/pull/10731))
- Fix missing SVG option in the file filters when adding an image fill (by @LuBoys) [#10756](https://github.com/penpot/penpot/issues/10756) (PR: [#10771](https://github.com/penpot/penpot/pull/10771))
- Update onboarding image [#10779](https://github.com/penpot/penpot/issues/10779) (PR: [#10783](https://github.com/penpot/penpot/pull/10783))
- Fix main toolbar overlapping the grid edition bar [#10788](https://github.com/penpot/penpot/issues/10788) (PR: [#10789](https://github.com/penpot/penpot/pull/10789))
- Fix WASM renderer panic when the WebGL context is restored mid-reload [#10810](https://github.com/penpot/penpot/issues/10810) (PR: [#10824](https://github.com/penpot/penpot/pull/10824))
- Fix nginx frontend forwarding the client Host header to backend/exporter, breaking Istio strict mTLS routing (by @yamila-moreno) [#10835](https://github.com/penpot/penpot/issues/10835) (PR: [#11233](https://github.com/penpot/penpot/pull/11233))
- Fix tutorial templates with components causing errors [#10839](https://github.com/penpot/penpot/issues/10839)
- Fix plugin 'Try out' flow crashing when projects have not loaded yet [#10858](https://github.com/penpot/penpot/issues/10858) (PR: [#10859](https://github.com/penpot/penpot/pull/10859))
- Fix collapsed Fill color section on the design panel for new texts [#10860](https://github.com/penpot/penpot/issues/10860) (PR: [#10972](https://github.com/penpot/penpot/pull/10972))
- Fix grid item date tooltip in the project view showing 'Will be deleted' instead of creation date (by @0xTHAC0) [#10873](https://github.com/penpot/penpot/issues/10873) (PR: [#11161](https://github.com/penpot/penpot/pull/11161))
- Merge stop and start measurement shortcut to match current behavior [#10884](https://github.com/penpot/penpot/issues/10884) (PR: [#10906](https://github.com/penpot/penpot/pull/10906))
- Fix shape size badge displayed twice when a user with Viewer permissions selects a shape [#10893](https://github.com/penpot/penpot/issues/10893) (PR: [#10985](https://github.com/penpot/penpot/pull/10985))
- Fix main menu being covered by the toolbar [#10902](https://github.com/penpot/penpot/issues/10902) (PR: [#10926](https://github.com/penpot/penpot/pull/10926))
- Fix font family typography asset persisting across files in newly created text layers [#10925](https://github.com/penpot/penpot/issues/10925) (PR: [#11134](https://github.com/penpot/penpot/pull/11134))
- Fix error raised when editing justified text [#10944](https://github.com/penpot/penpot/issues/10944) (PR: [#10945](https://github.com/penpot/penpot/pull/10945))
- Fix MCP WebSocket proxy failing after penpot-mcp container restarts due to stale nginx DNS resolution (by @780Farva) [#10946](https://github.com/penpot/penpot/issues/10946) (PR: [#10947](https://github.com/penpot/penpot/pull/10947))
- Fix verification email address being unreadable due to low-contrast text on the register success page [#10950](https://github.com/penpot/penpot/issues/10950) (PR: [#10965](https://github.com/penpot/penpot/pull/10965))
- Fix image swatches displaying a wrong format in the color picker list view [#10951](https://github.com/penpot/penpot/issues/10951) (PR: [#10975](https://github.com/penpot/penpot/pull/10975))
- Fix text editor crashing when dropping dragged text after selecting all content [#10954](https://github.com/penpot/penpot/issues/10954) (PR: [#10959](https://github.com/penpot/penpot/pull/10959))
- Fix MCP tokens being usable as API access tokens [#10960](https://github.com/penpot/penpot/issues/10960) (PR: [#10962](https://github.com/penpot/penpot/pull/10962))
- Add size limit and rate limiting to the send-user-feedback endpoint [#10979](https://github.com/penpot/penpot/issues/10979) (PR: [#10990](https://github.com/penpot/penpot/pull/10990))
- Fix main menu not keeping alignment when the left sidebar is expanded [#10981](https://github.com/penpot/penpot/issues/10981) (PR: [#10986](https://github.com/penpot/penpot/pull/10986))
- Fix update-profile-props RPC method accepting undocumented keys [#10991](https://github.com/penpot/penpot/issues/10991) (PR: [#10992](https://github.com/penpot/penpot/pull/10992))
- Fix import-binfile RPC method schema accepting a file-id parameter [#10993](https://github.com/penpot/penpot/issues/10993) (PR: [#10994](https://github.com/penpot/penpot/pull/10994))
- Fix assemble-chunks session lookup ignoring the profile-id scope [#11011](https://github.com/penpot/penpot/issues/11011) (PR: [#11012](https://github.com/penpot/penpot/pull/11012))
- Validate font-id team ownership in create-font-variant [#11013](https://github.com/penpot/penpot/issues/11013) (PR: [#11014](https://github.com/penpot/penpot/pull/11014))
- Validate team ownership on file library link endpoints [#11015](https://github.com/penpot/penpot/issues/11015) (PR: [#11016](https://github.com/penpot/penpot/pull/11016))
- Limit object size allocation in the V1 binfile parser [#11017](https://github.com/penpot/penpot/issues/11017) (PR: [#11018](https://github.com/penpot/penpot/pull/11018))
- Limit recursion depth in the Fressian reader [#11019](https://github.com/penpot/penpot/issues/11019) (PR: [#11020](https://github.com/penpot/penpot/pull/11020))
- Limit concurrent imports in the import-binfile RPC method [#11023](https://github.com/penpot/penpot/issues/11023) (PR: [#11024](https://github.com/penpot/penpot/pull/11024))
- Validate content-type on management upload endpoints [#11025](https://github.com/penpot/penpot/issues/11025) (PR: [#11026](https://github.com/penpot/penpot/pull/11026))
- Fix webhook endpoints allowing unauthorized access via creator-id fallback [#11028](https://github.com/penpot/penpot/issues/11028) (PR: [#11029](https://github.com/penpot/penpot/pull/11029))
- Escape markdown in user-controlled fields of Mattermost error notifications [#11033](https://github.com/penpot/penpot/issues/11033) (PR: [#11034](https://github.com/penpot/penpot/pull/11034))
- Enforce file read permission check on asset endpoints [#11035](https://github.com/penpot/penpot/issues/11035) (PR: [#11036](https://github.com/penpot/penpot/pull/11036))
- Add accumulated storage byte quota for media uploads [#11037](https://github.com/penpot/penpot/issues/11037) (PR: [#11038](https://github.com/penpot/penpot/pull/11038))
- Add bounding box dimension limit to exports [#11041](https://github.com/penpot/penpot/issues/11041) (PR: [#11042](https://github.com/penpot/penpot/pull/11042))
- Sanitize embedded scripts in SVG uploads [#11043](https://github.com/penpot/penpot/issues/11043) (PR: [#11044](https://github.com/penpot/penpot/pull/11044))
- Fix duplicate file ID returning inconsistent error responses [#11045](https://github.com/penpot/penpot/issues/11045) (PR: [#11050](https://github.com/penpot/penpot/pull/11050))
- Enforce permission checks in WebSocket subscription handlers [#11052](https://github.com/penpot/penpot/issues/11052) (PR: [#11054](https://github.com/penpot/penpot/pull/11054))
- Fix 'something went wrong' popup when using incremental numerical input interaction [#11053](https://github.com/penpot/penpot/issues/11053) (PR: [#10794](https://github.com/penpot/penpot/pull/10794))
- Enforce password complexity validation on the backend [#11055](https://github.com/penpot/penpot/issues/11055) (PR: [#11059](https://github.com/penpot/penpot/pull/11059))
- Normalize string inputs before processing [#11060](https://github.com/penpot/penpot/issues/11060) (PR: [#11061](https://github.com/penpot/penpot/pull/11061))
- Add cooldown to avoid sending duplicate invitation emails [#11062](https://github.com/penpot/penpot/issues/11062) (PR: [#11063](https://github.com/penpot/penpot/pull/11063))
- Enable SSRF protection for organization SSO validation [#11064](https://github.com/penpot/penpot/issues/11064) (PR: [#11065](https://github.com/penpot/penpot/pull/11065))
- Fix clone-file-media-object allowing to clone media objects from files without read access [#11087](https://github.com/penpot/penpot/issues/11087) (PR: [#11090](https://github.com/penpot/penpot/pull/11090))
- Fix 404 error page logo not visible in dark mode [#11091](https://github.com/penpot/penpot/issues/11091) (PR: [#11167](https://github.com/penpot/penpot/pull/11167))
- Fix incorrect permission handling when creating an invitation [#11098](https://github.com/penpot/penpot/issues/11098) (PR: [#11099](https://github.com/penpot/penpot/pull/11099))
- Reject zero or negative total-chunks values in upload sessions [#11103](https://github.com/penpot/penpot/issues/11103) (PR: [#11104](https://github.com/penpot/penpot/pull/11104))
- Fix import-binfile accepting unsupported version values without validation [#11105](https://github.com/penpot/penpot/issues/11105) (PR: [#11107](https://github.com/penpot/penpot/pull/11107))
- Fix sessions remaining active on other devices after account deletion [#11114](https://github.com/penpot/penpot/issues/11114) (PR: [#11115](https://github.com/penpot/penpot/pull/11115))
- Use random UUIDs for share link IDs instead of a predictable scheme [#11116](https://github.com/penpot/penpot/issues/11116) (PR: [#11117](https://github.com/penpot/penpot/pull/11117))
- Fix plugin manifest fetch hanging indefinitely without timeout [#11119](https://github.com/penpot/penpot/issues/11119) (PR: [#11120](https://github.com/penpot/penpot/pull/11120))
- Use constant-time comparison for shared key authentication [#11121](https://github.com/penpot/penpot/issues/11121) (PR: [#11122](https://github.com/penpot/penpot/pull/11122))
- Fix ESC key not closing the comment input box after posting a comment in the workspace [#11128](https://github.com/penpot/penpot/issues/11128) (PR: [#11131](https://github.com/penpot/penpot/pull/11131))
- Fix token edit modal crashing when resolving tokens with group nodes [#11143](https://github.com/penpot/penpot/issues/11143) (PR: [#11144](https://github.com/penpot/penpot/pull/11144))
- Fix text editor crashing when pasting into an empty text shape [#11149](https://github.com/penpot/penpot/issues/11149) (PR: [#11150](https://github.com/penpot/penpot/pull/11150))
- Fix comment avatars appearing on top of rulers when scrolling the canvas (by @filipsajdak) [#11163](https://github.com/penpot/penpot/issues/11163) (PR: [#11168](https://github.com/penpot/penpot/pull/11168))
- Fix infinite loop of get-teams and get-team-members calls when granting team access from an email link [#11215](https://github.com/penpot/penpot/issues/11215) (PR: [#11223](https://github.com/penpot/penpot/pull/11223))
- Fix RPC requests bypassing rate limiting with fractional bucket refill intervals [#11253](https://github.com/penpot/penpot/issues/11253) (PR: [#11254](https://github.com/penpot/penpot/pull/11254))
- Fix tempfile bucket serving objects to any authenticated user instead of only the uploader [#11269](https://github.com/penpot/penpot/issues/11269) (PR: [#11270](https://github.com/penpot/penpot/pull/11270))
- Fix increasing a value by clicking and dragging in a numeric input [#11274](https://github.com/penpot/penpot/issues/11274) (PR: [#11334](https://github.com/penpot/penpot/pull/11334))
- Fix notification pill rendering unescaped HTML in the detail section when importing tokens [#11276](https://github.com/penpot/penpot/issues/11276) (PR: [#11275](https://github.com/penpot/penpot/pull/11275))
- Fix share-link holders reading pages outside the authorized scope via the get-page RPC command [#11281](https://github.com/penpot/penpot/issues/11281) (PR: [#11284](https://github.com/penpot/penpot/pull/11284))
- Fix incorrect permission handling when managing share links on a file [#11289](https://github.com/penpot/penpot/issues/11289) (PR: [#11290](https://github.com/penpot/penpot/pull/11290))
- Fix backend session remaining valid after logout when the auth-token cookie is replayed [#11316](https://github.com/penpot/penpot/issues/11316) (PR: [#11317](https://github.com/penpot/penpot/pull/11317))
- Fix get-team-invitation-token requiring only read permissions [#11358](https://github.com/penpot/penpot/issues/11358) (PR: [#11359](https://github.com/penpot/penpot/pull/11359))
- Fix missing text in legacy SVG board thumbnails [#10182](https://github.com/penpot/penpot/issues/10182) (PR: [#11552](https://github.com/penpot/penpot/pull/11552))
- Fix workspace crash when applying transform modifiers in the WASM renderer [#10894](https://github.com/penpot/penpot/issues/10894) (PR: [#10896](https://github.com/penpot/penpot/pull/10896))
- Limit ZIP entry count and object size on V3 binfile import [#11021](https://github.com/penpot/penpot/issues/11021) (PR: [#11022](https://github.com/penpot/penpot/pull/11022))
- Block plugin UI iframe URLs targeting the Penpot domain [#11271](https://github.com/penpot/penpot/issues/11271) (PR: [#11273](https://github.com/penpot/penpot/pull/11273))
- Restrict the MCP REPL code execution endpoint to development environments [#11283](https://github.com/penpot/penpot/issues/11283) (PR: [#11282](https://github.com/penpot/penpot/pull/11282))
- Filter share-link tokens from the get-view-only-bundle response [#11285](https://github.com/penpot/penpot/issues/11285) (PR: [#11286](https://github.com/penpot/penpot/pull/11286))
- Disable MCP developer tools in multi-user mode [#11291](https://github.com/penpot/penpot/issues/11291) (PR: [#11310](https://github.com/penpot/penpot/pull/11310))
- Fix Hide comments setting being ignored after opening the Comments section [#11308](https://github.com/penpot/penpot/issues/11308) (PR: [#11492](https://github.com/penpot/penpot/pull/11492))
- Block NAT64/6to4/Teredo IPv6 transition addresses in the SSRF guard [#11319](https://github.com/penpot/penpot/issues/11319) (PR: [#11320](https://github.com/penpot/penpot/pull/11320))
- Prevent team admins from removing the team owner [#11367](https://github.com/penpot/penpot/issues/11367) (PR: [#11368](https://github.com/penpot/penpot/pull/11368))
- Enforce share-link comment permissions and page scope [#11370](https://github.com/penpot/penpot/issues/11370) (PR: [#11371](https://github.com/penpot/penpot/pull/11371))
- Clean up orphaned teams, projects and files on profile deletion [#11394](https://github.com/penpot/penpot/issues/11394) (PR: [#11395](https://github.com/penpot/penpot/pull/11395))
- Fix crash when pressing Ctrl+D with no shape selected [#11448](https://github.com/penpot/penpot/issues/11448) (PR: [#11491](https://github.com/penpot/penpot/pull/11491))
- Fix text layout not updating when auto-width is set by double-clicking the bounding box [#11480](https://github.com/penpot/penpot/issues/11480) (PR: [#11541](https://github.com/penpot/penpot/pull/11541))
- Fix boolean shapes rendering deformed in the WASM renderer and exports [#11482](https://github.com/penpot/penpot/issues/11482) (PR: [#11551](https://github.com/penpot/penpot/pull/11551))
### :sparkles: New features & Enhancements
- Group toolbar drawing tools into shape and free-draw flyouts [#9316](https://github.com/penpot/penpot/issues/9316) (PR: [#9480](https://github.com/penpot/penpot/pull/9480), [#10354](https://github.com/penpot/penpot/pull/10354))
- Add outline stroke to Paths [#9961](https://github.com/penpot/penpot/issues/9961) (PR: [#8677](https://github.com/penpot/penpot/pull/8677))
- Make throwValidationErrors default to true for v2 manifest plugins [#10401](https://github.com/penpot/penpot/issues/10401) (PR: [#10433](https://github.com/penpot/penpot/pull/10433))
- Add dedicated Line and Arrow drawing tools (by @davidv399) [#9145](https://github.com/penpot/penpot/issues/9145) (PR: [#9146](https://github.com/penpot/penpot/pull/9146))
- Refactor wasm rulers and UI state [#10116](https://github.com/penpot/penpot/issues/10116) (PR: [#10461](https://github.com/penpot/penpot/pull/10461))
- Improve team invitations modal in the dashboard [#10484](https://github.com/penpot/penpot/issues/10484) (PR: [#10459](https://github.com/penpot/penpot/pull/10459))
- Highlight the first matching font in the font list when searching (by @ai-mountain) [#3204](https://github.com/penpot/penpot/issues/3204) (PR: [#9512](https://github.com/penpot/penpot/pull/9512), [#10450](https://github.com/penpot/penpot/pull/10450))
- Preserve token references when copying and pasting properties instead of resolving them to values (by @AKnassa) [#9582](https://github.com/penpot/penpot/issues/9582) (PR: [#10665](https://github.com/penpot/penpot/pull/10665))
- Add waitForLayoutUpdate method to the plugin API [#10136](https://github.com/penpot/penpot/issues/10136) (PR: [#9898](https://github.com/penpot/penpot/pull/9898))
- Show and manage comments while designing in the workspace [#10239](https://github.com/penpot/penpot/issues/10239) (PR: [#10275](https://github.com/penpot/penpot/pull/10275))
- Simplify MCP server configuration for common MCP clients [#10355](https://github.com/penpot/penpot/issues/10355) (PR: [#10604](https://github.com/penpot/penpot/pull/10604))
- Remove misleading MCP client JSON snippet from the key-generated modal (by @Shlok1729) [#10399](https://github.com/penpot/penpot/issues/10399) (PR: [#10415](https://github.com/penpot/penpot/pull/10415))
- Preview font families in the font selector [#10403](https://github.com/penpot/penpot/issues/10403) (PR: [#10411](https://github.com/penpot/penpot/pull/10411))
- Remember expanded/collapsed state of token sets in the color tokens picker (session scope) [#10551](https://github.com/penpot/penpot/issues/10551) (PR: [#10864](https://github.com/penpot/penpot/pull/10864))
- Show token sets in reverse order by default in the color tokens picker (by @rhinocap) [#10552](https://github.com/penpot/penpot/issues/10552) (PR: [#10658](https://github.com/penpot/penpot/pull/10658))
- Add multi-selection and bulk delete support to pages in the workspace sitemap [#10580](https://github.com/penpot/penpot/issues/10580) (PR: [#10581](https://github.com/penpot/penpot/pull/10581))
- Add a grid/list view toggle for files in the dashboard [#10691](https://github.com/penpot/penpot/issues/10691) (PR: [#10692](https://github.com/penpot/penpot/pull/10692))
- Migrate Docker images to Docker Hardened Images (DHI) [#10720](https://github.com/penpot/penpot/issues/10720) (PR: [#10732](https://github.com/penpot/penpot/pull/10732), [#10733](https://github.com/penpot/penpot/pull/10733), [#10734](https://github.com/penpot/penpot/pull/10734))
- Adopt React Aria [#10802](https://github.com/penpot/penpot/issues/10802) (PR: [#10675](https://github.com/penpot/penpot/pull/10675))
- Add plugin API function for awaiting component updates beyond waitForLayoutUpdate [#10927](https://github.com/penpot/penpot/issues/10927) (PR: [#10964](https://github.com/penpot/penpot/pull/10964))
- Emit open-workspace-file audit event with file statistics on workspace load [#11106](https://github.com/penpot/penpot/issues/11106) (PR: [#11138](https://github.com/penpot/penpot/pull/11138))
## 2.17.2
### :bug: Bugs fixed
@@ -233,10 +62,6 @@
### :rocket: Epics and highlights
- Render prototype viewer with WASM (Skia) engine instead of SVG [#10037](https://github.com/penpot/penpot/issues/10037) (PR: [#10038](https://github.com/penpot/penpot/pull/10038))
- Add layer blur effect for visual depth and styling [#9844](https://github.com/penpot/penpot/issues/9844) (PR: [#10034](https://github.com/penpot/penpot/pull/10034))
- Render guides in WebGL for consistent viewer performance [#10068](https://github.com/penpot/penpot/issues/10068) (PR: [#10014](https://github.com/penpot/penpot/pull/10014))
- Add concurrency limiter and status indicators for MCP server communications [#9493](https://github.com/penpot/penpot/issues/9493) (PR: [#9748](https://github.com/penpot/penpot/pull/9748))
- Add typography token row to multiselected texts for better token visibility [#9336](https://github.com/penpot/penpot/issues/9336) (PR: [#9128](https://github.com/penpot/penpot/pull/9128))
### :sparkles: New features & Enhancements
@@ -412,7 +237,7 @@
### :rocket: Epics and highlights
- WebGL rendering (beta) user preference [#9683](https://github.com/penpot/penpot/issues/9683) (PR: [#9113](https://github.com/penpot/penpot/pull/9113))
- WebGL rendering (beta) user preference [#9683](https://github.com/penpot/penpot/issues/9683) (PR:[9113](https://github.com/penpot/penpot/pull/9113))
- Design Tokens at the design tab: numeric fields with token selection in place [#9358](https://github.com/penpot/penpot/issues/9358)
### :sparkles: New features & Enhancements
@@ -591,10 +416,6 @@
## 2.15.0
### :rocket: Epics and highlights
- Add MCP server integration for AI-assisted design workflows [#9174](https://github.com/penpot/penpot/issues/9174) (PR: [#9032](https://github.com/penpot/penpot/pull/9032), [#9321](https://github.com/penpot/penpot/pull/9321))
### :sparkles: New features & Enhancements
- Add MCP server integration [GH #9174](https://github.com/penpot/penpot/issues/9174)
@@ -3212,7 +3033,7 @@ is a number of cores)
- Enable penpot SVG metadata only when exporting complete files [Taiga #1914](https://tree.taiga.io/project/penpot/us/1914?milestone=295883)
- Export to PDF all artboards of one page [Taiga #1895](https://tree.taiga.io/project/penpot/us/1895)
- Go to a undo step clicking on a history element of the list [Taiga #1374](https://tree.taiga.io/project/penpot/us/1374)
- Increment font size by 10 with shift+arrows [#1047](https://github.com/penpot/penpot/issues/1047)
- Increment font size by 10 with shift+arrows [1047](https://github.com/penpot/penpot/issues/1047)
- New shortcut to detach components Ctrl+Shift+K [Taiga #1799](https://tree.taiga.io/project/penpot/us/1799)
- Set email inputs to type "email", to aid keyboard entry [Taiga #1921](https://tree.taiga.io/project/penpot/issue/1921)
- Use shift+move to move element orthogonally [#823](https://github.com/penpot/penpot/issues/823)
-3
View File
@@ -1,3 +0,0 @@
Read and follow the instructions in `AGENTS.md`.
Treat `AGENTS.md` as the canonical project instruction file.
-13
View File
@@ -14,7 +14,6 @@ Center](https://help.penpot.app/).
- [Reporting Bugs](#reporting-bugs)
- [Pull Requests](#pull-requests)
- [Workflow](#workflow)
- [Branch naming](#branch-naming)
- [Format](#format)
- [Title format](#title-format)
- [Description](#description)
@@ -74,18 +73,6 @@ Advisories](https://github.com/penpot/penpot/security/advisories)
4. **Format and lint** — run the checks described in
[Formatting and Linting](#formatting-and-linting) before submitting.
### Branch naming
Branch names are not enforced, but we recommend the following:
- **`issue-NNNN`** — when working from a GitHub issue, name the branch after
it (e.g. `issue-11525`). This makes each PR's origin self-evident.
- Otherwise, use a short, descriptive name with words separated by hyphens
and no slashes (e.g. `fix-ellipse-icon-typo`, `feat-auto-link-libraries`).
Since PRs are squash-merged, the branch name does not survive into the
commit history — what matters is the [PR title](#title-format).
### Format
#### Title
-7
View File
@@ -56,7 +56,6 @@ If your organization is scaling and needs extra support, were here to help. [
- [Why Penpot](#why-penpot)
- [Getting Started](#getting-started)
- [Penpot Enterprise](#penpot-enterprise)
- [Community](#community)
- [Contributing](#contributing)
- [Resources](#resources)
@@ -94,12 +93,6 @@ Penpot is the only design & prototype platform that is deployment agnostic. You
Learn how to install it with Docker, Kubernetes, Elestio or other options on [our website](https://penpot.app/self-host).
<img width="100%" height="1010" alt="2" src="https://github.com/user-attachments/assets/243e796e-a140-481a-b68f-b24be6a70e37" />
## Penpot Enterprise ##
Penpot Enterprise is our paid plan for organizations that need to scale their design work across multiple teams with advanced governance, security, and administration. Manage teams and access from a centralized **Admin Console**, configure advanced permissions, and connect your **identity provider through SSO**. Available for cloud and self-hosted environments, it combines enterprise controls with Penpots open-source foundation and open standards.
## Community ##
We love the Open Source software community. Contributing is our passion and if its yours too, participate and [improve](https://community.penpot.app/c/help-us-improve-penpot/7) Penpot. All your designs, code and ideas are welcome!
+11 -17
View File
@@ -17,7 +17,7 @@
io.prometheus/simpleclient_httpserver {:mvn/version "0.16.0"}
io.lettuce/lettuce-core {:mvn/version "7.7.0.RELEASE"}
io.lettuce/lettuce-core {:mvn/version "7.6.0.RELEASE"}
;; Minimal dependencies required by lettuce, we need to include them
;; explicitly because clojure dependency management does not support
;; yet the BOM format.
@@ -25,7 +25,7 @@
io.micrometer/micrometer-observation {:mvn/version "1.14.2"}
java-http-clj/java-http-clj {:mvn/version "0.4.3"}
com.google.guava/guava {:mvn/version "33.7.1-jre"}
com.google.guava/guava {:mvn/version "33.6.0-jre"}
funcool/yetti
{:git/tag "v11.10"
@@ -40,44 +40,38 @@
nrepl/nrepl {:mvn/version "1.7.0"}
org.postgresql/postgresql {:mvn/version "42.7.13"}
org.xerial/sqlite-jdbc {:mvn/version "3.53.4.0"}
org.xerial/sqlite-jdbc {:mvn/version "3.53.2.1"}
com.zaxxer/HikariCP {:mvn/version "7.1.0"}
io.whitfin/siphash {:mvn/version "3.0.0"}
io.whitfin/siphash {:mvn/version "2.0.0"}
buddy/buddy-hashers {:mvn/version "2.0.167"}
buddy/buddy-sign {:mvn/version "3.6.1-359"}
org.passay/passay {:mvn/version "2.0.0"}
org.passay/passay {:mvn/version "1.6.6"}
com.github.ben-manes.caffeine/caffeine {:mvn/version "3.2.4"}
org.jsoup/jsoup {:mvn/version "1.23.2"}
org.jsoup/jsoup {:mvn/version "1.23.1"}
at.yawk.lz4/lz4-java
{:mvn/version "1.11.2"}
{:mvn/version "1.11.1"}
org.clojars.pntblnk/clj-ldap {:mvn/version "0.0.17"}
dawran6/emoji {:mvn/version "0.2.0"}
markdown-clj/markdown-clj {:mvn/version "1.12.9"}
markdown-clj/markdown-clj {:mvn/version "1.12.8"}
;; Pretty Print specs
pretty-spec/pretty-spec {:mvn/version "0.1.4"}
software.amazon.awssdk/s3 {:mvn/version "2.54.5"}
software.amazon.awssdk/sts {:mvn/version "2.54.5"}
com.ladybugdb/lbug {:mvn/version "0.19.1"}
;; Required by Arrow RootAllocator (lbug only pulls arrow-memory-core).
org.apache.arrow/arrow-memory-netty {:mvn/version "18.2.0"}}
software.amazon.awssdk/s3 {:mvn/version "2.50.1"}
software.amazon.awssdk/sts {:mvn/version "2.50.1"}}
:paths ["src" "resources" "target/classes"]
:aliases
{:dev
{:jvm-opts ["--sun-misc-unsafe-memory-access=allow"
"--enable-native-access=ALL-UNNAMED"
;; Arrow jars are on the classpath (unnamed module), not module-path.
"--add-opens=java.base/java.nio=ALL-UNNAMED"]
"--enable-native-access=ALL-UNNAMED"]
:extra-deps
{com.bhauman/rebel-readline {:mvn/version "0.1.11"}
clojure-humanize/clojure-humanize {:mvn/version "0.2.2"}
+1 -1
View File
@@ -4,7 +4,7 @@
"license": "MPL-2.0",
"author": "Kaleidos INC Sucursal en España SL",
"private": true,
"packageManager": "pnpm@12.3.4+sha512.961aa41fb077da3a04a441d9f8e15ebc0c96da8ef710b2eb67bf9ee7cb0610eabd48f1fd85f51cffe73846785fa0f87c56a3a872a1d893f8446741b5cce45457",
"packageManager": "pnpm@11.20.0+sha512.9a6f330a95b66446ea088faf1521405a8a01f07fde7124cc9958dfed52d4bb436737e65b08f85f37b46fcba375092558ac51262b816844b22f63406ed166bfee",
"repository": {
"type": "git",
"url": "https://github.com/penpot/penpot"
-101
View File
@@ -1,104 +1,3 @@
---
lockfileVersion: '9.0'
importers:
.:
configDependencies: {}
packageManagerDependencies:
pnpm:
specifier: 12.3.4
version: 12.3.4
packages:
'@pnpm/exe.darwin-arm64@12.3.4':
resolution: {integrity: sha512-PAyUol8T1+/+ViOiXAt51ECA+QnfXCqz6foL4bW+LsoX0NcVd5XVEM2mRQu+LV4oc7uRz9zf9U0P+XFfuQeDAw==}
cpu: [arm64]
os: [darwin]
'@pnpm/exe.darwin-x64@12.3.4':
resolution: {integrity: sha512-fxP9JCk0Cdye+ePuj+GJJLMUMTqHGWRdb1dtv4How876uQ2ehxvenpgiYAir/ceO9PsYUZkFTtyZdx+rRu5QOA==}
cpu: [x64]
os: [darwin]
'@pnpm/exe.linux-arm64-musl@12.3.4':
resolution: {integrity: sha512-FBOt0/7ye6O6q4AllVV5QMviB6qE6fqkeczV/+MDWQsmo+QJrlfsh6X7CpH/tClVpBZEyIbjpUoT8bNhCYBxEg==}
cpu: [arm64]
os: [linux]
libc: [musl]
'@pnpm/exe.linux-arm64@12.3.4':
resolution: {integrity: sha512-t71AVA7LRqiKTyZ5xMYaZc2n5DfdpMbfokZuiIOXHBOM03ECnF0t4iYwaBDqJgVjlKYUOwaF/bRQajGNA4cJ4w==}
cpu: [arm64]
os: [linux]
libc: [glibc]
'@pnpm/exe.linux-x64-musl@12.3.4':
resolution: {integrity: sha512-RPmk7Jb/aYaFvL2iyDN/AtMY+hUEsue732WmXpcuQ9tBpMnGyA5py7Z3+e+qmQaJ0zY/4ni9jJiyPBQHujmv6w==}
cpu: [x64]
os: [linux]
libc: [musl]
'@pnpm/exe.linux-x64@12.3.4':
resolution: {integrity: sha512-2ZqOlSPkfwX1h5cR+FPiWf8+F+2hZT/3TvhUK5sigHqwaQCIiq8R7CGxhndKs63JtcLi2a1Qpo+wX/EoyfjyJQ==}
cpu: [x64]
os: [linux]
libc: [glibc]
'@pnpm/exe.win32-arm64@12.3.4':
resolution: {integrity: sha512-ANyrHqyqco6SXBysUTRF74itDyyraea7IbFsKFdNXTjcFnfycTDx37EwuhdpPYFNSIh2JhUG4fByclsRfiHX7w==}
cpu: [arm64]
os: [win32]
'@pnpm/exe.win32-x64@12.3.4':
resolution: {integrity: sha512-WH/KqBPY/hq2Tb7SgQltEZytimcjgKRaCRL/aM9CI0c67iKc5TVmHUhIiL3Ux9FB4bWn36i6XewUcScQI+zG8w==}
cpu: [x64]
os: [win32]
pnpm@12.3.4:
resolution: {integrity: sha512-lhqkH7B32joEpEHZ+OFevAyW2o73ELLrZ7+e58sGEOq9SPH9hfUc/+c4RnhfoPh8VqOocqHYk/hEZ0G1zORUVw==}
engines: {node: '>=18.*'}
hasBin: true
snapshots:
'@pnpm/exe.darwin-arm64@12.3.4':
optional: true
'@pnpm/exe.darwin-x64@12.3.4':
optional: true
'@pnpm/exe.linux-arm64-musl@12.3.4':
optional: true
'@pnpm/exe.linux-arm64@12.3.4':
optional: true
'@pnpm/exe.linux-x64-musl@12.3.4':
optional: true
'@pnpm/exe.linux-x64@12.3.4':
optional: true
'@pnpm/exe.win32-arm64@12.3.4':
optional: true
'@pnpm/exe.win32-x64@12.3.4':
optional: true
pnpm@12.3.4:
optionalDependencies:
'@pnpm/exe.darwin-arm64': 12.3.4
'@pnpm/exe.darwin-x64': 12.3.4
'@pnpm/exe.linux-arm64': 12.3.4
'@pnpm/exe.linux-arm64-musl': 12.3.4
'@pnpm/exe.linux-x64': 12.3.4
'@pnpm/exe.linux-x64-musl': 12.3.4
'@pnpm/exe.win32-arm64': 12.3.4
'@pnpm/exe.win32-x64': 12.3.4
---
lockfileVersion: '9.0'
settings:
-2
View File
@@ -1,4 +1,2 @@
storeDir: ../.pnpm-store
minimumReleaseAgeExclude:
- brace-expansion@5.0.8 || 5.0.9
@@ -191,7 +191,7 @@
file named “{{file-name|abbreviate:25}}”.
</p>
<p>
Since this file is in your Personal Projects, you can provide access by sending a view-only link.
Since this file is in your Penpot team, you can provide access by sending a view-only link.
This will allow {{requested-by|abbreviate:25}} to view the content without making any changes.
</p>
<p>To proceed, please click the button below to generate and send the view-only link:</p>
@@ -2,7 +2,7 @@ Hello!
{{requested-by|abbreviate:25}} ({{requested-by-email}}) wants to have view-only access to the file named “{{file-name|abbreviate:25}}”.
Since this file is in your Personal Projects, you can provide access by sending a view-only link. This will allow {{requested-by|abbreviate:25}} to view the content without making any changes.
Since this file is in your Penpot team, you can provide access by sending a view-only link. This will allow {{requested-by|abbreviate:25}} to view the content without making any changes.
To proceed, please click the link below to generate and send the view-only link:
@@ -191,7 +191,7 @@
“{{file-name|abbreviate:25}}”.
</p>
<p>
Please note that the file is currently in Personal Projects, so direct access cannot be
Please note that the file is currently in Your Penpot 's team, so direct access cannot be
granted. However, you have two options to provide the requested access:
</p>
<ul>
@@ -5,7 +5,7 @@ Hello!
{{requested-by|abbreviate:25}} ({{requested-by-email}}) has requested access to the file named “{{file-name|abbreviate:25}}”.
Please note that the file is currently in Personal Projects, so direct access cannot be granted. However, you have two options to provide the requested access:
Please note that the file is currently in Your Penpot 's team, so direct access cannot be granted. However, you have two options to provide the requested access:
- Move the File to Another Team:
@@ -236,23 +236,7 @@ Debug Main Page
</div>
</form>
</fieldset>
{% if graph-enabled %}
<fieldset>
<legend>Export graph (Ladybug):</legend>
<desc>Given a FILE-ID, builds the graph projection and downloads
the `.lbug` database file.</desc>
<form method="get" action="/dbg/actions/graph-export">
<div class="row">
<input type="text" style="width:300px" name="file-id" placeholder="file-id" />
</div>
<div class="row">
<input type="submit" value="Download .lbug" />
<a href="/dbg/graph">Open graph console</a>
</div>
</form>
</fieldset>
{% endif %}
<fieldset>
<legend>Import binfile:</legend>
<desc>Import penpot file in binary format.</desc>
@@ -296,60 +280,5 @@ Debug Main Page
</form>
</fieldset>
</section>
</main>
<main class="dashboard wide">
<section class="widget wide">
<fieldset>
<legend>Export jobs:</legend>
<desc>
Export jobs as the exporter left them in redis. Records expire an hour
after the export settles, so this is a live view, not a history.
</desc>
<form method="get" action="/dbg">
<div class="row">
<input type="text" style="width:300px" name="job-id"
placeholder="filter by job id" value="{{export-job-filter}}" />
<input type="submit" value="Filter" />
<a href="/dbg">clear</a>
</div>
</form>
<div class="scroll-box">
<table>
<thead>
<tr>
<th>JOB ID</th>
<th>STATE</th>
<th>PROGRESS</th>
<th>CMD</th>
<th>BACKEND</th>
<th>NAME</th>
<th>CREATED</th>
<th>ENDED</th>
</tr>
</thead>
<tbody>
{% for job in export-jobs %}
<tr>
<td><tt>{{job.id}}</tt></td>
<td>{{job.state}}{% if job.interrupted %} (interrupted){% endif %}</td>
<td>{{job.done}} / {{job.total}}</td>
<td>{{job.cmd}}</td>
<td>{{job.backend}}</td>
<td>{{job.name}}</td>
<td>{{job.created-at}}</td>
<td>{{job.ended-at}}</td>
</tr>
{% empty %}
<tr><td colspan="8">No export jobs.</td></tr>
{% endfor %}
</tbody>
</table>
</div>
</fieldset>
</section>
</main>
{% endblock %}
File diff suppressed because it is too large. Load diff
@@ -143,35 +143,6 @@ nav > div:not(:last-child) {
height: fit-content;
}
/* A widget that holds a table rather than a form: full width, and tall
enough to be worth scrolling inside. */
.dashboard.wide {
margin-top: 0px;
}
.widget.wide {
max-width: none;
width: 100%;
}
.widget.wide .scroll-box {
max-height: 320px;
overflow-y: auto;
margin-top: 10px;
}
.widget.wide table {
width: 100%;
border-collapse: collapse;
}
.widget.wide th {
text-align: left;
position: sticky;
top: 0;
background: white;
}
.widget input[type=submit] {
outline: none;
border: 1px solid gray;
+3 -3
View File
@@ -31,7 +31,7 @@ export PENPOT_MEDIA_PROCESSING_SERVICE_URI=http://localhost:6065
export PENPOT_FLAGS="\
$PENPOT_FLAGS \
enable-login-with-password \
enable-login-with-ldap \
disable-login-with-ldap \
disable-login-with-oidc \
disable-login-with-google \
disable-login-with-github \
@@ -48,6 +48,7 @@ export PENPOT_FLAGS="\
enable-user-feedback \
disable-secure-session-cookies \
enable-smtp \
enable-account-lockout \
enable-prepl-server \
enable-urepl-server \
enable-nrepl-server \
@@ -93,8 +94,7 @@ export JAVA_OPTS="\
-XX:-OmitStackTraceInFastThrow \
--sun-misc-unsafe-memory-access=allow \
--enable-preview \
--enable-native-access=ALL-UNNAMED \
--add-opens=java.base/java.nio=ALL-UNNAMED";
--enable-native-access=ALL-UNNAMED";
function setup_minio() {
if [ "${PENPOT_OBJECTS_STORAGE_BACKEND}" != "s3" ]; then
+1 -1
View File
@@ -18,7 +18,7 @@ if [ -f ./environ ]; then
source ./environ
fi
export JAVA_OPTS="-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Dlog4j2.configurationFile=log4j2.xml -XX:-OmitStackTraceInFastThrow --sun-misc-unsafe-memory-access=allow --enable-native-access=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --enable-preview $JVM_OPTS $JAVA_OPTS"
export JAVA_OPTS="-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager -Dlog4j2.configurationFile=log4j2.xml -XX:-OmitStackTraceInFastThrow --sun-misc-unsafe-memory-access=allow --enable-native-access=ALL-UNNAMED --enable-preview $JVM_OPTS $JAVA_OPTS"
ENTRYPOINT=${1:-app.main};
-11
View File
@@ -14,21 +14,10 @@
:iterations 3
:parallelism 2})
(def ^:private weak-options
{:alg :pbkdf2+sha256
:iterations 100})
(defn derive-password
[password]
(hashers/derive password default-options))
(defn derive-password-weak
"Derives a password using a fast algorithm (pbkdf2+sha256, 100 iterations).
Intended for demo users only — they are already gated behind the
`demo-users` config flag which is disabled in production."
[password]
(hashers/derive password weak-options))
(defn verify-password
[attempt password]
(try
+6 -24
View File
@@ -10,7 +10,7 @@
[app.common.logging :as l]
[app.common.schema :as sm]
[clj-ldap.client :as ldap]
[cuerdas.core :as str]
[clojure.string]
[integrant.core :as ig]))
(defn- prepare-params
@@ -36,22 +36,11 @@
:cause cause))))
(defn- replace-several [s & {:as replacements}]
(reduce-kv str/replace s replacements))
(defn- escape-ldap-filter-value
"Escapes special characters in a string for use in LDAP filter values,
per RFC 4515 section 3."
[s]
(-> s
(str/replace "\\" "\\5c")
(str/replace "*" "\\2a")
(str/replace "(" "\\28")
(str/replace ")" "\\29")
(str/replace "\u0000" "\\00")))
(reduce-kv clojure.string/replace s replacements))
(defn- search-user
[{:keys [::conn base-dn] :as cfg} email]
(let [query (replace-several (:query cfg) ":username" (escape-ldap-filter-value email))
(let [query (replace-several (:query cfg) ":username" email)
attrs [(:attrs-username cfg)
(:attrs-email cfg)
(:attrs-fullname cfg)]
@@ -60,19 +49,12 @@
:attributes attrs}]
(first (ldap/search conn base-dn params))))
(defn- get-attr
"Retrieves an attribute from an LDAP entry. Handles multi-valued
attributes by returning the first value."
[entry attr-key]
(let [v (get entry attr-key)]
(if (coll? v) (first v) v)))
(defn- retrieve-user
[{:keys [::conn] :as cfg} {:keys [email password]}]
(when-let [{:keys [dn] :as user} (search-user cfg email)]
(when (ldap/bind? conn dn password)
{:fullname (get-attr user (-> cfg :attrs-fullname keyword))
:email (get-attr user (-> cfg :attrs-email keyword))
{:fullname (get user (-> cfg :attrs-fullname keyword))
:email email
:backend "ldap"})))
(def ^:private schema:info-data
@@ -97,7 +79,7 @@
(l/warn :hint "invalid response from ldap, looks like ldap is not configured correctly" :data user)
(ex/raise :type :restriction
:code :wrong-ldap-response
::sm/explain explain)))
:explain explain)))
user)))
(defn- try-connectivity
+132
View File
@@ -0,0 +1,132 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.auth.login-lockout
"Brute-force protection: per-account failed login counter backed by Redis.
Uses an atomic Lua script for the increment operation to prevent
concurrent login bypass. Stores count:window_start in the value;
time is passed from Clojure (ct/now) via ARGV, keeping it injectable
for tests. When the counter reaches the configured threshold within
the time window, the account is locked out until the window expires."
(:require
[app.common.generic-pool :as gpool]
[app.common.logging :as l]
[app.common.time :as ct]
[app.config :as cf]
[app.redis :as rds]
[app.redis.script :as-alias rscript]
[clojure.string :as str])
(:import
java.lang.AutoCloseable))
(def ^:private key-prefix "penpot.login-lockout.")
(def ^:private lockout-script
{::rscript/name ::login-lockout
::rscript/path "app/auth/login_lockout.lua"})
(defn- get-pool
[cfg]
(:app.redis/pool cfg))
(defn- with-conn
[cfg f]
(let [pool (get-pool cfg)
conn (gpool/get pool)]
(try
(f @conn)
(finally
(.close ^AutoCloseable conn)))))
(defn- parse-value
"Parse stored \"count:window_start\" string. Returns nil if missing
or expired. Throws NumberFormatException for malformed values
(caught by caller → fail-open)."
[value window-ms now-ms]
(when value
(let [sep (str/index-of value ":")]
(when sep
(let [count (Long/parseLong (subs value 0 sep))
start (Long/parseLong (subs value (inc sep)))]
(when (> (+ start window-ms) now-ms)
{:count count :window-start start}))))))
(defn record-failed-attempt!
"Increment the failed-login counter for a profile-id (atomic via Lua).
Returns nil when the flag is disabled or on Redis error (fail-open).
Otherwise returns a map with :count (int), :ttl (int, seconds),
and :locked? (boolean)."
[cfg profile-id]
(when (contains? cf/flags :account-lockout)
(try
(let [threshold (cf/get :login-lockout-max-attempts)
window (cf/get :login-lockout-window)
window-ms (if (integer? window) window (.toMillis window))
_ (assert (>= threshold 1) "login-lockout-max-attempts must be >= 1")
_ (assert (>= window-ms 60000) "login-lockout-window must be >= 60000ms (1 minute)")
key (str key-prefix profile-id)
now-ms (inst-ms (ct/now))
result (with-conn cfg
(fn [conn]
(rds/eval conn
(assoc lockout-script
::rscript/keys [key]
::rscript/vals [threshold window-ms now-ms]))))
[count locked ttl] result]
{:count count
:locked? (= 1 locked)
:ttl (max 0 ttl)})
(catch Exception cause
(l/warn :hint "redis unavailable, failing open on login lockout"
:profile-id (str profile-id)
:cause cause)
nil))))
(defn clear-attempts!
"Clear the failed-login counter (on successful login or password reset).
No-op when the flag is disabled."
[cfg profile-id]
(when (contains? cf/flags :account-lockout)
(try
(with-conn cfg
(fn [conn]
(rds/del conn (str key-prefix profile-id))))
(catch Exception cause
(l/warn :hint "redis unavailable, failed to clear login lockout"
:profile-id (str profile-id)
:cause cause)))))
(defn locked?
"Check whether the account is currently locked out. Does not increment
the counter. Returns {:locked? false} when the flag is disabled or on
Redis error (fail-open). When locked, includes :ttl (seconds remaining)."
[cfg profile-id]
(if (contains? cf/flags :account-lockout)
(try
(let [threshold (cf/get :login-lockout-max-attempts)
window (cf/get :login-lockout-window)
window-ms (if (integer? window) window (.toMillis window))
_ (assert (>= threshold 1) "login-lockout-max-attempts must be >= 1")
_ (assert (>= window-ms 60000) "login-lockout-window must be >= 60000ms (1 minute)")
key (str key-prefix profile-id)
now-ms (inst-ms (ct/now))
result (with-conn cfg
(fn [conn]
(if-let [current (parse-value (rds/get conn key) window-ms now-ms)]
(let [elapsed (- now-ms (:window-start current))
ttl-ms (- window-ms elapsed)
locked? (>= (:count current) threshold)]
(cond-> {:locked? locked?}
locked?
(assoc :ttl (int (Math/ceil (/ ttl-ms 1000.0))))))
{:locked? false})))]
result)
(catch Exception cause
(l/warn :hint "redis unavailable, failing open on lockout check"
:profile-id (str profile-id)
:cause cause)
{:locked? false}))
{:locked? false}))
+46
View File
@@ -0,0 +1,46 @@
-- This Source Code Form is subject to the terms of the Mozilla Public
-- License, v. 2.0. If a copy of the MPL was not distributed with this
-- file, You can obtain one at http://mozilla.org/MPL/2.0/.
--
-- Copyright (c) KALEIDOS SUBSIDIARY SL
local key = KEYS[1]
local threshold = tonumber(ARGV[1])
local window_ms = tonumber(ARGV[2])
local now_ms = tonumber(ARGV[3])
local window_sec = math.ceil(window_ms / 1000)
local val = redis.call('GET', key)
local count
local window_start
if val then
local colon = string.find(val, ':')
if colon then
count = tonumber(string.sub(val, 1, colon - 1))
window_start = tonumber(string.sub(val, colon + 1))
else
count = 0
window_start = now_ms
end
else
count = 0
window_start = now_ms
end
if (now_ms - window_start) > window_ms then
count = 0
window_start = now_ms
end
count = count + 1
redis.call('SET', key, count .. ':' .. window_start, 'EX', window_sec)
local locked = 0
local ttl = 0
if count >= threshold then
locked = 1
ttl = math.ceil((window_ms - (now_ms - window_start)) / 1000)
end
return {count, locked, ttl}
+1 -1
View File
@@ -1037,7 +1037,7 @@
provider (prepare-organization-sso-provider cfg sso)
_info (get-info cfg provider state code)
session (session/get-session request)
exp (ct/in-future {:hours 4})]
exp (ct/in-future {:minutes 15})]
(when (and session organization-id)
(let [props (-> (or (:props session) {})
(update :sso assoc organization-id exp))]
+5 -8
View File
@@ -9,9 +9,7 @@
(:require
[app.common.exceptions :as ex])
(:import
[org.passay PasswordData]
[org.passay.data EnglishCharacterData]
[org.passay.rule CharacterCharacteristicsRule CharacterRule]))
[org.passay CharacterCharacteristicsRule CharacterRule EnglishCharacterData PasswordData]))
(defonce ^:private passay-code->translation-key
{"INSUFFICIENT_LOWERCASE" "errors.weak-password.insufficient-lowercase"
@@ -20,13 +18,12 @@
"INSUFFICIENT_SPECIAL" "errors.weak-password.insufficient-special"})
(defonce ^:private character-characteristics-rule
(CharacterCharacteristicsRule.
4
(into-array org.passay.rule.CharacterRule
[(CharacterRule. EnglishCharacterData/LowerCase 1)
(doto (CharacterCharacteristicsRule.)
(.setRules [(CharacterRule. EnglishCharacterData/LowerCase 1)
(CharacterRule. EnglishCharacterData/UpperCase 1)
(CharacterRule. EnglishCharacterData/Digit 1)
(CharacterRule. EnglishCharacterData/Special 1)])))
(CharacterRule. EnglishCharacterData/Special 1)])
(.setNumberOfCharacteristics 4)))
(defn validate-password
"Validates password strength.
+8 -55
View File
@@ -42,7 +42,6 @@
[datoteka.io :as io])
(:import
java.io.File
java.io.FilterInputStream
java.io.InputStream
java.io.OutputStreamWriter
java.lang.AutoCloseable
@@ -462,32 +461,6 @@
[^ZipFile input ^ZipEntry entry]
(.getInputStream input entry))
(defn- size-limiting-stream
"Wraps an InputStream to enforce a maximum number of decompressed bytes.
Raises :validation :max-file-size-reached when the limit is exceeded."
^InputStream
[^InputStream input ^long max-size]
(let [counter (atom 0)
on-read (fn [n]
(when (pos? n)
(when (> (swap! counter + (long n)) max-size)
(ex/raise :type :validation
:code :max-file-size-reached
:hint (str "stream exceeded max size: " max-size))))
n)]
(proxy [FilterInputStream] [input]
(read
([]
(let [b (.read input)]
(when (pos? b) (on-read 1))
b))
([^bytes buf]
(on-read (.read input buf 0 (alength buf))))
([^bytes buf off]
(on-read (.read input buf (int off) (- (alength buf) (int off)))))
([^bytes buf off len]
(on-read (.read input buf (int off) (int len))))))))
(defn- zip-entry-reader
[^ZipFile input ^ZipEntry entry]
(-> (zip-entry-stream input entry)
@@ -496,12 +469,10 @@
(defn- zip-entry-storage-content
"Wraps a ZipFile and ZipEntry into a penpot storage compatible
object and avoid creating temporal objects"
[input entry & {:keys [max-size]}]
(let [stream-fn (fn []
(cond-> (zip-entry-stream input entry)
max-size (size-limiting-stream max-size)))
hash (delay (->> (stream-fn)
(sto.impl/calculate-hash)))]
[input entry]
(let [hash (delay (->> entry
(zip-entry-stream input)
(sto.impl/calculate-hash)))]
(reify
sto.impl/IContentObject
(get-size [_]
@@ -518,7 +489,7 @@
(throw (UnsupportedOperationException. "not implemented")))
(make-input-stream [_ _]
(stream-fn))
(zip-entry-stream input entry))
(make-output-stream [_ _]
(throw (UnsupportedOperationException. "not implemented"))))))
@@ -913,9 +884,9 @@
ext (cmedia/mtype->extension (:content-type object))
path (str "objects/" id ext)
content (zip-entry-storage-content input
(get-zip-entry input path)
:max-size (::bfc/import-max-object-size cfg))]
content (->> path
(get-zip-entry input)
(zip-entry-storage-content input))]
(when (not= (:size object) (sto/get-size content))
(ex/raise :type :validation
@@ -925,15 +896,6 @@
:expected-size (:size object)
:found-size (sto/get-size content)))
(when-let [max (::bfc/import-max-object-size cfg)]
(when (> (sto/get-size content) max)
(ex/raise :type :validation
:code :max-file-size-reached
:hint (str "storage object exceeds maximum size: " (sto/get-size content))
:path path
:max max
:found (sto/get-size content))))
(when-let [hash (get object :hash)]
(when (not= hash (sto/get-hash content))
(ex/raise :type :validation
@@ -1155,15 +1117,6 @@
(let [manifest (-> (read-manifest input)
(validate-manifest))
entries (read-zip-entries input)
_ (when-let [max (::bfc/import-max-zip-entries cfg)]
(when (> (count entries) max)
(ex/raise :type :validation
:code :too-many-zip-entries
:hint (str "zip file has too many entries: " (count entries))
:max max
:found (count entries))))
cfg (-> cfg
(assoc ::entries entries)
(assoc ::manifest manifest)
+6 -10
View File
@@ -58,7 +58,6 @@
:objects-storage-fs-directory "assets"
:auth-token-cookie-name "auth-token"
:auth-token-cookie-max-age-absolute (ct/duration {:days 30})
:assets-path "/internal/assets/"
:smtp-default-reply-to "Penpot <no-reply@example.com>"
@@ -70,6 +69,9 @@
:profile-bounce-max-age (ct/duration {:days 7})
:profile-bounce-threshold 10
:login-lockout-max-attempts 5
:login-lockout-window (ct/duration "15m")
:telemetry-uri "https://telemetry.penpot.app/"
:media-max-file-size (* 1024 1024 30) ; 30MiB
@@ -95,11 +97,7 @@
;; SSRF protection
:ssrf-allowed-hosts #{}
:ssrf-extra-blocked-cidrs #{}
;; Binfile import limits
:binfile-import-max-object-size (* 1024 1024 100) ;; 100 MiB
:binfile-import-max-zip-entries (* 500 1000)}) ;; 500,000
:ssrf-extra-blocked-cidrs #{}})
(def schema:config
(do #_sm/optional-keys
@@ -156,9 +154,8 @@
[:media-processing-service-uri {:optional true} ::sm/uri]
[:media-processing-service-timeout {:optional true} ::sm/int]
;; Binfile import limits (PENPOT_BINFILE_IMPORT_*)
[:binfile-import-max-object-size {:optional true} ::sm/int]
[:binfile-import-max-zip-entries {:optional true} ::sm/int]
[:login-lockout-max-attempts {:optional true} ::sm/int]
[:login-lockout-window {:optional true} ::ct/duration]
[:deletion-delay {:optional true} ::ct/duration]
[:file-clean-delay {:optional true} ::ct/duration]
@@ -207,7 +204,6 @@
[:auth-token-cookie-name {:optional true} :string]
[:auth-token-cookie-max-age {:optional true} ::ct/duration]
[:auth-token-cookie-max-age-absolute {:optional true} ::ct/duration]
[:registration-domain-whitelist {:optional true} [::sm/set :string]]
[:email-verify-threshold {:optional true} ::ct/duration]
+5 -5
View File
@@ -31,8 +31,8 @@
com.zaxxer.hikari.HikariDataSource
com.zaxxer.hikari.HikariPoolMXBean
com.zaxxer.hikari.metrics.prometheus.PrometheusMetricsTrackerFactory
io.whitfin.siphash.SipHash
io.whitfin.siphash.SipHashContext
io.whitfin.siphash.SipHasher
io.whitfin.siphash.SipHasherContainer
java.io.InputStream
java.io.OutputStream
java.sql.Connection
@@ -701,12 +701,12 @@
;; --- Locks
(def ^:private siphash-state
(SipHash/context
(uuid/get-bytes uuid/zero)))
(SipHasher/container
(uuid/get-bytes uuid/zero)))
(defn uuid->hash-code
[o]
(.hash ^SipHashContext siphash-state
(.hash ^SipHasherContainer siphash-state
^bytes (uuid/get-bytes o)))
(defn- xact-check-param
+2 -2
View File
@@ -505,13 +505,13 @@
:schema schema:request-file-access))
(def request-file-access-yourpenpot
"File access on Personal Projects request email."
"File access on Your Penpot request email."
(template-factory
:id ::request-file-access-yourpenpot
:schema schema:request-file-access))
(def request-file-access-yourpenpot-view
"File access on Personal Projects view mode request email."
"File access on Your Penpot view mode request email."
(template-factory
:id ::request-file-access-yourpenpot-view
:schema schema:request-file-access))
-370
View File
@@ -1,370 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.graph.arrow
"Bulk Ladybug ingest through in-memory Arrow.
Rows are built as Arrow `VectorSchemaRoot`s in the JVM's off-heap memory,
handed to Ladybug as a virtual table, and `COPY`d into the real one. No file
is written and no value is rendered as text for the engine to re-parse, so
nothing in this path needs escaping. Arrow carries MAP, STRUCT, fixed-size
arrays and multi-line strings natively.
The type language is Ladybug's, read recursively by `app.graph.schema.values`;
this namespace adds the matching Arrow `Field` and a writer for each shape.
`values/coerce` shapes a value first — a matrix into six doubles, a colour
into a packed integer — exactly as it does for the Cypher path, so the two
writers cannot disagree.
Engine facts this file depends on, each verified against lbug 0.19.1:
- An Arrow table is **not** a `COPY` source identifier, but it *is* a
MATCH-able node label: `COPY T FROM (MATCH (n:stg) RETURN n.a AS a, …)`.
- A MAP vector's `entries` child struct must be non-nullable, and
`MapVector/getWriter` silently promotes it to a sparse union — so map
vectors are built from an explicit `Field` and filled child-first.
- Ladybug quotes the column and table names it interpolates into the staged
table's DDL, and does not quote a STRUCT member name. So a top-level field
arrives plain and a struct member whose name is a reserved word (`column`)
arrives backticked.
- `createArrowRelTable` resolves a UUID-keyed endpoint only from a
`FixedSizeBinary(16)` column carrying the `arrow.uuid` extension, so edges
are staged as a node table and joined by the `COPY` subquery instead."
(:require
[app.common.json :as json]
[app.graph.ladybug :as ladybug]
[app.graph.schema.nodes :as nodes]
[app.graph.schema.values :as values]
[clojure.string :as str])
(:import
com.ladybugdb.Connection
com.ladybugdb.QueryResult
java.nio.charset.StandardCharsets
java.util.ArrayList
java.util.List
org.apache.arrow.memory.BufferAllocator
org.apache.arrow.memory.RootAllocator
org.apache.arrow.vector.BigIntVector
org.apache.arrow.vector.BitVector
org.apache.arrow.vector.complex.ListVector
org.apache.arrow.vector.complex.MapVector
org.apache.arrow.vector.complex.StructVector
org.apache.arrow.vector.FieldVector
org.apache.arrow.vector.Float8Vector
org.apache.arrow.vector.TimeStampMicroVector
org.apache.arrow.vector.types.FloatingPointPrecision
org.apache.arrow.vector.types.pojo.ArrowType$Bool
org.apache.arrow.vector.types.pojo.ArrowType$FloatingPoint
org.apache.arrow.vector.types.pojo.ArrowType$Int
org.apache.arrow.vector.types.pojo.ArrowType$List
org.apache.arrow.vector.types.pojo.ArrowType$Map
org.apache.arrow.vector.types.pojo.ArrowType$Struct
org.apache.arrow.vector.types.pojo.ArrowType$Timestamp
org.apache.arrow.vector.types.pojo.ArrowType$Utf8
org.apache.arrow.vector.types.pojo.Field
org.apache.arrow.vector.types.pojo.FieldType
org.apache.arrow.vector.types.pojo.Schema
org.apache.arrow.vector.types.TimeUnit
org.apache.arrow.vector.UInt4Vector
org.apache.arrow.vector.VarCharVector
org.apache.arrow.vector.VectorSchemaRoot))
(set! *warn-on-reflection* true)
;; --------------------------------------------------------------- allocator
(defn with-allocator!
"Invoke `(f allocator)` with a fresh Arrow `RootAllocator`.
The allocator must outlive the Ladybug connection, because Ladybug releases
its references to the staged buffers only when the Arrow tables are dropped —
which happens on connection close at the latest. Closing it first surfaces as
`IllegalStateException: Memory was leaked`, *thrown while unwinding*, which
hides whatever actually failed. Any diagnostic here must catch inside this
scope."
[f]
(with-open [allocator (RootAllocator.)]
(f allocator)))
;; ------------------------------------------------------ Ladybug type → Field
(def ^:private scalar-arrow-type
"Ladybug scalar → Arrow type. `UUID` and `JSON` ride as UTF-8: Ladybug
accepts a string into either column and does the conversion itself, which is
cheaper than teaching this side two more binary layouts."
{"STRING" #(ArrowType$Utf8.)
"UUID" #(ArrowType$Utf8.)
"JSON" #(ArrowType$Utf8.)
"INT64" #(ArrowType$Int. 64 true)
"UINT32" #(ArrowType$Int. 32 false)
"DOUBLE" #(ArrowType$FloatingPoint. FloatingPointPrecision/DOUBLE)
"BOOLEAN" #(ArrowType$Bool.)
"TIMESTAMP" #(ArrowType$Timestamp. TimeUnit/MICROSECOND nil)})
(defn column-field
"Arrow `Field` for a column of `ladybug-type`, recursively.
`nullable?` is false only where Arrow's own invariants demand it — a MAP's
`entries` struct and its key."
(^Field [^String field-name ladybug-type]
(column-field field-name ladybug-type true))
(^Field [^String field-name ladybug-type nullable?]
(cond
;; A list first: `STRUCT(…)[]` starts with `STRUCT(` but is a list of them.
(ladybug/list-type? ladybug-type)
(Field. field-name (FieldType. nullable? (ArrowType$List.) nil)
[(column-field "item" (values/list-element ladybug-type))])
(ladybug/map-type? ladybug-type)
(let [[key-type value-type] (values/map-types ladybug-type)]
(Field. field-name (FieldType. nullable? (ArrowType$Map. false) nil)
[(Field. "entries" (FieldType. false (ArrowType$Struct.) nil)
[(column-field "key" key-type false)
(column-field "value" value-type)])]))
(ladybug/struct-type? ladybug-type)
(Field. field-name (FieldType. nullable? (ArrowType$Struct.) nil)
;; Backticks kept: Ladybug quotes none of these when it names the
;; staged struct's fields, so `column` has to arrive quoted.
(mapv (fn [[field field-type]] (column-field field field-type))
(values/struct-fields-quoted ladybug-type)))
:else
(if-let [mk (get scalar-arrow-type ladybug-type)]
(Field. field-name (FieldType. nullable? (mk) nil) nil)
(throw (ex-info (str "no Arrow mapping for Ladybug type: " ladybug-type)
{:ladybug-type ladybug-type}))))))
;; ------------------------------------------------------------------- writer
(defn- utf8
^bytes [v]
(.getBytes (if (keyword? v) (name v) (str v)) StandardCharsets/UTF_8))
(defn- epoch-micros
^long [v]
(let [^java.time.Instant inst
(cond
(instance? java.time.Instant v) v
(instance? java.util.Date v) (.toInstant ^java.util.Date v)
:else (java.time.Instant/parse (str v)))]
(+ (* (.getEpochSecond inst) 1000000) (long (quot (.getNano inst) 1000)))))
(defn- write-scalar!
[^FieldVector fv ladybug-type ^long idx v]
(case ladybug-type
("STRING" "UUID") (.setSafe ^VarCharVector fv idx (utf8 v))
;; A JSON column holds JSON, not a Clojure value's print form: `str` on a
;; map yields `{:fill-color "#000000"}`, which is EDN and which every
;; consumer of `fills`, `content` or `position_data` would fail to parse.
;; Same encoder the Cypher path uses (`app.graph.ladybug/format-json`).
"JSON" (.setSafe ^VarCharVector fv idx
(.getBytes ^String (json/encode v)
StandardCharsets/UTF_8))
"INT64" (.setSafe ^BigIntVector fv idx (long v))
"UINT32" (.setSafe ^UInt4Vector fv idx (unchecked-int (long v)))
"DOUBLE" (.setSafe ^Float8Vector fv idx (double v))
"BOOLEAN" (.setSafe ^BitVector fv idx (if v 1 0))
"TIMESTAMP" (.setSafe ^TimeStampMicroVector fv idx (epoch-micros v))
(throw (ex-info (str "no Arrow writer for Ladybug type: " ladybug-type)
{:ladybug-type ladybug-type}))))
(defn write-value!
"Write already-coerced `v` into `fv` at `idx`, per `ladybug-type`.
`map-key-fn` renders the keys of a `MAP(STRING, …)`, for the same reason
`app.graph.ladybug/format-typed-value` takes one: the right spelling is a
property of the column, not of the writer."
;; `idx` is deliberately unhinted: Clojure only accepts primitive args on fns
;; of four or fewer, and the map-key renderer has to travel with the value.
[^FieldVector fv ladybug-type idx v map-key-fn]
(if (nil? v)
(.setNull fv (int idx))
(cond
(ladybug/list-type? ladybug-type)
(let [^ListVector lv fv
child (.getDataVector lv)
element-type (values/list-element ladybug-type)
elements (vec (if (or (sequential? v) (set? v)) v [v]))
start (.startNewValue lv (int idx))]
(dotimes [i (count elements)]
(write-value! child element-type (+ start i) (nth elements i) map-key-fn))
(.endValue lv (int idx) (count elements)))
(ladybug/map-type? ladybug-type)
(let [^MapVector mv fv
^StructVector entries (.getDataVector mv)
[key-type value-type] (values/map-types ladybug-type)
key-vec (.getChild entries "key")
value-vec (.getChild entries "value")
render-key (if (and map-key-fn (= "STRING" key-type)) map-key-fn identity)
pairs (vec (seq v))
start (.startNewValue mv (int idx))]
(dotimes [i (count pairs)]
(let [[k mv'] (nth pairs i)
at (+ start i)]
;; The entries struct is non-nullable: every slot must be defined.
(.setIndexDefined entries (int at))
(write-value! key-vec key-type at (render-key k) nil)
(write-value! value-vec value-type at mv' map-key-fn)))
(.endValue mv (int idx) (count pairs)))
(ladybug/struct-type? ladybug-type)
(let [^StructVector sv fv]
(.setIndexDefined sv (int idx))
(doseq [[quoted-field field-type] (values/struct-fields-quoted ladybug-type)]
;; The child is named with its backticks; the coerced value is keyed
;; without them.
(write-value! (.getChild sv quoted-field) field-type idx
(get v (str/replace quoted-field "`" "")) map-key-fn)))
:else
(write-scalar! fv ladybug-type (long idx) v))))
;; ------------------------------------------------------------------ batches
(defn- fill-vector!
[^VectorSchemaRoot root ^String field-name ladybug-type rows value-fn map-key-fn]
(let [^FieldVector fv (.getVector root field-name)]
(.allocateNew fv)
(dotimes [i (count rows)]
(write-value! fv ladybug-type i
(values/coerce ladybug-type (value-fn (nth rows i)))
map-key-fn))
(.setValueCount fv (count rows))))
(defn- node-batch
"One `VectorSchemaRoot` holding every projected row of `table`.
Fields carry the plain column name. Ladybug quotes every identifier it
interpolates into the staged table's DDL, so a name that is a reserved word
(`Page.index`, `Document.options`) arrives unquoted and a name arriving
pre-quoted comes out doubly backticked and fails to parse. The `COPY`
projection below is Cypher, not DDL, so it quotes the same names itself."
^VectorSchemaRoot [^BufferAllocator allocator table rows]
(let [columns (nodes/column-keys table)
fields (mapv (fn [k] (column-field (nodes/column-name table k)
(nodes/column-ladybug-type table k)))
columns)
root (VectorSchemaRoot/create (Schema. ^List fields) allocator)]
(doseq [k columns]
(fill-vector! root (nodes/column-name table k)
(nodes/column-ladybug-type table k)
rows #(get % k) (nodes/column-map-key-fn table k)))
(.setRowCount root (count rows))
root))
(def ^:private edge-fields
"Edge staging columns. `id` is the staging table's own key — Ladybug wants a
first column to key the virtual table on — and `from`/`to` land as STRING,
hence the cast in the join."
[(Field. "id" (FieldType. true (ArrowType$Utf8.) nil) nil)
(Field. "from" (FieldType. true (ArrowType$Utf8.) nil) nil)
(Field. "to" (FieldType. true (ArrowType$Utf8.) nil) nil)
(Field. "position" (FieldType. true (ArrowType$Int. 64 true) nil) nil)])
(defn- edge-batch
^VectorSchemaRoot [^BufferAllocator allocator edges]
(let [root (VectorSchemaRoot/create (Schema. ^List edge-fields) allocator)
^VarCharVector iv (.getVector root "id")
^VarCharVector fv (.getVector root "from")
^VarCharVector tv (.getVector root "to")
^BigIntVector pv (.getVector root "position")
n (count edges)]
(doseq [^FieldVector v [iv fv tv pv]] (.allocateNew v))
(dotimes [i n]
(let [{:keys [from-id to-id position]} (nth edges i)]
(.setSafe iv i (utf8 i))
(.setSafe fv i (utf8 from-id))
(.setSafe tv i (utf8 to-id))
(if (nil? position) (.setNull pv i) (.setSafe pv i (long position)))))
(doseq [^FieldVector v [iv fv tv pv]] (.setValueCount v n))
(.setRowCount root n)
root))
;; ------------------------------------------------------------------ staging
(defn- batches
^List [^VectorSchemaRoot root]
(doto (ArrayList.) (.add root)))
(defn- check!
[^QueryResult result hint data]
(when-not (.isSuccess result)
(throw (ex-info (str hint ": " (.getErrorMessage result))
(assoc data :err (.getErrorMessage result))))))
(defn- with-staged-table!
"Create Arrow table `staging-name` from `root`, run `(f)`, always drop it."
[^Connection conn ^BufferAllocator allocator ^String staging-name
^VectorSchemaRoot root data f]
(try
(with-open [^QueryResult r (.createArrowTable conn staging-name (batches root) allocator)]
(check! r "createArrowTable failed" data))
(f)
(finally
;; Dropped even on failure: the staged buffers stay referenced by Ladybug
;; until it is, and the allocator's leak check fires on close otherwise.
(try (.close ^QueryResult (.dropArrowTable conn staging-name))
(catch Throwable _ nil)))))
(defn- copy-node-table!
[^Connection conn table ^String staging-name]
(let [projection (str/join ", " (for [k (nodes/column-keys table)
:let [c (nodes/cypher-property-key table k)]]
(str "n." c " AS " c)))
statement (str "COPY `" table "` FROM (MATCH (n:" staging-name ") "
"RETURN " projection ");")]
(with-open [^QueryResult r (.query conn statement)]
(check! r (str "COPY node table failed: " table)
{:table table :statement statement}))))
(defn- copy-edge-group!
"Load one FROM/TO pair of `IsChildOf`.
`createArrowRelTable` is unusable here — it cannot resolve endpoints against a
UUID-keyed node table — so the edge list is staged as a node table and the
endpoints are resolved by the subquery. The `WHERE` is clause-level because
this dialect prohibits an inline pattern `WHERE`, and both sides are pinned by
label so the join cannot reach outside the pair."
[^Connection conn from-table to-table ^String staging-name]
(let [statement (str "COPY `IsChildOf` FROM ("
"MATCH (e:" staging-name "), "
"(a:" (nodes/match-label from-table) "), "
"(b:" (nodes/match-label to-table) ") "
"WHERE a.id = cast(e.from AS UUID) "
"AND b.id = cast(e.to AS UUID) "
"RETURN a.id, b.id, e.position) "
"(from='" from-table "', to='" to-table "');")]
(with-open [^QueryResult r (.query conn statement)]
(check! r (str "COPY edge group failed: " from-table " -> " to-table)
{:from-table from-table :to-table to-table :statement statement}))))
(defn- staging-name
[prefix & parts]
(str/replace (str/join "_" (cons (str "stg_" prefix) parts)) #"[^A-Za-z0-9_]" "_"))
;; --------------------------------------------------------------------- load
(defn load-projection!
"Load projected nodes and edges into an open Ladybug connection.
`allocator` must outlive `conn` — see `with-allocator!`."
[^Connection conn {:keys [nodes edges]} ^BufferAllocator allocator]
(doseq [[table rows] (sort-by key nodes)
:when (seq rows)]
(let [name (staging-name "node" table)]
(with-open [root (node-batch allocator table rows)]
(with-staged-table! conn allocator name root {:table table}
#(copy-node-table! conn table name)))))
(doseq [[[from-table to-table] group]
(sort-by key (group-by (juxt :from-table :to-table) edges))
:when (seq group)]
(let [name (staging-name "edge" from-table to-table)]
(with-open [root (edge-batch allocator group)]
(with-staged-table! conn allocator name root
{:from-table from-table :to-table to-table}
#(copy-edge-group! conn from-table to-table name))))))
-383
View File
@@ -1,383 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.graph.debug
"In-memory Ladybug sessions for the debug graph console."
(:require
[app.common.exceptions :as ex]
[app.common.logging :as l]
[app.common.time :as ct]
[app.graph.ingest :as graph.ingest]
[app.graph.ladybug :as ladybug]
[app.graph.schema.nodes :as nodes]
[app.graph.sync :as graph.sync]
[app.msgbus :as mbus]
[clojure.java.io :as io]
[clojure.string :as str]
[promesa.exec.csp :as sp])
(:import
com.ladybugdb.Connection
com.ladybugdb.Database))
(set! *warn-on-reflection* true)
(def default-query
"Default console query, written to be self-explanatory in the textarea.
The `filter_*` columns carry node ids for the graph-view result filter;
the results table hides them (see `hide-filter-columns` and the
template's `renderQueryOutput`)."
(str "MATCH (s)-[r]->(t)\n"
"// WHERE some condition\n"
"RETURN label(s) AS src, s.name,\n"
" label(r) AS rel,\n"
" t.name, label(t) AS tgt,\n"
"\n"
"// filter_* columns omitted from table; these needed for graph view\n"
"s.id AS filter_src_id, t.id AS filter_tgt_id;"))
(defonce ^:private sessions
(atom {}))
(defn- session-key
[profile-id]
(str profile-id))
(defn- destroy-session!
[{:keys [conn db sync-ch msgbus]}]
(when sync-ch
(sp/close! sync-ch)
(when msgbus
(mbus/purge! msgbus [sync-ch])))
(when conn
(ex/ignoring (.close ^Connection conn)))
(when db
(ex/ignoring (.close ^Database db))))
(defn- slim-ingest-meta
"Drop full projection rows from session meta.
`build-index` needs `:nodes`/`:edges` once; keeping them in the session
duplicates the entire graph on the JVM heap for every Load."
[meta]
(update meta :projection #(select-keys % [:stats])))
(defn- format-cell
[value]
(cond
(nil? value) "NULL"
(string? value) value
:else (str value)))
(defn- format-query-result
[{:keys [columns rows truncated?]}]
{:columns (mapv str columns)
:rows (mapv (fn [row]
(mapv format-cell row))
rows)
:truncated? truncated?
:row-count (count rows)})
(defn- apply-file-change!
[conn profile-id {:keys [changes revn file-id]}]
(try
(some-> (get @sessions (session-key profile-id))
(as-> current
(when (= file-id (:file-id current))
(let [lock (:lock current)
result (locking lock
(graph.sync/apply-changes!
conn (:index current) changes revn))
sync-at (ct/now)]
(swap! sessions assoc-in [(session-key profile-id) :index]
(:index result))
(swap! sessions update-in [(session-key profile-id) :meta]
(fn [meta]
(cond-> (-> meta
(update :sync dissoc :error)
(assoc-in [:sync :last-at] sync-at)
(assoc-in [:sync :last-applied] (:applied result))
(assoc-in [:sync :last-skipped] (:skipped result)))
(seq (:applied result))
(assoc :revn (:revn result)))))
(when (seq (:skipped result))
(l/dbg :hint "graph sync skipped changes"
:file-id (str file-id)
:revn revn
:skipped (:skipped result)))))))
(catch Throwable cause
(l/wrn :hint "graph sync failed"
:file-id (str file-id)
:cause cause)
(swap! sessions assoc-in [(session-key profile-id) :meta :sync :error]
(ex-message cause)))))
(defn- start-sync-loop!
[{:keys [conn profile-id file-id] :as session}]
(if-let [msgbus (:msgbus session)]
(let [sync-ch (sp/chan :buf (sp/dropping-buffer 64))]
(mbus/sub! msgbus :topic file-id :chan sync-ch)
;; Recur ONLY while the channel is open. A bare `(recur)` after
;; `take!` returns nil would spin forever and pin this Connection
;; (and its Ladybug Database native memory) across every Load.
(sp/go-loop []
(when-let [message (sp/take! sync-ch)]
(when (= :file-change (:type message))
(apply-file-change! conn profile-id message))
(recur)))
(assoc session :sync-ch sync-ch))
session))
(defn session-info
"Return a public view of the current session for `profile-id`, if any."
[profile-id]
(when-let [{:keys [file-id meta loaded-at index]} (get @sessions (session-key profile-id))]
{:file-id file-id
:name (:name meta)
:revn (:revn meta)
:graph-revn (:revn index)
:schema-version (:schema-version meta)
:projection (:projection meta)
:sync (:sync meta)
:loaded-at (ct/format-inst loaded-at :iso)}))
(defn sync-status
"Return incremental sync status for the active session."
[profile-id]
(when-let [session (get @sessions (session-key profile-id))]
(let [{:keys [file-id meta index loaded-at]} session]
{:file-id file-id
:revn (:revn meta)
:graph-revn (:revn index)
:sync (:sync meta)
:loaded-at (ct/format-inst loaded-at :iso)})))
(defn unload-session!
"Close and discard the in-memory graph for `profile-id`."
[profile-id]
(when-let [session (get @sessions (session-key profile-id))]
(destroy-session! session))
(swap! sessions dissoc (session-key profile-id)))
(defn load-session!
"Ingest `file-id` into a new in-memory Ladybug database for `profile-id`."
[cfg profile-id file-id]
(unload-session! profile-id)
(let [^Database db (Database.)
^Connection conn (Connection. db)
msgbus (::mbus/msgbus cfg)]
(.setQueryTimeout conn 0)
(ladybug/ensure-extensions! conn)
(try
(let [meta (graph.ingest/ingest-on-connection! cfg conn file-id
:db-path ":memory:"
:skip-stats? true
:skip-validation? true)
index (graph.sync/build-index file-id (:revn meta) (:projection meta))
;; Discard projection rows after indexing — they are only needed
;; to seed the sync index and would otherwise leak heap on each Load.
meta (slim-ingest-meta meta)
session
;; :lock serializes access to the shared Connection between the
;; msgbus sync loop (writes) and HTTP handlers (reads); the Java
;; binding gives no thread-safety guarantee for one Connection.
(-> {:db db
:conn conn
:lock (Object.)
:file-id file-id
:meta meta
:index index
:msgbus msgbus
:profile-id profile-id
:loaded-at (ct/now)}
start-sync-loop!)]
(swap! sessions assoc (session-key profile-id) session)
meta)
(catch Throwable cause
(destroy-session! {:conn conn :db db :msgbus msgbus})
(throw cause)))))
(defn query-session!
"Run a read-only `statement` against the in-memory graph for `profile-id`.
The statement is bound against the live schema before it runs, so a query
naming a table or a property that does not exist reports the binder's own
message and executes nothing. The engine's read/write analysis then decides
whether it may run at all: the console is an inspection surface, and a
session graph is rebuilt from the file by Reload, so a mutation from here
would produce a graph no rebuild reproduces."
[profile-id statement]
(when (str/blank? statement)
(ex/raise :type :validation
:code :missing-query
:hint "cypher query is required"))
(if-let [{:keys [conn lock]} (get @sessions (session-key profile-id))]
(locking lock
(let [{:keys [ok? error read-only?]} (ladybug/validate-on-connection! conn statement)]
(when-not ok?
(ex/raise :type :validation
:code :graph-query-invalid
:hint error))
(when-not read-only?
(ex/raise :type :validation
:code :graph-query-not-read-only
:hint "the graph console runs read-only queries"))
(-> (ladybug/query-on-connection! conn statement)
format-query-result)))
(ex/raise :type :not-found
:code :graph-session-not-loaded
:hint "load a file graph before running queries")))
(def ^:private export-max-rows
"Row cap for graph-view export queries; far above expected per-file node
and edge counts. `:truncated` in the export signals when it was hit."
100000)
(defn- export-nodes
[conn]
(reduce
(fn [acc {:keys [table]}]
(let [stmt (str "MATCH (n:" (nodes/match-label table)
") RETURN n.id AS id, n.name AS name;")
{:keys [rows truncated?]}
(ladybug/query-on-connection! conn stmt :max-rows export-max-rows)]
(-> acc
(update :nodes into
(map (fn [[id label]]
{:id (str id) :label (str label) :table table}))
rows)
(update :truncated? #(or % truncated?)))))
{:nodes [] :truncated? false}
nodes/node-types))
(defn rel-tables
"Every relationship table in the open database, with whether it carries a
`position` property.
Read from the catalog rather than listed here, so a newly ported transform's
rel table appears in the graph view without the console being told about it."
[conn]
(for [[table] (:rows (ladybug/query-on-connection!
conn "CALL show_tables() WHERE type = 'REL' RETURN name;"
:max-rows 1000))
:let [props (->> (ladybug/query-on-connection!
conn (str "CALL table_info('" table "') RETURN *;")
:max-rows 1000)
:rows
(into #{} (map (comp str second))))]]
{:table table :position? (contains? props "position")}))
(defn- export-edges
[conn]
(reduce
(fn [acc {:keys [table position?]}]
(let [stmt (str "MATCH (a)-[r:`" table "`]->(b) "
"RETURN a.id AS source, b.id AS target, "
(if position? "r.position" "NULL") " AS position, "
"'" table "' AS rel;")
{:keys [rows truncated?]}
(ladybug/query-on-connection! conn stmt :max-rows export-max-rows)]
(-> acc
(update :edges into
(map (fn [[source target position rel]]
(cond-> {:source (str source)
:target (str target)
:rel (str rel)}
(some? position) (assoc :position position))))
rows)
(update :truncated? #(or % truncated?)))))
{:edges [] :truncated? false}
(rel-tables conn)))
(defn- bm-usage-bytes
"Buffer-manager memory in use by this session's in-memory database
(`CALL bm_info()` → [mem_limit mem_usage]); nil if the call fails."
[conn]
(ex/ignoring
(-> (ladybug/query-on-connection! conn "CALL bm_info() RETURN *;" :max-rows 1)
:rows first second)))
(defn export-graph-data!
"Export the node/edge inventory of the in-memory graph for `profile-id`
as plain data for the debug graph view. Returns nil when no session is
loaded. Queries the Ladybug database (not the sync index) so the view
reflects actual DB state, including drift."
[profile-id]
(when-let [{:keys [conn lock file-id index]} (get @sessions (session-key profile-id))]
(locking lock
(let [{:keys [nodes] nodes-truncated? :truncated?} (export-nodes conn)
{:keys [edges] edges-truncated? :truncated?} (export-edges conn)]
{:file-id (str file-id)
:revn (:revn index)
:truncated (boolean (or nodes-truncated? edges-truncated?))
:bm-bytes (bm-usage-bytes conn)
:nodes nodes
:edges edges}))))
(defn- delete-tree!
[^java.io.File file]
(when (.exists file)
(doseq [f (reverse (file-seq file))]
(.delete ^java.io.File f))))
(defn export-session-database!
"Materialize the in-memory session graph of `profile-id` as a `.lbug` file.
The console's graph is in-memory and live-synced, so it can differ from a
fresh projection of the same file — which is exactly when someone wants to
take it away and query it elsewhere. There is no \"save this database\"
primitive, so the transfer goes through Ladybug's `EXPORT DATABASE` (Parquet
per table) into a fresh on-disk database via `IMPORT DATABASE`.
Note the round trip drops table comments. Nothing in the graph is addressed
by a table comment: every table is resolved by name, so the loss costs
nothing.
Returns the path of the written database, or nil when no session is loaded.
The caller owns the file and must delete it once streamed."
[profile-id]
(when-let [{:keys [conn lock file-id]} (get @sessions (session-key profile-id))]
(let [stamp (System/nanoTime)
staging (io/file (System/getProperty "java.io.tmpdir")
(str "penpot-graph-session-" file-id "-" stamp))
db-path (str (io/file (System/getProperty "java.io.tmpdir")
(str file-id "-session-" stamp ".lbug")))]
(try
(locking lock
(ladybug/exec-on-connection!
conn [(str "EXPORT DATABASE '" (.getAbsolutePath staging)
"' (format='parquet');")]))
(ladybug/with-connection! db-path
(fn [target]
(ladybug/exec-on-connection!
target [(str "IMPORT DATABASE '" (.getAbsolutePath staging) "';")
"CHECKPOINT;"])))
db-path
(finally
(delete-tree! staging))))))
(defn- hide-filter-columns
"Drop `filter_*` columns from a query result before HTML table render;
they exist to feed node ids to the graph-view filter, not for reading.
The JSON response path keeps the full result."
[{:keys [columns rows] :as result}]
(let [idxs (vec (keep-indexed
(fn [i c] (when-not (str/starts-with? (str c) "filter_") i))
columns))]
(if (or (empty? idxs) (= (count idxs) (count columns)))
result
(assoc result
:columns (mapv (vec columns) idxs)
:rows (mapv (fn [row] (mapv (vec row) idxs)) rows)))))
(defn console-context
"Build template data for the graph debug console page."
[profile-id & {:keys [query query-result error message]}]
{:session (session-info profile-id)
:query (or query default-query)
:query-result (some-> query-result hide-filter-columns)
:error error
:message message
:default-query default-query})
-106
View File
@@ -1,106 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.graph.ingest
"Penpot file -> Ladybug graph projection."
(:require
[app.binfile.common :as bfc]
[app.common.exceptions :as ex]
[app.common.logging :as l]
[app.common.types.file :as ctf]
[app.db :as db]
[app.graph.arrow :as graph.arrow]
[app.graph.ladybug :as ladybug]
[app.graph.meta :as graph.meta]
[app.graph.projection.document :as projection.document]
[app.graph.projection.transforms :as projection.transforms]
[app.graph.schema :as schema]
[app.graph.stats :as stats]
[app.srepl.helpers :as h])
(:import
com.ladybugdb.Connection
org.apache.arrow.memory.BufferAllocator))
(defn- fetch-file!
[system file-id]
(let [file-id (h/parse-uuid file-id)
file (db/run! system #(bfc/get-file % file-id :realize? true))]
(when-not file
(ex/raise :type :not-found
:code :file-not-found
:file-id (str file-id)))
(when-not (:data file)
(ex/raise :type :validation
:code :file-without-data
:hint "file has no data to project"
:file-id (str file-id)))
[file-id file]))
(defn- ingest-on-connection*!
[system ^Connection conn file-id ^BufferAllocator allocator
{:keys [db-path skip-stats? skip-validation?] :or {skip-stats? true}}]
(let [[file-id file] (fetch-file! system file-id)
db-path (or db-path (ladybug/db-path-for-file file-id))
data (:data file)]
(when-not skip-validation?
(ctf/check-file-data data))
(l/inf :hint "graph ingest"
:file-id (str file-id)
:revn (:revn file)
:db-path db-path
:schema schema/schema-version)
(let [ddl (schema/ddl-statements)
{:keys [nodes edges stats]}
(projection.document/projection-data data file)]
(ladybug/exec-on-connection! conn ddl)
(graph.arrow/load-projection! conn {:nodes nodes :edges edges} allocator)
(ladybug/exec-on-connection! conn ["CHECKPOINT;"])
(let [transforms (projection.transforms/apply-transforms! system conn data file)]
;; Written last: its presence doubles as the build-complete marker.
(graph.meta/write! conn {:file-id file-id
:revn (:revn file)})
{:file-id file-id
:revn (:revn file)
:name (or (:name data) (:name file))
:db-path db-path
:schema-version schema/schema-version
:projection {:stats stats
:nodes nodes
:edges edges}
:transforms transforms
:stats (when-not skip-stats?
(stats/summarize-connection conn))}))))
(defn ingest-on-connection!
"Project `file-id` into an already open Ladybug `conn`.
Takes an `:arrow-alloc` when the caller already owns one; otherwise it makes
a short-lived allocator around this call. A caller that opened the connection
itself should pass its own, because the allocator has to be closed *after*
the connection — see `app.graph.arrow/with-allocator!`."
[system ^Connection conn file-id & {:keys [arrow-alloc] :as opts}]
(if arrow-alloc
(ingest-on-connection*! system conn file-id arrow-alloc opts)
(graph.arrow/with-allocator!
(fn [allocator] (ingest-on-connection*! system conn file-id allocator opts)))))
(defn ingest-file!
[system file-id & {:keys [db-path reset-db? skip-stats? skip-validation?]
:or {reset-db? true}}]
(let [db-path (or db-path (ladybug/db-path-for-file (h/parse-uuid file-id)))]
(when reset-db?
(ladybug/reset-db-path! db-path))
;; Allocator outermost: Ladybug holds the staged Arrow buffers until its
;; tables are dropped, which is no later than connection close, so the
;; allocator must be closed after the connection and the database.
(graph.arrow/with-allocator!
(fn [allocator]
(ladybug/with-connection! db-path
(fn [conn]
(ingest-on-connection*! system conn file-id allocator
{:db-path db-path
:skip-stats? skip-stats?
:skip-validation? skip-validation?})))))))
-504
View File
@@ -1,504 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.graph.ladybug
"Ladybug access layer for graph-backed Penpot.
Uses the embedded Java API (`com.ladybugdb/lbug`)."
(:require
[app.common.exceptions :as ex]
[app.common.json :as json]
[app.graph.schema.values :as values]
[clojure.string :as str]
[datoteka.fs :as fs])
(:import
com.ladybugdb.Connection
com.ladybugdb.Database
com.ladybugdb.FlatTuple
com.ladybugdb.PreparedStatement
com.ladybugdb.QueryResult
com.ladybugdb.Value))
(set! *warn-on-reflection* true)
(defn default-graph-dir
[]
(or (System/getenv "PENPOT_GRAPH_DIR") "/tmp/penpot-graph"))
(defn db-path-for-file
[file-id]
(str (fs/path (default-graph-dir) (str file-id ".lbug"))))
(defn- memory-db-path?
[db-path]
(= db-path ":memory:"))
(defn reset-db-path!
[db-path]
(when-not (memory-db-path? db-path)
(when (fs/exists? db-path)
(fs/delete db-path))))
(defn escape-cypher-string
[s]
(-> (str s)
(str/replace "\\" "\\\\")
(str/replace "'" "\\'")))
(defn format-uuid
[id]
(str "uuid('" (str id) "')"))
(defn format-string
[s]
(str "'" (escape-cypher-string s) "'"))
(defn format-int
[n]
(str (long n)))
(defn format-number
[n]
(if (== n (long n))
(format-int n)
(str (double n))))
(defn format-json
[v]
(str "json('" (escape-cypher-string (json/encode v)) "')"))
(defn format-timestamp
"Ladybug TIMESTAMP literal of the form `timestamp('<ISO-8601 instant>')`."
[v]
(let [s (cond
(instance? java.time.Instant v)
(.toString ^java.time.Instant v)
(instance? java.util.Date v)
(.toString (.toInstant ^java.util.Date v))
(string? v)
v
:else
(str v))]
(str "timestamp('" (escape-cypher-string s) "')")))
(defn format-value
[v]
(cond
(nil? v) "NULL"
(uuid? v) (format-uuid v)
(instance? java.time.Instant v) (format-timestamp v)
(instance? java.util.Date v) (format-timestamp v)
(string? v) (format-string v)
(number? v) (format-number v)
(boolean? v) (if v "true" "false")
(keyword? v) (format-string (name v))
(map? v) (format-json v)
(coll? v) (format-json v)
:else (format-string (str v))))
(defn map-type?
"Is `ladybug-type` a MAP column?"
[ladybug-type]
(and (string? ladybug-type)
(str/starts-with? ladybug-type "MAP(")
(not (str/ends-with? ladybug-type "]"))))
(defn list-type?
"Is this a list or fixed-size array type? Checked before MAP and STRUCT,
since `STRUCT(…)[]` starts with `STRUCT(` but is a list of them."
[ladybug-type]
(and (string? ladybug-type)
(some? (re-matches #".+\[\d*\]$" ladybug-type))))
(defn struct-type?
[ladybug-type]
(and (string? ladybug-type)
(str/starts-with? ladybug-type "STRUCT(")
(not (list-type? ladybug-type))))
(declare format-typed-value)
(defn- format-typed-list
"Cypher LIST literal, elements formatted by the element type.
Handles `T[]` and the fixed-size `T[n]` alike: the size constrains the column,
not the literal."
[ladybug-type v]
(let [element (second (re-matches #"(.+?)\[\d*\]$" ladybug-type))
elems (if (or (sequential? v) (set? v)) (seq v) [v])]
(str "[" (str/join ", " (map #(format-typed-value element %) elems)) "]")))
(defn- format-struct
"Cypher STRUCT literal, `{field: value, …}`.
*Every* declared field is emitted, NULL where the value has none: a struct
literal's type is its field list, so omitting a field yields a different type
and Ladybug refuses the implicit cast (`STRUCT(m2 DOUBLE, m4 DOUBLE)` cannot
be assigned to `STRUCT(m1 …, m2 …, m3 …, m4 …)`). Penpot's layout margins are
exactly that case — a shape sets only the sides it overrides."
[ladybug-type v]
(let [fields (values/struct-fields ladybug-type)]
(str "{"
(str/join ", "
(for [[field field-type] fields
:let [fv (get v field)]]
;; Backticked for the same reason as in the DDL: a field
;; named `column` is a keyword and will not parse bare.
;; A bare NULL is typed STRING, which changes the struct's
;; type as surely as omitting the field would, so absent
;; fields get a NULL cast to their declared type.
(str "`" field "`: "
(if (nil? fv)
(str "cast(NULL, '" field-type "')")
(format-typed-value field-type fv)))))
"}")))
(defn format-typed-value
"Cypher literal for `v` in a column of `ladybug-type`.
Recursive over the type language, because the types are: a
`MAP(UUID, STRUCT(…))` needs its keys, its fields and each field's own type
honoured. `app.graph.schema.values/coerce` shapes the value first — turning a
matrix record into six doubles, a hex colour into a packed integer — so this
function only has to escape plain data.
`map-key-fn` renders the keys of a `MAP(STRING, …)`; the caller supplies it
because the right form is a property of the column, not of this function
(`app.graph.schema.contract/map-key-fn`)."
([ladybug-type v] (format-typed-value ladybug-type v nil))
([ladybug-type v map-key-fn]
(let [v (values/coerce ladybug-type v)]
(cond
(nil? v)
"NULL"
(list-type? ladybug-type)
(format-typed-list ladybug-type v)
(map-type? ladybug-type)
(let [[key-type value-type] (values/map-types ladybug-type)
entries (seq v)
format-key (if (and map-key-fn (= "STRING" key-type))
#(format-string (map-key-fn (key %)))
#(format-typed-value key-type (key %)))]
(str "map([" (str/join ", " (map format-key entries))
"], ["
(str/join ", " (map #(format-typed-value value-type (val %)) entries))
"])"))
(struct-type? ladybug-type)
(format-struct ladybug-type v)
(= ladybug-type "JSON")
(format-json v)
;; Coerce string ids from transit edge-cases into UUID literals.
(= ladybug-type "UUID")
(format-uuid v)
(= ladybug-type "TIMESTAMP")
(format-timestamp v)
:else
(format-value v)))))
(defn- ensure-semicolon
[statement]
(let [s (str/trim (str statement))]
(if (str/ends-with? s ";") s (str s ";"))))
(defn- value->clj
[^Value value]
(when-not (.isNull value)
(let [v (try
(.getValue value)
(catch Exception _
;; LIST/STRUCT values are not supported by the binding's
;; getValue (\"value_get_value\"); fall back to the textual
;; representation so console queries do not crash.
(.toString value)))]
(cond
(instance? Long v) v
(instance? Integer v) (long v)
(instance? Double v) v
:else v))))
(defn- check-success!
[^QueryResult result statement]
(when-not (.isSuccess result)
(let [err (.getErrorMessage result)]
(ex/raise :type :internal
:code :ladybug-query-failed
:hint (str "Ladybug query failed: " err)
:statement statement
:err err))))
(defn- query-columns
[^QueryResult result]
(let [ncols (.getNumColumns result)]
(vec (for [i (range ncols)]
(.getColumnName result (long i))))))
(defn- query-row
[^FlatTuple tuple ncols]
(vec (for [i (range ncols)]
(with-open [^Value value (.getValue tuple (long i))]
(value->clj value)))))
(def ^:private default-query-max-rows 200)
(defn- read-query-rows
[^QueryResult result ncols max-rows]
(loop [rows [] n 0]
(if (and (< n max-rows) (.hasNext result))
(let [row (with-open [^FlatTuple tuple (.getNext result)]
(query-row tuple ncols))]
(recur (conj rows row) (inc n)))
rows)))
(defn query-on-connection!
"Execute a Cypher query on `conn` and return tabular results.
Returns `{:columns [...] :rows [[...] ...] :truncated? bool}`."
[^Connection conn statement & {:keys [max-rows]
:or {max-rows default-query-max-rows}}]
(let [cypher (ensure-semicolon statement)]
(with-open [^QueryResult result (.query conn cypher)]
(check-success! result cypher)
(let [ncols (long (.getNumColumns result))
columns (query-columns result)
rows (read-query-rows result ncols max-rows)
total (long (.getNumTuples result))]
{:columns columns
:rows rows
:truncated? (and (pos? total) (> total (count rows)))}))))
(def ^:private default-query-timeout-ms
"0 disables query timeout (recommended for bulk COPY ingest)."
0)
(defn- scalar-value
[^Connection conn statement]
(let [cypher (ensure-semicolon statement)]
(with-open [^QueryResult result (.query conn cypher)]
(check-success! result cypher)
(when (.hasNext result)
(with-open [^FlatTuple tuple (.getNext result)]
(with-open [^Value value (.getValue tuple 0)]
(value->clj value)))))))
(defn- extension-statement-ok?
[err-msg]
(let [err (str/lower-case (or err-msg ""))]
(or (str/includes? err "already loaded")
(str/includes? err "already installed"))))
(defn- run-extension-statement!
[^Connection conn statement]
(let [cypher (ensure-semicolon statement)]
(with-open [^QueryResult result (.query conn cypher)]
(when-not (.isSuccess result)
(let [err (.getErrorMessage result)]
(when-not (extension-statement-ok? err)
(check-success! result cypher)))))))
(defn ensure-extensions!
"Install and load Ladybug extensions required by graph ingest and sync."
[^Connection conn]
(run-extension-statement! conn "INSTALL json;")
(run-extension-statement! conn "LOAD json;"))
(defn- run-statements!
[^Connection conn statements]
(doseq [statement statements]
(let [cypher (ensure-semicolon statement)]
(with-open [^QueryResult result (.query conn cypher)]
(check-success! result cypher)))))
(defn- ensure-db-path!
[db-path]
(when-not (memory-db-path? db-path)
(fs/create-dir (fs/parent db-path))))
(defn with-connection!
"Open a Ladybug connection for `db-path` and invoke `(f conn)`.
Options:
- `:query-timeout-ms` query timeout in milliseconds (default 0, disabled)
For `:memory:`, the database only lives for the duration of this call;
all reads and writes must happen inside `f`."
[db-path f & {:keys [query-timeout-ms]
:or {query-timeout-ms default-query-timeout-ms}}]
(ensure-db-path! db-path)
(let [^Database db (if (memory-db-path? db-path)
(Database.)
(Database. (str db-path)))]
(try
(let [^Connection conn (Connection. db)]
(try
(.setQueryTimeout conn (long query-timeout-ms))
(ensure-extensions! conn)
(f conn)
(finally
(.close conn))))
(finally
(.close db)))))
(defn exec-on-connection!
"Execute Cypher statements on an open Ladybug connection."
[^Connection conn statements]
(assert (sequential? statements) "statements should be a sequential collection")
(run-statements! conn statements))
;; --- prepared statements
(defn- ->param-value
"Clojure scalar → `Value` for prepared-statement binding.
This is the only `Value` constructor on the write path, so every parameter
is wrapped here. Parameters are scalars: the `Value` constructor takes no
list or map, so `MAP`, `STRUCT` and `T[]` columns stay literal-rendered
(`format-typed-value`) and the `:else` raise below means a caller tried to
bind one."
^Value [v]
(cond
(nil? v) (Value/createNull) ; no explicit type needed
(uuid? v) (Value. ^Object v) ; native UUID
(string? v) (Value. ^Object v)
(boolean? v) (Value. ^Object v)
(integer? v) (Value. ^Object (long v))
(number? v) (Value. ^Object (double v))
(keyword? v) (Value. ^Object (name v))
(instance? java.time.Instant v) ; native TIMESTAMP
(Value. ^Object v)
(instance? java.util.Date v)
(Value. ^Object (.toInstant ^java.util.Date v))
:else
(ex/raise :type :internal
:code :ladybug-unsupported-param
:hint (str "cannot bind a " (type v) " as a Ladybug parameter; "
"compound columns must be literal-rendered")
:value v)))
(defn- as-statement
"Normalize a statement to `{:cypher … :params …}`.
A bare string binds nothing, so the sync builders can convert to bound
parameters one family at a time."
[stmt]
(if (map? stmt)
(update stmt :params #(or % {}))
{:cypher stmt :params {}}))
(defn prepare-on-connection!
"Parse and bind `statement` on `conn` without executing it.
The returned `PreparedStatement` is a JNI resource: the caller closes it."
^PreparedStatement [^Connection conn statement]
(let [cypher (ensure-semicolon statement)
ps (.prepare conn cypher)]
(when-not (.isSuccess ps)
(let [err (.getErrorMessage ps)]
(.close ps)
(ex/raise :type :internal
:code :ladybug-prepare-failed
:hint (str "Ladybug prepare failed: " err)
:statement cypher
:err err)))
ps))
(defn execute-prepared!
"Bind `params` into `ps` and execute it on `conn`.
`params` keys are parameter names without the `$` (keyword or string);
values are scalars. Every bound `Value` is closed, including the ones built
before a later parameter is rejected."
[^Connection conn ^PreparedStatement ps params]
(let [vmap (java.util.HashMap.)]
(try
(doseq [[k v] params]
(.put vmap (name k) (->param-value v)))
(with-open [^QueryResult result (.execute conn ps vmap)]
(check-success! result "<prepared>"))
(finally
(run! #(.close ^Value %) (.values vmap))))))
(defn exec-prepared-on-connection!
"Prepare all statements, then execute all of them.
A parse or bind failure in *any* statement aborts the batch before the first
mutation runs — the bind-level batch gate. Statements are
`{:cypher … :params {…}}` maps or bare strings."
[^Connection conn stmts]
(assert (sequential? stmts) "statements should be a sequential collection")
(let [prepared (volatile! [])]
(try
(doseq [stmt stmts]
(let [{:keys [cypher params]} (as-statement stmt)]
(vswap! prepared conj {:ps (prepare-on-connection! conn cypher)
:params params})))
(doseq [{:keys [ps params]} @prepared]
(execute-prepared! conn ps params))
(finally
(run! #(.close ^PreparedStatement (:ps %)) @prepared)))))
(defn validate-on-connection!
"Binder gate: parse and semantic-check `statement` against the live schema,
without executing it.
Returns `{:ok? … :error … :read-only? …}`. Unlike `prepare-on-connection!`
a failure is a return value rather than a raise: the callers are gates (the
CI binder gate, the console read-only gate) that report it. `:read-only?` is
the engine's own read/write analysis."
[^Connection conn statement]
(with-open [^PreparedStatement ps (.prepare conn (ensure-semicolon statement))]
(let [ok? (.isSuccess ps)]
{:ok? ok?
:error (when-not ok? (.getErrorMessage ps))
:read-only? (when ok? (.isReadOnly ps))})))
(defn query-scalar-on-connection!
"Execute a query expected to return a single scalar value on `conn`."
[^Connection conn statement]
(scalar-value conn statement))
(defn exec!
"Execute Cypher statements against a Ladybug database.
`db-path` is either `:memory:` or a filesystem path to a `.lbug` database."
[db-path statements]
(with-connection! db-path
(fn [conn]
(exec-on-connection! conn statements))))
(defn query-scalar!
"Execute a query expected to return a single scalar value."
[db-path statement]
(with-connection! db-path
(fn [conn]
(query-scalar-on-connection! conn statement))))
(defn smoke-test!
"Run a minimal CREATE + count against Ladybug."
[& {:keys [db-path] :or {db-path ":memory:"}}]
(when-not (memory-db-path? db-path)
(reset-db-path! db-path))
(with-connection! db-path
(fn [^Connection conn]
(run-statements! conn
["CREATE NODE TABLE Person(name STRING, age INT64, PRIMARY KEY(name));"
"CREATE (:Person {name: 'Alice', age: 25});"
"CREATE (:Person {name: 'Bob', age: 30});"])
{:db-path db-path
:person-count (scalar-value conn
"MATCH (a:Person) RETURN count(a) AS c;")})))
-59
View File
@@ -1,59 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.graph.meta
"`GraphMeta`: the graph's own account of who built it and from what.
A projected graph is a cache of a file at a revision, built by a known
schema. The row records both, so a reader can decide whether to reuse the
database or rebuild it: a `schema_version` that no longer matches the
registry, or a `source_revn` behind the file's, means the cache is stale.
The row is written *last* in a build, so its presence also marks the build
complete.
Keyed by `source_file_id` rather than holding a single row: a closure graph
is a union of per-file builds, and each contributing file keeps its own
provenance."
(:require
[app.common.time :as ct]
[app.graph.ladybug :as ladybug]
[app.graph.schema.nodes :as nodes])
(:import
com.ladybugdb.Connection))
(set! *warn-on-reflection* true)
(def table
"GraphMeta")
(def producer
"penpot")
(def ddl
"DDL for the provenance table."
(str "CREATE NODE TABLE `" table "` ("
"`source_file_id` UUID, "
"`producer` STRING, "
"`producer_version` STRING, "
"`schema_version` STRING, "
"`source_revn` INT64, "
"`built_at` TIMESTAMP, "
"PRIMARY KEY (`source_file_id`));"))
(defn write!
"Record what this build produced for `file-id`."
[^Connection conn {:keys [file-id revn]}]
(ladybug/exec-on-connection! conn [ddl])
(ladybug/exec-on-connection!
conn
[(str "MERGE (m:`" table "` {source_file_id: " (ladybug/format-uuid file-id) "}) "
"SET m.producer = " (ladybug/format-string producer) ", "
"m.producer_version = " (ladybug/format-string (or (System/getenv "PENPOT_BUILD") "devenv")) ", "
"m.schema_version = " (ladybug/format-string nodes/schema-version) ", "
"m.source_revn = " (ladybug/format-int (or revn 0)) ", "
"m.built_at = " (ladybug/format-timestamp (ct/now)) ";")]))
@@ -1,214 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.graph.projection.document
"Project a Penpot file-data map into Ladybug nodes and structural edges.
Projects Document, Page, Component, the full shape tree (skipping the root
frame), and `IsChildOf` edges from shapes/pages/components to their parent.
Two denormalizations happen here rather than in a later pass, because the
walk already has both answers in hand and a post-ingest statement would have
to rediscover them:
- `page-id` on every shape, from the page the walk is currently in;
- `component-id` propagated from an instance head down to its descendants,
from the head context the walk carries."
(:require
[app.common.logging :as l]
[app.common.uuid :as uuid]
[app.graph.schema.nodes :as nodes]))
(def root-frame-id
uuid/zero)
(defn- document-attrs
"The Document node's attrs: the file row, minus its data blob.
`:options` is lifted out of the blob before it goes: it is file-level
configuration a consumer wants without opening `:data`."
[file data]
(-> file
(assoc :id (or (:id data) (:id file)))
(cond-> (:options data) (assoc :options (:options data)))
(dissoc :data)))
(defn- page-attrs
[page index]
(-> page
(dissoc :objects)
(cond-> (some? index) (assoc :index (long index)))))
(defn- component-attrs
[component]
(-> component
(dissoc :objects)
;; schema:component requires :path; some legacy rows omit it
(update :path #(or % ""))))
(defn- shape-table
[shape]
(nodes/table-for-type (:type shape)))
(defn denormalized-shape
"`shape` with `page-id` set and an inherited `component-id` filled in.
A shape that carries its own `component-id` keeps it; `component-ctx` only
fills the gap for descendants (see `descend-component-ctx`)."
[shape page-id component-ctx]
(cond-> (assoc shape :page-id page-id)
(and (uuid? component-ctx) (nil? (:component-id shape)))
(assoc :component-id component-ctx)))
(defn- shape-node-attrs
[table shape page-id component-ctx]
(nodes/project-attrs table (denormalized-shape shape page-id component-ctx)))
(defn descend-component-ctx
"The component context to pass to `shape`'s children.
Inheritance stops at the nearest ancestor Frame carrying a `component-id`,
and any intermediate shape that carries one is a barrier:
- a Frame with its own `component-id` becomes the new context (it is an
instance head, and its descendants belong to *it*, not to an outer head);
- any other shape carrying a `component-id` blocks inheritance below it
without being able to supply one, since only Frames are heads;
- otherwise the context passes through unchanged."
[table shape ctx]
(let [own (:component-id shape)]
(cond
(and (some? own) (= table "Frame")) own
(some? own) ::blocked
:else ctx)))
(defn- container-table?
[table]
(contains? nodes/container-tables table))
(defn- child-shape-ids
"Child ids in Penpot z-order (reversed from the stored :shapes list)."
[parent]
(when-let [shapes (:shapes parent)]
(vec (reverse shapes))))
(defn- initial-acc
[]
{:nodes {}
:edges []
:stats {:documents 0 :pages 0 :components 0 :shapes 0}})
(declare project-shape-ids)
(defn- project-shape
[objects acc table shape parent-table parent-id position page-id component-ctx]
(let [shape-id (:id shape)
acc' (-> acc
(update-in [:nodes table] (fnil conj [])
(shape-node-attrs table shape page-id component-ctx))
(update :edges conj {:from-table table
:from-id shape-id
:to-table parent-table
:to-id parent-id
:position position})
(update-in [:stats :shapes] inc))]
(if-let [child-ids (when (container-table? table)
(child-shape-ids shape))]
(project-shape-ids objects acc' table shape-id child-ids page-id
(descend-component-ctx table shape component-ctx))
acc')))
(defn- project-shape-ids
[objects acc parent-table parent-id child-ids page-id component-ctx]
(reduce
(fn [acc [position shape-id]]
(if-let [shape (get objects shape-id)]
(if-let [table (shape-table shape)]
(project-shape objects acc table shape parent-table parent-id position
page-id component-ctx)
(do
(l/wrn :hint "unsupported shape type for graph slice"
:shape-id (str shape-id)
:type (:type shape))
acc))
(do
(l/wrn :hint "missing shape in page objects"
:shape-id (str shape-id))
acc)))
acc
(map-indexed vector child-ids)))
(defn- project-page
[acc doc-id page position]
(let [page-id (:id page)
objects (:objects page)
root (get objects root-frame-id)
page-node (nodes/project-attrs "Page" (page-attrs page position))
acc' (-> acc
(update-in [:nodes "Page"] (fnil conj []) page-node)
(update :edges conj {:from-table "Page"
:from-id page-id
:to-table "Document"
:to-id doc-id
:position position})
(update-in [:stats :pages] inc))]
(if-let [top-level-ids (child-shape-ids root)]
(project-shape-ids objects acc' "Page" page-id top-level-ids page-id nil)
acc')))
(defn- project-component
[acc doc-id component position]
(if (:deleted component)
acc
(let [comp-id (:id component)
node (nodes/project-attrs "Component" (component-attrs component))]
(-> acc
(update-in [:nodes "Component"] (fnil conj []) node)
(update :edges conj {:from-table "Component"
:from-id comp-id
:to-table "Document"
:to-id doc-id
:position position})
(update-in [:stats :components] inc)))))
(defn- project-components
[acc doc-id components]
(reduce (fn [acc [position [_id component]]]
(project-component acc doc-id component position))
acc
(map-indexed vector components)))
(defn projection-data
"Build node/edge rows for projecting `data` into Ladybug.
Returns `{:nodes {table [attrs ...]} :edges [...] :stats {...}}`."
[data file]
(let [doc-id (or (:id data) (:id file))
doc-node (nodes/project-attrs "Document" (document-attrs file data))
;; `:pages` is the tab order the user sees, and `Page.index` and the
;; page's `IsChildOf.position` are that order. Child shapes are
;; reversed on the way in (`child-shape-ids`) because their stored
;; list runs bottom to top; pages have no such second ordering.
pages (seq (:pages data))
comps (seq (:components data))
acc0 (-> (initial-acc)
(update-in [:nodes "Document"] (fnil conj []) doc-node)
(assoc-in [:stats :documents] 1))
acc (cond-> acc0
(seq comps)
(project-components doc-id comps))
acc (if (empty? pages)
acc
(reduce (fn [acc [position page-id]]
(if-let [page (get-in data [:pages-index page-id])]
(project-page acc doc-id page position)
(do
(l/wrn :hint "missing page in pages-index"
:page-id (str page-id))
acc)))
acc
(map-indexed vector pages)))]
(select-keys acc [:nodes :edges :stats])))
@@ -1,149 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.graph.projection.transforms
"Derived graph links: edges a reader could compute from the projected
columns, materialized once at build time so a query does not have to.
Each entry in `registry` names the transform, the relationship it produces,
and the function that produces it, so adding one is a single entry and
nothing else has to be told about it."
(:require
[app.common.logging :as l]
[app.graph.ladybug :as ladybug]
[app.graph.schema.nodes :as nodes])
(:import
com.ladybugdb.Connection))
(set! *warn-on-reflection* true)
(defn- run-scalar!
[^Connection conn statement]
(or (ladybug/query-scalar-on-connection! conn statement) 0))
(defn- link-component-instances!
"`IsInstanceOf` from Frame instance heads to their Component.
Every head is linked, the main instance and any copy root alike.
`component-file` is what makes a head a head here, not `component-id` alone.
`app.common.types.component/instance-of?` requires both, and the projection
denormalizes `component-id` down the shape tree
(`app.graph.projection.document`), so on its own it no longer distinguishes a
head from a shape that merely lives inside one. `component-file` is not
denormalized and remains the head marker Penpot itself uses."
[^Connection conn]
(run-scalar! conn
(str "MATCH (f:Frame), (c:Component) "
"WHERE f.component_id = c.id "
"AND f.component_file IS NOT NULL "
"AND NOT COALESCE(c.deleted, false) "
"MERGE (f)-[:IsInstanceOf]->(c) "
"RETURN count(*);")))
(defn- shape-pair-statements
"One statement per (from, to) shape-table pair.
Ladybug cannot create a relationship bound by multiple node labels in a
single `MERGE`, a constraint inherited from Kùzu, which it forks (upstream
issue kuzudb/kuzu#5841). The loop over label pairs is that dialect
constraint, not a modelling choice."
[f]
(for [from nodes/shape-tables
to nodes/shape-tables]
(f from to)))
(defn- link-shape-refs!
"`RefersTo` from an instance shape to its homologue in the main instance,
driven by `shape-ref`."
[^Connection conn]
(reduce
(fn [total statement] (+ total (run-scalar! conn statement)))
0
(shape-pair-statements
(fn [from to]
(str "MATCH (s:" (nodes/match-label from) "), (t:" (nodes/match-label to) ") "
"WHERE s.shape_ref = t.id "
"MERGE (s)-[:RefersTo]->(t) "
"RETURN count(*);")))))
(def ^:private swap-slot-prefix "swap-slot-")
(def ^:private slot-uuid-expr
;; Ladybug `substring` is 1-indexed; 36 = RFC 4122 UUID text length.
(str "substring(touched_key, " (inc (count swap-slot-prefix)) ", 36)"))
(defn- link-swap-slots!
"`FillsSwapSlot` from a swapped-in shape to the slot it replaces.
Penpot records a component sub-shape swap as a `swap-slot-<uuid>` entry in
the *replacing* shape's `touched` set, where `<uuid>` names the replaced
slot shape in the main instance. The entries are then stripped from
`touched`, as `app.common.types.component/normal-touched-groups` does, so a
reader of `touched` sees design edits rather than swap bookkeeping.
Stripping makes this the one transform that writes a column another
transform could read. Anything reading `touched` has to run before it."
[^Connection conn]
(let [linked
(reduce
(fn [total statement] (+ total (run-scalar! conn statement)))
0
(shape-pair-statements
(fn [from to]
(str "MATCH (s:" (nodes/match-label from) ") "
"WHERE size(s.touched) > 0 "
"UNWIND s.touched AS touched_key "
"WITH s, touched_key "
"WHERE STARTS_WITH(touched_key, '" swap-slot-prefix "') "
"WITH s, CAST(" slot-uuid-expr ", 'UUID') AS slot_id "
"MATCH (t:" (nodes/match-label to) ") "
"WHERE t.id = slot_id AND s.id <> t.id "
"MERGE (s)-[r:FillsSwapSlot {slot_id: slot_id}]->(t) "
"RETURN count(r);"))))]
;; Strip unconditionally: an entry may name a slot that was garbage
;; collected, so "no edge created" does not mean "nothing to strip".
(doseq [table nodes/shape-tables]
(ladybug/exec-on-connection!
conn
[(str "MATCH (s:" (nodes/match-label table) ") "
"WHERE size(s.touched) > 0 "
"SET s.touched = list_filter(s.touched, x -> "
"NOT STARTS_WITH(x, '" swap-slot-prefix "'));")]))
linked))
(def registry
"Every transform this backend applies.
`:id` names the transform in the ingest report and the log. `:rel` names
the relationship it produces. The three registered here read disjoint
columns, so the vector order is not load-bearing. The one ordering
constraint that exists is stated on `link-swap-slots!`."
[{:id "link-component-instances" :rel :IsInstanceOf :run link-component-instances!}
{:id "link-shape-refs" :rel :RefersTo :run link-shape-refs!}
{:id "link-swap-slots" :rel :FillsSwapSlot :run link-swap-slots!}])
(defn apply-transforms!
"Apply every registered transform to an already loaded graph.
Returns `{:ids [...] :counts {...} :transforms n}`, where `:ids` names what
ran and `:counts` gives the edges each one produced."
[_system ^Connection conn _data _file]
(reduce
(fn [acc {:keys [id rel run]}]
(let [n (run conn)]
(l/inf :hint "graph transform" :transform id :edges n)
(-> acc
(update :ids conj id)
(update :counts assoc rel n)
(assoc rel n))))
{:ids [] :counts {} :transforms (count registry)}
registry))
(defn transform-ids
"Ids of every transform in the registry."
[]
(mapv :id registry))
-65
View File
@@ -1,65 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.graph.report
(:require
[clojure.core :as c]
[clojure.string :as str]))
(defn- println!
[& lines]
(doseq [line lines]
(println line)))
(defn- section-title
[title]
(println! (str "\n" title)
(str (apply str (repeat (count title) "─")))))
(defn- kv-line
[k v]
(format " %-14s %s" (str k ":") v))
(defn- print-node-counts
[nodes]
(doseq [[table count] (sort-by first nodes)
:when (pos? (long count))]
(println! (kv-line table count))))
(defn print-ingest!
"Pretty-print the result map returned by `app.graph.ingest/ingest-file!`."
[{:keys [file-id revn name db-path schema-version projection transforms stats]}]
(section-title "Graph ingest")
(println! (kv-line "File" (str name " (" file-id ")"))
(kv-line "Revision" revn)
(kv-line "Schema" schema-version)
(kv-line "Database" db-path))
(when-let [pstats (:stats projection)]
(section-title "Projection")
(doseq [[k v] (sort-by key pstats)]
(println! (kv-line (c/name k) v))))
(section-title "Transforms")
(println! (kv-line "Applied" (or (:transforms transforms) 0)))
(doseq [[rel count] (sort-by key (:counts transforms))]
(println! (kv-line (c/name rel) count)))
(when-let [ids (seq (:ids transforms))]
(println! (kv-line "Recorded" (str/join ", " ids))))
(when stats
(section-title "Graph counts")
(when-let [nodes (:nodes stats)]
(println! " Nodes")
(print-node-counts nodes))
(when-let [edges (:edges stats)]
(println! " Edges")
(doseq [[rel count] (sort-by key edges)
:when (pos? (long count))]
(println! (kv-line (c/name rel) count)))))
(println!)
nil)
-30
View File
@@ -1,30 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.graph.schema
"Ladybug DDL facade for the graph-backed Penpot vertical slice.
Node metadata and DDL generation live in `app.graph.schema.nodes`."
(:require
[app.graph.schema.nodes :as nodes]))
(def schema-version
nodes/schema-version)
(def container-node-tables
nodes/container-tables)
(def shape-node-tables
nodes/shape-tables)
(def node-tables
(mapv (fn [{:keys [table schema]}]
{:name table :schema schema})
nodes/node-types))
(defn ddl-statements
[]
(nodes/ddl-statements))
-150
View File
@@ -1,150 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.graph.schema.contract
"Deliberate choices in Penpot's graph schema, recorded as data.
Penpot must pick a spelling and a type for every graph column. A Ladybug
column gets both once, at table creation, and neither widens afterwards. The
choices are therefore worth making deliberately and worth recording.
Three of them live here:
- `column-name` maps a Penpot key to its column. The rule is snake_case of
the key, and `renames` records every exception.
- `dropped-keys` and `per-table-dropped` name Penpot keys that deliberately
get no column.
- `type-overrides` pins the Ladybug type where the Malli-derived one
(`app.graph.schema.types`) is coarser than the column deserves.
Each entry carries its reason. A divergence from the default rule is then a
diff to review rather than a silent rename."
(:require
[app.common.json :as json]
[clojure.string :as str]))
(def ^:private renames
"Penpot key to column name, where the column is not snake_case of the key.
Keyed by the Penpot key alone: no shape type gives one of these a second
meaning, so a per-table map would only add ceremony."
{;; `bool` collides with the Ladybug type name, so the column is named after
;; the table (`Boolean`) rather than after Penpot's `:bool` shape type.
:bool-type "boolean_type"
;; The column records what the file saved, which can lag what the shape
;; tree implies. The `saved_` prefix marks it as the stored value rather
;; than a derivation.
:component-root "saved_component_root"
;; The value is a list, so the plural is accurate.
:shadow "shadows"
;; The column spells the revision number out.
:revn "revision"})
(def dropped-keys
"Penpot keys projected by the Malli registry that get no column.
Dropping is right only when the column would be dead weight for every reader
of the graph. A key a reader might learn from belongs in `unprojected-keys`
instead."
{:deleted-at
"Only non-nil for a soft-deleted file, and a deleted file is never ingested."
:pixel-grid-color
"Viewer chrome: the color of the editor's pixel grid, not design content."
:pixel-grid-opacity
"Viewer chrome, as above."})
(def unprojected-keys
"Penpot keys that should become graph columns and do not have one yet.
Distinct from `dropped-keys` on purpose: these are a debt the projection
owes, not a decision to discard data. Keeping the two apart means a new
upstream attribute cannot be quietly buried in the drop list."
{:background-blur
"Landed upstream behind a default-on flag. No column for it yet."})
(def ^:private per-table-dropped
"Keys dropped only on certain tables.
`:grids` is the standing case: Penpot's shape schema admits it on every
shape, but only a Frame ever carries one. Emitting an always-null column on
ten other tables would widen every multi-table scan for nothing."
{:grids #{"Boolean" "Circle" "Group" "Image" "Path" "Rectangle" "SVGRaw" "Text"}})
(def type-overrides
"Ladybug column type per column name, where the derived type is too coarse.
`app.graph.schema.types` derives a type from the Malli schema, which is the
right default but coarser than the column deserves in places: a Malli `:map`
becomes `JSON`, where a native Ladybug MAP or a fixed-size array lets a
consumer read a tensor row without parsing.
Only load-bearing divergences are pinned here, in the order they became
load-bearing."
{;; Must be a native MAP: a JSON blob cannot be indexed by key in Cypher, so
;; `map_keys` and `map_extract` cannot reach a single token at all.
"applied_tokens" "MAP(STRING, STRING)"
;; `grc/schema:rect` is an inline `:and` over a map, not the registered
;; `::grc/rect`, so `app.graph.schema.types` cannot recognize it by type.
;; Four doubles rather than the eight-field struct: `x1`/`y1`/`x2`/`y2` are
;; derivable from `x`/`y`/`width`/`height`, and a fixed-size array is a
;; tensor row a consumer reads without parsing.
"selrect" "DOUBLE[4]"
;; The SVG provenance attributes are typed `:map` in the shape schema on
;; purpose. Legacy files hold them as plain maps rather than as
;; `::grc/rect` and `::gmt/matrix` records, and a tighter *schema* would
;; reject those files
;; (`app.common.types.shape/schema:shape-generic-attrs`). A tighter
;; *column* is free: `app.graph.schema.values/coerce` reads either form.
"svg_viewbox" "DOUBLE[4]"
"svg_transform" "DOUBLE[6]"
;; `:fills` is an `:or` over the packed `app.common.types.fills` value and
;; a plain vector of fill maps, so the schema alone cannot say it is a
;; collection. It always is one, and a fill has enough optional shape
;; (solid, gradient, image) that JSON per element is the honest element
;; type.
"fills" "JSON[]"})
(def ^:private map-key-fns
"How to render the *keys* of a MAP column, per column.
A column name is schema, so it is snake_case. The keys inside a MAP are
values, so they keep the spelling their producer used. `applied_tokens` is
keyed by shape attribute in the camelCase form
`app.common.json/write-camel-key` produces: `strokeWidth`, not
`stroke-width`."
{"applied_tokens" json/write-camel-key})
(defn map-key-fn
"Key renderer for a MAP column. `name` unless the column says otherwise."
[column]
(get map-key-fns column name))
(defn column-name
"The graph column name for Penpot key `k`.
Default: snake_case of the key. `renames` overrides."
[k]
(or (get renames k)
(str/replace (name k) "-" "_")))
(defn drop-key?
"Should key `k` be omitted from `table`'s columns?"
[table k]
(or (contains? dropped-keys k)
(contains? (get per-table-dropped k #{}) table)))
(defn ladybug-type
"The pinned Ladybug type for `column`, or `fallback` when nothing is pinned."
[column fallback]
(get type-overrides column fallback))
-343
View File
@@ -1,343 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.graph.schema.nodes
"Single source of truth for graph node tables.
Each registry entry declares Penpot Malli sources plus projection
options (`:drop`, optional `:extra`). Derived artifacts — Ladybug
DDL, Arrow fields, validation, type dispatch — all flow from that.
This registry is the single source of the graph schema. A Ladybug column
gets its name and its type once, at table creation, and there is no
widening afterwards. Every divergence between a Penpot key and its column
is recorded in `app.graph.schema.contract`."
(:require
[app.common.exceptions :as ex]
[app.common.schema :as sm]
[app.common.time :as ct]
[app.common.types.component :as ctk]
[app.common.types.file :as ctf]
[app.common.types.page :as ctp]
[app.graph.ladybug :as ladybug]
[app.graph.schema.contract :as contract]
[app.graph.schema.projection :as projection]
[app.graph.schema.types :as types]
[clojure.string :as str]))
(def schema-version
"penpot-graph-slice-4")
(def ^:private document-projection
{:source ctf/schema:file
:drop [:data]
;; Attributes a file map carries that `ctf/schema:file` does not declare.
;;
;; They belong here rather than in that schema, even though the graph wants
;; them, because `schema:file` is on the *write* path too:
;; `app.binfile.common/update-file!` derives its UPDATE columns from a file
;; map's keys, so declaring `:backend` there made it try to write a `backend`
;; column, which the `file` table does not have — it is synthesized on read.
;; A projection `:extra` is local to the graph and cannot reach a write.
;;
;; `:options` is lifted out of `:data` before the blob is dropped
;; (`app.graph.projection.document/document-attrs`); the rest come off the file
;; map as `get-file` returns it.
:extra [:map
[:options {:optional true} [:maybe :map]]
[:backend {:optional true} [:maybe :string]]
[:comment-thread-seqn {:optional true} [:maybe :int]]
[:ignore-sync-until {:optional true} [:maybe ::ct/inst]]]})
(def ^:private page-projection
{:source ctp/schema:page
:drop [:objects]})
(def ^:private component-projection
{:source ctk/schema:component
:drop [:objects]
;; Soft-delete flag used at runtime; not in schema:component.
:extra [:map
[:deleted {:optional true} :boolean]
[:annotation {:optional true} :string]]})
(def ^:private shape-projection
{:drop [:type]})
(def ^:private shape-node-types
[{:table "Frame" :penpot-type :frame :container? true}
{:table "Group" :penpot-type :group :container? true}
{:table "Boolean" :penpot-type :bool :container? true}
{:table "SVGRaw" :penpot-type :svg-raw :container? true}
{:table "Rectangle" :penpot-type :rect}
{:table "Circle" :penpot-type :circle}
{:table "Path" :penpot-type :path}
{:table "Text" :penpot-type :text}
{:table "Image" :penpot-type :image}])
(defn- resolve-schema
[{:keys [schema source drop extra penpot-type]}]
(or schema
(when penpot-type
(projection/project-shape-schema penpot-type
{:drop drop
:extra extra}))
(projection/project-schema source
{:drop drop
:extra extra})))
(defn- shape-node-entry
[{:keys [table penpot-type container?] :as entry}]
(let [projection (-> shape-projection
(merge (:projection entry))
(assoc :penpot-type penpot-type))]
{:table table
:pk :id
:penpot-type penpot-type
:container? container?
:projection projection
:schema (resolve-schema projection)}))
(def node-types
"Ordered node registry."
(into [{:table "Document"
:pk :id
:projection document-projection
:schema (resolve-schema document-projection)}
{:table "Page"
:pk :id
:projection page-projection
:schema (resolve-schema page-projection)}
{:table "Component"
:pk :id
:projection component-projection
:schema (resolve-schema component-projection)}]
(map shape-node-entry shape-node-types)))
(def ^:private by-table
(into {} (map (juxt :table identity) node-types)))
(def ^:private by-penpot-type
(into {} (keep (fn [{:keys [penpot-type table]}]
(when penpot-type [penpot-type table]))
node-types)))
(def container-tables
(into #{} (comp (filter :container?) (map :table)) node-types))
(def shape-tables
(into [] (comp (filter :penpot-type) (map :table)) node-types))
(defn table-for-type
"Map a Penpot shape `:type` keyword to a Ladybug node table name."
[penpot-type]
(get by-penpot-type (keyword penpot-type)))
(defn node-entry
[table]
(get by-table table))
(defn projection-for
"Return the projection options map for `table`."
[table]
(:projection (node-entry table)))
(defn- entry-child-schema
"Return the value schema from a Malli map entry (`[k s]` or `[k props s]`)."
[entry]
(if (> (count entry) 2)
(nth entry 2)
(nth entry 1)))
(defn column-name
"Graph column name for projected key `k` on `table`."
[_table k]
(contract/column-name k))
(defn column-ladybug-type
"Ladybug column type for projected key `k` on `table`."
[table k]
(some (fn [entry]
(when (= k (first entry))
(contract/ladybug-type (column-name table k)
(types/ladybug-type (entry-child-schema entry)))))
(projection/schema-map-entries (:schema (node-entry table)))))
(defn column-keys
"Projected column keys for `table`, in registry order.
Keys the contract drops on this table are omitted, so the column order, the
Arrow batch, and the DDL cannot disagree about what exists."
[table]
(into []
(comp (map first)
(remove #(contract/drop-key? table %)))
(projection/schema-map-entries (:schema (node-entry table)))))
(defn columns
"Projected column names for `table`, in registry order."
[table]
(mapv #(column-name table %) (column-keys table)))
(def ^:private validate-node-fn
(memoize
(fn [table]
(let [{:keys [schema]} (node-entry table)]
(sm/check-fn schema
:type :validation
:code (keyword "graph-node-projection" (str/lower-case table))
:hint (str "invalid graph node projection for " table))))))
(defn- projection-error-hint
[table explain]
(str "invalid graph node projection for " table
(when explain
(str "\n" (sm/humanize-explain explain)))))
(defn validate-node
"Validate and return projected node attrs for `table`."
[table value]
(let [{:keys [schema]} (node-entry table)]
(try
((validate-node-fn table) value)
(catch clojure.lang.ExceptionInfo e
(let [data (ex-data e)
explain (or (::sm/explain data)
(sm/explain schema value))]
(ex/raise :type :validation
:code (keyword "graph-node-projection" (str/lower-case table))
:hint (projection-error-hint table explain)
:table table
::sm/explain explain
:cause e))))))
(defn- get-projected-attr
"The attribute under `k`, keyword or string key.
`if-some`, not `or`: `false` and `0` are values, and falling through on them
is how `opacity 0` became `nil` and then the column default."
[attrs k]
(if-some [v (get attrs k)]
v
(when (keyword? k) (get attrs (name k)))))
(defn- raise-empty-projection!
[table attrs]
(ex/raise :type :validation
:code (keyword "graph-node-projection" (str/lower-case table))
:hint (str "empty graph node projection for " table
"; columns=" (count (column-keys table))
" shape-keys=" (vec (keys attrs)))))
(defn project-attrs
"Select and validate the projected columns for `table` from `attrs`."
[table attrs]
;; `some?`, not truthiness: `false` and `0` are values. Dropping them sent
;; `opacity 0` to the column default of 1.0 — a fully transparent shape
;; projected as opaque.
(let [projected (into {}
(keep (fn [k]
(let [v (get-projected-attr attrs k)]
(when (some? v) [k v])))
(column-keys table)))]
(when (empty? projected)
(raise-empty-projection! table attrs))
(validate-node table projected)))
(defn match-label
"Cypher node label for MATCH; backtick-wrapped when required by Ladybug."
[table]
(if (#{"Group" "Boolean"} table)
(str "`" table "`")
table))
(defn cypher-property-key
"Backtick-wrapped column name for inline Cypher literals."
[table k]
(str "`" (column-name table k) "`"))
(defn column-map-key-fn
"How a MAP column of `table` renders its keys.
A MAP's keys are values, not schema, so they keep the spelling their consumer
parsed — `applied_tokens` is keyed in camelCase. Both writers need this, so it
lives next to the column's type rather than in either of them."
[table k]
(contract/map-key-fn (column-name table k)))
(defn format-column-value
"Cypher literal for `v` in column `k` of `table`.
The single place that knows both the column's Ladybug type and the contract
detail that a MAP column may render its keys differently from `name` — used
by the bulk loader's post-COPY fixups and by the incremental sync alike, so
the two cannot disagree about a value's shape."
[table k v]
(ladybug/format-typed-value (column-ladybug-type table k)
v
(column-map-key-fn table k)))
(defn- create-node-table-ddl
[{:keys [table pk]}]
(let [cols (for [k (column-keys table)]
(str "`" (column-name table k) "` " (column-ladybug-type table k)))]
(str "CREATE NODE TABLE `" table "` ("
(str/join ", " (concat cols
[(str "PRIMARY KEY (`" (column-name table pk) "`)")]))
");")))
(defn is-child-of-ddl
[]
(str "CREATE REL TABLE `IsChildOf` ("
"FROM `Page` TO `Document`, "
"FROM `Component` TO `Document`, "
(str/join ", "
(concat
(map (fn [shape]
(str "FROM `" shape "` TO `Page`"))
shape-tables)
(for [shape shape-tables
container container-tables]
(str "FROM `" shape "` TO `" container "`"))))
", `position` INT64);"))
(defn is-instance-of-ddl
"Frame instance heads → Component."
[]
"CREATE REL TABLE `IsInstanceOf` (FROM `Frame` TO `Component`);")
(defn- shape-to-shape-rel-ddl
"A rel table over the full shape × shape product.
Created up-front rather than on demand: the bulk loader must never race on
lazy table creation, and a consumer can then tell \"this producer cannot
emit that pair\" from \"this document happens to have none\"."
[rel props]
(str "CREATE REL TABLE `" rel "` ("
(str/join ", " (for [from shape-tables
to shape-tables]
(str "FROM `" from "` TO `" to "`")))
(when (seq props) (str ", " (str/join ", " props)))
");"))
(defn refers-to-ddl
"Instance shape → its homologue in the component main instance, resolved
from `shape-ref`."
[]
(shape-to-shape-rel-ddl "RefersTo" nil))
(defn fills-swap-slot-ddl
"Swapped-in shape → the slot shape it replaces."
[]
(shape-to-shape-rel-ddl "FillsSwapSlot" ["`slot_id` UUID"]))
(defn ddl-statements
[]
(-> (mapv create-node-table-ddl node-types)
(conj (is-child-of-ddl))
(conj (is-instance-of-ddl))
(conj (refers-to-ddl))
(conj (fills-swap-slot-ddl))))
@@ -1,85 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.graph.schema.projection
"Derive Ladybug node column schemas from Penpot Malli sources.
Start from the canonical schema and remove the keys that must not become
graph columns."
(:require
[app.common.exceptions :as ex]
[app.common.schema :as sm]
[app.common.types.shape :as cts]
[malli.core :as m]))
(def ^:private malli-opts sm/default-options)
(defn- coerce-schema
"Normalize Malli sources to a compiled schema, unwrapping `:val` nodes."
[schema]
(loop [s (cond
(sm/schema? schema) schema
:else (sm/schema schema))]
(if (= :malli.core/val (sm/type s))
(recur (first (sm/children s)))
s)))
(defn- unsupported-projection-schema!
[schema]
(ex/raise :type :internal
:code :unsupported-projection-schema
:hint (str "unsupported projection schema type: "
(sm/type (coerce-schema schema)))))
(defn schema-map-entries
"Map entries for `schema`, flattening `:merge` composites."
[schema]
(let [s (coerce-schema schema)]
(or (seq (sm/entries s))
(unsupported-projection-schema! schema))))
(defn- select-projected-keys
"Project `schema` to a flat map schema, optionally dropping keys."
[schema drop-keys]
(let [s (coerce-schema schema)
keys (if (seq drop-keys)
(remove (set drop-keys) (sm/keys s))
(sm/keys s))]
(sm/select-keys s (vec keys))))
(defn shape-type-schema
"Return the compiled Penpot Malli branch for shape type `penpot-type`.
`m/entries` on the shape `:multi` yields MapEntries whose values are
compiled branch schemas (wrapped in `:val`). `m/children` returns raw
entry forms and must not be used here."
[penpot-type]
(let [kw (keyword penpot-type)
multi (sm/schema cts/schema:shape-attrs)]
(or (some (fn [entry]
(when (= kw (key entry))
(val entry)))
(m/entries multi malli-opts))
(ex/raise :type :validation
:code :unknown-shape-type
:hint (str "unknown penpot shape type: " kw)))))
(defn project-schema
"Build a graph node schema from canonical Malli `source`.
Options:
- `:drop` - keys removed from the source
- `:extra` - optional extra `[:map ...]` merged on top"
[source {:keys [drop extra]}]
(let [projected (select-projected-keys source drop)]
(if extra
(sm/merge projected (coerce-schema extra))
projected)))
(defn project-shape-schema
"Project `:drop` from the Penpot schema for `penpot-type`."
[penpot-type opts]
(project-schema (shape-type-schema penpot-type) opts))
-174
View File
@@ -1,174 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.graph.schema.types
"Map Malli schemas to Ladybug column types.
Ladybug is schema-first and strongly typed: every property key gets its type
at table-creation time, and there is no widening later. That makes this
mapping the whole of the graph's typing, and it is worth being tight — a
column typed `DOUBLE[4]` is four numbers a consumer reads as a tensor row,
where the same value as `JSON` is text somebody has to parse and trust. So
JSON is the fallback of last resort, taken only where the Malli schema
genuinely admits shapes no single column can hold.
Three groups, in the order the mapping tries them:
1. **Scalars** (`base-type->ladybug`) — the leaf Malli types.
2. **Registered composites** (`custom-type->ladybug`) — Penpot's own value
types whose *layout* is fixed even though Malli only sees a map or a
string: a matrix is six doubles, a point two, a rect four, a hex colour
one packed integer. These are named explicitly because the tight encoding
is a modelling decision, not something derivable from the schema.
3. **Structure** — collections become `T[]`, `:map-of` becomes `MAP(k, v)`,
and a closed map of scalars becomes a `STRUCT`. Anything that could be
more than one shape (a `:multi`, an `:or`, an optional-keyed map) becomes
`JSON`, because a Ladybug column cannot be two types.
Every encoding here has a matching value formatter in `app.graph.ladybug`.
The two must move together: a column type with no case there falls back to
guessing the literal from the runtime value."
(:require
[app.common.logging :as l]
[app.common.schema :as sm]
[app.common.time :as ct]
[clojure.string :as str]
[malli.core :as m]))
(def ^:private malli-opts sm/default-options)
(def ^:private base-type->ladybug
{::sm/uuid "UUID"
::sm/safe-number "DOUBLE"
::sm/safe-double "DOUBLE"
::sm/safe-int "INT64"
::sm/number "DOUBLE"
::sm/boolean "BOOLEAN"
::sm/int "INT64"
::ct/inst "TIMESTAMP"
:uuid "UUID"
:string "STRING"
:int "INT64"
:double "DOUBLE"
:float "DOUBLE"
:boolean "BOOLEAN"
:keyword "STRING"
:inst "TIMESTAMP"})
(def ^:private custom-type->ladybug
"Penpot value types with a fixed layout Malli does not express.
Fixed-size arrays are the point of each: they are dense, they need no
parsing, and a consumer can read a whole column as a tensor.
- `::gmt/matrix` — the affine transform, `[a b c d e f]`.
- `::gpt/point` — `[x y]`.
- `::grc/rect` — `[x y width height]`. `x1`/`y1`/`x2`/`y2` are dropped: they
are derivable from those four, and carrying them would double the column.
- `::clr/hex-color` — `#RRGGBB` packed as `0xRRGGBBAA`, so colours compare
and group without string handling."
{:app.common.geom.matrix/matrix "DOUBLE[6]"
:app.common.geom.point/point "DOUBLE[2]"
:app.common.geom.rect/rect "DOUBLE[4]"
:app.common.types.color/hex-color "UINT32"})
(def ^:private collection-types
#{:vector :sequential :set ::sm/vec ::sm/set ::sm/coll})
(def ^:private string-collection-types
"Registered collection schemas whose element type is not in `children`."
{::sm/set-of-strings "STRING[]"
::sm/set-of-keywords "STRING[]"
::sm/set-of-uuid "UUID[]"
::sm/vec-of-uuid "UUID[]"})
(defn- normalize-schema
"Resolve refs, but stop at a schema this namespace maps explicitly.
Order matters: `::grc/rect` derefs to an `:and` over a map, and following
that would lose the fixed-size-array encoding."
[schema]
(let [s (sm/schema schema)]
(if (and (m/-ref-schema? s)
(not (contains? custom-type->ladybug (m/type s)))
(not (contains? string-collection-types (m/type s))))
(recur (m/deref s malli-opts))
s)))
(declare ladybug-type)
(defn- entry-child
"The value schema of a Malli map entry (`[k s]` or `[k props s]`)."
[entry]
(if (> (count entry) 2) (nth entry 2) (nth entry 1)))
(defn- entry-optional?
[entry]
(and (> (count entry) 2)
(:optional (nth entry 1))))
(defn- struct-type
"`STRUCT(...)` for a closed map of scalars, or nil when JSON is the honest answer.
A struct is a fixed layout: every field present, every field a single type.
An optional key would make the column's shape depend on the row, and a nested
collection or map makes it recursive — Ladybug allows nesting, but a consumer
reading such a column gains nothing over JSON, so the line is drawn at
scalars."
[s]
(let [entries (m/entries s malli-opts)]
(when (and (seq entries)
(not-any? entry-optional? entries))
(let [fields (for [entry entries
:let [t (ladybug-type (entry-child entry))]]
(when (and t
(not= "JSON" t)
(not (str/includes? t "(")))
;; snake_case like a column name, and always
;; backtick-quoted: a grid cell has a field called
;; `column`, which is a Ladybug keyword, and an unquoted
;; one fails to parse in the DDL *and* in every literal.
;; The catalog reports them unquoted.
(str "`" (str/replace (name (key entry)) "-" "_") "` " t)))]
(when (every? some? fields)
(str "STRUCT(" (str/join ", " fields) ")"))))))
(defn ladybug-type
"Return the Ladybug column type for a Malli child schema."
[schema]
(let [s (normalize-schema schema)
t (m/type s)]
(or (base-type->ladybug t)
(custom-type->ladybug t)
(string-collection-types t)
(when (contains? collection-types t)
(when-let [child (first (m/children s malli-opts))]
(str (ladybug-type child) "[]")))
(case t
(:maybe :and) (ladybug-type (first (m/children s malli-opts)))
;; `::sm/one-of` is how Penpot spells a closed set of keywords —
;; `:blend-mode`, `:grow-type`, every `:layout-*`. One keyword, one
;; string.
(:enum ::sm/one-of) "STRING"
:map-of
(let [[key-schema value-schema] (m/children s malli-opts)]
(str "MAP(" (ladybug-type key-schema) ", "
(ladybug-type value-schema) ")"))
:map (or (struct-type s) "JSON")
;; A schema we do not recognize. If it has no children it is a leaf —
;; one of Penpot's registered keyword or enum schemas, say — and a
;; string holds it exactly. If it has children it is a composite whose
;; shape we cannot pin down, and JSON is the honest answer.
(if (empty? (m/children s malli-opts))
"STRING"
(do
(l/wrn :hint "unmapped composite malli type, defaulting to JSON"
:malli-type t)
"JSON"))))))
-202
View File
@@ -1,202 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.graph.schema.values
"Shape a Penpot value into the plain data its Ladybug column type wants.
Ladybug is strongly typed, and `app.graph.schema.types` maps Penpot's Malli
schemas onto types as tight as it can — a matrix is `DOUBLE[6]`, a rect
`DOUBLE[4]`, a colour `UINT32`, a closed map a `STRUCT`. A tight column is
only worth having if the writer actually fills it in that shape, which is
what this namespace does: it turns records and maps into the numbers, vectors
and plain maps the type names.
It deliberately stops there. Serialization belongs to the writer — Cypher
literals in `app.graph.ladybug`, Arrow vectors in `app.graph.arrow` — so that
shaping a value and writing it are separate concerns and each has one home.
The type language is the Ladybug one, read recursively: `T[]`, `T[n]`,
`MAP(k, v)`, `STRUCT(name t, …)`. Anything else is passed through."
(:require
[app.common.geom.matrix :as gmt]
[app.common.geom.point :as gpt]
[app.common.types.color :as clr]
[clojure.string :as str]))
(defn- split-args
"Split a comma-separated type argument list, respecting nesting.
`\"UUID, STRUCT(a INT64, b INT64)\"` → `[\"UUID\" \"STRUCT(a INT64, b INT64)\"]`."
[s]
(loop [chars (seq s) depth 0 current (StringBuilder.) out []]
(if-let [c (first chars)]
(cond
(and (= c \,) (zero? depth))
(recur (rest chars) depth (StringBuilder.) (conj out (str/trim (str current))))
(or (= c \() (= c \[))
(recur (rest chars) (inc depth) (.append current c) out)
(or (= c \)) (= c \]))
(recur (rest chars) (dec depth) (.append current c) out)
:else
(recur (rest chars) depth (.append current c) out))
(let [last-arg (str/trim (str current))]
(cond-> out (seq last-arg) (conj last-arg))))))
(defn- parse-list
"`[element-type]` when `t` is a list or fixed-size array type, else nil.
`DOUBLE[]` and `DOUBLE[4]` are both lists of doubles as far as shaping goes;
the size only matters to the DDL."
[t]
(when-let [[_ element] (re-matches #"(.+?)\[\d*\]$" t)]
[element]))
(defn- parse-map
"`[key-type value-type]` when `t` is a MAP type, else nil."
[t]
(when-let [[_ args] (re-matches #"MAP\((.*)\)$" t)]
(let [[k v] (split-args args)]
(when (and k v) [k v]))))
(defn- parse-struct
"`[[field-name field-type] …]` when `t` is a STRUCT type, else nil.
Field names arrive backtick-quoted (see `app.graph.schema.types`). The
quoting is syntax, so it is stripped by default and re-applied by the writer —
except for the Arrow writer, which needs it kept (`keep-quotes?`)."
[t keep-quotes?]
(when-let [[_ args] (re-matches #"STRUCT\((.*)\)$" t)]
(for [arg (split-args args)
:let [idx (str/index-of arg " ")]
:when idx]
[(cond-> (subs arg 0 idx) (not keep-quotes?) (str/replace "`" ""))
(str/trim (subs arg (inc idx)))])))
(def ^:private struct-field-keys
"Field name → the Penpot keys that may hold it.
A STRUCT field name is the snake_case of the Penpot key, but a value arrives
with its original key, and some arrive from JSON with the string form. Both
are tried before giving up."
(memoize
(fn [field]
[(keyword (str/replace field "_" "-"))
(keyword field)
field
(str/replace field "_" "-")])))
(defn- struct-field
[value field]
(some (fn [k] (when (contains? value k) (get value k)))
(struct-field-keys field)))
(defn- fixed-vector
"`v` as a plain vector of numbers, for a `DOUBLE[n]` column.
Records come first because they are what a realized snapshot holds; the map
forms are what a JSON round-trip leaves behind."
[v]
(cond
(gmt/matrix? v) [(:a v) (:b v) (:c v) (:d v) (:e v) (:f v)]
(gpt/point? v) [(:x v) (:y v)]
;; A rect: four of the eight fields, the rest being derivable.
(and (map? v) (contains? v :width) (contains? v :height))
[(:x v) (:y v) (:width v) (:height v)]
(and (map? v) (contains? v :x) (contains? v :y))
[(:x v) (:y v)]
(and (map? v) (contains? v :a) (contains? v :f))
[(:a v) (:b v) (:c v) (:d v) (:e v) (:f v)]
(sequential? v) (vec v)
:else nil))
(defn- packed-color
"`#RRGGBB` as the packed integer `0xRRGGBBAA`.
Alpha defaults to opaque: the column holds a colour, and any opacity Penpot
keeps alongside it is a separate attribute."
[v]
(cond
(integer? v) v
(and (string? v) (clr/valid-hex-color? v))
(let [rgb (Long/parseLong (subs v 1) 16)]
(bit-or (bit-shift-left rgb 8) 0xFF))
:else nil))
(def struct-fields
"`[[field-name field-type] …]` for a STRUCT type, memoized.
Public because the writers need the same field list to emit a literal."
(memoize (fn [ladybug-type] (vec (parse-struct ladybug-type false)))))
(def struct-fields-quoted
"`struct-fields` with the DDL's backticks intact.
Only the Arrow writer wants this: Ladybug names a staged struct's fields from
the Arrow child names and quotes none of them, so a field whose name is a
reserved word — a layout grid cell's `column` — has to arrive already quoted
or `createArrowTable` fails outright."
(memoize (fn [ladybug-type] (vec (parse-struct ladybug-type true)))))
(def map-types
"`[key-type value-type]` for a MAP type, memoized."
(memoize (fn [ladybug-type] (parse-map ladybug-type))))
(def list-element
"Element type of a `T[]` / `T[n]` column, memoized; nil when not a list."
(memoize (fn [ladybug-type] (first (parse-list ladybug-type)))))
(declare coerce)
(defn- coerce-struct
[fields v]
(when (map? v)
(into {}
(keep (fn [[field field-type]]
(when-some [fv (struct-field v field)]
[field (coerce field-type fv)])))
fields)))
(defn coerce
"`v` as the plain data a column of `ladybug-type` holds.
Returns `nil` when the value cannot be shaped that way, which callers treat
as \"write NULL\" — a wrong shape in a strongly typed column fails the whole
load, so declining is better than guessing."
[ladybug-type v]
(cond
(nil? v) nil
(not (string? ladybug-type)) v
(= "UINT32" ladybug-type) (packed-color v)
;; Fixed-size numeric arrays are records: matrix, point, rect.
(re-matches #"DOUBLE\[\d+\]" ladybug-type) (fixed-vector v)
:else
(if-let [[element] (parse-list ladybug-type)]
(when (or (sequential? v) (set? v))
;; A set has no order, so its column would otherwise vary between
;; builds of the same file. Sorting makes it deterministic — which is
;; what lets two builds be diffed at all, and what a stable golden
;; needs. Sequential values keep their order: for `shapes` and
;; `points`, the order *is* the content.
(let [elements (mapv #(coerce element %) v)]
(if (set? v) (vec (sort-by str elements)) elements)))
(if-let [[key-type value-type] (parse-map ladybug-type)]
(when (map? v)
(into {}
(map (fn [[k mv]] [(coerce key-type k) (coerce value-type mv)]))
v))
(if-let [fields (seq (parse-struct ladybug-type false))]
(coerce-struct fields v)
v)))))
-48
View File
@@ -1,48 +0,0 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.graph.stats
(:require
[app.graph.ladybug :as ladybug]
[app.graph.schema.nodes :as nodes]))
(defn- count-on-connection
[conn statement]
(or (ladybug/query-scalar-on-connection! conn statement) 0))
(defn- rel-table-names
"Relationship tables present in the open database.
Read from the catalog so a newly ported transform's edges are counted
without this namespace being told about it."
[conn]
(->> (ladybug/query-on-connection!
conn "CALL show_tables() WHERE type = 'REL' RETURN name;" :max-rows 1000)
:rows
(map first)))
(defn summarize-connection
"Return node/edge counts using an open Ladybug connection."
[conn]
{:nodes (into {}
(map (fn [table]
[table (count-on-connection
conn
(str "MATCH (n:" (nodes/match-label table) ") "
"RETURN count(n) AS " table "_c;"))])
(map :table nodes/node-types)))
:edges (into {}
(map (fn [rel]
[(keyword rel)
(count-on-connection
conn
(str "MATCH ()-[e:`" rel "`]->() RETURN count(e) AS c;"))]))
(rel-table-names conn))})
(defn summarize
"Return node/edge counts from the graph database."
[db-path]
(ladybug/with-connection! db-path summarize-connection))
Loaded 100 of 646 files, more files were not shown because too many files have changed in this diff. Show more