Compare commits

...
Author SHA1 Message Date
Andrey Antukh 334322e6db ♻️ Simplify storage target helpers and refusal logging
Drop the bang from the pure storage-routes validators,
inline the refusal log into park-unresolvable in both GC
tasks, and read the storage target from object metadata
only. Tests now use the production object shape.

AI-assisted-by: muse-spark-1.3-contributor
2026-09-11 17:25:03 +00:00
Andrey Antukh e96b8a4798 📎 Test the serving path rejects unconfigured storage targets
The read/serve path (get-object-data) must fail with
:invalid-storage-target when an object references a target id that is
not configured, matching get-object-url and bulk delete. Add the
missing test so removing a target id cannot silently serve from the
wrong place.

Closes #11630

AI-assisted-by: deepseek-flash
2026-09-11 07:29:26 +00:00
Andrey Antukh 6eece49f69 🐛 Keep objects whose storage target is no longer configured
An object whose :storage-target id is not in the S3 routes configuration
can no longer be located. Reads and serving now fail with
:invalid-storage-target instead of silently using the default bucket,
and garbage collection refuses to delete the row: it logs an error and
parks the row (deleted_at +1 day, no deletion attempts, no give-up)
until the target is configured again. Legacy rows (no target) and the
default target keep working and deleting normally.

Also addresses review findings: per-target dedup isolation and
stale-repair coverage, legacy-row GC coverage through the real task
paths, a single declared-target schema, closing already-built S3
clients when target init fails, and loader edge pins. Docs and memory
updated.

Closes #11630

AI-assisted-by: deepseek-flash
2026-09-10 21:38:38 +00:00
Andrey Antukh 0d855b3436 Add per-bucket S3 target routing to asset storage
Introduce optional named S3 targets and route internal semantic buckets
(for example tempfile) to them, keeping the fs/s3 backend choice and the
storage_object.backend value unchanged.

Targets and routes are read from an EDN file referenced by
PENPOT_OBJECTS_STORAGE_S3_ROUTES_FILE. The chosen target id is stored in
the object metadata as :storage-target and resolved on reads, URL
signing, deduplication, individual deletes and both GC tasks. The
implicit :default target is built from PENPOT_OBJECTS_STORAGE_S3_*, so
behavior is unchanged when the file is absent. Bulk deletion now carries
the target, and one S3 client/presigner pair is shared per distinct
region/endpoint.

Closes #11630

AI-assisted-by: deepseek-flash
2026-09-10 19:27:47 +00:00
Andrey Antukh 37dab75e1a Merge remote-tracking branch 'origin/staging' into develop 2026-09-10 20:29:44 +02:00
Andrey Antukh f9c02926b9 Merge remote-tracking branch 'origin/main' into staging 2026-09-10 20:21:41 +02:00
bameda bae3900537 ♻️ Rebalance CI runners and drop pinned ubuntu-24.04
Move build-docker and build-docker-devenv jobs from penpot-extended-runner
to penpot-standar-runner, point tests-exporter at the canonical
penpot-extended-runner label instead of the stale penpot-runner-02 alias,
and switch build-tag/release notify jobs from ubuntu-24.04 to ubuntu-latest.

Signed-off-by: David Barragán Merino <david.barragan@kaleidos.net>
2026-09-10 19:23:13 +02:00
bameda 757a5bd479 ♻️ Rebalance CI runners and drop pinned ubuntu-24.04
Move build-docker and build-docker-devenv jobs from penpot-extended-runner
to penpot-standar-runner, point tests-exporter at the canonical
penpot-extended-runner label instead of the stale penpot-runner-02 alias,
and switch build-tag/release notify jobs from ubuntu-24.04 to ubuntu-latest.

Signed-off-by: David Barragán Merino <david.barragan@kaleidos.net>
2026-09-10 19:22:46 +02:00
bameda 9c07dd124a ♻️ Rebalance CI runners and drop pinned ubuntu-24.04
Move build-docker and build-docker-devenv jobs from penpot-extended-runner
to penpot-standar-runner, point tests-exporter at the canonical
penpot-extended-runner label instead of the stale penpot-runner-02 alias,
and switch build-tag/release notify jobs from ubuntu-24.04 to ubuntu-latest.

Signed-off-by: David Barragán Merino <david.barragan@kaleidos.net>
2026-09-10 19:14:55 +02:00
makesomethingshitandAndrey Antukh 99c036feac 🐛 Close nitrate modal when navigating to current plan (#11615)
Signed-off-by: makesomethingshit <junsoo1172@gmail.com>
Co-authored-by: Andrey Antukh <niwi@niwi.nz>
2026-09-10 18:17:11 +02:00
Alejandro Alonso aa78ca0be8 🐛 Keep text image fills aligned during drag (#11610)
Cached Skia paragraphs bake absolute image/gradient shaders at layout
time. On move, clones reuse those paragraphs while painting at the new
selrect, so glyphs move and the fill stays put. Record the paint origin
when layout is built and translate the canvas when painting from cache
so shaders track the text. Also sync bounds before update_layout so
fills bake against the current container.
2026-09-10 16:55:45 +02:00
Andrey Antukh 8952d70fd2 Optimize get-profiles-for-file-comments query (#11622)
Rewrite sql:file-comment-users to join comment with
comment_thread and union the requesting profile id, then
join the resulting small id set against profile.

The previous "id IN (subquery) OR id = ?" forced a
sequential scan over the whole profile table with a hashed
subplan filter, taking ~1.9s on large instances. The
semi-join lets the planner use profile_pkey, dropping the
query to sub-millisecond time. UNION (not UNION ALL) keeps
the previous dedup semantics when the requesting profile is
also a commenter.

AI-assisted-by: deepseek-flash
2026-09-10 16:45:22 +02:00
Andrey Antukh 7c27ed812a ♻️ Consolidate HIGHLIGHTS.md into CHANGES.md 🚀 section (#11531)
* ♻️ Consolidate HIGHLIGHTS.md into CHANGES.md 🚀 section

Eliminate the redundant HIGHLIGHTS.md file and make CHANGES.md
the single source of truth for version highlights.

- Add 🚀 section for 2.15.0 (MCP server integration)
- Add 4 missing highlight entries to 2.17.0 🚀 section
- Rewrite frontend parser to extract from CHANGES.md 🚀
  subsections instead of flat HIGHLIGHTS.md format
- Decouple parse-latest-released-version from highlights
  extraction so it works independently of 🚀 content
- Conditionally render highlights section in modal when non-empty
- Rewrite tests for new parser behavior (11 tests, 21 assertions)
- Delete HIGHLIGHTS.md and remove .gitignore exception
- Add step 8b to update-changelog skill for proactively
  proposing highlights during release workflows
- Add missing-highlights and missing-highlight-reference
  anomaly types to the changelog anomaly report script

Closes #11530

AI-assisted-by: qwen3.7-plus

* ♻️ Use consistent string library and add multi-version test

Address code review findings:

- Use str/split (cuerdas) consistently in extract-rocket-items
  instead of mixing cstr/split (clojure.string)
- Add parse-highlights-extracts-multiple-versions test to verify
  the parser correctly extracts 🚀 items from multiple
  versions in a single CHANGES.md body

AI-assisted-by: qwen3.7-plus

* ♻️ Scope 🚀 checks to X.Y.0 and split gaps from anomalies

Type C now only checks released X.Y.0 versions, since patches never carry 🚀 subsections by design. Type D requires both issue AND PR references with exact format, accepting multi-PR entries. C/D are reported as highlight gaps in their own section and no longer count toward the anomaly total. Key Principles and anomaly definitions updated to match. Addresses review comments on PR #11531.

AI-assisted-by: muse-spark-1.3-contributor

*  Render markdown links and bold in check-updates highlights

The highlights modal showed raw markdown from CHANGES.md 🚀 lines (brackets and URLs). Add a pure parse-highlight-item parser for inline links and bold, render fragments with literal hiccup in the modal (links open in a new tab), and style links and strong elements. Non-http URLs and malformed markup degrade to plain text. Adds 12 unit tests.

AI-assisted-by: muse-spark-1.3-contributor

* 🐛 Point full changelog link to main instead of staging

The view-changelog button in the check-updates modal linked to the staging branch. Point it to main, which holds the published changelog. Version detection still fetches from staging.

AI-assisted-by: muse-spark-1.3-contributor
2026-09-10 16:40:32 +02:00
Andrey Antukh 4ce459d720 🐛 Escape LDAP filter values and use directory email in retrieve-user (#11085)
Fix LDAP injection vulnerability (T5-N1-03) where the client-supplied email was used directly in the LDAP search filter without escaping RFC 4515 special characters (*, (, ), \, NUL), and the profile email was taken from client input instead of the LDAP directory attribute.

Changes:
- Add escape-ldap-filter-value per RFC 4515 section 3
- Apply escaping in search-user before building LDAP filter
- Add get-attr helper for multi-valued LDAP attributes
- Fix retrieve-user to use directory email (attrs-email) instead of client email
- Use cuerdas.core instead of clojure.string

Closes #11084

AI-assisted-by: mimo-v2.5-pro
2026-09-10 16:39:35 +02:00
Shlok Goyal 286ccb03fa 🐛 Preserve stroke dash and gap values on color change (#11557)
Signed-off-by: Shlok Goyal <shlokgoyal1279@gmail.com>
2026-09-10 14:36:51 +02:00
Alejandro Alonso dbe5941a23 Export layer blur to WASM SVG (#11580)
SkSVGDevice drops paint image-filters, so re-emit visible layer blur as a
native feGaussianBlur filter on the composite <g>. Match canvas sigma via
radius_to_sigma(value * scale), and skip Skia blur filters on the SVG
VectorRenderer path so shapes do not vanish.

Closes #11380
2026-09-10 14:25:47 +02:00
makesomethingshit 30849babcc 🐛 Fix fontFamilies token property mapping in Plugin API (#11566)
* 🐛 Fix fontFamilies token property mapping in Plugin API

The Plugin API exposes the font-family token property as `fontFamilies`,
while Penpot stores the canonical applied-token attribute as
`:font-family`. The bidirectional plugin/internal attribute map did not
contain that alias, so explicit `applyToken(..., ["fontFamilies"])`
validation rejected the property and applied-token readback exposed the
undocumented singular `fontFamily`.

Add `:font-family -> :font-families` to the existing canonical alias
map. The reverse mapping is derived automatically, keeping application
and readback symmetric without introducing a font-specific code path.

Closes #11405

AI-assisted-by: Omen Alpha
Signed-off-by: 최준수 <junsoo1172@gmail.com>

* 🐛 Fix fontFamilies e2e test to target a text shape

The fontFamilies end-to-end regression created a flex layout frame,
whose attribute set (frame-with-layout-attributes) excludes
:font-family. The workspace token application filters such shapes,
so the internal binding and readback assertions would pass vacuously
without exercising the alias.

Target an actual `:text` shape (ctho/add-text) instead, so the test
verifies the full JS "fontFamilies" -> schema -> alias -> canonical
:font-family -> camelCase readback path.

AI-assisted-by: Omen Alpha
Signed-off-by: 최준수 <junsoo1172@gmail.com>

* 🐛 Fix fontFamilies test WASM error and add changelog entry

The text-shape fontFamilies e2e applies a layout-affecting token via
wasm renderer path, hitting missing WASM exports under Node. Merge
thw/setup-wasm-mocks! into the :each fixture and add plugins
CHANGELOG entry for the fontFamilies alias fix.

AI-assisted-by: muse-spark-1.3-contributor
Related to #11566

Signed-off-by: makesomethingshit <junsoo1172@gmail.com>

---------

Signed-off-by: 최준수 <junsoo1172@gmail.com>
Signed-off-by: makesomethingshit <junsoo1172@gmail.com>
2026-09-10 13:43:41 +02:00
Alejandro Alonso dc12f1db91 Export image-filled strokes to WASM SVG as linked images (#11559)
Closes #11384

Skia's SVG backend drops save_layer+SrcIn, so image strokes are re-emitted
as a linked <image> clipped to an opaque stroke silhouette (filled outline,
clip-rule evenodd). Open-path caps join the silhouette and grow the image
dest by cap_bounds_margin so markers stay textured.
2026-09-10 13:29:14 +02:00
Luis de Dios c589563912 ♻️ Replace digit with number in password validations (#11609) 2026-09-10 12:15:58 +02:00
andrés gonzález 0eb3179016 💄 Adjust release notes 2.18 titles (#11608) 2026-09-10 11:50:57 +02:00
Andrey Antukh d1ebf4cda2 📎 Update changelog 2026-09-10 10:41:10 +02:00
Andrey Antukh fdb9e97572 📎 Update planner skill and AGENTS.md 2026-09-10 10:12:09 +02:00
Elenzakaleidos 9cd3b63eea 📚 Update README.md (#11602)
Added a new section for Penpot Enterprise detailing its features and benefits for organizations.

Signed-off-by: Elenzakaleidos <elena.scilinguo@kaleidos.net>
2026-09-10 09:52:06 +02:00
David Barragán Merino 94555c027e 🔧 Sync .github/workflows with develop
Signed-off-by: David Barragán Merino <david.barragan@kaleidos.net>
2026-09-10 09:47:10 +02:00
Elena Torró 9b24907992 🐛 Fix render text-decoration on spans and update regression tests (#11583) 2026-09-10 09:42:09 +02:00
Juan de la CruzandLuis de Dios b283d952a8 Add new slides content for 2.18 release (#11222)
*  Add new slides content for 2.18 release

* ♻️ Use buttons from DS

* ♻️ Use new SCSS guidelines

* ♻️ Use a base stylesheet for all version files

* ♻️ Use new SCSS guidelines

*  Add new images and wording

---------

Co-authored-by: Luis de Dios <luis.dedios@kaleidos.net>
2026-09-10 09:35:34 +02:00
andrés gonzálezandCursor 6f63a5fcbf Persist hide resolved comments preference (#10694)
Store the hide-resolved filter in user storage and restore it when
entering the workspace or viewer, consistent with canvas comment
visibility from #10239. Match the comments filter separator styling to
the main menu and add the missing mentions option in the viewer
dropdown.

Closes #10686

Signed-off-by: Andres Gonzalez <andres.gonzalez79@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-09-10 08:14:37 +02:00
Elena Torró 61caefbcf3 🔧 Support independent image bounds on wasm export (#11590) 2026-09-09 18:27:22 +02:00
Elena Torró 6586631293 🐛 Fix wasm boolean paths with opposite winding operand (#11551) 2026-09-09 18:19:29 +02:00
Andrey Antukh 66fb4a69ba 📎 Update copyright headers 2026-09-09 17:58:09 +02:00
Alejandro Alonso ac84557740 🐛 Re-upload WASM text after WebGL context restore (#11589)
During reload-renderer!, reloading? keeps initialized?/ready? false
while set-objects runs (especially the sync path for small files).
Text content used that guard and was skipped; geometry already used
live?. Gate use-shape, has-shape, and set-shape-text-content on
wasm/live? so text is restored with the rest of the shapes.
2026-09-09 17:25:18 +02:00
Elena Torró 3d4a5ca2aa 🔧 Use mutex so only one test owns global at a time (#11585) 2026-09-09 17:10:04 +02:00
Danny ShirelyandAndrey Antukh d45c6710b7 🎉 Implement independent image bounds resizing (#11430)
* 🎉 Implement independent image bounds resizing

Add canvas resize interaction mode that allows users to resize an
image object's bounding box independently from the underlying bitmap
content without scaling or distortion while holding the Mod key.

AI-assisted-by: gemini-2.5-pro

* ♻️ Address reviewer feedback from elenatorro

- Remove legacy cfh/image-shape? check in shape-has-image-fill?
- Guard bounds-resize with positive dimensions instead of clamping scalev to preserve flipping
- Remove :metadata from transform-attrs in modifiers.cljs
- Restore preserveAspectRatio logic based on keep-ar? in fills.cljs
- Compute source rect against destination rect for raster and SVG fills in WASM renderer

* 🔧 Fix clippy needless borrow warnings in wasm image fills

---------

Co-authored-by: Andrey Antukh <niwi@niwi.nz>
2026-09-09 15:16:56 +02:00
Andrey Antukh 43f7e49aa0 📎 Update changelog 2026-09-09 11:48:37 +02:00
Andrey Antukh eca1d81692 🔧 Remove legacy pnpm build key and clarify updating doc
Drop the ignored-since-pnpm-11 onlyBuiltDependencies entry from
render-wasm/pnpm-workspace.yaml, keeping allowBuilds as the single
source of build approvals. Clarify the updating-pnpm gotcha so it no
longer claims pnpm writes ignoredBuiltDependencies.

AI-assisted-by: muse-spark-1.3-contributor
2026-09-09 11:45:00 +02:00
Pablo Alba d263c23a58 🐛 Add ssrf check for nitrate sso and add timeouts to http client (#11576) 2026-09-09 11:23:53 +02:00
218 changed files with 7453 additions and 2904 deletions

No files matched your search

+57 -232
View File
@@ -1,13 +1,11 @@
---
name: planner
description: Read-only planning and architecture analysis for Penpot — produce a structured implementation plan with task breakdown, acceptance criteria, sizing, and checkpoints. Always output to the user with the plan's save path (saved or suggested) and the next steps.
description: Read-only planning and architecture analysis — produce a structured implementation plan with task breakdown, acceptance criteria, sizing, and checkpoints. Always output to the user with the plan, suggested save path and the next steps.
---
# Planner
Read-only senior software architect role for Penpot. Produces structured
implementation plans with task breakdowns that engineers or other agents can
execute. Never writes or modifies code.
Produce a plan that another engineer or agent can execute without guessing.
## When to Use
@@ -21,24 +19,7 @@ execute. Never writes or modifies code.
- A task feels too large or vague to start.
- Work needs to be parallelized across multiple agents or sessions.
Do **not** use this skill to actually implement anything — it is read-only.
**When NOT to use:** Single-file changes with obvious scope, or when the spec
already contains well-defined tasks.
## Role
You help users understand the Penpot codebase, design solutions, and produce
implementation plans that other agents or developers can execute. The plan
tells them what to build and how to verify it, task by task.
The implementer reads the project's agent docs (`AGENTS.md`, project memories
such as `mem:critical-info`, `mem:testing`, and each module's core memory)
before working. Reference those memories instead of re-explaining tooling,
conventions, or test design — explain in the plan only what they do not cover.
Do **not** suggest commit messages or commit names anywhere in your plans or
responses — committing is the implementer's responsibility.
Do not use for a small change with obvious scope or an existing executable plan.
## CRITICAL: Required Reading Before Planning
@@ -55,67 +36,36 @@ Before drafting any plan, work through the project's own guidance:
Skipping this step is the #1 cause of incorrect or incomplete plans.
---
## Constraints
## The Planning Process
- You are **analysis-only** — never create, edit, or delete source code. The
only file you may write is the plan itself, and only when the command or
user explicitly instructs you to save it.
- You do **not** run builds, tests, linters, or any commands that modify state.
- You do **not** create git commits or interact with version control.
- You do **not** execute shell commands beyond read-only searches (`rg`, `ls`,
`find`, `cat`, `bat`).
- Your output is a structured plan or analysis, ready for handoff to an
engineer agent or developer.
### Phase 1: Architecture Analysis
## Planning Process
1. Read the spec, requirements, or feature request.
2. Analyze the codebase architecture and identify affected modules.
3. Read project conventions (starting with `critical-info` and module core
memories) before drafting.
4. Map dependencies between components (see the dependency graph in
`critical-info`).
5. Identify risks, edge cases, performance implications, and breaking changes.
1. Define the problem, desired outcome, constraints, and exclusions.
2. Trace the current behavior through the affected modules.
3. Map dependencies and choose an implementation order that builds foundations
before their consumers.
4. Identify open product or architecture decisions. Resolve implementation
details from existing conventions when they do not affect public behavior.
5. Identify edge cases, security and data risks, performance bounds, breaking
changes, and external dependencies.
6. Split the work into small, ordered tasks. Prefer complete testable slices
over unrelated layer-wide batches. Apply DRY and KISS to the proposed
implementation.
7. Define exact acceptance criteria and verification for every task.
8. Add a checkpoint after every two or three tasks in a longer plan.
9. State which tasks can run in parallel and which must remain sequential.
### Phase 2: Task Breakdown
#### Identify the Dependency Graph
Map what depends on what, following the monorepo's module dependency graph:
```
common (shared types, schemas — no deps)
├── backend (depends common)
│ ├── RPC handlers
│ └── persistence / migrations
├── frontend (depends common, render-wasm)
│ ├── UI components
│ └── state / API integration
├── exporter (depends common)
└── render-wasm (consumed by frontend)
```
Implementation order follows the dependency graph bottom-up: build shared
foundations first, then layer consumers on top.
#### Slice Vertically
Instead of building all of common, then all of backend, then all of frontend —
build one complete feature path at a time:
**Bad (horizontal slicing):**
```
Task 1: Build all common types
Task 2: Build all backend handlers
Task 3: Build all frontend components
```
**Good (vertical slicing):**
```
Task 1: common data types + schema ← foundation
Task 2: backend RPC handler + persistence
Task 3: frontend UI component + API integration
```
Each vertical slice delivers working, testable functionality.
#### Write Tasks
## Task Format
Each task follows this structure:
@@ -152,17 +102,16 @@ implementation. Omit when the task is mechanical.
**Estimated scope:** [XS: 1 file | S: 1-2 files | M: 3-5 files | L: 5+ files]
```
Replace "module-specific test command" with the actual commands for the module
(e.g. `clojure -M:dev:test` for backend/common,
`npx shadow-cljs compile test && npx karma start` for frontend, or the
commands noted in the module's core memory).
Use commands from `mem:testing` and affected module memories. Never substitute
generic text such as "run the tests" when the project documents an exact
command.
When possible, design each task with TDD in mind: acceptance criteria double
as a test list, and the natural first step of the task is writing those tests
before the implementation. Some tasks resist this (config, migrations, pure
wiring) — for those, keep the usual verification steps.
When possible, design each task with TDD in mind: acceptance criteria double as a test
list, and the natural first step of the task is writing those tests before the
implementation. Some tasks resist this (config, migrations, pure wiring) — for those, keep
the usual verification steps.
#### Estimate Scope
## Task Sizing
| Size | Files | Scope | Example |
|------|-------|-------|---------|
@@ -172,16 +121,11 @@ wiring) — for those, keep the usual verification steps.
| **L** | 5-8 | Multi-component feature | Search with filtering and pagination |
| **XL** | 8+ | **Too large — break it down further** | — |
If a task is XL, it should be broken into smaller tasks. Agents perform best
on S and M tasks.
Split a task when it contains independent outcomes, spans unrelated systems, or cannot be
completed and verified in one focused session (if a task is XL, it should be broken into
smaller tasks; agents perform best on S and M tasks).
**When to break a task down further:**
- It would take more than one focused session
- You cannot describe the acceptance criteria in 3 or fewer bullet points
- It touches two or more independent subsystems
- You find yourself writing "and" in the task title (a sign it is two tasks)
#### Order and Checkpoints
## Task order and checkpoints
Arrange tasks so that:
@@ -197,153 +141,43 @@ Add explicit checkpoints with the relevant module commands:
- [ ] Relevant tests pass (module-specific command).
- [ ] The relevant build or compilation passes, if applicable.
- [ ] The core flow works end-to-end.
- [ ] Review with human before proceeding.
```
## Requirements
- Analyze the codebase architecture and identify affected modules.
- Read project conventions before drafting (start with `critical-info` and
affected module core memories).
- Break down complex features or bugs into atomic, actionable steps.
- Propose solutions with clear rationale, trade-offs, and sequencing.
- Identify risks, edge cases, performance implications, and breaking changes.
- Apply DRY and KISS principles to the proposed implementation.
- Define a testing strategy aligned with each affected module's tooling.
- Every task must have acceptance criteria and verification steps.
- Checkpoints must exist after every 2-3 tasks.
## Constraints
- You are **analysis-only** — never create, edit, or delete source code. The
only file you may write is the plan itself, and only when the command or
user explicitly instructs you to save it.
- You do **not** run builds, tests, linters, or any commands that modify state.
- You do **not** create git commits or interact with version control.
- You do **not** execute shell commands beyond read-only searches (`rg`, `ls`,
`find`, `cat`, `bat`).
- Your output is a structured plan or analysis, ready for handoff to an
engineer agent or developer.
## Output Format
The plan is always delivered in the response so the user sees it regardless
of which agent is running the skill. By default you never write the plan file;
announce the path instead. Write the file only when the command or user
explicitly instructs you to save it — and then only that file.
of which agent is running the skill. File writes follow `Constraints`
by default announce the path instead of writing.
Announce the suggested save path:
```
.agents/plans/YYYY-MM-DD-<plan-one-line-title>.md
```
Use today's date in the user's local timezone. The `<plan-one-line-title>`
slug is lowercase, hyphen-separated, and a short summary of the task
(e.g. `add-batch-get-profiles-for-file-comments`). If the user explicitly
provides a target file path, announce that path instead of the default.
Announce the save path `.agents/plans/YYYY-MM-DD-<slug>.md` (today's date,
lowercase hyphen-separated slug, e.g. `2026-09-10-add-batch-get-profiles`;
an explicit user path wins).
End the response by suggesting the next steps: `/review-plan` to get a second
opinion on the plan and `/implement-plan` to execute it.
### Plan Document Template
### Plan Structure
Use this document shape:
```markdown
# Plan: [Feature/Project Name]
# Plan: Title
## Context
[One paragraph: what is the problem or feature request? Why is it needed?]
## Affected Modules
[Which modules of the monorepo are involved? Reference module paths and any
`mem:` memories that were consulted.]
## Architecture Decisions
- [Key decision 1 and rationale]
- [Key decision 2 and rationale]
## Risks & Considerations
[Edge cases, performance implications, breaking changes, migration concerns,
security implications.]
## Risks and Considerations
## Approach
[A short strategy summary: 3-5 sentences describing the overall approach and
the shape of the dependency graph (what depends on what, what gets built
first). High-level only — the task-by-task detail lives in the Task List.]
## Task List
Each task uses the full task structure defined in
[Write Tasks](#write-tasks) — description, rationale, acceptance criteria,
verification, dependencies, files, estimated scope, and optional code sketch.
Never reduce a task to a one-line checkbox; the plan must be self-contained
and executable without other context.
Tasks are a flat, ordered list — a plan is not a roadmap. Do not group tasks
into phases, milestones, or sprints; ordering and dependencies are already
captured per task. Insert a checkpoint after every 2-3 tasks.
## Task 1: [Short descriptive title]
**Description:** [What this task accomplishes.]
**Rationale:** [Why this approach over the alternatives.]
**Acceptance criteria:**
- [ ] [Specific, testable condition]
**Verification:**
- [ ] Relevant tests pass (module-specific command).
**Dependencies:** None
**Files likely touched:**
- `path/to/file`
**Estimated scope:** [XS: 1 file | S: 1-2 files | M: 3-5 files | L: 5+ files]
**Code sketch (optional):** [Short contract-level example, only if the shape
is non-obvious.]
## Task 2: [Short descriptive title]
[Same structure as Task 1.]
## Task 3: [Short descriptive title]
[Same structure as Task 1.]
### Checkpoint: After Tasks 1-3
- [ ] Relevant tests pass (module-specific command).
- [ ] The relevant build or compilation passes, if applicable.
- [ ] The core flow works end-to-end.
- [ ] Review with human before proceeding.
## Task 4: [Short descriptive title]
[Same structure as Task 1.]
## Task 5: [Short descriptive title]
[Same structure as Task 1.]
## Verification & Testing
[How to verify each task and the whole plan: the project's real test, lint,
build, and run commands (extracted during Required Reading), coverage
expectations, and manual checks. Consult each module's core memory for the
exact commands.]
## Parallelization Opportunities
- **Safe to parallelize:** Independent feature slices across separate
modules, tests for already-implemented features, documentation
- **Must be sequential:** Shared common schema changes, database migrations
- **Needs coordination:** Features that share a contract (define the contract
first, then parallelize)
## Verification and Testing
## Parallelization
## Open Questions
- [Question needing human input]
```
Omit empty sections only when they do not apply. Every implementation task
still requires acceptance criteria, verification, dependencies, likely files,
and scope.
When the plan is purely analytical (e.g. a code review or feasibility study
with no implementation), skip the **Approach** and **Task List** sections and
lead with **Findings** instead, keeping the rest of the structure.
@@ -357,15 +191,6 @@ lead with **Findings** instead, keeping the rest of the structure.
| "Planning is overhead" | Planning is the task. Implementation without a plan is just typing. |
| "I can hold it all in my head" | Context windows are finite. Written plans survive session boundaries and compaction. |
## Red Flags
- Delivering prose without a task breakdown
- Tasks that say "implement the feature" without acceptance criteria
- No verification steps in the plan
- All tasks are XL-sized
- No checkpoints between tasks
- Dependency order isn't considered
## Verification Checklist
Before delivering the plan, confirm:
+125 -11
View File
@@ -357,6 +357,39 @@ Insert the new version section right after the `# CHANGELOG` header (before
the previous version entry). Use the `edit` tool with enough context to make
a unique match.
### 8b. Propose and populate the `:rocket: Epics and highlights` subsection
After inserting the version section, proactively create or populate the
`### :rocket: Epics and highlights` subsection. This section surfaces the
most impactful changes for self-hosted users checking for updates.
**When to create:** If the version section does not already have a
`### :rocket: Epics and highlights` subsection, create one. Place it before
`### :sparkles:` (matching existing order in CHANGES.md).
**How to identify highlights:** Review the `:sparkles:` entries for the
version and select 25 of the most impactful/user-visible ones. Criteria:
- New user-visible features (not internal refactors)
- Significant capability additions
- Items that create "FOMO" for self-hosted users on older versions
**Use release notes as hints:** Check
`frontend/src/app/main/ui/releases/v2_<MINOR>.cljs` for the corresponding
version. The slide titles and feature descriptions there are curated
marketing content indicating what the team considers highlight-worthy. Match
those themes to changelog entries. Treat these files as optional hints — they
may not exist for every version.
**Format requirement:** Every `:rocket:` entry MUST follow the standard
changelog format with issue/PR references:
```
- <description> [#<ISSUE>](https://github.com/penpot/penpot/issues/<ISSUE>) (PR: [#<PR>](https://github.com/penpot/penpot/pull/<PR>))
```
An entry without issue AND PR references is a highlight gap (warning, not an anomaly).
**Preserve existing entries:** If the `:rocket:` section already exists from
a prior run, preserve its entries. Do not remove or rewrite them.
### 9. Verify
Read the top of `CHANGES.md` and confirm:
@@ -468,9 +501,8 @@ Markdown viewer.
## What is an anomaly
**An anomaly is a milestone-mismatch between an issue and its referenced
PR.** It indicates that the changelog claim "this issue is fixed by this PR,
all in milestone M" is inconsistent with the actual milestone assignments.
There are exactly two types:
PR.** There are two anomaly types, plus two highlight gaps (warnings that
do not count toward the anomaly total):
1. **Issue is in the milestone, but its referenced PR is in a different
milestone (or has no milestone).** The changelog claims a fix in this
@@ -486,6 +518,13 @@ There are exactly two types:
PR that closes an issue with no milestone references an issue from
another (probably private) project; that is expected and the issue is
not part of this changelog. Do not report it.
3. **missing-highlights (gap):** A released X.Y.0 version section has no
`### :rocket: Epics and highlights` subsection. Patches (X.Y.Z) never
carry highlights, so only minors/majors are checked.
4. **missing-highlight-reference (gap):** A `:rocket:` entry lacks the
required issue AND PR references. Every highlight entry must follow the
standard changelog format with `[#ISSUE]` and `(PR: [#PR])` links
(multi-PR `(PR: [#A](...), [#B](...))` accepted).
**Anything else is not an anomaly.** Other discrepancies (exclusion
labels on in-changelog issues, missing valid issues, unmerged PR
@@ -653,6 +692,40 @@ for pr_num in sorted(changelog_prs):
'issue_milestone': issue_ms, # may be None
})
# --- Type C: released X.Y.0 version sections without :rocket: subsection ---
# Patches (X.Y.Z with Z != 0) never carry :rocket: by design — only minors/majors (X.Y.0).
anomalies_c = [] # list of version strings
rocket_heading_re = re.compile(r'^### :rocket:', re.MULTILINE)
version_sections = re.split(r'(?=^## \d+\.\d+\.\d+)', content, flags=re.MULTILINE)
for vs in version_sections:
m = re.match(r'^## (\d+\.\d+\.\d+)(.*)', vs)
if not m: continue
ver, suffix = m.group(1), m.group(2)
if 'unreleased' in suffix.lower(): continue
if ver.split('.')[2] != '0': continue
if not rocket_heading_re.search(vs):
anomalies_c.append(ver)
# --- Type D: :rocket: entries without issue AND PR references ---
# Both are required: `[#ISSUE](.../issues/N)` and `(PR: [#PR](.../pull/M))`.
# Multi-PR entries `(PR: [#A](...), [#B](...))` are accepted.
anomalies_d = [] # list of dicts: {version, line}
issue_ref_re = re.compile(r'\[#\d+\]\(https://github\.com/penpot/penpot/issues/\d+\)')
pr_ref_re = re.compile(r'\(PR:\s*\[#\d+\]\(https://github\.com/penpot/penpot/pull/\d+\)(\s*,\s*\[#\d+\]\(https://github\.com/penpot/penpot/pull/\d+\))*\)')
for vs in version_sections:
m = re.match(r'^## (\d+\.\d+\.\d+)(.*)', vs)
if not m: continue
ver = m.group(1)
rocket_match = rocket_heading_re.search(vs)
if not rocket_match: continue
# Extract the :rocket: subsection body (up to next ### or ##)
rocket_body = vs[rocket_match.end():]
rocket_body = re.split(r'(?m)^#{2,3}\s', rocket_body)[0]
for line in rocket_body.splitlines():
line = line.strip()
if line.startswith('- ') and not (issue_ref_re.search(line) and pr_ref_re.search(line)):
anomalies_d.append({'version': ver, 'line': line[:100]})
# --- Write report ---
def fmt_ms(ms):
return ms if ms else "_none_"
@@ -664,13 +737,17 @@ with open(OUTPUT, 'w') as f:
n_a = len(anomalies_a)
n_b = len(anomalies_b)
n_c = len(anomalies_c)
n_d = len(anomalies_d)
f.write('## Summary\n\n')
f.write(f'- **Issue in {MILESTONE}, referenced PR in different milestone or no milestone:** {n_a}\n')
f.write(f'- **PR in {MILESTONE}, closing issue in a different milestone:** {n_b}\n')
f.write(f'- **Total anomalies:** {n_a + n_b}\n\n')
f.write(f'- **Total anomalies:** {n_a + n_b}\n')
f.write(f'- **Released X.Y.0 version missing :rocket: section (gap):** {n_c}\n')
f.write(f'- **:rocket: entry without issue AND PR references (gap):** {n_d}\n\n')
# --- Anomalies section ---
# --- Anomalies section (milestone mismatches only) ---
if n_a or n_b:
f.write('## Anomalies\n\n')
f.write('These are milestone mismatches between an issue in the changelog '
@@ -709,9 +786,37 @@ with open(OUTPUT, 'w') as f:
badge = '🔴' if e['issue_milestone'] is None else '⚠️'
f.write(f' - {badge} Closing {issue_link(e["issue"])} is in milestone **{ms_label}** (expected: {MILESTONE})\n')
f.write('\n')
else:
f.write('✅ No anomalies found. All (issue, PR) pairs in the changelog have aligned milestone assignments.\n\n')
# --- Highlight gaps (warnings, not anomalies) ---
if n_c or n_d:
f.write('## Highlight gaps\n\n')
f.write('These are warnings, not anomalies: they do not affect the '
'milestone-mismatch total above. They track `:rocket:` coverage '
'across all released X.Y.0 versions. Historical entries (e.g. '
'Taiga links) predate the current reference convention and are '
'expected to appear here.\n\n')
if n_c:
f.write(f'### Released X.Y.0 version missing :rocket: section\n\n')
f.write('These released minors/majors have no `### :rocket: Epics and highlights` subsection. '
'Add highlights to help self-hosted users understand what they are missing.\n\n')
for ver in anomalies_c:
f.write(f'- Version **{ver}**\n')
f.write('\n')
if n_d:
f.write(f'### :rocket: entry without issue AND PR references\n\n')
f.write('These highlight entries lack the required issue AND PR references. '
'Add `[#ISSUE](...)` and `(PR: [#PR](...))` links.\n\n')
for d in anomalies_d:
f.write(f'- **{d["version"]}**: `{d["line"]}`\n')
f.write('\n')
elif not (n_a or n_b):
f.write('✅ No highlight gaps found. All released X.Y.0 versions have properly referenced :rocket: entries.\n\n')
# --- Context ---
f.write('---\n\n')
f.write('## Context\n\n')
@@ -726,8 +831,7 @@ print(f"Anomaly report written to {OUTPUT}")
PYEOF
```
This generates `CHANGES-ISSUES.md` containing **only the anomalies**
milestone mismatches between issues and their referenced PRs:
This generates `CHANGES-ISSUES.md` containing anomalies and highlight gaps:
1. **Issue in milestone, referenced PR in different milestone or no milestone**
the changelog claims a fix here, but the PR is released elsewhere.
@@ -736,6 +840,13 @@ milestone mismatches between issues and their referenced PRs:
(An issue with *no* milestone belongs to another, probably private,
project — milestones are only required on the "Main" project — so it is
neither an anomaly nor a changelog candidate.)
3. **missing-highlights (gap, warning)** — a released X.Y.0 version section
has no `### :rocket: Epics and highlights` subsection. Patches (X.Y.Z)
never carry highlights.
4. **missing-highlight-reference (gap, warning)** — a `:rocket:` entry lacks
the required issue AND PR references.
Gaps do not count toward the anomaly total.
**Rule violations are not in the report** — they are workflow errors the
LLM must fix directly in `CHANGES.md` during step 6a (pre-flight checks).
@@ -809,10 +920,13 @@ self-contained and clickable in any Markdown viewer.
issue from a different project or context. If the PR title and issue title
are clearly unrelated, or the PR predates the issue by years, treat it as a
data glitch and skip it.
- **Anomaly = milestone mismatch only.** The report contains only milestone
mismatches: (1) the issue is in this milestone but the referenced PR is
in a different milestone (or unassigned), and (2) the PR is in this
milestone but the issue it closes is in a different milestone. An
- **Anomaly = milestone mismatch only; gaps are warnings.** The report's
anomaly total counts only milestone mismatches: (1) the issue is in this
milestone but the referenced PR is in a different milestone (or unassigned),
and (2) the PR is in this milestone but the issue it closes is in a
different milestone. `:rocket:` highlight gaps (missing section on a
released X.Y.0, entry without issue AND PR references) are reported in a
separate `Highlight gaps` section and never count toward the anomaly total. An
*unassigned* (milestone-less) issue closed by a milestone PR is **not**
an anomaly: milestones are required only for the "Main" project, so such
issues come from another (probably private) project and are not changelog
+1 -1
View File
@@ -6,7 +6,7 @@ on:
jobs:
build-and-push:
name: Build and push DevEnv Docker image
runs-on: penpot-extended-runner
runs-on: penpot-standar-runner
steps:
- name: Set common environment variables
+4 -4
View File
@@ -46,7 +46,7 @@ jobs:
# ── 1. Resolve the build key and check the whole set at once ───────────
prepare:
name: Prepare
runs-on: penpot-extended-runner
runs-on: penpot-standar-runner
timeout-minutes: 15
outputs:
gh_ref: ${{ steps.vars.outputs.gh_ref }}
@@ -135,7 +135,7 @@ jobs:
# ── 2. One build per image, in parallel, only when needed ──────────────
build:
name: Build ${{ matrix.image }}
runs-on: penpot-extended-runner
runs-on: penpot-standar-runner
timeout-minutes: 60
needs: prepare
if: needs.prepare.outputs.exists == 'false'
@@ -248,7 +248,7 @@ jobs:
# the S3 marker guarantees the branch tags were already moved.
promote:
name: Promote image set
runs-on: penpot-extended-runner
runs-on: penpot-standar-runner
timeout-minutes: 10
needs: [prepare, build]
@@ -302,7 +302,7 @@ jobs:
# ── 4. Single failure notification for the whole workflow ─────────────
notify:
name: Notify failure
runs-on: penpot-extended-runner
runs-on: penpot-standar-runner
timeout-minutes: 5
needs: [prepare, build, promote]
if: failure()
+1 -1
View File
@@ -46,7 +46,7 @@ jobs:
notify:
name: Notifications
runs-on: ubuntu-24.04
runs-on: ubuntu-latest
needs:
- build-docker
- build-docker-admin-console
+1 -1
View File
@@ -19,7 +19,7 @@ permissions:
jobs:
release:
runs-on: ubuntu-24.04
runs-on: ubuntu-latest
outputs:
version: ${{ steps.vars.outputs.gh_ref }}
release_notes: ${{ steps.extract_release_notes.outputs.release_notes }}
+1 -1
View File
@@ -32,7 +32,7 @@ jobs:
test-exporter:
if: ${{ !github.event.pull_request.draft }}
name: "Exporter Tests"
runs-on: penpot-runner-02
runs-on: penpot-extended-runner
container:
image: penpotapp/devenv:latest
volumes:
-1
View File
@@ -24,7 +24,6 @@ opencode.json
!AGENTS.md
!CODE_OF_CONDUCT.md
!SECURITY.md
!HIGHLIGHTS.md
/*.png
/*.svg
/*.sql
+21 -2
View File
@@ -8,13 +8,30 @@
- The backend stores the binary content.
- Supported backends are `:fs` and `:s3`.
- FS uses one root directory and a UUID-derived path.
- S3 uses one configured bucket and an optional prefix.
- S3 uses a default configured bucket and an optional prefix, plus optional named targets.
- A Penpot bucket is metadata. It is not an S3 bucket or a filesystem directory.
- FS and S3 use the same UUID-derived object path. The bucket does not change the path.
- `PENPOT_OBJECTS_STORAGE_*` configures the current object backend.
- Deprecated asset-storage config keys remain supported for migration.
- Database rows keep the backend name. Keep the legacy `:assets-fs` and `:assets-s3` aliases.
## S3 Targets and Routing
- The `:s3` backend keeps `storage_object.backend = 's3'`; routing lives inside the backend.
- Routing maps a Penpot semantic bucket to a named target (own bucket, optional prefix/region/endpoint).
- Targets are declared in an EDN file referenced by `PENPOT_OBJECTS_STORAGE_S3_ROUTES_FILE` (`app.storage.config/load`).
- Schema: `{:targets {<id> {:bucket ... :prefix? ... :region? ... :endpoint? ...}} :routes {"<semantic-bucket>" <id>}}`.
- The reserved `:default` target is implicit and built from `PENPOT_OBJECTS_STORAGE_S3_*`; declared targets inherit missing region/endpoint/prefix from it.
- Without a routes file, `::sto/bucket->target` is nil and every object uses `:default` (unchanged behavior).
- The chosen target id is stored in object metadata as `:storage-target` (plain string) by `put-object!`.
- `app.storage.s3/resolve-target` reads the object metadata; `nil` (legacy) and `"default"` use the default target, an unknown non-nil id raises `:invalid-storage-target` (no fallback) on reads/serving/deletes.
- `impl/target-resolvable?` (wrapped as `sto/target-resolvable?`) reports whether a target id is configured; `:fs` is always true.
- GC-deleted and pending-gc refuse to delete rows whose target is not resolvable: they log `:err`, park the row (`deleted_at = now()+1d`, no attempts, no give-up) and never remove it until the target is configured again.
- `deleted_at` doubles as the pending-gc park marker; the pending selection skips rows whose `deleted_at` is in the future.
- One S3 client/presigner is built per distinct `[region endpoint]` and shared by targets; a failed init closes the already-built pairs.
- Target ids are stored in metadata, so they must stay stable; removing one makes its old rows unreadable and unGC-able by design.
- `:storage-target` metadata is load-bearing: `pending-gc` passes it via `with-meta` so `del-object` resolves the right target.
## Object Lifecycle
- `put-object!` creates the database row before it writes backend content.
@@ -64,7 +81,7 @@ Since `put-object!` uses backend-specific operations (`impl/resolve-backend` + `
## Deduplication
- Deduplication requires `::sto/deduplicate?`, a content hash, and bucket metadata.
- The lookup matches hash, bucket, backend, and `deleted_at IS NULL`.
- The lookup matches hash, bucket, backend, storage target (`:storage-target`, coalesced to `default`), and `deleted_at IS NULL`.
- The lookup only considers rows with `status='valid'`; pending rows are invisible.
- A hit whose blob is missing is repaired in place: the same row/id is kept,
and `put-object!` rewrites the blob under that id. This heals all existing
@@ -93,6 +110,8 @@ Since `put-object!` uses backend-specific operations (`impl/resolve-backend` + `
- The valid bucket set lives in `app.storage/valid-buckets`.
- `file-media-object` is the default bucket for old rows without bucket metadata.
- Under `:s3`, any valid bucket may be routed to a named target; unrouted buckets use `:default`.
- GC resolves the target from `metadata.:storage-target` for deleted and pending rows.
- Do not assign a new bucket without adding its access and cleanup behavior.
- The touched-object collector raises an internal error for an unknown bucket.
- It supports `file-media-object`, `team-font-variant`, `file-object-thumbnail`, `file-thumbnail`, `profile`, `file-data`, `tempfile`, and `organization`.
+6 -4
View File
@@ -52,10 +52,12 @@ file (never pipe tool output through filters).
then re-run `corepack use pnpm@<tag>` in that directory.
- A workspace may fail with `ERR_PNPM_IGNORED_BUILDS`, and pnpm then writes
a placeholder scaffold into its `pnpm-workspace.yaml`:
`allowBuilds: esbuild: set this to true or false` plus
`ignoredBuiltDependencies`. Repo convention is `allowBuilds: esbuild: true`.
Replace the placeholder and drop the `ignoredBuiltDependencies` entry,
then re-run.
`allowBuilds: esbuild: set this to true or false`. Current pnpm writes
only the `allowBuilds` placeholder; any legacy key still present
(`ignoredBuiltDependencies`, `onlyBuiltDependencies`,
`neverBuiltDependencies`) is ignored since pnpm 11. Repo convention is
`allowBuilds: esbuild: true`. Replace the placeholder and drop the
legacy entry, then re-run.
- `plugins/apps/composable-test-suite` once had its own
`pnpm-workspace.yaml` and acted as a nested workspace root. That state is
gone on purpose: pnpm picks the nearest `pnpm-workspace.yaml` walking up,
+3
View File
@@ -14,6 +14,9 @@
- **Never pipe test output directly to filters** (`| head`, `| tail`, `| grep`, etc.).
Always redirect to a file first: `command > /tmp/output.txt 2>&1`, then read/grep the file.
This prevents hiding test failures. See `mem:testing` for details.
- **`.claude/skills` is a symlink to `.agents/skills`.**
Edit skills only in their canonical location (`.agents/skills`); never edit
through `.claude/skills`.
- **Read the workflow memory BEFORE the corresponding action**:
- Before `git commit``mem:workflow/creating-commits` (commit format, AI-assisted-by trailer)
- Before `gh issue create``mem:workflow/creating-issues` (title derivation, body template, Issue Type)
+23
View File
@@ -156,6 +156,21 @@
- Fix incorrect permission handling when managing share links on a file [#11289](https://github.com/penpot/penpot/issues/11289) (PR: [#11290](https://github.com/penpot/penpot/pull/11290))
- Fix backend session remaining valid after logout when the auth-token cookie is replayed [#11316](https://github.com/penpot/penpot/issues/11316) (PR: [#11317](https://github.com/penpot/penpot/pull/11317))
- Fix get-team-invitation-token requiring only read permissions [#11358](https://github.com/penpot/penpot/issues/11358) (PR: [#11359](https://github.com/penpot/penpot/pull/11359))
- Fix missing text in legacy SVG board thumbnails [#10182](https://github.com/penpot/penpot/issues/10182) (PR: [#11552](https://github.com/penpot/penpot/pull/11552))
- Fix workspace crash when applying transform modifiers in the WASM renderer [#10894](https://github.com/penpot/penpot/issues/10894) (PR: [#10896](https://github.com/penpot/penpot/pull/10896))
- Limit ZIP entry count and object size on V3 binfile import [#11021](https://github.com/penpot/penpot/issues/11021) (PR: [#11022](https://github.com/penpot/penpot/pull/11022))
- Block plugin UI iframe URLs targeting the Penpot domain [#11271](https://github.com/penpot/penpot/issues/11271) (PR: [#11273](https://github.com/penpot/penpot/pull/11273))
- Restrict the MCP REPL code execution endpoint to development environments [#11283](https://github.com/penpot/penpot/issues/11283) (PR: [#11282](https://github.com/penpot/penpot/pull/11282))
- Filter share-link tokens from the get-view-only-bundle response [#11285](https://github.com/penpot/penpot/issues/11285) (PR: [#11286](https://github.com/penpot/penpot/pull/11286))
- Disable MCP developer tools in multi-user mode [#11291](https://github.com/penpot/penpot/issues/11291) (PR: [#11310](https://github.com/penpot/penpot/pull/11310))
- Fix Hide comments setting being ignored after opening the Comments section [#11308](https://github.com/penpot/penpot/issues/11308) (PR: [#11492](https://github.com/penpot/penpot/pull/11492))
- Block NAT64/6to4/Teredo IPv6 transition addresses in the SSRF guard [#11319](https://github.com/penpot/penpot/issues/11319) (PR: [#11320](https://github.com/penpot/penpot/pull/11320))
- Prevent team admins from removing the team owner [#11367](https://github.com/penpot/penpot/issues/11367) (PR: [#11368](https://github.com/penpot/penpot/pull/11368))
- Enforce share-link comment permissions and page scope [#11370](https://github.com/penpot/penpot/issues/11370) (PR: [#11371](https://github.com/penpot/penpot/pull/11371))
- Clean up orphaned teams, projects and files on profile deletion [#11394](https://github.com/penpot/penpot/issues/11394) (PR: [#11395](https://github.com/penpot/penpot/pull/11395))
- Fix crash when pressing Ctrl+D with no shape selected [#11448](https://github.com/penpot/penpot/issues/11448) (PR: [#11491](https://github.com/penpot/penpot/pull/11491))
- Fix text layout not updating when auto-width is set by double-clicking the bounding box [#11480](https://github.com/penpot/penpot/issues/11480) (PR: [#11541](https://github.com/penpot/penpot/pull/11541))
- Fix boolean shapes rendering deformed in the WASM renderer and exports [#11482](https://github.com/penpot/penpot/issues/11482) (PR: [#11551](https://github.com/penpot/penpot/pull/11551))
### :sparkles: New features & Enhancements
@@ -218,6 +233,10 @@
### :rocket: Epics and highlights
- Render prototype viewer with WASM (Skia) engine instead of SVG [#10037](https://github.com/penpot/penpot/issues/10037) (PR: [#10038](https://github.com/penpot/penpot/pull/10038))
- Add layer blur effect for visual depth and styling [#9844](https://github.com/penpot/penpot/issues/9844) (PR: [#10034](https://github.com/penpot/penpot/pull/10034))
- Render guides in WebGL for consistent viewer performance [#10068](https://github.com/penpot/penpot/issues/10068) (PR: [#10014](https://github.com/penpot/penpot/pull/10014))
- Add concurrency limiter and status indicators for MCP server communications [#9493](https://github.com/penpot/penpot/issues/9493) (PR: [#9748](https://github.com/penpot/penpot/pull/9748))
- Add typography token row to multiselected texts for better token visibility [#9336](https://github.com/penpot/penpot/issues/9336) (PR: [#9128](https://github.com/penpot/penpot/pull/9128))
### :sparkles: New features & Enhancements
@@ -572,6 +591,10 @@
## 2.15.0
### :rocket: Epics and highlights
- Add MCP server integration for AI-assisted design workflows [#9174](https://github.com/penpot/penpot/issues/9174) (PR: [#9032](https://github.com/penpot/penpot/pull/9032), [#9321](https://github.com/penpot/penpot/pull/9321))
### :sparkles: New features & Enhancements
- Add MCP server integration [GH #9174](https://github.com/penpot/penpot/issues/9174)
-26
View File
@@ -1,26 +0,0 @@
# HIGHLIGHTS
## 2.17.0
- Background blur is here
- WebGL rendering gets stronger
- MCP connection status and more
- Design tokens: more visible, more user-friendly
## 2.16.0
- Design tokens in the design panel
- Major community contributions
- WebGL rendering (beta)
## 2.15.0
- AI connected to real design context
- Multi-directional workflow
- Your stack, your model, your decision
+7
View File
@@ -56,6 +56,7 @@ If your organization is scaling and needs extra support, were here to help. [
- [Why Penpot](#why-penpot)
- [Getting Started](#getting-started)
- [Penpot Enterprise](#penpot-enterprise)
- [Community](#community)
- [Contributing](#contributing)
- [Resources](#resources)
@@ -93,6 +94,12 @@ Penpot is the only design & prototype platform that is deployment agnostic. You
Learn how to install it with Docker, Kubernetes, Elestio or other options on [our website](https://penpot.app/self-host).
<img width="100%" height="1010" alt="2" src="https://github.com/user-attachments/assets/243e796e-a140-481a-b68f-b24be6a70e37" />
## Penpot Enterprise ##
Penpot Enterprise is our paid plan for organizations that need to scale their design work across multiple teams with advanced governance, security, and administration. Manage teams and access from a centralized **Admin Console**, configure advanced permissions, and connect your **identity provider through SSO**. Available for cloud and self-hosted environments, it combines enterprise controls with Penpots open-source foundation and open standards.
## Community ##
We love the Open Source software community. Contributing is our passion and if its yours too, participate and [improve](https://community.penpot.app/c/help-us-improve-penpot/7) Penpot. All your designs, code and ideas are welcome!
+1 -1
View File
@@ -31,7 +31,7 @@ export PENPOT_MEDIA_PROCESSING_SERVICE_URI=http://localhost:6065
export PENPOT_FLAGS="\
$PENPOT_FLAGS \
enable-login-with-password \
disable-login-with-ldap \
enable-login-with-ldap \
disable-login-with-oidc \
disable-login-with-google \
disable-login-with-github \
+24 -6
View File
@@ -10,7 +10,7 @@
[app.common.logging :as l]
[app.common.schema :as sm]
[clj-ldap.client :as ldap]
[clojure.string]
[cuerdas.core :as str]
[integrant.core :as ig]))
(defn- prepare-params
@@ -36,11 +36,22 @@
:cause cause))))
(defn- replace-several [s & {:as replacements}]
(reduce-kv clojure.string/replace s replacements))
(reduce-kv str/replace s replacements))
(defn- escape-ldap-filter-value
"Escapes special characters in a string for use in LDAP filter values,
per RFC 4515 section 3."
[s]
(-> s
(str/replace "\\" "\\5c")
(str/replace "*" "\\2a")
(str/replace "(" "\\28")
(str/replace ")" "\\29")
(str/replace "\u0000" "\\00")))
(defn- search-user
[{:keys [::conn base-dn] :as cfg} email]
(let [query (replace-several (:query cfg) ":username" email)
(let [query (replace-several (:query cfg) ":username" (escape-ldap-filter-value email))
attrs [(:attrs-username cfg)
(:attrs-email cfg)
(:attrs-fullname cfg)]
@@ -49,12 +60,19 @@
:attributes attrs}]
(first (ldap/search conn base-dn params))))
(defn- get-attr
"Retrieves an attribute from an LDAP entry. Handles multi-valued
attributes by returning the first value."
[entry attr-key]
(let [v (get entry attr-key)]
(if (coll? v) (first v) v)))
(defn- retrieve-user
[{:keys [::conn] :as cfg} {:keys [email password]}]
(when-let [{:keys [dn] :as user} (search-user cfg email)]
(when (ldap/bind? conn dn password)
{:fullname (get user (-> cfg :attrs-fullname keyword))
:email email
{:fullname (get-attr user (-> cfg :attrs-fullname keyword))
:email (get-attr user (-> cfg :attrs-email keyword))
:backend "ldap"})))
(def ^:private schema:info-data
@@ -79,7 +97,7 @@
(l/warn :hint "invalid response from ldap, looks like ldap is not configured correctly" :data user)
(ex/raise :type :restriction
:code :wrong-ldap-response
:explain explain)))
::sm/explain explain)))
user)))
(defn- try-connectivity
+1
View File
@@ -293,6 +293,7 @@
[:objects-storage-s3-bucket {:optional true} :string]
[:objects-storage-s3-region {:optional true} :keyword]
[:objects-storage-s3-endpoint {:optional true} ::sm/uri]
[:objects-storage-s3-routes-file {:optional true} :string]
;; SSRF protection
[:ssrf-allowed-hosts {:optional true} [::sm/set :string]]
+1 -1
View File
@@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.graph.arrow
"Bulk Ladybug ingest through in-memory Arrow.
+1 -1
View File
@@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.graph.debug
"In-memory Ladybug sessions for the debug graph console."
+1 -1
View File
@@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.graph.ingest
"Penpot file -> Ladybug graph projection."
+1 -1
View File
@@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.graph.ladybug
"Ladybug access layer for graph-backed Penpot.
+1 -1
View File
@@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.graph.meta
"`GraphMeta`: the graph's own account of who built it and from what.
@@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.graph.projection.document
"Project a Penpot file-data map into Ladybug nodes and structural edges.
@@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.graph.projection.transforms
"Derived graph links: edges a reader could compute from the projected
+1 -1
View File
@@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.graph.report
(:require
+1 -1
View File
@@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.graph.schema
"Ladybug DDL facade for the graph-backed Penpot vertical slice.
+1 -1
View File
@@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.graph.schema.contract
"Deliberate choices in Penpot's graph schema, recorded as data.
+1 -1
View File
@@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.graph.schema.nodes
"Single source of truth for graph node tables.
+1 -1
View File
@@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.graph.schema.projection
"Derive Ladybug node column schemas from Penpot Malli sources.
+1 -1
View File
@@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.graph.schema.types
"Map Malli schemas to Ladybug column types.
+1 -1
View File
@@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.graph.schema.values
"Shape a Penpot value into the plain data its Ladybug column type wants.
+1 -1
View File
@@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.graph.stats
(:require
+1 -1
View File
@@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.graph.sync
"Incremental Ladybug graph updates from Penpot file-change events."
+8 -3
View File
@@ -15,6 +15,7 @@
(:require
[app.common.schema :as sm]
[app.util.ssrf :as ssrf]
[app.worker :as-alias wrk]
[cuerdas.core :as str]
[integrant.core :as ig]
[java-http-clj.core :as http])
@@ -23,6 +24,8 @@
java.net.URI))
(def default-max-redirects 5)
(def default-connect-timeout 30000)
(def default-request-timeout 30000)
(defn client?
[o]
@@ -33,15 +36,17 @@
:pred client?})
(defmethod ig/init-key ::client
[_ _]
(http/build-client {:connect-timeout 30000
[_ {:keys [::wrk/executor]}]
(http/build-client {:connect-timeout default-connect-timeout
:executor executor
:follow-redirects :never}))
(defn send!
([client req] (send! client req {}))
([client req {:keys [response-type] :or {response-type :string}}]
(assert (client? client) "expected valid http client")
(http/send req {:client client :as response-type})))
(http/send (merge {:timeout default-request-timeout} req)
{:client client :as response-type})))
(defn- resolve-client
[params]
+7 -1
View File
@@ -60,7 +60,13 @@
(defmethod handle-error :restriction
[err request _]
(let [{:keys [code] :as data} (ex-data err)]
(let [data (ex-data err)
code (get data :code)
explain (ex/explain data)
data (-> data
(dissoc ::sm/explain)
(cond-> explain (assoc :explain explain)))]
(if (= code :method-not-allowed)
{::yres/status 405
::yres/body data}
+10 -2
View File
@@ -34,6 +34,7 @@
[app.setup :as-alias setup]
[app.srepl :as-alias srepl]
[app.storage :as-alias sto]
[app.storage.config :as sto.config]
[app.storage.fs :as-alias sto.fs]
[app.storage.gc-deleted :as-alias sto.gc-deleted]
[app.storage.gc-touched :as-alias sto.gc-touched]
@@ -148,6 +149,11 @@
::mdef/labels []
::mdef/type :histogram}})
(def ^:private storage-routing
"Optional S3 storage targets and per semantic-bucket routing. Loaded once
from the external routes file. Empty when the feature is not configured."
(sto.config/load))
(def system-config
{::db/pool
{::db/uri (cf/get :database-uri)
@@ -205,7 +211,7 @@
::sto/storage (ig/ref ::sto/storage)}
::http.client/client
{}
{::wrk/executor (ig/ref ::wrk/executor)}
::session/manager
{::db/pool (ig/ref ::db/pool)}
@@ -522,7 +528,8 @@
;; explicit migration because the database objects/rows will
;; still reference the old names).
:assets-s3 (ig/ref :app.storage.s3/backend)
:assets-fs (ig/ref :app.storage.fs/backend)}}
:assets-fs (ig/ref :app.storage.fs/backend)}
::sto/bucket->target (:routes storage-routing)}
:app.storage.s3/backend
{::sto.s3/region (or (cf/get :storage-assets-s3-region)
@@ -533,6 +540,7 @@
(cf/get :objects-storage-s3-bucket))
::sto.s3/io-threads (or (cf/get :storage-assets-s3-io-threads)
(cf/get :objects-storage-s3-io-threads))
::sto.s3/targets (:targets storage-routing)
::wrk/netty-io-executor
(ig/ref ::wrk/netty-io-executor)}
+3 -3
View File
@@ -75,10 +75,10 @@
{:method method
:uri uri
:body body
:headers headers}
:headers headers
:timeout timeout}
{:response-type :input-stream
:skip-ssrf-check? true
:timeout timeout})
:skip-ssrf-check? true})
status (:status resp)]
(when (not (<= 200 status 299))
(let [body (:body resp)]
+14 -6
View File
@@ -390,18 +390,26 @@
(def ^:private sql:file-comment-users
"WITH available_profiles AS (
SELECT DISTINCT owner_id AS id
FROM comment
WHERE thread_id IN (SELECT id FROM comment_thread WHERE file_id=?)
SELECT DISTINCT c.owner_id AS id
FROM comment c
JOIN comment_thread ct
ON ct.id = c.thread_id
WHERE ct.file_id = ?::uuid
),
profile_ids AS (
SELECT id FROM available_profiles
UNION
SELECT ?::uuid
)
SELECT p.id,
p.email,
p.fullname AS name,
p.fullname AS fullname,
p.fullname,
p.photo_id,
p.is_active
FROM profile AS p
WHERE p.id IN (SELECT id FROM available_profiles) OR p.id=?")
FROM profile p
JOIN profile_ids AS x
ON x.id = p.id;")
(defn get-file-comments-users
[conn file-id profile-id]
+1 -1
View File
@@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.rpc.commands.plugins
(:require
+8 -2
View File
@@ -41,6 +41,7 @@
[app.rpc.notifications :as notifications]
[app.storage :as sto]
[app.util.services :as sv]
[app.util.ssrf :as ssrf]
[app.worker :as wrk]
[cuerdas.core :as str]))
@@ -960,13 +961,18 @@ RETURNING id, deleted_at;")
(sv/defmethod ::check-organization-sso
"Validate an organization SSO configuration by generating a login redirect URL.
Nitrate calls this while configuring SSO to verify client credentials and OIDC
discovery before saving the settings."
discovery before saving the settings. The issuer URL is nitrate-supplied
(customer-configured), so it is checked against the SSRF blocklist before
any outbound request is attempted."
{::doc/added "2.18"
::sm/params cto/schema:nitrate-sso
::sm/result schema:check-organization-sso-result
::rpc/auth false}
[cfg params]
{:valid (oidc/is-organization-sso-config-valid? cfg params)})
(let [issuer (oidc/organization-sso-discovery-uri params)]
{:valid (boolean (and issuer
(ssrf/safe-url? issuer)
(oidc/is-organization-sso-config-valid? cfg params)))}))
;; ---- API: notify-organization-sso-change
(sv/defmethod ::notify-organization-sso-change
+32 -5
View File
@@ -70,6 +70,7 @@
[:map {:title "storage"}
[::backends schema:backends]
[::backend [:enum :s3 :fs]]
[::bucket->target {:optional true} [:map-of :string :keyword]]
::db/pool])
(def valid-storage?
@@ -112,17 +113,18 @@
params))
(defn- get-database-object-by-hash
[connectable backend bucket hash]
[connectable backend bucket target hash]
(let [sql (str "select * from storage_object "
" where (metadata->>'~:hash') = ? "
" and (metadata->>'~:bucket') = ? "
" and coalesce(metadata->>'~:storage-target', 'default') = ? "
" and backend = ?"
" and deleted_at is null"
" and status = 'valid'"
" limit 1")]
;; NOTE: metadata is left encoded; row->storage-object is
;; responsible for decoding it.
(db/exec-one! connectable [sql hash bucket (name backend)])))
(db/exec-one! connectable [sql hash bucket target (name backend)])))
(defn- promote-object!
[storage object]
@@ -185,6 +187,13 @@
(let [ds (db/get-connectable storage)]
(get-database-object ds id)))
(defn- resolve-target-id
"Returns the storage target id for the given semantic bucket, or nil when
the routing does not apply (non-S3 backends)."
[storage bucket]
(when (= :s3 (::backend storage))
(get (::bucket->target storage) bucket :default)))
(defn put-object!
"Creates a new object with the provided content."
[{:keys [::backend ::db/pool] :as storage}
@@ -193,9 +202,16 @@
(assert (impl/content? content) "expected an instance of content")
(let [id (or (::id params) (uuid/random))
mdata (cond-> (get-metadata params)
base-mdata (get-metadata params)
bucket (:bucket base-mdata)
target (resolve-target-id storage bucket)
target-str (or (some-> target name) "default")
mdata (cond-> base-mdata
(satisfies? impl/IContentHash content)
(assoc :hash (impl/get-hash content)))
(assoc :hash (impl/get-hash content))
(some? target)
(assoc :storage-target target-str))
touched-at (if touch
(or touched-at (ct/now))
@@ -214,10 +230,11 @@
(not= tempfile-bucket (:bucket mdata)))
(get-database-object-by-hash pool backend
(:bucket mdata)
target-str
(:hash mdata)))]
;; PHASE 2: an existing reference is found: reuse or repair it.
(if (impl/exists-object? backend' hit)
(if (impl/exists-object? backend' (row->storage-object hit))
;; PHASE 2a: healthy reference. Optionally refresh touched_at
;; and reuse the object as it is.
@@ -312,6 +329,16 @@
(ct/is-after? (:expired-at object) (ct/now))))
(-> (impl/get-object-url backend object nil) file-url->path))))
(defn target-resolvable?
"Returns true when the backend referenced by `backend-id` can resolve
`target` to a real destination. GC callers must refuse to delete objects
whose target is not resolvable, so a misconfigured target never removes
the database row while orphaning the blob."
[storage backend-id target]
(assert (valid-storage? storage))
(-> (impl/resolve-backend storage backend-id)
(impl/target-resolvable? target)))
(defn del-object!
[storage object-or-id]
(assert (valid-storage? storage))
+143
View File
@@ -0,0 +1,143 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.storage.config
"Configuration of the optional S3 storage targets and their per
semantic-bucket routing.
The routing is read from an external EDN file referenced by the
`PENPOT_OBJECTS_STORAGE_S3_ROUTES_FILE` environment variable. The file
declares additional S3 targets and, optionally, a map from Penpot
semantic bucket to target id:
{:targets
{:temp {:bucket \"penpot-temp\"}
:cold {:bucket \"penpot-cold\" :region :us-east-1}}
:routes
{\"tempfile\" :temp
\"file-data\" :temp}}
The implicit `:default` target always exists and is built from the
existing `PENPOT_OBJECTS_STORAGE_S3_*` configuration, so the feature is
fully backward compatible when the file is absent."
(:refer-clojure :exclude [load])
(:require
[app.common.exceptions :as ex]
[app.common.schema :as sm]
[app.common.uri :as u]
[app.config :as cf]
[app.storage :as sto]
[app.storage.s3 :as sto.s3]
[clojure.edn :as edn]
[clojure.java.io :as io]))
(def ^:private schema:target
"The EDN declares targets with the same shape as the S3 backend, except
the endpoint is accepted as a plain string and normalized to a URI by
`normalize-target`."
[:merge
sto.s3/schema:target
[:map
[:endpoint {:optional true} [:or :string ::sm/uri]]]])
(def ^:private schema:file
[:map {:title "storage-routes"}
[:targets [:map-of :keyword schema:target]]
[:routes {:optional true} [:map-of :string :keyword]]])
(def ^:private valid-file?
(sm/validator schema:file))
(def ^:private explain-file
(sm/explainer schema:file))
(defn- read-file
[path]
(try
(-> (io/file path) slurp (edn/read-string))
(catch Exception cause
(ex/raise :type :validation
:code :invalid-storage-routes-file
:hint "unable to read storage routes file"
:path (str path)
:cause cause))))
(defn- assert-backend-s3
"The routing only makes sense on top of the S3 backend."
[]
(let [backend (or (sto/get-legacy-backend)
(cf/get :objects-storage-backend)
:fs)]
(when-not (= :s3 backend)
(ex/raise :type :validation
:code :invalid-storage-routes-backend
:hint "storage routes file requires the :s3 backend"
:backend (keyword backend)))))
(defn- validate
[data path]
(when-not (valid-file? data)
(ex/raise :type :validation
:code :invalid-storage-routes
:hint "invalid storage routes file"
:path (str path)
:explain (explain-file data)))
(let [targets (:targets data)
routes (:routes data)]
(when (contains? targets :default)
(ex/raise :type :validation
:code :reserved-storage-target
:hint "`:default` is a reserved storage target id"
:path (str path)))
(doseq [[bucket target] routes]
(when-not (contains? sto/valid-buckets bucket)
(ex/raise :type :validation
:code :invalid-storage-routes-bucket
:hint "unknown semantic bucket in storage routes"
:bucket bucket
:path (str path)))
(when-not (contains? targets target)
(ex/raise :type :validation
:code :unknown-storage-target
:hint "storage route points to an undeclared target"
:bucket bucket
:target target
:path (str path))))
data))
(defn- normalize-target
[target]
(cond-> target
(string? (:endpoint target))
(update :endpoint u/uri)))
(defn- normalize-targets
[targets]
(persistent!
(reduce-kv (fn [acc id target]
(assoc! acc id (normalize-target target)))
(transient {})
targets)))
(defn load
"Reads and validates the optional S3 routing file.
Returns a map with `:targets` (id -> target definition) and `:routes`
(semantic bucket -> target id), or `{:targets nil :routes nil}` when the
configuration key is unset."
[]
(if-let [path (not-empty (cf/get :objects-storage-s3-routes-file))]
(let [data (-> (read-file path) (validate path))]
(assert-backend-s3)
{:targets (normalize-targets (:targets data))
:routes (:routes data)})
{:targets nil
:routes nil}))
+5 -1
View File
@@ -141,7 +141,7 @@
(Files/deleteIfExists ^Path path)))
(defmethod impl/del-objects-in-bulk :fs
[backend ids]
[backend _target ids]
(assert (valid-backend? backend) "expected a valid backend instance")
(let [base (fs/path (::directory backend))]
(reduce (fn [fail-ids id]
@@ -153,3 +153,7 @@
(conj fail-ids id)))))
#{} ids)))
(defmethod impl/target-resolvable? :fs
[_backend _target]
true)
+51 -21
View File
@@ -78,6 +78,24 @@
(let [ids (db/create-array conn "uuid" ids)]
(db/exec-one! conn [sql:delete-give-up ids max-attempts])))
(def ^:private sql:defer-unresolvable
"UPDATE storage_object
SET deleted_at = NOW() + INTERVAL '1 day'
WHERE id = ANY(?::uuid[])")
(defn- park-unresolvable!
"Refuses to delete rows whose target id is not configured: logs the
misconfiguration and pushes `deleted_at` forward so the rows leave the
selection window without being deleted and without counting a deletion
attempt (they are therefore never subject to the give-up window)."
[conn backend-id target ids]
(l/err :hint "storage target is not configured, deletion refused"
:backend (name backend-id)
:target target
:ids (mapv str ids))
(let [ids (db/create-array conn "uuid" ids)]
(db/exec-one! conn [sql:defer-unresolvable ids])))
(defn- process-chunk
"Attempt to delete a chunk of storage objects from a specific backend.
@@ -87,11 +105,11 @@
Returns the number of successfully deleted objects, or 0 if no rows
could be locked."
[conn storage backend-id ids]
[conn storage backend-id target ids]
(if-let [locked-ids (lock-ids conn ids)]
(let [fail-ids (try
(-> (impl/resolve-backend storage backend-id)
(impl/del-objects-in-bulk locked-ids))
(impl/del-objects-in-bulk target locked-ids))
(catch Throwable cause
(l/err :hint "error on physical deletion, will retry"
:ids locked-ids
@@ -118,12 +136,15 @@
(count ok-ids))
0))
(defn- group-by-backend
(defn- group-by-route
[items]
(d/group-by (comp keyword :backend) :id #{} items))
(d/group-by (fn [item]
[(keyword (:backend item)) (:target item)])
:id #{} items))
(def ^:private sql:get-deleted-chunk
"SELECT id, backend
"SELECT id, backend,
coalesce(metadata->>'~:storage-target', 'default') as target
FROM storage_object
WHERE deleted_at IS NOT NULL
AND deleted_at <= ?
@@ -139,19 +160,27 @@
(defn- clean-deleted!
[cfg]
(loop [total 0]
(let [deleted (db/tx-run! cfg
(fn [{:keys [::db/conn ::sto/storage]}]
(let [chunk (get-deleted-chunk conn chunk-size)]
(when (seq chunk)
(let [by-backend (group-by-backend chunk)]
(reduce-kv (fn [acc backend-id ids]
(+ acc (process-chunk conn storage backend-id ids)))
0
by-backend))))))]
(if deleted
(recur (+ total deleted))
total))))
(loop [deleted 0
parked 0]
(let [result (db/tx-run! cfg
(fn [{:keys [::db/conn ::sto/storage]}]
(let [chunk (get-deleted-chunk conn chunk-size)]
(when (seq chunk)
(let [by-route (group-by-route chunk)]
(reduce-kv
(fn [acc [backend-id target] ids]
(if (sto/target-resolvable? storage backend-id target)
(update acc :deleted + (process-chunk conn storage backend-id target ids))
(do
(park-unresolvable! conn backend-id target ids)
(update acc :parked + (count ids)))))
{:deleted 0 :parked 0}
by-route))))))]
(if result
(recur (+ deleted (:deleted result))
(+ parked (:parked result)))
{:deleted deleted
:parked parked}))))
(defmethod ig/assert-key ::handler
[_ params]
@@ -161,6 +190,7 @@
(defmethod ig/init-key ::handler
[_ cfg]
(fn [_]
(let [total (clean-deleted! cfg)]
(l/inf :hint "task finished" :total total)
{:deleted total})))
(let [{:keys [deleted parked]} (clean-deleted! cfg)]
(l/inf :hint "task finished" :total deleted :parked parked)
{:deleted deleted
:parked parked})))
+17 -4
View File
@@ -72,12 +72,14 @@
:context cfg))
(defmulti del-objects-in-bulk
"Delete multiple objects in bulk. Returns #{fail-ids} — the set of ids
whose blob deletion failed. Empty set = all succeeded."
(fn [cfg _] (::sto/type cfg)))
"Delete multiple objects in bulk. `target` is an optional backend-specific
destination id (used by the S3 storage targets); backends that do not route
ignore it. Returns #{fail-ids} — the set of ids whose blob deletion failed.
Empty set = all succeeded."
(fn [cfg _ _] (::sto/type cfg)))
(defmethod del-objects-in-bulk :default
[cfg _]
[cfg _ _]
(ex/raise :type :internal
:code :invalid-storage-backend
:context cfg))
@@ -90,6 +92,17 @@
:code :invalid-storage-backend
:context cfg))
(defmulti target-resolvable?
"Returns true when the backend can resolve `target` to a real destination.
GC callers must refuse deletion (and keep the row) when this is false."
(fn [cfg _] (::sto/type cfg)))
(defmethod target-resolvable? :default
[cfg _]
(ex/raise :type :internal
:code :invalid-storage-backend
:context cfg))
;; --- HELPERS
(defn uuid->hex
+67 -19
View File
@@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.storage.pending-gc
"A maintenance task that reclaims storage objects created in 'pending'
@@ -20,10 +20,12 @@
[integrant.core :as ig]))
(def ^:private sql:get-pending-sobjects
"SELECT id, backend
"SELECT id, backend,
coalesce(metadata->>'~:storage-target', 'default') as target
FROM storage_object
WHERE status = 'pending'
AND created_at <= now() - interval '24 hours'
AND (deleted_at IS NULL OR deleted_at <= now())
ORDER BY created_at ASC
LIMIT ?
FOR UPDATE
@@ -36,30 +38,71 @@
(def ^:private sql:delete-pending-sobject
"DELETE FROM storage_object WHERE id = ? AND status = 'pending'")
(def ^:private sql:park-unresolvable
"UPDATE storage_object
SET deleted_at = NOW() + INTERVAL '1 day'
WHERE id = ANY(?::uuid[])
AND status = 'pending'")
(defn- park-unresolvable!
"Refuses to delete rows whose target id is not configured: logs the
misconfiguration and pushes `deleted_at` forward so the rows are excluded
from the next selection without being deleted."
[conn backend-id target ids]
(l/err :hint "storage target is not configured, deletion refused"
:backend (name backend-id)
:target target
:ids (mapv str ids))
(let [ids (db/create-array conn "uuid" ids)]
(db/exec-one! conn [sql:park-unresolvable ids])))
(def ^:private chunk-size
100)
(defn- group-by-route
[rows]
(group-by (fn [{:keys [backend target]}]
[(keyword backend) target])
rows))
(defn- delete-pending-rows!
"Select, lock and delete a chunk of pending rows in a single transaction.
Returns the deleted rows or nil when there is nothing left to reclaim."
Rows whose target id is not configured are never deleted: they are parked
(error logged, `deleted_at` pushed forward) so the loop terminates and the
rows survive.
Returns a map `{:deleted rows :parked count}`, or nil when there is nothing
left to reclaim."
[cfg]
(db/tx-run! cfg
(fn [{:keys [::db/conn]}]
(fn [{:keys [::db/conn ::sto/storage]}]
;; NOTE: db/exec! returns an empty vector when there are no
;; rows left; use not-empty to detect it.
(when-let [chunk (not-empty (get-pending-chunk conn chunk-size))]
(doseq [{:keys [id]} chunk]
(db/exec-one! conn [sql:delete-pending-sobject id]))
chunk))))
(reduce-kv
(fn [acc [backend-id target] rows]
(if (sto/target-resolvable? storage backend-id target)
(do
(doseq [{:keys [id]} rows]
(db/exec-one! conn [sql:delete-pending-sobject id]))
(update acc :deleted into rows))
(do
(park-unresolvable! conn backend-id target (mapv :id rows))
(update acc :parked + (count rows)))))
{:deleted [] :parked 0}
(group-by-route chunk))))))
(defn- delete-blobs!
"Best-effort removal of the orphaned blobs. Runs after the pending rows
have been committed so a failure here never blocks their reclamation."
have been committed so a failure here never blocks their reclamation.
The `:storage-target` metadata is load-bearing: the S3 backend reads it
(`s3/target-id`) to resolve the bucket/prefix/client for `del-object`."
[storage rows]
(doseq [{:keys [id backend]} rows]
(doseq [{:keys [id backend target]} rows]
(try
(-> (impl/resolve-backend storage (keyword backend))
(impl/del-object {:id id}))
(impl/del-object (with-meta {:id id} {:storage-target target})))
(catch Throwable cause
(l/err :hint "error deleting orphaned pending blob"
:id (str id)
@@ -68,12 +111,16 @@
(defn- process!
[{::sto/keys [storage] :as cfg}]
(loop [total 0]
(if-let [rows (delete-pending-rows! cfg)]
(do
(delete-blobs! storage rows)
(recur (long (+ total (count rows)))))
total)))
(loop [total-deleted 0
total-parked 0]
(if-let [result (delete-pending-rows! cfg)]
(let [removed (:deleted result)
parked (:parked result)]
(delete-blobs! storage removed)
(recur (long (+ total-deleted (count removed)))
(long (+ total-parked parked))))
{:processed total-deleted
:parked total-parked})))
(defmethod ig/assert-key ::handler
[_ params]
@@ -83,6 +130,7 @@
(defmethod ig/init-key ::handler
[_ cfg]
(fn [_]
(let [total (process! cfg)]
(l/inf :hint "task finished" :total total)
{:processed total})))
(let [{:keys [processed parked]} (process! cfg)]
(l/inf :hint "task finished" :total processed :parked parked)
{:processed processed
:parked parked})))
+136 -32
View File
@@ -88,13 +88,21 @@
;; --- BACKEND INIT
(def schema:target
[:map {:title "s3-target"}
[:bucket ::sm/text]
[:region {:optional true} :keyword]
[:endpoint {:optional true} ::sm/uri]
[:prefix {:optional true} ::sm/text]])
(def ^:private schema:config
[:map {:title "s3-backend-config"}
::wrk/netty-io-executor
[::region {:optional true} :keyword]
[::bucket {:optional true} ::sm/text]
[::prefix {:optional true} ::sm/text]
[::endpoint {:optional true} ::sm/uri]])
[::endpoint {:optional true} ::sm/uri]
[::targets {:optional true} [:map-of :keyword schema:target]]])
(defmethod ig/expand-key ::backend
[k v]
@@ -104,42 +112,131 @@
[_ params]
(assert (sm/check schema:config params)))
(defn- build-client-pair
[{:keys [::wrk/netty-io-executor]} region endpoint]
(let [params {::region region
::endpoint endpoint
::wrk/netty-io-executor netty-io-executor}
client (build-s3-client params)
presigner (build-s3-presigner params)]
{:client @client
:presigner presigner
:close-fn #(.close ^java.lang.AutoCloseable client)}))
(defn- build-client-pair-or-cleanup
"Builds a client pair, closing the pairs already built when the build
fails so a failed backend init does not leak clients."
[acc params region endpoint]
(try
(build-client-pair params region endpoint)
(catch Throwable cause
(doseq [f (:close-fns acc)]
(ex/ignoring (f)))
(throw cause))))
(defn- build-targets
"Resolves the implicit `:default` target plus the declared targets, sharing
one S3 client/presigner pair per distinct `[region endpoint]`."
[{:keys [::region ::endpoint ::bucket ::prefix ::targets] :as params}]
(let [defs (merge {:default {:region region :endpoint endpoint
:bucket bucket :prefix prefix}}
(into {}
(map (fn [[id target]]
[id {:region (or (:region target) region)
:endpoint (or (:endpoint target) endpoint)
:bucket (:bucket target)
:prefix (or (:prefix target) prefix)}]))
targets))
result (reduce-kv
(fn [acc id {:keys [region endpoint bucket prefix]}]
(let [k [region endpoint]
pair (or (get-in acc [:pairs k])
(build-client-pair-or-cleanup acc params region endpoint))
acc (cond-> acc
(nil? (get-in acc [:pairs k]))
(-> (assoc-in [:pairs k] pair)
(update :close-fns conj (:close-fn pair))))]
(assoc-in acc [:targets id]
{::client (:client pair)
::presigner (:presigner pair)
::bucket bucket
::prefix prefix})))
{:pairs {} :targets {} :close-fns []}
defs)]
(select-keys result [:targets :close-fns])))
(defmethod ig/init-key ::backend
[_ params]
(when (and (contains? params ::region)
(contains? params ::bucket))
(let [client (build-s3-client params)
presigner (build-s3-presigner params)]
(let [{:keys [targets close-fns]} (build-targets params)]
(assoc params
::sto/type :s3
::counter (AtomicLong. 0)
::client @client
::presigner presigner
::close-fn #(.close ^java.lang.AutoCloseable client)))))
::default-target :default
::targets targets
::close-fns (vec close-fns)))))
(defmethod ig/resolve-key ::backend
[_ params]
(dissoc params ::close-fn))
(dissoc params ::close-fns))
(defmethod ig/halt-key! ::backend
[_ {:keys [::close-fn]}]
(when (fn? close-fn)
(close-fn)))
[_ {:keys [::close-fns]}]
(doseq [f close-fns]
(when (fn? f)
(f))))
(def ^:private schema:backend
[:map {:title "s3-backend"}
;; [::region :keyword]
;; [::bucket ::sm/text]
[::client [:fn #(instance? S3AsyncClient %)]]
[::presigner [:fn #(instance? S3Presigner %)]]
[::prefix {:optional true} ::sm/text]
#_[::sto/type [:= :s3]]])
[::default-target :keyword]
[::targets
[:map-of :keyword
[:map
[::client [:fn #(instance? S3AsyncClient %)]]
[::presigner [:fn #(instance? S3Presigner %)]]
[::bucket ::sm/text]
[::prefix {:optional true} ::sm/text]]]]])
(sm/register! ::backend schema:backend)
(def ^:private valid-backend?
(sm/validator schema:backend))
;; --- TARGET RESOLUTION
(defn- target-id
"Returns the configured target id stored on the object, or the default
target name when the object predates the routing feature."
[backend object]
(or (some-> (meta object) :storage-target)
(name (::default-target backend))))
(defn- resolve-target-by-id
[backend target-id]
(let [tid (cond
(nil? target-id) (::default-target backend)
(keyword? target-id) target-id
:else (keyword target-id))]
(or (get (::targets backend) tid)
(ex/raise :type :internal
:code :invalid-storage-target
:hint "storage target is not configured"
:target target-id
:available (vec (keys (::targets backend)))))))
(defn- resolve-target
[backend object]
(resolve-target-by-id backend (target-id backend object)))
(defmethod impl/target-resolvable? :s3
[backend target-id]
(let [tid (cond
(nil? target-id) (::default-target backend)
(keyword? target-id) target-id
:else (keyword target-id))]
(contains? (::targets backend) tid)))
;; --- API IMPL
(defmethod impl/put-object :s3
@@ -210,13 +307,14 @@
true)))
(defmethod impl/del-objects-in-bulk :s3
[backend ids]
[backend target ids]
(assert (valid-backend? backend) "expected a valid backend instance")
(let [key->id (into {} (map (fn [id]
[(str (::prefix backend) (impl/id->path id)) id]))
(let [target (resolve-target-by-id backend target)
key->id (into {} (map (fn [id]
[(str (::prefix target) (impl/id->path id)) id]))
ids)
result (try
(p/await! (del-object-in-bulk backend ids))
(p/await! (del-object-in-bulk target ids))
(catch Throwable cause
(l/err :hint "error on s3 bulk deletion"
:ids ids
@@ -320,8 +418,9 @@
^Subscriber subscriber))))))
(defn- put-object
[{:keys [::client ::bucket ::prefix ::counter]} {:keys [id] :as object} content]
(let [path (dm/str prefix (impl/id->path id))
[{:keys [::counter] :as backend} {:keys [id] :as object} content]
(let [{:keys [::client ::bucket ::prefix]} (resolve-target backend object)
path (dm/str prefix (impl/id->path id))
mdata (meta object)
mtype (:content-type mdata "application/octet-stream")
rbody (make-request-body counter content)
@@ -344,8 +443,9 @@
(proxy-super close))))
(defn- get-object-data
[{:keys [::client ::bucket ::prefix]} {:keys [id size]}]
(let [gor (.. (GetObjectRequest/builder)
[backend {:keys [id size] :as object}]
(let [{:keys [::client ::bucket ::prefix]} (resolve-target backend object)
gor (.. (GetObjectRequest/builder)
(bucket bucket)
(key (str prefix (impl/id->path id)))
(build))]
@@ -369,16 +469,18 @@
(p/fmap #(.asInputStream ^ResponseBytes %)))))))
(defn- head-object
[{:keys [::client ::bucket ::prefix]} {:keys [id]}]
(let [hor (.. (HeadObjectRequest/builder)
[backend {:keys [id] :as object}]
(let [{:keys [::client ::bucket ::prefix]} (resolve-target backend object)
hor (.. (HeadObjectRequest/builder)
(bucket bucket)
(key (str prefix (impl/id->path id)))
(build))]
(.headObject ^S3AsyncClient client ^HeadObjectRequest hor)))
(defn- get-object-bytes
[{:keys [::client ::bucket ::prefix]} {:keys [id]}]
(let [gor (.. (GetObjectRequest/builder)
[backend {:keys [id] :as object}]
(let [{:keys [::client ::bucket ::prefix]} (resolve-target backend object)
gor (.. (GetObjectRequest/builder)
(bucket bucket)
(key (str prefix (impl/id->path id)))
(build))
@@ -392,7 +494,7 @@
(ct/duration {:minutes 10}))
(defn- get-object-url
[{:keys [::presigner ::bucket ::prefix]} {:keys [id]}
[backend {:keys [id] :as object}
{:keys [max-age content-disposition] :or {max-age default-max-age}}]
(assert (ct/duration? max-age) "expected valid duration instance")
@@ -400,7 +502,8 @@
;; object store sets that header on the response the client fetches after
;; following the redirect. It is only set when asked for, so urls for
;; objects served inline stay byte identical to before.
(let [gorb (.. (GetObjectRequest/builder)
(let [{:keys [::presigner ::bucket ::prefix]} (resolve-target backend object)
gorb (.. (GetObjectRequest/builder)
(bucket bucket)
(key (dm/str prefix (impl/id->path id))))
gorb (cond-> gorb
@@ -415,8 +518,9 @@
(u/uri (str (.url ^PresignedGetObjectRequest pgor)))))
(defn- del-object
[{:keys [::bucket ::client ::prefix]} {:keys [id] :as obj}]
(let [dor (.. (DeleteObjectRequest/builder)
[backend {:keys [id] :as object}]
(let [{:keys [::bucket ::client ::prefix]} (resolve-target backend object)
dor (.. (DeleteObjectRequest/builder)
(bucket bucket)
(key (dm/str prefix (impl/id->path id)))
(build))]
+1 -1
View File
@@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.tasks.demo-purge
"Task handler for delayed demo profile deletion. Submitted at demo
@@ -0,0 +1,76 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns backend-tests.auth-ldap-test
(:require
[app.auth.ldap :as ldap-auth]
[clj-ldap.client :as ldap]
[clojure.test :as t]))
;; --- search-user: filter must be escaped (RED: currently not escaped)
(t/deftest search-user-escapes-email-in-filter
(t/testing "wildcard * is escaped before building LDAP filter"
(let [captured-query (atom nil)
fake-search (fn [_conn _base-dn params]
(reset! captured-query (:filter params))
[])]
(with-redefs [ldap/search fake-search]
(#'ldap-auth/search-user {:query "(mail=:username)" :sizelimit 1
:attrs-username "uid" :attrs-email "mail"
:attrs-fullname "cn"}
"fry*@planetexpress.com"))
;; After fix: * should be escaped as \2a
(t/is (= "(mail=fry\\2a@planetexpress.com)" @captured-query)
"filter must have * escaped per RFC 4515"))))
;; --- retrieve-user: email must come from directory, not client (RED)
(t/deftest retrieve-user-uses-directory-email
(t/testing "returned email is from LDAP directory, not client input"
(let [fake-search (fn [_conn _base-dn _params]
[{:dn "cn=fry,ou=people,dc=planetexpress,dc=com"
:mail "fry@planetexpress.com"
:cn "Philip J. Fry"
:uid "fry"}])
fake-bind? (fn [_conn _dn _password] true)]
(with-redefs [ldap/search fake-search
ldap/bind? fake-bind?]
(let [cfg {:query "(mail=:username)" :sizelimit 1
:attrs-username "uid" :attrs-email "mail"
:attrs-fullname "cn"}
result (#'ldap-auth/retrieve-user cfg {:email "fry*@planetexpress.com" :password "fry"})]
;; After fix: email should be from directory (fry@planetexpress.com)
;; BUG: email is client input (fry*@planetexpress.com)
(t/is (= "fry@planetexpress.com" (:email result))
"email must come from LDAP directory attribute, not client input"))))))
;; --- authenticate: full flow with directory email (RED)
(t/deftest authenticate-returns-directory-email
(t/testing "authenticate returns directory email for profile"
(let [fake-search (fn [_conn _base-dn _params]
[{:dn "cn=amy,ou=people,dc=planetexpress,dc=com"
:mail "amy@planetexpress.com"
:cn "Amy Wong"
:uid "amy"}])
fake-bind? (fn [_conn _dn _password] true)]
(with-redefs [ldap/search fake-search
ldap/bind? fake-bind?
ldap/connect (fn [_cfg] (reify java.lang.AutoCloseable (close [_] nil)))]
(let [cfg {:query "(mail=:username)" :sizelimit 1
:attrs-username "uid" :attrs-email "mail"
:attrs-fullname "cn"
:bind-dn "cn=admin,dc=planetexpress,dc=com"
:bind-password "GoodNewsEveryone"
:host "localhost" :port 10389
:ssl false :tls false
:base-dn "ou=people,dc=planetexpress,dc=com"}
result (ldap-auth/authenticate cfg {:email "*@planetexpress.com" :password "amy"})]
;; After fix: email should be amy@planetexpress.com (directory)
;; BUG: email is *@planetexpress.com (client)
(t/is (= "amy@planetexpress.com" (:email result))
"authenticate must return directory email, not client-supplied wildcard"))))))
+1 -1
View File
@@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns backend-tests.demo-test
(:require
@@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns backend-tests.graph-binder-gate-test
"Binder gate for the incremental-sync statement templates.
@@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns backend-tests.graph-sync-parity-test
"Cold projection and incremental sync are two implementations of one mapping,
@@ -0,0 +1,30 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns backend-tests.http-client-test
(:require
[app.http.client :as http]
[clojure.test :as t]
[java-http-clj.core :as jhttp]
[mockery.core :refer [with-mocks]]))
(t/deftest send-injects-default-timeout-when-absent
(with-mocks [mock {:target 'java-http-clj.core/send
:return {:status 200 :body ""}}]
(let [client (jhttp/build-client {})]
(http/send! client {:method :get :uri "https://example.com/"})
(let [[req _opts] (:call-args @mock)]
(t/is (= http/default-request-timeout (:timeout req)))))))
(t/deftest send-preserves-caller-supplied-timeout
(with-mocks [mock {:target 'java-http-clj.core/send
:return {:status 200 :body ""}}]
(let [client (jhttp/build-client {})]
(http/send! client {:method :get
:uri "https://example.com/"
:timeout 5000})
(let [[req _opts] (:call-args @mock)]
(t/is (= 5000 (:timeout req)))))))
@@ -8,6 +8,7 @@
(:require
[app.common.exceptions :as ex]
[app.config :as cf]
[app.http.client :as http]
[app.media.remote :as media.remote]
[app.setup :as-alias setup]
[app.util.json :as json]
@@ -500,6 +501,22 @@
:headers {}})]
(t/is (= 200 (:status resp))))))))
(t/deftest service-request-puts-configured-timeout-in-request
(t/testing "service-request puts media-processing-service-timeout on the http request"
(let [captured (atom nil)]
(with-redefs [cf/get (th/config-get-mock config-mock)
http/req (fn [_client request _opts]
(reset! captured request)
{:status 200
:body (json-stream {:width 100 :height 100})})]
(media.remote/service-request
(mk-system)
{:method :post
:uri "http://localhost:6065/api/image/info"
:body nil
:headers {}})
(t/is (= 5000 (:timeout @captured)))))))
;; ---------------------------------------------------------------------------
;; Shared key
;; ---------------------------------------------------------------------------
@@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns backend-tests.passwords-test
(:require
+1 -1
View File
@@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns backend-tests.rpc-demo-test
(:require
@@ -17,6 +17,7 @@
[app.msgbus :as mbus]
[app.nitrate :as nitrate]
[app.rpc :as-alias rpc]
[app.util.ssrf :as ssrf]
[app.worker :as wrk]
[backend-tests.helpers :as th]
[clojure.set :as set]
@@ -1806,13 +1807,14 @@
(t/deftest check-organization-sso-returns-valid-true
(let [organization-id (uuid/random)
out (with-redefs [oidc/is-organization-sso-config-valid? (constantly true)]
(th/management-command!
{::th/type :check-organization-sso
:organization-id organization-id
:client-id "test-client"
:client-secret "test-secret"
:issuer "https://idp.example.com"}))]
out (with-redefs [ssrf/safe-url? (constantly true)
oidc/is-organization-sso-config-valid? (constantly true)]
(th/management-command!
{::th/type :check-organization-sso
:organization-id organization-id
:client-id "test-client"
:client-secret "test-secret"
:issuer "https://idp.example.com"}))]
(t/is (th/success? out))
(t/is (true? (-> out :result :valid)))))
@@ -1827,19 +1829,36 @@
(t/deftest check-organization-sso-passes-issuer-to-validation
(let [organization-id (uuid/random)
out (with-redefs [oidc/is-organization-sso-config-valid?
(fn [_cfg sso]
(and (= "test-client" (:client-id sso))
(= "https://idp.example.com/" (:issuer sso))))]
(th/management-command!
{::th/type :check-organization-sso
:organization-id organization-id
:client-id "test-client"
:client-secret "test-secret"
:issuer "https://idp.example.com/"}))]
out (with-redefs [ssrf/safe-url? (constantly true)
oidc/is-organization-sso-config-valid?
(fn [_cfg sso]
(and (= "test-client" (:client-id sso))
(= "https://idp.example.com/" (:issuer sso))))]
(th/management-command!
{::th/type :check-organization-sso
:organization-id organization-id
:client-id "test-client"
:client-secret "test-secret"
:issuer "https://idp.example.com/"}))]
(t/is (th/success? out))
(t/is (true? (-> out :result :valid)))))
(t/deftest check-organization-sso-returns-valid-false-on-ssrf-blocked-issuer
(t/testing "an SSRF-blocked issuer must not reach the OIDC validation flow"
(let [called? (atom false)
out (with-redefs [oidc/is-organization-sso-config-valid?
(fn [_cfg _sso] (reset! called? true) true)]
(th/management-command!
{::th/type :check-organization-sso
:organization-id (uuid/random)
:client-id "test-client"
:client-secret "test-secret"
:issuer "http://127.0.0.1/idp"}))]
(t/is (th/success? out))
(t/is (false? (-> out :result :valid)))
(t/is (false? @called?)
"OIDC validation should not run when the issuer is SSRF-blocked"))))
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
;; PUSH AUDIT EVENTS
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
@@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns backend-tests.rpc-plugins-test
(:require
@@ -0,0 +1,97 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns backend-tests.storage-config-test
(:require
[app.common.uri :as u]
[app.config :as cf]
[app.storage.config :as sto.config]
[clojure.java.io :as io]
[clojure.test :as t]))
(defn- write-routes!
[content]
(let [file (java.io.File/createTempFile "storage-routes" ".edn")]
(spit file content)
file))
(defn- delete!
[file]
(io/delete-file file true))
(defn- load-with
[config]
(binding [cf/config (merge cf/config
{:objects-storage-backend :s3}
config)]
(sto.config/load)))
(defn- load-error
"Runs `load` against a routes file with `content` and returns the raised
throwable, or nil when it succeeds."
[content config]
(let [file (write-routes! content)]
(try
(try
(load-with (assoc config :objects-storage-s3-routes-file (.getAbsolutePath file)))
nil
(catch Throwable cause cause))
(finally
(delete! file)))))
(t/deftest returns-empty-when-unset
(t/is (= {:targets nil :routes nil}
(load-with {:objects-storage-s3-routes-file nil}))))
(t/deftest loads-and-normalizes-routes
(let [file (write-routes!
"{:targets {:temp {:bucket \"penpot-temp\" :endpoint \"https://s3.example.com\"}} :routes {\"tempfile\" :temp}}")]
(try
(let [result (load-with {:objects-storage-s3-routes-file (.getAbsolutePath file)})]
(t/is (= #{:temp} (set (keys (:targets result)))))
(t/is (= "penpot-temp" (get-in result [:targets :temp :bucket])))
(t/is (u/uri? (get-in result [:targets :temp :endpoint])))
(t/is (= {"tempfile" :temp} (:routes result))))
(finally
(delete! file)))))
(t/deftest rejects-reserved-default-target
(t/is (some? (load-error "{:targets {:default {:bucket \"x\"}}}"
{}))))
(t/deftest rejects-unknown-route-target
(t/is (some? (load-error
"{:targets {:temp {:bucket \"x\"}} :routes {\"tempfile\" :missing}}"
{}))))
(t/deftest rejects-invalid-semantic-bucket
(t/is (some? (load-error
"{:targets {:temp {:bucket \"x\"}} :routes {\"not-a-bucket\" :temp}}"
{}))))
(t/deftest rejects-unreadable-file
(t/is (some? (load-error "{:targets" {}))))
(t/deftest rejects-routing-when-backend-is-not-s3
(t/is (some? (load-error
"{:targets {:temp {:bucket \"x\"}} :routes {\"tempfile\" :temp}}"
{:objects-storage-backend :fs}))))
(t/deftest rejects-route-pointing-to-default
(let [err (load-error
"{:targets {:temp {:bucket \"x\"}} :routes {\"tempfile\" :default}}"
{})]
(t/is (some? err))
(t/is (= :unknown-storage-target (:code (ex-data err))))))
(t/deftest loads-empty-targets-map
(let [file (write-routes! "{:targets {}}")]
(try
(let [result (load-with {:objects-storage-s3-routes-file (.getAbsolutePath file)})]
(t/is (= {} (:targets result)))
(t/is (nil? (:routes result))))
(finally
(delete! file)))))
+462 -4
View File
@@ -13,13 +13,16 @@
[app.rpc :as-alias rpc]
[app.storage :as sto]
[app.storage.fs :as-alias sto.fs]
[app.storage.gc-deleted :as sto.gc-deleted]
[app.storage.impl :as impl]
[app.storage.pending-gc :as sto.pending-gc]
[app.storage.s3 :as-alias sto.s3]
[backend-tests.helpers :as th]
[clojure.test :as t]
[cuerdas.core :as str]
[datoteka.fs :as fs]
[datoteka.io :as io]
[integrant.core :as ig]
[mockery.core :refer [with-mocks]]
[promesa.core :as p])
(:import
@@ -41,6 +44,17 @@
[storage]
(assoc storage ::sto/backend :fs))
(declare fake-s3-backend-with-targets)
(defn configure-s3-storage
"Returns a storage configured with the fake S3 backend and the given
semantic-bucket -> target routing."
[bucket->target]
(-> (:app.storage/storage th/*system*)
(assoc ::sto/backend :s3)
(assoc-in [::sto/backends :s3] (fake-s3-backend-with-targets))
(assoc ::sto/bucket->target bucket->target)))
(t/deftest put-and-retrieve-object
(let [storage (-> (:app.storage/storage th/*system*)
(configure-storage-backend))
@@ -708,7 +722,7 @@
{:id (:id object)})
(with-mocks [_mock {:target 'app.storage.impl/del-objects-in-bulk
:return (fn [_ ids] (set ids))}]
:return (fn [_ _ ids] (set ids))}]
(let [res (th/run-task! :storage-gc-deleted {})]
(t/is (= 0 (:deleted res)))))
@@ -832,9 +846,145 @@
(defn- fake-s3-backend
[]
{::sto/type :s3
::sto.s3/client (reify S3AsyncClient)
::sto.s3/presigner (reify S3Presigner)})
{::sto/type :s3
::sto.s3/default-target :default
::sto.s3/targets
{:default {::sto.s3/client (reify S3AsyncClient)
::sto.s3/presigner (reify S3Presigner)
::sto.s3/bucket "test-bucket"
::sto.s3/prefix nil}}})
(defn- fake-s3-target
[bucket prefix]
{::sto.s3/client (reify S3AsyncClient)
::sto.s3/presigner (reify S3Presigner)
::sto.s3/bucket bucket
::sto.s3/prefix prefix})
(defn- fake-s3-backend-with-targets
[]
(assoc (fake-s3-backend)
::sto.s3/targets
{:default (fake-s3-target "default" nil)
:temp (fake-s3-target "temp" "tmp/")}))
(t/deftest s3-bulk-delete-resolves-target
(let [backend (fake-s3-backend-with-targets)
captured (atom nil)]
(with-mocks [_mock {:target 'app.storage.s3/del-object-in-bulk
:return (fn [target _ids]
(reset! captured target)
(p/resolved nil))}]
(t/is (= #{} (impl/del-objects-in-bulk backend :temp #{(uuid/next)})))
(t/is (= "temp" (::sto.s3/bucket @captured)))
(t/is (= "tmp/" (::sto.s3/prefix @captured))))))
(t/deftest s3-bulk-delete-rejects-unknown-target
(let [backend (fake-s3-backend-with-targets)
captured (atom nil)]
(with-mocks [_mock {:target 'app.storage.s3/del-object-in-bulk
:return (fn [target _ids]
(reset! captured target)
(p/resolved nil))}]
(let [ex (try
(impl/del-objects-in-bulk backend :missing #{(uuid/next)})
nil
(catch Throwable cause cause))]
(t/is (some? ex))
(t/is (= :invalid-storage-target (:code (ex-data ex))))
(t/is (nil? @captured))))))
(t/deftest s3-get-object-url-rejects-unknown-target
(let [backend (fake-s3-backend-with-targets)
ex (try
(impl/get-object-url backend (with-meta {:id (uuid/next)}
{:storage-target "ghost"}) {})
nil
(catch Throwable cause cause))]
(t/is (some? ex))
(t/is (= :invalid-storage-target (:code (ex-data ex))))))
(t/deftest s3-get-object-data-rejects-unknown-target
(let [backend (fake-s3-backend-with-targets)
ex (try
(impl/get-object-data backend (with-meta {:id (uuid/next) :size 1}
{:storage-target "ghost"}))
nil
(catch Throwable cause cause))]
(t/is (some? ex))
(t/is (= :invalid-storage-target (:code (ex-data ex))))))
(t/deftest s3-target-resolvable-checks-configured-targets
(let [backend (fake-s3-backend-with-targets)]
(t/is (true? (impl/target-resolvable? backend nil)))
(t/is (true? (impl/target-resolvable? backend "default")))
(t/is (true? (impl/target-resolvable? backend :temp)))
(t/is (false? (impl/target-resolvable? backend "ghost")))))
(t/deftest fs-target-is-always-resolvable
(t/is (true? (impl/target-resolvable? {::sto/type :fs} "anything"))))
(t/deftest s3-build-targets-shares-clients-and-closes-them
(let [clients (atom [])
close-count (atom 0)
mk-client (fn [_]
(let [c (reify
clojure.lang.IDeref
(deref [_] (Object.))
java.lang.AutoCloseable
(close [_] (swap! close-count inc)))]
(swap! clients conj c)
c))]
(with-mocks [_c {:target 'app.storage.s3/build-s3-client
:return mk-client}
_p {:target 'app.storage.s3/build-s3-presigner
:return (fn [_] (reify S3Presigner))}]
(let [backend (ig/init-key :app.storage.s3/backend
{:app.storage.s3/region :eu-central-1
:app.storage.s3/bucket "main"
:app.worker/netty-io-executor :executor
:app.storage.s3/targets
{:same {:bucket "same"}
:other {:bucket "other" :region :us-east-1}}})]
;; one client pair per distinct [region endpoint]
(t/is (= 2 (count @clients)))
;; same region reuses the default client
(t/is (= (get-in backend [:app.storage.s3/targets :default :app.storage.s3/client])
(get-in backend [:app.storage.s3/targets :same :app.storage.s3/client])))
;; different region gets its own client
(t/is (not= (get-in backend [:app.storage.s3/targets :default :app.storage.s3/client])
(get-in backend [:app.storage.s3/targets :other :app.storage.s3/client])))
(ig/halt-key! :app.storage.s3/backend backend)
(t/is (= 2 @close-count))))))
(t/deftest s3-build-targets-closes-clients-on-failure
(let [close-count (atom 0)
mk-client (fn [params]
(when (= :us-east-1 (:app.storage.s3/region params))
(throw (RuntimeException. "boom")))
(reify
clojure.lang.IDeref
(deref [_] (Object.))
java.lang.AutoCloseable
(close [_] (swap! close-count inc))))]
(with-mocks [_c {:target 'app.storage.s3/build-s3-client
:return mk-client}
_p {:target 'app.storage.s3/build-s3-presigner
:return (fn [_] (reify S3Presigner))}]
(let [ex (try
(ig/init-key :app.storage.s3/backend
{:app.storage.s3/region :eu-central-1
:app.storage.s3/bucket "main"
:app.worker/netty-io-executor :executor
:app.storage.s3/targets
{:same {:bucket "same"}
:other {:bucket "other" :region :us-east-1}}})
nil
(catch Throwable cause cause))]
(t/is (some? ex))
;; the pair built before the failing one is closed
(t/is (= 1 @close-count))))))
(t/deftest s3-exists-object-returns-true-on-found
(with-mocks [mock {:target 'app.storage.s3/head-object
@@ -874,3 +1024,311 @@
(t/is (= "boom" (ex-message (ex-cause ex)))))
;; one initial attempt plus max-retries
(t/is (= 4 (:call-count @mock)))))
;; --- Storage target metadata / dedup
(t/deftest put-object-routes-bucket-to-storage-target
(let [storage (configure-s3-storage {"tempfile" :temp})]
(with-mocks [_mock {:target 'app.storage.impl/put-object
:return (fn [_ object _] object)}]
(let [object (sto/put-object! storage {::sto/content (sto/content "content")
:bucket "tempfile"
:content-type "text/plain"})
row (th/db-exec-one!
["select backend, metadata->>'~:storage-target' as target
from storage_object where id = ?" (:id object)])]
(t/is (= "s3" (:backend row)))
(t/is (= "temp" (:target row)))))))
(t/deftest put-object-falls-back-to-default-storage-target
(let [storage (configure-s3-storage {"tempfile" :temp})]
(with-mocks [_mock {:target 'app.storage.impl/put-object
:return (fn [_ object _] object)}]
(let [object (sto/put-object! storage {::sto/content (sto/content "content")
:bucket "file-media-object"
:content-type "text/plain"})
row (th/db-exec-one!
["select metadata->>'~:storage-target' as target
from storage_object where id = ?" (:id object)])]
(t/is (= "default" (:target row)))))))
(t/deftest put-object-fs-does-not-set-storage-target
(let [storage (configure-storage-backend (:app.storage/storage th/*system*))
object (sto/put-object! storage {::sto/content (sto/content "content")
:bucket "file-media-object"
:content-type "text/plain"})
row (th/db-exec-one!
["select metadata->>'~:storage-target' as target
from storage_object where id = ?" (:id object)])]
(t/is (nil? (:target row)))))
(t/deftest dedup-is-isolated-per-storage-target
(let [routed (configure-s3-storage {"file-data" :temp})
unrouted (configure-s3-storage {})
content (-> (sto/content "content")
(sto/wrap-with-hash "same-hash"))]
(with-mocks [_mock {:target 'app.storage.impl/put-object
:return (fn [_ object _] object)}]
(let [object1 (sto/put-object! routed {::sto/content content
::sto/deduplicate? true
:bucket "file-data"
:content-type "text/plain"})
object2 (sto/put-object! unrouted {::sto/content content
::sto/deduplicate? true
:bucket "file-data"
:content-type "text/plain"})
row (th/db-exec-one! ["select count(*) from storage_object"])]
;; same semantic bucket, different target: no dedup hit, two rows
(t/is (not= (:id object1) (:id object2)))
(t/is (= 2 (:count row)))))))
(t/deftest dedup-reuses-object-within-same-storage-target
(let [storage (configure-s3-storage {"file-data" :temp})
content (-> (sto/content "content")
(sto/wrap-with-hash "same-hash"))]
(with-mocks [_p {:target 'app.storage.impl/put-object
:return (fn [_ object _] object)}
_e {:target 'app.storage.impl/exists-object?
:return (fn [_ _] true)}]
(let [object1 (sto/put-object! storage {::sto/content content
::sto/deduplicate? true
:bucket "file-data"
:content-type "text/plain"})
object2 (sto/put-object! storage {::sto/content content
::sto/deduplicate? true
:bucket "file-data"
:content-type "text/plain"})]
;; same semantic bucket and target: dedup reuses the object
(t/is (= (:id object1) (:id object2)))))))
(t/deftest dedup-hit-carries-storage-target-metadata
(let [storage (configure-s3-storage {"file-data" :temp})
content (-> (sto/content "content")
(sto/wrap-with-hash "same-hash"))
captured (atom nil)]
(with-mocks [_p {:target 'app.storage.impl/put-object
:return (fn [_ object _] object)}
_e {:target 'app.storage.impl/exists-object?
:return (fn [_ object]
(reset! captured object)
true)}]
(sto/put-object! storage {::sto/content content
::sto/deduplicate? true
:bucket "file-data"
:content-type "text/plain"})
(sto/put-object! storage {::sto/content content
::sto/deduplicate? true
:bucket "file-data"
:content-type "text/plain"})
(t/is (some? @captured))
(t/is (= "temp" (:storage-target (meta @captured)))))))
(t/deftest dedup-repair-carries-storage-target-metadata
(let [storage (configure-s3-storage {"file-data" :temp})
content (-> (sto/content "content")
(sto/wrap-with-hash "same-hash"))
calls (atom [])]
(with-mocks [_p {:target 'app.storage.impl/put-object
:return (fn [_ object _]
(swap! calls conj object)
object)}
_h {:target 'app.storage.s3/head-object
:return (p/rejected (-> (NoSuchKeyException/builder)
(.message "no key")
(.build)))}]
(let [object1 (sto/put-object! storage {::sto/content content
::sto/deduplicate? true
:bucket "file-data"
:content-type "text/plain"})
;; second put finds the row but the real exists-object? sees a
;; missing blob and repairs it in place
object2 (sto/put-object! storage {::sto/content content
::sto/deduplicate? true
:bucket "file-data"
:content-type "text/plain"})
row (th/db-exec-one!
["select status from storage_object where id = ?" (:id object1)])
count (th/db-exec-one! ["select count(*) from storage_object"])]
(t/is (= (:id object1) (:id object2)))
(t/is (= "valid" (:status row)))
(t/is (= 1 (:count count)))
(t/is (= "temp" (:storage-target (meta (last @calls)))))))))
;; --- GC target routing
(defn- storage-with-s3-targets
[]
(assoc (:app.storage/storage th/*system*)
::sto/backends
{:s3 (fake-s3-backend-with-targets)}))
(t/deftest gc-deleted-deletes-from-routed-target
(let [storage (storage-with-s3-targets)
cfg {::db/pool th/*pool* ::sto/storage storage}
id (uuid/next)
captured (atom nil)]
(th/db-exec! ["insert into storage_object (id, size, backend, metadata, deleted_at, status)
values (?, 1, 's3', ?, ?, 'valid')"
id
(db/tjson {:bucket "file-data" :storage-target "temp"})
(ct/in-past {:minutes 1})])
(with-mocks [_mock {:target 'app.storage.impl/del-objects-in-bulk
:return (fn [_ target _ids]
(reset! captured target)
#{})}]
(t/is (= 1 (:deleted (#'sto.gc-deleted/clean-deleted! cfg))))
(t/is (= "temp" @captured)))))
(t/deftest gc-deleted-refuses-unknown-target-and-keeps-row
(let [storage (storage-with-s3-targets)
cfg {::db/pool th/*pool* ::sto/storage storage}
id (uuid/next)]
(th/db-exec! ["insert into storage_object (id, size, backend, metadata, deleted_at, status)
values (?, 1, 's3', ?, ?, 'valid')"
id
(db/tjson {:bucket "file-data" :storage-target "ghost"})
(ct/in-past {:minutes 1})])
(with-mocks [mock {:target 'app.storage.impl/del-objects-in-bulk
:return (fn [_ _ _] #{})}]
(let [result (#'sto.gc-deleted/clean-deleted! cfg)
row (th/db-exec-one!
["select status, deleted_at, deletion_attempts
from storage_object where id = ?" id])]
(t/is (= 0 (:deleted result)))
(t/is (= 1 (:parked result)))
(t/is (= 0 (:call-count @mock)))
(t/is (= "valid" (:status row)))
(t/is (ct/is-after? (:deleted-at row) (ct/now)))
(t/is (= 0 (:deletion-attempts row)))))))
(t/deftest gc-deleted-give-up-not-applied-to-unknown-target
(let [storage (storage-with-s3-targets)
cfg {::db/pool th/*pool* ::sto/storage storage}
id (uuid/next)]
(th/db-exec! ["insert into storage_object (id, size, backend, metadata, deleted_at, status, deletion_attempts)
values (?, 1, 's3', ?, ?, 'valid', 10)"
id
(db/tjson {:bucket "file-data" :storage-target "ghost"})
(ct/in-past {:minutes 1})])
(with-mocks [_mock {:target 'app.storage.impl/del-objects-in-bulk
:return (fn [_ _ _] #{})}]
(#'sto.gc-deleted/clean-deleted! cfg))
(let [row (th/db-exec-one! ["select count(*) from storage_object where id = ?" id])]
(t/is (= 1 (:count row))))))
(t/deftest gc-deleted-normal-failure-defers-and-gives-up
(let [storage (storage-with-s3-targets)
cfg {::db/pool th/*pool* ::sto/storage storage}
id (uuid/next)]
(th/db-exec! ["insert into storage_object (id, size, backend, metadata, deleted_at, status)
values (?, 1, 's3', ?, ?, 'valid')"
id
(db/tjson {:bucket "file-data" :storage-target "temp"})
(ct/in-past {:minutes 1})])
(with-mocks [_mock {:target 'app.storage.impl/del-objects-in-bulk
:return (fn [_ _ ids] (set ids))}]
(let [result (#'sto.gc-deleted/clean-deleted! cfg)
row (th/db-exec-one!
["select deleted_at, deletion_attempts
from storage_object where id = ?" id])]
(t/is (= 0 (:deleted result)))
(t/is (= 0 (:parked result)))
(t/is (ct/is-after? (:deleted-at row) (ct/now)))
(t/is (= 1 (:deletion-attempts row))))
;; force the give-up threshold and let the next pass remove the row
(th/db-update! :storage-object
{:deletion-attempts 7
:deleted-at (ct/in-past {:minutes 1})}
{:id id})
(let [result (#'sto.gc-deleted/clean-deleted! cfg)]
(t/is (= 0 (:deleted result)))
(let [row (th/db-exec-one! ["select count(*) from storage_object where id = ?" id])]
(t/is (= 0 (:count row))))))))
(t/deftest pending-gc-deletes-resolvable-target
(let [storage (storage-with-s3-targets)
cfg {::db/pool th/*pool* ::sto/storage storage}
id (uuid/next)
captured (atom nil)]
(th/db-exec! ["insert into storage_object (id, size, backend, metadata, created_at, status)
values (?, 1, 's3', ?, ?, 'pending')"
id
(db/tjson {:storage-target "temp"})
(ct/in-past {:days 2})])
(with-mocks [_mock {:target 'app.storage.impl/del-object
:return (fn [_ object]
(reset! captured object)
nil)}]
(let [result (#'sto.pending-gc/process! cfg)]
(t/is (= 1 (:processed result)))
(t/is (= 0 (:parked result)))
(t/is (= "temp" (:storage-target (meta @captured))))
(let [row (th/db-exec-one! ["select count(*) from storage_object where id = ?" id])]
(t/is (= 0 (:count row))))))))
(t/deftest pending-gc-refuses-unknown-target-and-keeps-row
(let [storage (storage-with-s3-targets)
cfg {::db/pool th/*pool* ::sto/storage storage}
id (uuid/next)
captured (atom nil)]
(th/db-exec! ["insert into storage_object (id, size, backend, metadata, created_at, status)
values (?, 1, 's3', ?, ?, 'pending')"
id
(db/tjson {:storage-target "ghost"})
(ct/in-past {:days 2})])
(with-mocks [mock {:target 'app.storage.impl/del-object
:return (fn [_ object]
(reset! captured object)
nil)}]
(let [result (#'sto.pending-gc/process! cfg)
row (th/db-exec-one!
["select status, deleted_at from storage_object where id = ?" id])]
(t/is (= 0 (:processed result)))
(t/is (= 1 (:parked result)))
(t/is (= 0 (:call-count @mock)))
(t/is (nil? @captured))
(t/is (= "pending" (:status row)))
(t/is (ct/is-after? (:deleted-at row) (ct/now)))))))
(t/deftest gc-deleted-legacy-rows-delete-from-default-target
(let [storage (storage-with-s3-targets)
cfg {::db/pool th/*pool* ::sto/storage storage}
id (uuid/next)
captured (atom nil)]
(th/db-exec! ["insert into storage_object (id, size, backend, metadata, deleted_at, status)
values (?, 1, 's3', ?, ?, 'valid')"
id
(db/tjson {:bucket "file-data"})
(ct/in-past {:minutes 1})])
(with-mocks [_mock {:target 'app.storage.impl/del-objects-in-bulk
:return (fn [_ target _ids]
(reset! captured target)
#{})}]
(let [result (#'sto.gc-deleted/clean-deleted! cfg)]
(t/is (= 1 (:deleted result)))
(t/is (= 0 (:parked result)))
(t/is (= "default" @captured))
(let [row (th/db-exec-one! ["select count(*) from storage_object where id = ?" id])]
(t/is (= 0 (:count row))))))))
(t/deftest pending-gc-legacy-rows-delete-from-default-target
(let [storage (storage-with-s3-targets)
cfg {::db/pool th/*pool* ::sto/storage storage}
id (uuid/next)
captured (atom nil)]
(th/db-exec! ["insert into storage_object (id, size, backend, metadata, created_at, status)
values (?, 1, 's3', ?, ?, 'pending')"
id
(db/tjson {:bucket "file-data"})
(ct/in-past {:days 2})])
(with-mocks [_mock {:target 'app.storage.impl/del-object
:return (fn [_ object]
(reset! captured object)
nil)}]
(let [result (#'sto.pending-gc/process! cfg)]
(t/is (= 1 (:processed result)))
(t/is (= 0 (:parked result)))
(t/is (= "default" (:storage-target (meta @captured))))
(let [row (th/db-exec-one! ["select count(*) from storage_object where id = ?" id])]
(t/is (= 0 (:count row))))))))
+106
View File
@@ -0,0 +1,106 @@
import { describe, it } from "node:test";
import assert from "node:assert/strict";
import { rpcPost, extractCookie } from "./helpers/client.mjs";
async function loginWithLdap(email, password) {
const res = await rpcPost("login-with-ldap", { email, password });
if (res.status !== 200 || res.body.type) {
throw new Error(
`LDAP login failed: ${JSON.stringify(res.body)}`
);
}
const cookie = extractCookie(res.setCookie);
return { profile: res.body, cookie };
}
describe("LDAP injection — T5-N1-03", () => {
it("normal LDAP login works with valid credentials", async () => {
const { profile, cookie } = await loginWithLdap(
"fry@planetexpress.com",
"fry"
);
assert.equal(profile.email, "fry@planetexpress.com");
assert.ok(profile.id, "profile should have id");
assert.ok(cookie, "cookie should be set");
});
it("wildcard injection: *@planetexpress.com must not return client literal as email", async () => {
// ATTACK SCENARIO (from Criptored audit):
// 1. Attacker (amy) sends email="*@planetexpress.com" with her own password
// 2. LDAP filter becomes (mail=*@planetexpress.com) — * is a wildcard
// 3. With sizelimit=1, LDAP returns amy's entry (first match)
// 4. Bind succeeds: amy's DN + amy's password = valid
//
// EXPECTED BEHAVIOR AFTER FIX (two valid outcomes):
// A) If * is escaped: LDAP finds no match → wrong-credentials (injection blocked)
// B) If * matches: profile email must be "amy@planetexpress.com" (directory), not "*@planetexpress.com" (client)
//
// Either outcome is correct — the vulnerability is fixed.
try {
const { profile } = await loginWithLdap("*@planetexpress.com", "amy");
// Outcome B: login succeeded, verify email is from directory
assert.equal(
profile.email,
"amy@planetexpress.com",
"email must come from LDAP directory, not client input"
);
} catch (e) {
// Outcome A: injection blocked — * is escaped, no LDAP match
assert.ok(
e.message.includes("wrong-credentials"),
"wildcard should be rejected or return directory email"
);
}
});
it("identity swap: alternate email must return primary directory email", async () => {
// Professor has two emails in LDAP: professor@ and hubert@.
// Login with hubert@ — the profile email should be the one
// the LDAP directory returns as attrs-email, not what the client typed.
//
// EXPECTED BEHAVIOR AFTER FIX:
// Profile email should be "professor@planetexpress.com" (primary directory email),
// NOT "hubert@planetexpress.com" (client literal).
//
// CURRENT BUG: email is "hubert@planetexpress.com" (client literal) — test FAILS
const { profile, cookie } = await loginWithLdap(
"hubert@planetexpress.com",
"professor"
);
assert.ok(profile.id, "profile should have id");
assert.ok(cookie, "cookie should be set");
// This assertion FAILS with current code (RED) — proves the vulnerability
assert.equal(
profile.email,
"professor@planetexpress.com",
"email must come from LDAP directory, not client input"
);
});
it("wrong password fails", async () => {
try {
await loginWithLdap("fry@planetexpress.com", "wrong-password");
assert.fail("should have thrown");
} catch (e) {
assert.ok(
e.message.includes("LDAP login failed") ||
e.message.includes("wrong-credentials"),
"should fail with wrong credentials"
);
}
});
it("non-existent user fails", async () => {
try {
await loginWithLdap("nobody@planetexpress.com", "password");
assert.fail("should have thrown");
} catch (e) {
assert.ok(
e.message.includes("LDAP login failed") ||
e.message.includes("wrong-credentials"),
"should fail for non-existent user"
);
}
});
});
+9 -1
View File
@@ -72,6 +72,13 @@
[:map {:title "PlainColorAttrs"}
[:color schema:hex-color]])
(def schema:image-transform
[:map {:title "ImageTransform" :closed true}
[:x {:optional true} ::sm/safe-number]
[:y {:optional true} ::sm/safe-number]
[:width {:optional true} ::sm/safe-number]
[:height {:optional true} ::sm/safe-number]])
(def schema:image
[:map {:title "ImageColor" :closed true}
[:width [::sm/int {:min 0 :gen/gen sg/int}]]
@@ -79,7 +86,8 @@
[:mtype {:gen/gen (sg/elements cm/image-types)} ::sm/text]
[:id ::sm/uuid]
[:name {:optional true} ::sm/text]
[:keep-aspect-ratio {:optional true} :boolean]])
[:keep-aspect-ratio {:optional true} :boolean]
[:transform {:optional true} schema:image-transform]])
(def image-attrs
"A set of attrs that corresponds to image data type"
+49 -27
View File
@@ -119,12 +119,15 @@
(defn write-image-fill
[offset buffer opacity image]
(let [image-id (get image :id)
image-width (get image :width)
image-height (get image :height)
alpha (mth/floor (* opacity 0xff))
keep-aspect-ratio (if (get image :keep-aspect-ratio false) 0x01 0x00)
flags (bit-or keep-aspect-ratio 0x00)]
(let [image-id (get image :id)
image-width (get image :width)
image-height (get image :height)
alpha (mth/floor (* opacity 0xff))
keep-aspect-ratio (if (get image :keep-aspect-ratio false) 0x01 0x00)
transform (get image :transform)
has-transform? (some? transform)
transform-flag (if has-transform? 0x02 0x00)
flags (bit-or keep-aspect-ratio transform-flag)]
(buf/write-byte buffer (+ offset 0) 0x03)
(buf/write-uuid buffer (+ offset 4) image-id)
(buf/write-byte buffer (+ offset 20) alpha)
@@ -132,6 +135,17 @@
(buf/write-short buffer (+ offset 22) 0) ;; 2-byte padding (reserved for future use)
(buf/write-int buffer (+ offset 24) image-width)
(buf/write-int buffer (+ offset 28) image-height)
(if has-transform?
(do
(buf/write-float buffer (+ offset 32) (double (get transform :x 0.0)))
(buf/write-float buffer (+ offset 36) (double (get transform :y 0.0)))
(buf/write-float buffer (+ offset 40) (double (get transform :width 1.0)))
(buf/write-float buffer (+ offset 44) (double (get transform :height 1.0))))
(do
(buf/write-float buffer (+ offset 32) 0.0)
(buf/write-float buffer (+ offset 36) 0.0)
(buf/write-float buffer (+ offset 40) 1.0)
(buf/write-float buffer (+ offset 44) 1.0)))
(+ offset FILL-U8-SIZE)))
(defn- write-metadata
@@ -208,28 +222,36 @@
:type type}})
3 ;; image fill
(let [id (buf/read-uuid dbuffer (+ doffset 4))
alpha (buf/read-unsigned-byte dbuffer (+ doffset 20))
opacity (mth/precision (/ alpha 0xff) 2)
flags (buf/read-unsigned-byte dbuffer (+ doffset 21))
ratio (boolean (bit-and flags 0x01))
width (buf/read-int dbuffer (+ doffset 24))
height (buf/read-int dbuffer (+ doffset 28))
mtype (buf/read-short mbuffer (+ moffset 2))
mtype (case mtype
0x01 "image/jpeg"
0x02 "image/png"
0x03 "image/gif"
0x04 "image/webp"
0x05 "image/svg+xml")]
(let [id (buf/read-uuid dbuffer (+ doffset 4))
alpha (buf/read-unsigned-byte dbuffer (+ doffset 20))
opacity (mth/precision (/ alpha 0xff) 2)
flags (buf/read-unsigned-byte dbuffer (+ doffset 21))
ratio (not (zero? (bit-and flags 0x01)))
has-tf (not (zero? (bit-and flags 0x02)))
width (buf/read-int dbuffer (+ doffset 24))
height (buf/read-int dbuffer (+ doffset 28))
transform (when has-tf
{:x (buf/read-float dbuffer (+ doffset 32))
:y (buf/read-float dbuffer (+ doffset 36))
:width (buf/read-float dbuffer (+ doffset 40))
:height (buf/read-float dbuffer (+ doffset 44))})
mtype (buf/read-short mbuffer (+ moffset 2))
mtype (case mtype
0x01 "image/jpeg"
0x02 "image/png"
0x03 "image/gif"
0x04 "image/webp"
0x05 "image/svg+xml")]
{:fill-opacity opacity
:fill-image {:id id
:width width
:height height
:mtype mtype
:keep-aspect-ratio ratio
;; FIXME: we are not encodign the name, looks useless
:name "sample"}}))]
:fill-image (cond-> {:id id
:width width
:height height
:mtype mtype
:keep-aspect-ratio ratio
;; FIXME: we are not encodign the name, looks useless
:name "sample"}
(some? transform)
(assoc :transform transform))}))]
(if refs?
(let [ref-file (buf/read-uuid mbuffer (+ moffset 4))
+1 -1
View File
@@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.common.types.path.fit
"Curve fitting helpers."
@@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.common.types.path.selection
"Transforms selected path nodes and handlers."
@@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.common.types.tokens-status
(:require
@@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns common-tests.files-migrations-0026-test
(:require
@@ -0,0 +1,275 @@
;; This Source Code Form is subject to the terms of the Mozilla Public
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns common-tests.geom-image-bounds-resize-test
(:require
#?(:clj [clojure.test :refer [deftest is testing]]
:cljs [cljs.test :refer-macros [deftest is testing]])
[app.common.math :as mth]
[app.common.schema :as sm]
[app.common.types.color :as clr]
[app.common.types.fills :as fills]
[app.common.types.fills.impl :as fills.impl]
[app.common.uuid :as uuid]))
(deftest test-image-transform-schema
(testing "validates image with transform"
(let [img {:id (uuid/custom 1)
:width 400
:height 300
:mtype "image/png"
:keep-aspect-ratio true
:transform {:x 0.1 :y -0.2 :width 1.5 :height 2.0}}]
(is (sm/validate clr/schema:image img))))
(testing "validates image without transform"
(let [img {:id (uuid/custom 1)
:width 400
:height 300
:mtype "image/png"
:keep-aspect-ratio true}]
(is (sm/validate clr/schema:image img))))
(testing "validates fill with image transform"
(let [fill {:fill-opacity 0.8
:fill-image {:id (uuid/custom 1)
:width 400
:height 300
:mtype "image/png"
:keep-aspect-ratio true
:transform {:x -0.5 :y -0.5 :width 2.0 :height 2.0}}}]
(is (sm/validate fills/schema:fill fill)))))
(deftest test-image-fill-buffer-roundtrip
(testing "roundtrip image fill without transform"
(let [fill-vec [{:fill-opacity 0.9
:fill-image {:id (uuid/custom 1)
:width 800
:height 600
:mtype "image/jpeg"
:keep-aspect-ratio true
:name "sample"}}]
coerced (fills/from-plain fill-vec)
plain (into [] coerced)]
(is (= 1 (count plain)))
(is (= 0.9 (:fill-opacity (first plain))))
(is (= 800 (-> plain first :fill-image :width)))
(is (= 600 (-> plain first :fill-image :height)))
(is (true? (-> plain first :fill-image :keep-aspect-ratio)))
(is (nil? (-> plain first :fill-image :transform)))))
(testing "roundtrip image fill with transform"
(let [fill-vec [{:fill-opacity 0.75
:fill-image {:id (uuid/custom 2)
:width 1920
:height 1080
:mtype "image/webp"
:keep-aspect-ratio false
:name "sample"
:transform {:x 0.25 :y -0.15 :width 1.5 :height 2.0}}}]
coerced (fills/from-plain fill-vec)
plain (into [] coerced)
tf (-> plain first :fill-image :transform)]
(is (= 1 (count plain)))
(is (= 0.75 (:fill-opacity (first plain))))
(is (= 1920 (-> plain first :fill-image :width)))
(is (= 1080 (-> plain first :fill-image :height)))
(is (false? (-> plain first :fill-image :keep-aspect-ratio)))
(is (some? tf))
(is (mth/close? 0.25 (double (:x tf))))
(is (mth/close? -0.15 (double (:y tf))))
(is (mth/close? 1.5 (double (:width tf))))
(is (mth/close? 2.0 (double (:height tf)))))))
(defn compute-bounds-resize-transform
"Mathematical model for independent image bounds resizing"
[{:keys [width height handler center? sx sy transform]}]
(let [w-new (* width sx)
h-new (* height sy)
[dx dy] (if ^boolean center?
[(/ (* width (- 1.0 sx)) 2.0)
(/ (* height (- 1.0 sy)) 2.0)]
[(case handler
(:left :bottom-left :top-left) (* width (- 1.0 sx))
0.0)
(case handler
(:top :top-left :top-right) (* height (- 1.0 sy))
0.0)])
nx0 (get transform :x 0.0)
ny0 (get transform :y 0.0)
nw0 (get transform :width 1.0)
nh0 (get transform :height 1.0)
nx' (/ (- (* nx0 width) dx) w-new)
ny' (/ (- (* ny0 height) dy) h-new)
nw' (/ nw0 sx)
nh' (/ nh0 sy)]
{:transform {:x nx' :y ny' :width nw' :height nh'}
:rendered-pixel-rect {:x (* nx' w-new)
:y (* ny' h-new)
:width (* nw' w-new)
:height (* nh' h-new)}}))
(deftest test-handle-anchoring-mathematics
(testing "Right handle crop (shrinking width to 50%)"
(let [res (compute-bounds-resize-transform
{:width 200 :height 100 :handler :right :center? false :sx 0.5 :sy 1.0})]
(is (mth/close? 0.0 (-> res :transform :x)))
(is (mth/close? 0.0 (-> res :transform :y)))
(is (mth/close? 2.0 (-> res :transform :width)))
(is (mth/close? 1.0 (-> res :transform :height)))
;; Rendered pixel content remains 200x100 starting at (0, 0)
(is (mth/close? 0.0 (-> res :rendered-pixel-rect :x)))
(is (mth/close? 0.0 (-> res :rendered-pixel-rect :y)))
(is (mth/close? 200.0 (-> res :rendered-pixel-rect :width)))
(is (mth/close? 100.0 (-> res :rendered-pixel-rect :height)))))
(testing "Left handle crop (shrinking width to 50% from left)"
(let [res (compute-bounds-resize-transform
{:width 200 :height 100 :handler :left :center? false :sx 0.5 :sy 1.0})]
(is (mth/close? -1.0 (-> res :transform :x)))
(is (mth/close? 0.0 (-> res :transform :y)))
(is (mth/close? 2.0 (-> res :transform :width)))
(is (mth/close? 1.0 (-> res :transform :height)))
;; Rendered pixel content has left at -100, width 200 -> right edge at +100 (matches right edge of 100px container!)
(is (mth/close? -100.0 (-> res :rendered-pixel-rect :x)))
(is (mth/close? 200.0 (-> res :rendered-pixel-rect :width)))))
(testing "Top handle crop (shrinking height to 50% from top)"
(let [res (compute-bounds-resize-transform
{:width 200 :height 100 :handler :top :center? false :sx 1.0 :sy 0.5})]
(is (mth/close? 0.0 (-> res :transform :x)))
(is (mth/close? -1.0 (-> res :transform :y)))
(is (mth/close? 1.0 (-> res :transform :width)))
(is (mth/close? 2.0 (-> res :transform :height)))
;; Rendered pixel content has top at -50, height 100 -> bottom edge at +50 (matches bottom edge of 50px container!)
(is (mth/close? -50.0 (-> res :rendered-pixel-rect :y)))
(is (mth/close? 100.0 (-> res :rendered-pixel-rect :height)))))
(testing "Top-Left handle crop (shrinking both dimensions to 50%)"
(let [res (compute-bounds-resize-transform
{:width 200 :height 100 :handler :top-left :center? false :sx 0.5 :sy 0.5})]
(is (mth/close? -1.0 (-> res :transform :x)))
(is (mth/close? -1.0 (-> res :transform :y)))
(is (mth/close? 2.0 (-> res :transform :width)))
(is (mth/close? 2.0 (-> res :transform :height)))
(is (mth/close? -100.0 (-> res :rendered-pixel-rect :x)))
(is (mth/close? -50.0 (-> res :rendered-pixel-rect :y)))
(is (mth/close? 200.0 (-> res :rendered-pixel-rect :width)))
(is (mth/close? 100.0 (-> res :rendered-pixel-rect :height)))))
(testing "Center resize (Alt modifier)"
(let [res (compute-bounds-resize-transform
{:width 200 :height 100 :handler :right :center? true :sx 0.5 :sy 0.5})]
(is (mth/close? -0.5 (-> res :transform :x)))
(is (mth/close? -0.5 (-> res :transform :y)))
(is (mth/close? 2.0 (-> res :transform :width)))
(is (mth/close? 2.0 (-> res :transform :height)))
(is (mth/close? -50.0 (-> res :rendered-pixel-rect :x)))
(is (mth/close? -25.0 (-> res :rendered-pixel-rect :y)))
(is (mth/close? 200.0 (-> res :rendered-pixel-rect :width)))
(is (mth/close? 100.0 (-> res :rendered-pixel-rect :height)))))
(testing "Bottom handle crop (shrinking height to 50% from bottom)"
(let [res (compute-bounds-resize-transform
{:width 200 :height 100 :handler :bottom :center? false :sx 1.0 :sy 0.5})]
(is (mth/close? 0.0 (-> res :transform :x)))
(is (mth/close? 0.0 (-> res :transform :y)))
(is (mth/close? 1.0 (-> res :transform :width)))
(is (mth/close? 2.0 (-> res :transform :height)))
(is (mth/close? 0.0 (-> res :rendered-pixel-rect :y)))
(is (mth/close? 100.0 (-> res :rendered-pixel-rect :height)))))
(testing "Top-Right handle crop (shrinking both dimensions to 50%)"
(let [res (compute-bounds-resize-transform
{:width 200 :height 100 :handler :top-right :center? false :sx 0.5 :sy 0.5})]
(is (mth/close? 0.0 (-> res :transform :x)))
(is (mth/close? -1.0 (-> res :transform :y)))
(is (mth/close? 2.0 (-> res :transform :width)))
(is (mth/close? 2.0 (-> res :transform :height)))
(is (mth/close? 0.0 (-> res :rendered-pixel-rect :x)))
(is (mth/close? -50.0 (-> res :rendered-pixel-rect :y)))
(is (mth/close? 200.0 (-> res :rendered-pixel-rect :width)))
(is (mth/close? 100.0 (-> res :rendered-pixel-rect :height)))))
(testing "Bottom-Left handle crop (shrinking both dimensions to 50%)"
(let [res (compute-bounds-resize-transform
{:width 200 :height 100 :handler :bottom-left :center? false :sx 0.5 :sy 0.5})]
(is (mth/close? -1.0 (-> res :transform :x)))
(is (mth/close? 0.0 (-> res :transform :y)))
(is (mth/close? 2.0 (-> res :transform :width)))
(is (mth/close? 2.0 (-> res :transform :height)))
(is (mth/close? -100.0 (-> res :rendered-pixel-rect :x)))
(is (mth/close? 0.0 (-> res :rendered-pixel-rect :y)))
(is (mth/close? 200.0 (-> res :rendered-pixel-rect :width)))
(is (mth/close? 100.0 (-> res :rendered-pixel-rect :height)))))
(testing "Expanding bounds beyond original size (empty space exposure)"
(let [res (compute-bounds-resize-transform
{:width 200 :height 100 :handler :right :center? false :sx 2.0 :sy 1.0})]
(is (mth/close? 0.0 (-> res :transform :x)))
(is (mth/close? 0.0 (-> res :transform :y)))
(is (mth/close? 0.5 (-> res :transform :width)))
(is (mth/close? 1.0 (-> res :transform :height)))
;; Rendered pixel content is 200px wide in a 400px container -> exposes 200px empty space
(is (mth/close? 0.0 (-> res :rendered-pixel-rect :x)))
(is (mth/close? 200.0 (-> res :rendered-pixel-rect :width))))))
(deftest test-sequential-resize-operations
(testing "Sequential crops: crop right then crop left"
;; Initial shape: 200x100, transform: {:x 0 :y 0 :width 1 :height 1}
;; Step 1: Crop right handle from 200 to 150 (sx = 0.75)
(let [step1 (compute-bounds-resize-transform
{:width 200 :height 100 :handler :right :center? false :sx 0.75 :sy 1.0})
tf1 (:transform step1)]
(is (mth/close? 0.0 (:x tf1)))
(is (mth/close? (/ 1.0 0.75) (:width tf1)))
;; Step 2: Now shape is 150x100 with tf1. Crop left handle from 150 to 100 (sx = 100/150 = 2/3)
(let [step2 (compute-bounds-resize-transform
{:width 150 :height 100 :handler :left :center? false :sx (/ 2.0 3.0) :sy 1.0 :transform tf1})
tf2 (:transform step2)]
;; The final 100x100 container has bitmap with width 200px
(is (mth/close? 200.0 (-> step2 :rendered-pixel-rect :width)))
;; The bitmap left edge is at -50px in the 100px container, so right edge is at -50 + 200 = 150px
(is (mth/close? -50.0 (-> step2 :rendered-pixel-rect :x))))))
(testing "Bounds resize followed by standard proportional scaling"
;; Step 1: Bounds resize crops width from 200 to 100
(let [step1 (compute-bounds-resize-transform
{:width 200 :height 100 :handler :right :center? false :sx 0.5 :sy 1.0})
tf1 (:transform step1)]
(is (mth/close? 2.0 (:width tf1)))
(is (mth/close? 1.0 (:height tf1)))
;; Step 2: Standard proportional scale of the 100x100 cropped shape to 200x200 (scale 2x)
;; During standard scale, normalized transform tf1 is kept constant!
(let [scaled-w (* 100.0 2.0)
scaled-h (* 100.0 2.0)
rendered-w (* (:width tf1) scaled-w)
rendered-h (* (:height tf1) scaled-h)]
;; The underlying bitmap scaled from 200x100 to 400x200, matching the 2x scale of the cropped frame!
(is (mth/close? 400.0 rendered-w))
(is (mth/close? 200.0 rendered-h))))))
(deftest test-proportion-lock-invariance
(testing "Shape proportion-lock attribute remains unchanged"
(let [shape {:id (uuid/custom 10)
:type :rect
:width 200
:height 100
:proportion-lock true
:fills [{:fill-image {:id (uuid/custom 1)
:width 800
:height 600
:keep-aspect-ratio true}}]}
;; Simulate bounds resize interaction
has-img? (boolean (or (some :fill-image (:fills shape)) (:fill-image shape)))
mod-pressed? true
bounds-resize? (and has-img? mod-pressed?)
lock-during-drag (if bounds-resize? false (:proportion-lock shape))]
;; During drag, lock is bypassed (unless Shift is pressed)
(is (false? lock-during-drag))
;; Shape's persistent setting is completely preserved
(is (true? (:proportion-lock shape))))))
+2
View File
@@ -29,6 +29,7 @@
[common-tests.geom-flex-layout-test]
[common-tests.geom-grid-layout-test]
[common-tests.geom-grid-test]
[common-tests.geom-image-bounds-resize-test]
[common-tests.geom-line-test]
[common-tests.geom-modif-tree-test]
[common-tests.geom-modifiers-test]
@@ -108,6 +109,7 @@
'common-tests.geom-flex-layout-test
'common-tests.geom-grid-layout-test
'common-tests.geom-grid-test
'common-tests.geom-image-bounds-resize-test
'common-tests.geom-line-test
'common-tests.geom-modif-tree-test
'common-tests.geom-modifiers-test
@@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns common-tests.types.tokens-status-test
(:require
+5
View File
@@ -157,6 +157,11 @@ services:
# PENPOT_OBJECTS_STORAGE_S3_ENDPOINT: <ENDPOINT>
# PENPOT_OBJECTS_STORAGE_S3_BUCKET: <BUKET_NAME>
## Optional: route specific internal semantic buckets (for example
## tempfile) to additional S3 targets. The file is EDN; see
## docs/technical-guide/configuration.md for the format.
# PENPOT_OBJECTS_STORAGE_S3_ROUTES_FILE: /opt/data/storage-routes.edn
## Telemetry. When enabled, a periodical process will send anonymous data about this
## instance. Telemetry data will enable us to learn how the application is used,
## based on real scenarios. If you want to help us, please leave it enabled. You can
+45
View File
@@ -549,6 +549,51 @@ PENPOT_OBJECTS_STORAGE_S3_ENDPOINT: <endpoint-uri>
These settings are equally useful if you have a Minio storage system.
</p>
#### S3 targets and per-bucket routing
__Since version 2.19.0__
By default the S3 backend stores every object in the configured bucket. You can
route specific internal semantic buckets (for example `tempfile` or `file-data`)
to additional S3 targets. Each target has its own bucket and, optionally, its
own key prefix, region and endpoint.
Targets and routing are declared in an external EDN file referenced by the
`PENPOT_OBJECTS_STORAGE_S3_ROUTES_FILE` environment variable:
```clojure
{:targets
{:temp {:bucket "penpot-temp"}
:cold {:bucket "penpot-cold"
:region :us-east-1
:endpoint "https://s3.us-east-1.amazonaws.com"
:prefix "objects/"}}
:routes
{"tempfile" :temp
"file-data" :temp}}
```
- `:targets` declares the named destinations. `:bucket` is required; `:prefix`,
`:region` and `:endpoint` are optional and inherit the values from the default
target (`PENPOT_OBJECTS_STORAGE_S3_*`) when omitted.
- `:routes` maps a Penpot semantic bucket to a target id. Semantic buckets that
are not listed use the default target (the configured
`PENPOT_OBJECTS_STORAGE_S3_BUCKET`).
- The `:default` target id is reserved and implicit; do not declare it.
- The file is optional and only applies to the `s3` backend. Without it, all
objects use the default bucket and the behavior is unchanged.
The target id is stored in the object metadata, so it must stay stable. If you
remove or rename a target id, objects already written with it cannot be located:
reads and serving of those objects fail, and the garbage-collection tasks refuse
to delete them. Those rows are parked for one day and retried on later runs,
without counting deletion attempts and without ever reaching the give-up window,
so the row is never removed until the routes file declares the target id again.
All backend replicas must mount the same routes file with the same target ids. A
replica with a stale file misroutes new writes and fails reads and garbage
collection for the targets it does not declare.
### File Data Storage
__Since version 2.11.0__
@@ -16,11 +16,12 @@ that may be used for any kind of user uploaded files. Currently:
There is an abstract interface and several implementations (or **backends**),
depending on where the objects are actually stored:
* <code class="language-clojure">:assets-fs</code> stores ojects in the file system, under a given base path.
* <code class="language-clojure">:assets-s3</code> stores them in any cloud storage with an AWS-S3 compatible
* <code class="language-clojure">:fs</code> stores objects in the file system, under a given base path.
* <code class="language-clojure">:s3</code> stores them in any cloud storage with an AWS-S3 compatible
interface.
* <code class="language-clojure">:assets-db</code> stores them inside the PostgreSQL database, in a special table
with a binary column.
Legacy rows may still reference the deprecated <code class="language-clojure">:assets-fs</code> and
<code class="language-clojure">:assets-s3</code> names, which alias the current backends.
## Storage API
@@ -83,6 +84,17 @@ The storage module may use the bucket (hardcoded) to make special treatment to
object, such as storing in a different path, or guessing how to know if an object
is referenced from other place.
When the <code class="language-clojure">:s3</code> backend is used, a semantic bucket may also be routed to a
named S3 **target** (a different bucket and/or key prefix). The chosen target id
is stored in the object metadata (<code class="language-clojure">:storage-target</code>) and resolved again on
reads, URL signing, deduplication and garbage collection. Objects without a
stored target use the default target. See the configuration guide for the
routing file format.
If an object references a target id that is not configured anymore, it cannot be
located: reads and serving fail, and garbage collection refuses to delete the
object, keeping the database row until the target is declared again.
## Sharing and deleting objects
To save storage space, duplicated objects wre shared. So, if for example
+1 -1
View File
@@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.auth
"Resolves the caller's session cookie to a real profile id.
+1 -1
View File
@@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.handlers.export
"Handle export jobs"
+1 -1
View File
@@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.handlers.jobs
"REST surface for export jobs, under `/api/export/jobs`.
+1 -1
View File
@@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.jobs
"Export job model and lifecycle.
+1 -1
View File
@@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.jobs.scheduler
"Admission control for export jobs.
+1 -1
View File
@@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.jobs.store
"Redis persistence for export jobs.
+1 -1
View File
@@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.jobs.utils
"Temp file ownership for export jobs.
+1 -1
View File
@@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.router
"Method + path dispatch.
+1 -1
View File
@@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.wasm.pool
"Pool of headless render workers.
+1 -1
View File
@@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.wasm.render
"Headless render pipeline: renders exports with the render-wasm Skia pipeline,
+1 -1
View File
@@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns app.wasm.worker
"Render worker entry point.
@@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns exporter-tests.export-shapes-test
"Chunking of the browser backend."
+1 -1
View File
@@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns exporter-tests.jobs-test
"Job state machine. Runs without redis: a store write with no connection is
@@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns exporter-tests.scheduler-test
"Admission control. A headless job leases one render worker for its whole run,
@@ -2,7 +2,7 @@
;; License, v. 2.0. If a copy of the MPL was not distributed with this
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
;;
;; Copyright (c) KALEIDOS INC Sucursal en España SL
;; Copyright (c) KALEIDOS SUBSIDIARY SL
(ns exporter-tests.wasm-pool-test
"Worker leasing, against a stub pool: `with-worker` must give the worker back
File diff suppressed because it is too large. Load diff
@@ -274,6 +274,25 @@ test("Renders a file with different text leaves decoration", async ({
await expect(workspace.canvas).toHaveScreenshot();
});
// Both paragraphs decorate the same spans; the first one paints every span with
// the same fill, which used to collapse the decorated spans into their
// neighbours and drop their underline / line-through.
test("Renders text spans decorated independently of their fill", async ({
page,
}) => {
const workspace = new WasmWorkspacePage(page);
await workspace.setupEmptyFile();
await workspace.mockGetFile("render-wasm/get-file-text-span-decoration.json");
await workspace.goToWorkspace({
id: "1d0f6a4c-0000-8000-8006-000000000001",
pageId: "1d0f6a4c-0000-8000-8006-000000000002",
});
await workspace.waitForFirstRenderWithoutUI();
await expect(workspace.canvas).toHaveScreenshot();
});
test("Renders a file with different text shadows combinations", async ({
page,
}) => {
Binary file not shown.

Before

Width:  |  Height:  |  Size: 241 KiB

After

Width:  |  Height:  |  Size: 299 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 175 KiB

After

Width:  |  Height:  |  Size: 220 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 168 KiB

After

Width:  |  Height:  |  Size: 181 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 80 KiB

After

Width:  |  Height:  |  Size: 117 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 450 KiB

After

Width:  |  Height:  |  Size: 471 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 519 KiB

After

Width:  |  Height:  |  Size: 665 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 143 KiB

After

Width:  |  Height:  |  Size: 185 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 88 KiB

After

Width:  |  Height:  |  Size: 118 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 116 KiB

After

Width:  |  Height:  |  Size: 152 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 124 KiB

After

Width:  |  Height:  |  Size: 139 KiB

Loaded 100 of 218 files, more files were not shown because too many files have changed in this diff. Show more