Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
334322e6db | ||
|
|
e96b8a4798 | ||
|
|
6eece49f69 | ||
|
|
0d855b3436 | ||
|
|
37dab75e1a | ||
|
|
f9c02926b9 | ||
|
|
bae3900537 | ||
|
|
757a5bd479 | ||
|
|
9c07dd124a | ||
|
|
99c036feac | ||
|
|
aa78ca0be8 | ||
|
|
8952d70fd2 | ||
|
|
7c27ed812a | ||
|
|
4ce459d720 | ||
|
|
286ccb03fa | ||
|
|
dbe5941a23 | ||
|
|
30849babcc | ||
|
|
dc12f1db91 | ||
|
|
c589563912 | ||
|
|
0eb3179016 | ||
|
|
d1ebf4cda2 | ||
|
|
fdb9e97572 | ||
|
|
9cd3b63eea | ||
|
|
94555c027e | ||
|
|
9b24907992 | ||
|
|
b283d952a8 | ||
|
|
6f63a5fcbf | ||
|
|
61caefbcf3 | ||
|
|
6586631293 | ||
|
|
66fb4a69ba | ||
|
|
ac84557740 | ||
|
|
3d4a5ca2aa | ||
|
|
d45c6710b7 | ||
|
|
43f7e49aa0 | ||
|
|
eca1d81692 | ||
|
|
d263c23a58 |
No files matched your search
@@ -1,13 +1,11 @@
|
||||
---
|
||||
name: planner
|
||||
description: Read-only planning and architecture analysis for Penpot — produce a structured implementation plan with task breakdown, acceptance criteria, sizing, and checkpoints. Always output to the user with the plan's save path (saved or suggested) and the next steps.
|
||||
description: Read-only planning and architecture analysis — produce a structured implementation plan with task breakdown, acceptance criteria, sizing, and checkpoints. Always output to the user with the plan, suggested save path and the next steps.
|
||||
---
|
||||
|
||||
# Planner
|
||||
|
||||
Read-only senior software architect role for Penpot. Produces structured
|
||||
implementation plans with task breakdowns that engineers or other agents can
|
||||
execute. Never writes or modifies code.
|
||||
Produce a plan that another engineer or agent can execute without guessing.
|
||||
|
||||
## When to Use
|
||||
|
||||
@@ -21,24 +19,7 @@ execute. Never writes or modifies code.
|
||||
- A task feels too large or vague to start.
|
||||
- Work needs to be parallelized across multiple agents or sessions.
|
||||
|
||||
Do **not** use this skill to actually implement anything — it is read-only.
|
||||
|
||||
**When NOT to use:** Single-file changes with obvious scope, or when the spec
|
||||
already contains well-defined tasks.
|
||||
|
||||
## Role
|
||||
|
||||
You help users understand the Penpot codebase, design solutions, and produce
|
||||
implementation plans that other agents or developers can execute. The plan
|
||||
tells them what to build and how to verify it, task by task.
|
||||
|
||||
The implementer reads the project's agent docs (`AGENTS.md`, project memories
|
||||
such as `mem:critical-info`, `mem:testing`, and each module's core memory)
|
||||
before working. Reference those memories instead of re-explaining tooling,
|
||||
conventions, or test design — explain in the plan only what they do not cover.
|
||||
|
||||
Do **not** suggest commit messages or commit names anywhere in your plans or
|
||||
responses — committing is the implementer's responsibility.
|
||||
Do not use for a small change with obvious scope or an existing executable plan.
|
||||
|
||||
## CRITICAL: Required Reading Before Planning
|
||||
|
||||
@@ -55,67 +36,36 @@ Before drafting any plan, work through the project's own guidance:
|
||||
|
||||
Skipping this step is the #1 cause of incorrect or incomplete plans.
|
||||
|
||||
---
|
||||
## Constraints
|
||||
|
||||
## The Planning Process
|
||||
- You are **analysis-only** — never create, edit, or delete source code. The
|
||||
only file you may write is the plan itself, and only when the command or
|
||||
user explicitly instructs you to save it.
|
||||
- You do **not** run builds, tests, linters, or any commands that modify state.
|
||||
- You do **not** create git commits or interact with version control.
|
||||
- You do **not** execute shell commands beyond read-only searches (`rg`, `ls`,
|
||||
`find`, `cat`, `bat`).
|
||||
- Your output is a structured plan or analysis, ready for handoff to an
|
||||
engineer agent or developer.
|
||||
|
||||
### Phase 1: Architecture Analysis
|
||||
## Planning Process
|
||||
|
||||
1. Read the spec, requirements, or feature request.
|
||||
2. Analyze the codebase architecture and identify affected modules.
|
||||
3. Read project conventions (starting with `critical-info` and module core
|
||||
memories) before drafting.
|
||||
4. Map dependencies between components (see the dependency graph in
|
||||
`critical-info`).
|
||||
5. Identify risks, edge cases, performance implications, and breaking changes.
|
||||
1. Define the problem, desired outcome, constraints, and exclusions.
|
||||
2. Trace the current behavior through the affected modules.
|
||||
3. Map dependencies and choose an implementation order that builds foundations
|
||||
before their consumers.
|
||||
4. Identify open product or architecture decisions. Resolve implementation
|
||||
details from existing conventions when they do not affect public behavior.
|
||||
5. Identify edge cases, security and data risks, performance bounds, breaking
|
||||
changes, and external dependencies.
|
||||
6. Split the work into small, ordered tasks. Prefer complete testable slices
|
||||
over unrelated layer-wide batches. Apply DRY and KISS to the proposed
|
||||
implementation.
|
||||
7. Define exact acceptance criteria and verification for every task.
|
||||
8. Add a checkpoint after every two or three tasks in a longer plan.
|
||||
9. State which tasks can run in parallel and which must remain sequential.
|
||||
|
||||
### Phase 2: Task Breakdown
|
||||
|
||||
#### Identify the Dependency Graph
|
||||
|
||||
Map what depends on what, following the monorepo's module dependency graph:
|
||||
|
||||
```
|
||||
common (shared types, schemas — no deps)
|
||||
│
|
||||
├── backend (depends common)
|
||||
│ ├── RPC handlers
|
||||
│ └── persistence / migrations
|
||||
│
|
||||
├── frontend (depends common, render-wasm)
|
||||
│ ├── UI components
|
||||
│ └── state / API integration
|
||||
│
|
||||
├── exporter (depends common)
|
||||
│
|
||||
└── render-wasm (consumed by frontend)
|
||||
```
|
||||
|
||||
Implementation order follows the dependency graph bottom-up: build shared
|
||||
foundations first, then layer consumers on top.
|
||||
|
||||
#### Slice Vertically
|
||||
|
||||
Instead of building all of common, then all of backend, then all of frontend —
|
||||
build one complete feature path at a time:
|
||||
|
||||
**Bad (horizontal slicing):**
|
||||
```
|
||||
Task 1: Build all common types
|
||||
Task 2: Build all backend handlers
|
||||
Task 3: Build all frontend components
|
||||
```
|
||||
|
||||
**Good (vertical slicing):**
|
||||
```
|
||||
Task 1: common data types + schema ← foundation
|
||||
Task 2: backend RPC handler + persistence
|
||||
Task 3: frontend UI component + API integration
|
||||
```
|
||||
|
||||
Each vertical slice delivers working, testable functionality.
|
||||
|
||||
#### Write Tasks
|
||||
## Task Format
|
||||
|
||||
Each task follows this structure:
|
||||
|
||||
@@ -152,17 +102,16 @@ implementation. Omit when the task is mechanical.
|
||||
**Estimated scope:** [XS: 1 file | S: 1-2 files | M: 3-5 files | L: 5+ files]
|
||||
```
|
||||
|
||||
Replace "module-specific test command" with the actual commands for the module
|
||||
(e.g. `clojure -M:dev:test` for backend/common,
|
||||
`npx shadow-cljs compile test && npx karma start` for frontend, or the
|
||||
commands noted in the module's core memory).
|
||||
Use commands from `mem:testing` and affected module memories. Never substitute
|
||||
generic text such as "run the tests" when the project documents an exact
|
||||
command.
|
||||
|
||||
When possible, design each task with TDD in mind: acceptance criteria double
|
||||
as a test list, and the natural first step of the task is writing those tests
|
||||
before the implementation. Some tasks resist this (config, migrations, pure
|
||||
wiring) — for those, keep the usual verification steps.
|
||||
When possible, design each task with TDD in mind: acceptance criteria double as a test
|
||||
list, and the natural first step of the task is writing those tests before the
|
||||
implementation. Some tasks resist this (config, migrations, pure wiring) — for those, keep
|
||||
the usual verification steps.
|
||||
|
||||
#### Estimate Scope
|
||||
## Task Sizing
|
||||
|
||||
| Size | Files | Scope | Example |
|
||||
|------|-------|-------|---------|
|
||||
@@ -172,16 +121,11 @@ wiring) — for those, keep the usual verification steps.
|
||||
| **L** | 5-8 | Multi-component feature | Search with filtering and pagination |
|
||||
| **XL** | 8+ | **Too large — break it down further** | — |
|
||||
|
||||
If a task is XL, it should be broken into smaller tasks. Agents perform best
|
||||
on S and M tasks.
|
||||
Split a task when it contains independent outcomes, spans unrelated systems, or cannot be
|
||||
completed and verified in one focused session (if a task is XL, it should be broken into
|
||||
smaller tasks; agents perform best on S and M tasks).
|
||||
|
||||
**When to break a task down further:**
|
||||
- It would take more than one focused session
|
||||
- You cannot describe the acceptance criteria in 3 or fewer bullet points
|
||||
- It touches two or more independent subsystems
|
||||
- You find yourself writing "and" in the task title (a sign it is two tasks)
|
||||
|
||||
#### Order and Checkpoints
|
||||
## Task order and checkpoints
|
||||
|
||||
Arrange tasks so that:
|
||||
|
||||
@@ -197,153 +141,43 @@ Add explicit checkpoints with the relevant module commands:
|
||||
- [ ] Relevant tests pass (module-specific command).
|
||||
- [ ] The relevant build or compilation passes, if applicable.
|
||||
- [ ] The core flow works end-to-end.
|
||||
- [ ] Review with human before proceeding.
|
||||
```
|
||||
|
||||
## Requirements
|
||||
|
||||
- Analyze the codebase architecture and identify affected modules.
|
||||
- Read project conventions before drafting (start with `critical-info` and
|
||||
affected module core memories).
|
||||
- Break down complex features or bugs into atomic, actionable steps.
|
||||
- Propose solutions with clear rationale, trade-offs, and sequencing.
|
||||
- Identify risks, edge cases, performance implications, and breaking changes.
|
||||
- Apply DRY and KISS principles to the proposed implementation.
|
||||
- Define a testing strategy aligned with each affected module's tooling.
|
||||
- Every task must have acceptance criteria and verification steps.
|
||||
- Checkpoints must exist after every 2-3 tasks.
|
||||
|
||||
## Constraints
|
||||
|
||||
- You are **analysis-only** — never create, edit, or delete source code. The
|
||||
only file you may write is the plan itself, and only when the command or
|
||||
user explicitly instructs you to save it.
|
||||
- You do **not** run builds, tests, linters, or any commands that modify state.
|
||||
- You do **not** create git commits or interact with version control.
|
||||
- You do **not** execute shell commands beyond read-only searches (`rg`, `ls`,
|
||||
`find`, `cat`, `bat`).
|
||||
- Your output is a structured plan or analysis, ready for handoff to an
|
||||
engineer agent or developer.
|
||||
|
||||
## Output Format
|
||||
|
||||
The plan is always delivered in the response so the user sees it regardless
|
||||
of which agent is running the skill. By default you never write the plan file;
|
||||
announce the path instead. Write the file only when the command or user
|
||||
explicitly instructs you to save it — and then only that file.
|
||||
of which agent is running the skill. File writes follow `Constraints` —
|
||||
by default announce the path instead of writing.
|
||||
|
||||
Announce the suggested save path:
|
||||
|
||||
```
|
||||
.agents/plans/YYYY-MM-DD-<plan-one-line-title>.md
|
||||
```
|
||||
|
||||
Use today's date in the user's local timezone. The `<plan-one-line-title>`
|
||||
slug is lowercase, hyphen-separated, and a short summary of the task
|
||||
(e.g. `add-batch-get-profiles-for-file-comments`). If the user explicitly
|
||||
provides a target file path, announce that path instead of the default.
|
||||
Announce the save path `.agents/plans/YYYY-MM-DD-<slug>.md` (today's date,
|
||||
lowercase hyphen-separated slug, e.g. `2026-09-10-add-batch-get-profiles`;
|
||||
an explicit user path wins).
|
||||
|
||||
End the response by suggesting the next steps: `/review-plan` to get a second
|
||||
opinion on the plan and `/implement-plan` to execute it.
|
||||
|
||||
### Plan Document Template
|
||||
### Plan Structure
|
||||
|
||||
Use this document shape:
|
||||
|
||||
```markdown
|
||||
# Plan: [Feature/Project Name]
|
||||
# Plan: Title
|
||||
|
||||
## Context
|
||||
[One paragraph: what is the problem or feature request? Why is it needed?]
|
||||
|
||||
## Affected Modules
|
||||
[Which modules of the monorepo are involved? Reference module paths and any
|
||||
`mem:` memories that were consulted.]
|
||||
|
||||
## Architecture Decisions
|
||||
- [Key decision 1 and rationale]
|
||||
- [Key decision 2 and rationale]
|
||||
|
||||
## Risks & Considerations
|
||||
[Edge cases, performance implications, breaking changes, migration concerns,
|
||||
security implications.]
|
||||
|
||||
## Risks and Considerations
|
||||
## Approach
|
||||
[A short strategy summary: 3-5 sentences describing the overall approach and
|
||||
the shape of the dependency graph (what depends on what, what gets built
|
||||
first). High-level only — the task-by-task detail lives in the Task List.]
|
||||
|
||||
## Task List
|
||||
|
||||
Each task uses the full task structure defined in
|
||||
[Write Tasks](#write-tasks) — description, rationale, acceptance criteria,
|
||||
verification, dependencies, files, estimated scope, and optional code sketch.
|
||||
Never reduce a task to a one-line checkbox; the plan must be self-contained
|
||||
and executable without other context.
|
||||
|
||||
Tasks are a flat, ordered list — a plan is not a roadmap. Do not group tasks
|
||||
into phases, milestones, or sprints; ordering and dependencies are already
|
||||
captured per task. Insert a checkpoint after every 2-3 tasks.
|
||||
|
||||
## Task 1: [Short descriptive title]
|
||||
|
||||
**Description:** [What this task accomplishes.]
|
||||
|
||||
**Rationale:** [Why this approach over the alternatives.]
|
||||
|
||||
**Acceptance criteria:**
|
||||
- [ ] [Specific, testable condition]
|
||||
|
||||
**Verification:**
|
||||
- [ ] Relevant tests pass (module-specific command).
|
||||
|
||||
**Dependencies:** None
|
||||
|
||||
**Files likely touched:**
|
||||
- `path/to/file`
|
||||
|
||||
**Estimated scope:** [XS: 1 file | S: 1-2 files | M: 3-5 files | L: 5+ files]
|
||||
|
||||
**Code sketch (optional):** [Short contract-level example, only if the shape
|
||||
is non-obvious.]
|
||||
|
||||
## Task 2: [Short descriptive title]
|
||||
|
||||
[Same structure as Task 1.]
|
||||
|
||||
## Task 3: [Short descriptive title]
|
||||
|
||||
[Same structure as Task 1.]
|
||||
|
||||
### Checkpoint: After Tasks 1-3
|
||||
- [ ] Relevant tests pass (module-specific command).
|
||||
- [ ] The relevant build or compilation passes, if applicable.
|
||||
- [ ] The core flow works end-to-end.
|
||||
- [ ] Review with human before proceeding.
|
||||
|
||||
## Task 4: [Short descriptive title]
|
||||
|
||||
[Same structure as Task 1.]
|
||||
|
||||
## Task 5: [Short descriptive title]
|
||||
|
||||
[Same structure as Task 1.]
|
||||
|
||||
## Verification & Testing
|
||||
[How to verify each task and the whole plan: the project's real test, lint,
|
||||
build, and run commands (extracted during Required Reading), coverage
|
||||
expectations, and manual checks. Consult each module's core memory for the
|
||||
exact commands.]
|
||||
|
||||
## Parallelization Opportunities
|
||||
- **Safe to parallelize:** Independent feature slices across separate
|
||||
modules, tests for already-implemented features, documentation
|
||||
- **Must be sequential:** Shared common schema changes, database migrations
|
||||
- **Needs coordination:** Features that share a contract (define the contract
|
||||
first, then parallelize)
|
||||
|
||||
## Verification and Testing
|
||||
## Parallelization
|
||||
## Open Questions
|
||||
- [Question needing human input]
|
||||
```
|
||||
|
||||
Omit empty sections only when they do not apply. Every implementation task
|
||||
still requires acceptance criteria, verification, dependencies, likely files,
|
||||
and scope.
|
||||
|
||||
When the plan is purely analytical (e.g. a code review or feasibility study
|
||||
with no implementation), skip the **Approach** and **Task List** sections and
|
||||
lead with **Findings** instead, keeping the rest of the structure.
|
||||
@@ -357,15 +191,6 @@ lead with **Findings** instead, keeping the rest of the structure.
|
||||
| "Planning is overhead" | Planning is the task. Implementation without a plan is just typing. |
|
||||
| "I can hold it all in my head" | Context windows are finite. Written plans survive session boundaries and compaction. |
|
||||
|
||||
## Red Flags
|
||||
|
||||
- Delivering prose without a task breakdown
|
||||
- Tasks that say "implement the feature" without acceptance criteria
|
||||
- No verification steps in the plan
|
||||
- All tasks are XL-sized
|
||||
- No checkpoints between tasks
|
||||
- Dependency order isn't considered
|
||||
|
||||
## Verification Checklist
|
||||
|
||||
Before delivering the plan, confirm:
|
||||
|
||||
@@ -357,6 +357,39 @@ Insert the new version section right after the `# CHANGELOG` header (before
|
||||
the previous version entry). Use the `edit` tool with enough context to make
|
||||
a unique match.
|
||||
|
||||
### 8b. Propose and populate the `:rocket: Epics and highlights` subsection
|
||||
|
||||
After inserting the version section, proactively create or populate the
|
||||
`### :rocket: Epics and highlights` subsection. This section surfaces the
|
||||
most impactful changes for self-hosted users checking for updates.
|
||||
|
||||
**When to create:** If the version section does not already have a
|
||||
`### :rocket: Epics and highlights` subsection, create one. Place it before
|
||||
`### :sparkles:` (matching existing order in CHANGES.md).
|
||||
|
||||
**How to identify highlights:** Review the `:sparkles:` entries for the
|
||||
version and select 2–5 of the most impactful/user-visible ones. Criteria:
|
||||
- New user-visible features (not internal refactors)
|
||||
- Significant capability additions
|
||||
- Items that create "FOMO" for self-hosted users on older versions
|
||||
|
||||
**Use release notes as hints:** Check
|
||||
`frontend/src/app/main/ui/releases/v2_<MINOR>.cljs` for the corresponding
|
||||
version. The slide titles and feature descriptions there are curated
|
||||
marketing content indicating what the team considers highlight-worthy. Match
|
||||
those themes to changelog entries. Treat these files as optional hints — they
|
||||
may not exist for every version.
|
||||
|
||||
**Format requirement:** Every `:rocket:` entry MUST follow the standard
|
||||
changelog format with issue/PR references:
|
||||
```
|
||||
- <description> [#<ISSUE>](https://github.com/penpot/penpot/issues/<ISSUE>) (PR: [#<PR>](https://github.com/penpot/penpot/pull/<PR>))
|
||||
```
|
||||
An entry without issue AND PR references is a highlight gap (warning, not an anomaly).
|
||||
|
||||
**Preserve existing entries:** If the `:rocket:` section already exists from
|
||||
a prior run, preserve its entries. Do not remove or rewrite them.
|
||||
|
||||
### 9. Verify
|
||||
|
||||
Read the top of `CHANGES.md` and confirm:
|
||||
@@ -468,9 +501,8 @@ Markdown viewer.
|
||||
## What is an anomaly
|
||||
|
||||
**An anomaly is a milestone-mismatch between an issue and its referenced
|
||||
PR.** It indicates that the changelog claim "this issue is fixed by this PR,
|
||||
all in milestone M" is inconsistent with the actual milestone assignments.
|
||||
There are exactly two types:
|
||||
PR.** There are two anomaly types, plus two highlight gaps (warnings that
|
||||
do not count toward the anomaly total):
|
||||
|
||||
1. **Issue is in the milestone, but its referenced PR is in a different
|
||||
milestone (or has no milestone).** The changelog claims a fix in this
|
||||
@@ -486,6 +518,13 @@ There are exactly two types:
|
||||
PR that closes an issue with no milestone references an issue from
|
||||
another (probably private) project; that is expected and the issue is
|
||||
not part of this changelog. Do not report it.
|
||||
3. **missing-highlights (gap):** A released X.Y.0 version section has no
|
||||
`### :rocket: Epics and highlights` subsection. Patches (X.Y.Z) never
|
||||
carry highlights, so only minors/majors are checked.
|
||||
4. **missing-highlight-reference (gap):** A `:rocket:` entry lacks the
|
||||
required issue AND PR references. Every highlight entry must follow the
|
||||
standard changelog format with `[#ISSUE]` and `(PR: [#PR])` links
|
||||
(multi-PR `(PR: [#A](...), [#B](...))` accepted).
|
||||
|
||||
**Anything else is not an anomaly.** Other discrepancies (exclusion
|
||||
labels on in-changelog issues, missing valid issues, unmerged PR
|
||||
@@ -653,6 +692,40 @@ for pr_num in sorted(changelog_prs):
|
||||
'issue_milestone': issue_ms, # may be None
|
||||
})
|
||||
|
||||
# --- Type C: released X.Y.0 version sections without :rocket: subsection ---
|
||||
# Patches (X.Y.Z with Z != 0) never carry :rocket: by design — only minors/majors (X.Y.0).
|
||||
anomalies_c = [] # list of version strings
|
||||
rocket_heading_re = re.compile(r'^### :rocket:', re.MULTILINE)
|
||||
version_sections = re.split(r'(?=^## \d+\.\d+\.\d+)', content, flags=re.MULTILINE)
|
||||
for vs in version_sections:
|
||||
m = re.match(r'^## (\d+\.\d+\.\d+)(.*)', vs)
|
||||
if not m: continue
|
||||
ver, suffix = m.group(1), m.group(2)
|
||||
if 'unreleased' in suffix.lower(): continue
|
||||
if ver.split('.')[2] != '0': continue
|
||||
if not rocket_heading_re.search(vs):
|
||||
anomalies_c.append(ver)
|
||||
|
||||
# --- Type D: :rocket: entries without issue AND PR references ---
|
||||
# Both are required: `[#ISSUE](.../issues/N)` and `(PR: [#PR](.../pull/M))`.
|
||||
# Multi-PR entries `(PR: [#A](...), [#B](...))` are accepted.
|
||||
anomalies_d = [] # list of dicts: {version, line}
|
||||
issue_ref_re = re.compile(r'\[#\d+\]\(https://github\.com/penpot/penpot/issues/\d+\)')
|
||||
pr_ref_re = re.compile(r'\(PR:\s*\[#\d+\]\(https://github\.com/penpot/penpot/pull/\d+\)(\s*,\s*\[#\d+\]\(https://github\.com/penpot/penpot/pull/\d+\))*\)')
|
||||
for vs in version_sections:
|
||||
m = re.match(r'^## (\d+\.\d+\.\d+)(.*)', vs)
|
||||
if not m: continue
|
||||
ver = m.group(1)
|
||||
rocket_match = rocket_heading_re.search(vs)
|
||||
if not rocket_match: continue
|
||||
# Extract the :rocket: subsection body (up to next ### or ##)
|
||||
rocket_body = vs[rocket_match.end():]
|
||||
rocket_body = re.split(r'(?m)^#{2,3}\s', rocket_body)[0]
|
||||
for line in rocket_body.splitlines():
|
||||
line = line.strip()
|
||||
if line.startswith('- ') and not (issue_ref_re.search(line) and pr_ref_re.search(line)):
|
||||
anomalies_d.append({'version': ver, 'line': line[:100]})
|
||||
|
||||
# --- Write report ---
|
||||
def fmt_ms(ms):
|
||||
return ms if ms else "_none_"
|
||||
@@ -664,13 +737,17 @@ with open(OUTPUT, 'w') as f:
|
||||
|
||||
n_a = len(anomalies_a)
|
||||
n_b = len(anomalies_b)
|
||||
n_c = len(anomalies_c)
|
||||
n_d = len(anomalies_d)
|
||||
|
||||
f.write('## Summary\n\n')
|
||||
f.write(f'- **Issue in {MILESTONE}, referenced PR in different milestone or no milestone:** {n_a}\n')
|
||||
f.write(f'- **PR in {MILESTONE}, closing issue in a different milestone:** {n_b}\n')
|
||||
f.write(f'- **Total anomalies:** {n_a + n_b}\n\n')
|
||||
f.write(f'- **Total anomalies:** {n_a + n_b}\n')
|
||||
f.write(f'- **Released X.Y.0 version missing :rocket: section (gap):** {n_c}\n')
|
||||
f.write(f'- **:rocket: entry without issue AND PR references (gap):** {n_d}\n\n')
|
||||
|
||||
# --- Anomalies section ---
|
||||
# --- Anomalies section (milestone mismatches only) ---
|
||||
if n_a or n_b:
|
||||
f.write('## Anomalies\n\n')
|
||||
f.write('These are milestone mismatches between an issue in the changelog '
|
||||
@@ -709,9 +786,37 @@ with open(OUTPUT, 'w') as f:
|
||||
badge = '🔴' if e['issue_milestone'] is None else '⚠️'
|
||||
f.write(f' - {badge} Closing {issue_link(e["issue"])} is in milestone **{ms_label}** (expected: {MILESTONE})\n')
|
||||
f.write('\n')
|
||||
|
||||
else:
|
||||
f.write('✅ No anomalies found. All (issue, PR) pairs in the changelog have aligned milestone assignments.\n\n')
|
||||
|
||||
# --- Highlight gaps (warnings, not anomalies) ---
|
||||
if n_c or n_d:
|
||||
f.write('## Highlight gaps\n\n')
|
||||
f.write('These are warnings, not anomalies: they do not affect the '
|
||||
'milestone-mismatch total above. They track `:rocket:` coverage '
|
||||
'across all released X.Y.0 versions. Historical entries (e.g. '
|
||||
'Taiga links) predate the current reference convention and are '
|
||||
'expected to appear here.\n\n')
|
||||
|
||||
if n_c:
|
||||
f.write(f'### Released X.Y.0 version missing :rocket: section\n\n')
|
||||
f.write('These released minors/majors have no `### :rocket: Epics and highlights` subsection. '
|
||||
'Add highlights to help self-hosted users understand what they are missing.\n\n')
|
||||
for ver in anomalies_c:
|
||||
f.write(f'- Version **{ver}**\n')
|
||||
f.write('\n')
|
||||
|
||||
if n_d:
|
||||
f.write(f'### :rocket: entry without issue AND PR references\n\n')
|
||||
f.write('These highlight entries lack the required issue AND PR references. '
|
||||
'Add `[#ISSUE](...)` and `(PR: [#PR](...))` links.\n\n')
|
||||
for d in anomalies_d:
|
||||
f.write(f'- **{d["version"]}**: `{d["line"]}`\n')
|
||||
f.write('\n')
|
||||
elif not (n_a or n_b):
|
||||
f.write('✅ No highlight gaps found. All released X.Y.0 versions have properly referenced :rocket: entries.\n\n')
|
||||
|
||||
# --- Context ---
|
||||
f.write('---\n\n')
|
||||
f.write('## Context\n\n')
|
||||
@@ -726,8 +831,7 @@ print(f"Anomaly report written to {OUTPUT}")
|
||||
PYEOF
|
||||
```
|
||||
|
||||
This generates `CHANGES-ISSUES.md` containing **only the anomalies** —
|
||||
milestone mismatches between issues and their referenced PRs:
|
||||
This generates `CHANGES-ISSUES.md` containing anomalies and highlight gaps:
|
||||
|
||||
1. **Issue in milestone, referenced PR in different milestone or no milestone** —
|
||||
the changelog claims a fix here, but the PR is released elsewhere.
|
||||
@@ -736,6 +840,13 @@ milestone mismatches between issues and their referenced PRs:
|
||||
(An issue with *no* milestone belongs to another, probably private,
|
||||
project — milestones are only required on the "Main" project — so it is
|
||||
neither an anomaly nor a changelog candidate.)
|
||||
3. **missing-highlights (gap, warning)** — a released X.Y.0 version section
|
||||
has no `### :rocket: Epics and highlights` subsection. Patches (X.Y.Z)
|
||||
never carry highlights.
|
||||
4. **missing-highlight-reference (gap, warning)** — a `:rocket:` entry lacks
|
||||
the required issue AND PR references.
|
||||
|
||||
Gaps do not count toward the anomaly total.
|
||||
|
||||
**Rule violations are not in the report** — they are workflow errors the
|
||||
LLM must fix directly in `CHANGES.md` during step 6a (pre-flight checks).
|
||||
@@ -809,10 +920,13 @@ self-contained and clickable in any Markdown viewer.
|
||||
issue from a different project or context. If the PR title and issue title
|
||||
are clearly unrelated, or the PR predates the issue by years, treat it as a
|
||||
data glitch and skip it.
|
||||
- **Anomaly = milestone mismatch only.** The report contains only milestone
|
||||
mismatches: (1) the issue is in this milestone but the referenced PR is
|
||||
in a different milestone (or unassigned), and (2) the PR is in this
|
||||
milestone but the issue it closes is in a different milestone. An
|
||||
- **Anomaly = milestone mismatch only; gaps are warnings.** The report's
|
||||
anomaly total counts only milestone mismatches: (1) the issue is in this
|
||||
milestone but the referenced PR is in a different milestone (or unassigned),
|
||||
and (2) the PR is in this milestone but the issue it closes is in a
|
||||
different milestone. `:rocket:` highlight gaps (missing section on a
|
||||
released X.Y.0, entry without issue AND PR references) are reported in a
|
||||
separate `Highlight gaps` section and never count toward the anomaly total. An
|
||||
*unassigned* (milestone-less) issue closed by a milestone PR is **not**
|
||||
an anomaly: milestones are required only for the "Main" project, so such
|
||||
issues come from another (probably private) project and are not changelog
|
||||
|
||||
@@ -6,7 +6,7 @@ on:
|
||||
jobs:
|
||||
build-and-push:
|
||||
name: Build and push DevEnv Docker image
|
||||
runs-on: penpot-extended-runner
|
||||
runs-on: penpot-standar-runner
|
||||
|
||||
steps:
|
||||
- name: Set common environment variables
|
||||
|
||||
@@ -46,7 +46,7 @@ jobs:
|
||||
# ── 1. Resolve the build key and check the whole set at once ───────────
|
||||
prepare:
|
||||
name: Prepare
|
||||
runs-on: penpot-extended-runner
|
||||
runs-on: penpot-standar-runner
|
||||
timeout-minutes: 15
|
||||
outputs:
|
||||
gh_ref: ${{ steps.vars.outputs.gh_ref }}
|
||||
@@ -135,7 +135,7 @@ jobs:
|
||||
# ── 2. One build per image, in parallel, only when needed ──────────────
|
||||
build:
|
||||
name: Build ${{ matrix.image }}
|
||||
runs-on: penpot-extended-runner
|
||||
runs-on: penpot-standar-runner
|
||||
timeout-minutes: 60
|
||||
needs: prepare
|
||||
if: needs.prepare.outputs.exists == 'false'
|
||||
@@ -248,7 +248,7 @@ jobs:
|
||||
# the S3 marker guarantees the branch tags were already moved.
|
||||
promote:
|
||||
name: Promote image set
|
||||
runs-on: penpot-extended-runner
|
||||
runs-on: penpot-standar-runner
|
||||
timeout-minutes: 10
|
||||
needs: [prepare, build]
|
||||
|
||||
@@ -302,7 +302,7 @@ jobs:
|
||||
# ── 4. Single failure notification for the whole workflow ─────────────
|
||||
notify:
|
||||
name: Notify failure
|
||||
runs-on: penpot-extended-runner
|
||||
runs-on: penpot-standar-runner
|
||||
timeout-minutes: 5
|
||||
needs: [prepare, build, promote]
|
||||
if: failure()
|
||||
|
||||
@@ -46,7 +46,7 @@ jobs:
|
||||
|
||||
notify:
|
||||
name: Notifications
|
||||
runs-on: ubuntu-24.04
|
||||
runs-on: ubuntu-latest
|
||||
needs:
|
||||
- build-docker
|
||||
- build-docker-admin-console
|
||||
|
||||
@@ -19,7 +19,7 @@ permissions:
|
||||
|
||||
jobs:
|
||||
release:
|
||||
runs-on: ubuntu-24.04
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
version: ${{ steps.vars.outputs.gh_ref }}
|
||||
release_notes: ${{ steps.extract_release_notes.outputs.release_notes }}
|
||||
|
||||
@@ -32,7 +32,7 @@ jobs:
|
||||
test-exporter:
|
||||
if: ${{ !github.event.pull_request.draft }}
|
||||
name: "Exporter Tests"
|
||||
runs-on: penpot-runner-02
|
||||
runs-on: penpot-extended-runner
|
||||
container:
|
||||
image: penpotapp/devenv:latest
|
||||
volumes:
|
||||
|
||||
@@ -24,7 +24,6 @@ opencode.json
|
||||
!AGENTS.md
|
||||
!CODE_OF_CONDUCT.md
|
||||
!SECURITY.md
|
||||
!HIGHLIGHTS.md
|
||||
/*.png
|
||||
/*.svg
|
||||
/*.sql
|
||||
|
||||
@@ -8,13 +8,30 @@
|
||||
- The backend stores the binary content.
|
||||
- Supported backends are `:fs` and `:s3`.
|
||||
- FS uses one root directory and a UUID-derived path.
|
||||
- S3 uses one configured bucket and an optional prefix.
|
||||
- S3 uses a default configured bucket and an optional prefix, plus optional named targets.
|
||||
- A Penpot bucket is metadata. It is not an S3 bucket or a filesystem directory.
|
||||
- FS and S3 use the same UUID-derived object path. The bucket does not change the path.
|
||||
- `PENPOT_OBJECTS_STORAGE_*` configures the current object backend.
|
||||
- Deprecated asset-storage config keys remain supported for migration.
|
||||
- Database rows keep the backend name. Keep the legacy `:assets-fs` and `:assets-s3` aliases.
|
||||
|
||||
## S3 Targets and Routing
|
||||
|
||||
- The `:s3` backend keeps `storage_object.backend = 's3'`; routing lives inside the backend.
|
||||
- Routing maps a Penpot semantic bucket to a named target (own bucket, optional prefix/region/endpoint).
|
||||
- Targets are declared in an EDN file referenced by `PENPOT_OBJECTS_STORAGE_S3_ROUTES_FILE` (`app.storage.config/load`).
|
||||
- Schema: `{:targets {<id> {:bucket ... :prefix? ... :region? ... :endpoint? ...}} :routes {"<semantic-bucket>" <id>}}`.
|
||||
- The reserved `:default` target is implicit and built from `PENPOT_OBJECTS_STORAGE_S3_*`; declared targets inherit missing region/endpoint/prefix from it.
|
||||
- Without a routes file, `::sto/bucket->target` is nil and every object uses `:default` (unchanged behavior).
|
||||
- The chosen target id is stored in object metadata as `:storage-target` (plain string) by `put-object!`.
|
||||
- `app.storage.s3/resolve-target` reads the object metadata; `nil` (legacy) and `"default"` use the default target, an unknown non-nil id raises `:invalid-storage-target` (no fallback) on reads/serving/deletes.
|
||||
- `impl/target-resolvable?` (wrapped as `sto/target-resolvable?`) reports whether a target id is configured; `:fs` is always true.
|
||||
- GC-deleted and pending-gc refuse to delete rows whose target is not resolvable: they log `:err`, park the row (`deleted_at = now()+1d`, no attempts, no give-up) and never remove it until the target is configured again.
|
||||
- `deleted_at` doubles as the pending-gc park marker; the pending selection skips rows whose `deleted_at` is in the future.
|
||||
- One S3 client/presigner is built per distinct `[region endpoint]` and shared by targets; a failed init closes the already-built pairs.
|
||||
- Target ids are stored in metadata, so they must stay stable; removing one makes its old rows unreadable and unGC-able by design.
|
||||
- `:storage-target` metadata is load-bearing: `pending-gc` passes it via `with-meta` so `del-object` resolves the right target.
|
||||
|
||||
## Object Lifecycle
|
||||
|
||||
- `put-object!` creates the database row before it writes backend content.
|
||||
@@ -64,7 +81,7 @@ Since `put-object!` uses backend-specific operations (`impl/resolve-backend` + `
|
||||
## Deduplication
|
||||
|
||||
- Deduplication requires `::sto/deduplicate?`, a content hash, and bucket metadata.
|
||||
- The lookup matches hash, bucket, backend, and `deleted_at IS NULL`.
|
||||
- The lookup matches hash, bucket, backend, storage target (`:storage-target`, coalesced to `default`), and `deleted_at IS NULL`.
|
||||
- The lookup only considers rows with `status='valid'`; pending rows are invisible.
|
||||
- A hit whose blob is missing is repaired in place: the same row/id is kept,
|
||||
and `put-object!` rewrites the blob under that id. This heals all existing
|
||||
@@ -93,6 +110,8 @@ Since `put-object!` uses backend-specific operations (`impl/resolve-backend` + `
|
||||
|
||||
- The valid bucket set lives in `app.storage/valid-buckets`.
|
||||
- `file-media-object` is the default bucket for old rows without bucket metadata.
|
||||
- Under `:s3`, any valid bucket may be routed to a named target; unrouted buckets use `:default`.
|
||||
- GC resolves the target from `metadata.:storage-target` for deleted and pending rows.
|
||||
- Do not assign a new bucket without adding its access and cleanup behavior.
|
||||
- The touched-object collector raises an internal error for an unknown bucket.
|
||||
- It supports `file-media-object`, `team-font-variant`, `file-object-thumbnail`, `file-thumbnail`, `profile`, `file-data`, `tempfile`, and `organization`.
|
||||
|
||||
@@ -52,10 +52,12 @@ file (never pipe tool output through filters).
|
||||
then re-run `corepack use pnpm@<tag>` in that directory.
|
||||
- A workspace may fail with `ERR_PNPM_IGNORED_BUILDS`, and pnpm then writes
|
||||
a placeholder scaffold into its `pnpm-workspace.yaml`:
|
||||
`allowBuilds: esbuild: set this to true or false` plus
|
||||
`ignoredBuiltDependencies`. Repo convention is `allowBuilds: esbuild: true`.
|
||||
Replace the placeholder and drop the `ignoredBuiltDependencies` entry,
|
||||
then re-run.
|
||||
`allowBuilds: esbuild: set this to true or false`. Current pnpm writes
|
||||
only the `allowBuilds` placeholder; any legacy key still present
|
||||
(`ignoredBuiltDependencies`, `onlyBuiltDependencies`,
|
||||
`neverBuiltDependencies`) is ignored since pnpm 11. Repo convention is
|
||||
`allowBuilds: esbuild: true`. Replace the placeholder and drop the
|
||||
legacy entry, then re-run.
|
||||
- `plugins/apps/composable-test-suite` once had its own
|
||||
`pnpm-workspace.yaml` and acted as a nested workspace root. That state is
|
||||
gone on purpose: pnpm picks the nearest `pnpm-workspace.yaml` walking up,
|
||||
|
||||
@@ -14,6 +14,9 @@
|
||||
- **Never pipe test output directly to filters** (`| head`, `| tail`, `| grep`, etc.).
|
||||
Always redirect to a file first: `command > /tmp/output.txt 2>&1`, then read/grep the file.
|
||||
This prevents hiding test failures. See `mem:testing` for details.
|
||||
- **`.claude/skills` is a symlink to `.agents/skills`.**
|
||||
Edit skills only in their canonical location (`.agents/skills`); never edit
|
||||
through `.claude/skills`.
|
||||
- **Read the workflow memory BEFORE the corresponding action**:
|
||||
- Before `git commit` → `mem:workflow/creating-commits` (commit format, AI-assisted-by trailer)
|
||||
- Before `gh issue create` → `mem:workflow/creating-issues` (title derivation, body template, Issue Type)
|
||||
|
||||
@@ -156,6 +156,21 @@
|
||||
- Fix incorrect permission handling when managing share links on a file [#11289](https://github.com/penpot/penpot/issues/11289) (PR: [#11290](https://github.com/penpot/penpot/pull/11290))
|
||||
- Fix backend session remaining valid after logout when the auth-token cookie is replayed [#11316](https://github.com/penpot/penpot/issues/11316) (PR: [#11317](https://github.com/penpot/penpot/pull/11317))
|
||||
- Fix get-team-invitation-token requiring only read permissions [#11358](https://github.com/penpot/penpot/issues/11358) (PR: [#11359](https://github.com/penpot/penpot/pull/11359))
|
||||
- Fix missing text in legacy SVG board thumbnails [#10182](https://github.com/penpot/penpot/issues/10182) (PR: [#11552](https://github.com/penpot/penpot/pull/11552))
|
||||
- Fix workspace crash when applying transform modifiers in the WASM renderer [#10894](https://github.com/penpot/penpot/issues/10894) (PR: [#10896](https://github.com/penpot/penpot/pull/10896))
|
||||
- Limit ZIP entry count and object size on V3 binfile import [#11021](https://github.com/penpot/penpot/issues/11021) (PR: [#11022](https://github.com/penpot/penpot/pull/11022))
|
||||
- Block plugin UI iframe URLs targeting the Penpot domain [#11271](https://github.com/penpot/penpot/issues/11271) (PR: [#11273](https://github.com/penpot/penpot/pull/11273))
|
||||
- Restrict the MCP REPL code execution endpoint to development environments [#11283](https://github.com/penpot/penpot/issues/11283) (PR: [#11282](https://github.com/penpot/penpot/pull/11282))
|
||||
- Filter share-link tokens from the get-view-only-bundle response [#11285](https://github.com/penpot/penpot/issues/11285) (PR: [#11286](https://github.com/penpot/penpot/pull/11286))
|
||||
- Disable MCP developer tools in multi-user mode [#11291](https://github.com/penpot/penpot/issues/11291) (PR: [#11310](https://github.com/penpot/penpot/pull/11310))
|
||||
- Fix Hide comments setting being ignored after opening the Comments section [#11308](https://github.com/penpot/penpot/issues/11308) (PR: [#11492](https://github.com/penpot/penpot/pull/11492))
|
||||
- Block NAT64/6to4/Teredo IPv6 transition addresses in the SSRF guard [#11319](https://github.com/penpot/penpot/issues/11319) (PR: [#11320](https://github.com/penpot/penpot/pull/11320))
|
||||
- Prevent team admins from removing the team owner [#11367](https://github.com/penpot/penpot/issues/11367) (PR: [#11368](https://github.com/penpot/penpot/pull/11368))
|
||||
- Enforce share-link comment permissions and page scope [#11370](https://github.com/penpot/penpot/issues/11370) (PR: [#11371](https://github.com/penpot/penpot/pull/11371))
|
||||
- Clean up orphaned teams, projects and files on profile deletion [#11394](https://github.com/penpot/penpot/issues/11394) (PR: [#11395](https://github.com/penpot/penpot/pull/11395))
|
||||
- Fix crash when pressing Ctrl+D with no shape selected [#11448](https://github.com/penpot/penpot/issues/11448) (PR: [#11491](https://github.com/penpot/penpot/pull/11491))
|
||||
- Fix text layout not updating when auto-width is set by double-clicking the bounding box [#11480](https://github.com/penpot/penpot/issues/11480) (PR: [#11541](https://github.com/penpot/penpot/pull/11541))
|
||||
- Fix boolean shapes rendering deformed in the WASM renderer and exports [#11482](https://github.com/penpot/penpot/issues/11482) (PR: [#11551](https://github.com/penpot/penpot/pull/11551))
|
||||
|
||||
### :sparkles: New features & Enhancements
|
||||
|
||||
@@ -218,6 +233,10 @@
|
||||
### :rocket: Epics and highlights
|
||||
|
||||
- Render prototype viewer with WASM (Skia) engine instead of SVG [#10037](https://github.com/penpot/penpot/issues/10037) (PR: [#10038](https://github.com/penpot/penpot/pull/10038))
|
||||
- Add layer blur effect for visual depth and styling [#9844](https://github.com/penpot/penpot/issues/9844) (PR: [#10034](https://github.com/penpot/penpot/pull/10034))
|
||||
- Render guides in WebGL for consistent viewer performance [#10068](https://github.com/penpot/penpot/issues/10068) (PR: [#10014](https://github.com/penpot/penpot/pull/10014))
|
||||
- Add concurrency limiter and status indicators for MCP server communications [#9493](https://github.com/penpot/penpot/issues/9493) (PR: [#9748](https://github.com/penpot/penpot/pull/9748))
|
||||
- Add typography token row to multiselected texts for better token visibility [#9336](https://github.com/penpot/penpot/issues/9336) (PR: [#9128](https://github.com/penpot/penpot/pull/9128))
|
||||
|
||||
### :sparkles: New features & Enhancements
|
||||
|
||||
@@ -572,6 +591,10 @@
|
||||
|
||||
## 2.15.0
|
||||
|
||||
### :rocket: Epics and highlights
|
||||
|
||||
- Add MCP server integration for AI-assisted design workflows [#9174](https://github.com/penpot/penpot/issues/9174) (PR: [#9032](https://github.com/penpot/penpot/pull/9032), [#9321](https://github.com/penpot/penpot/pull/9321))
|
||||
|
||||
### :sparkles: New features & Enhancements
|
||||
|
||||
- Add MCP server integration [GH #9174](https://github.com/penpot/penpot/issues/9174)
|
||||
|
||||
@@ -1,26 +0,0 @@
|
||||
# HIGHLIGHTS
|
||||
|
||||
## 2.17.0
|
||||
|
||||
- Background blur is here
|
||||
- WebGL rendering gets stronger
|
||||
- MCP connection status and more
|
||||
- Design tokens: more visible, more user-friendly
|
||||
|
||||
|
||||
## 2.16.0
|
||||
|
||||
- Design tokens in the design panel
|
||||
- Major community contributions
|
||||
- WebGL rendering (beta)
|
||||
|
||||
|
||||
## 2.15.0
|
||||
|
||||
- AI connected to real design context
|
||||
- Multi-directional workflow
|
||||
- Your stack, your model, your decision
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -56,6 +56,7 @@ If your organization is scaling and needs extra support, we’re here to help. [
|
||||
|
||||
- [Why Penpot](#why-penpot)
|
||||
- [Getting Started](#getting-started)
|
||||
- [Penpot Enterprise](#penpot-enterprise)
|
||||
- [Community](#community)
|
||||
- [Contributing](#contributing)
|
||||
- [Resources](#resources)
|
||||
@@ -93,6 +94,12 @@ Penpot is the only design & prototype platform that is deployment agnostic. You
|
||||
|
||||
Learn how to install it with Docker, Kubernetes, Elestio or other options on [our website](https://penpot.app/self-host).
|
||||
|
||||
<img width="100%" height="1010" alt="2" src="https://github.com/user-attachments/assets/243e796e-a140-481a-b68f-b24be6a70e37" />
|
||||
|
||||
## Penpot Enterprise ##
|
||||
|
||||
Penpot Enterprise is our paid plan for organizations that need to scale their design work across multiple teams with advanced governance, security, and administration. Manage teams and access from a centralized **Admin Console**, configure advanced permissions, and connect your **identity provider through SSO**. Available for cloud and self-hosted environments, it combines enterprise controls with Penpot’s open-source foundation and open standards.
|
||||
|
||||
## Community ##
|
||||
|
||||
We love the Open Source software community. Contributing is our passion and if it’s yours too, participate and [improve](https://community.penpot.app/c/help-us-improve-penpot/7) Penpot. All your designs, code and ideas are welcome!
|
||||
|
||||
@@ -31,7 +31,7 @@ export PENPOT_MEDIA_PROCESSING_SERVICE_URI=http://localhost:6065
|
||||
export PENPOT_FLAGS="\
|
||||
$PENPOT_FLAGS \
|
||||
enable-login-with-password \
|
||||
disable-login-with-ldap \
|
||||
enable-login-with-ldap \
|
||||
disable-login-with-oidc \
|
||||
disable-login-with-google \
|
||||
disable-login-with-github \
|
||||
|
||||
@@ -10,7 +10,7 @@
|
||||
[app.common.logging :as l]
|
||||
[app.common.schema :as sm]
|
||||
[clj-ldap.client :as ldap]
|
||||
[clojure.string]
|
||||
[cuerdas.core :as str]
|
||||
[integrant.core :as ig]))
|
||||
|
||||
(defn- prepare-params
|
||||
@@ -36,11 +36,22 @@
|
||||
:cause cause))))
|
||||
|
||||
(defn- replace-several [s & {:as replacements}]
|
||||
(reduce-kv clojure.string/replace s replacements))
|
||||
(reduce-kv str/replace s replacements))
|
||||
|
||||
(defn- escape-ldap-filter-value
|
||||
"Escapes special characters in a string for use in LDAP filter values,
|
||||
per RFC 4515 section 3."
|
||||
[s]
|
||||
(-> s
|
||||
(str/replace "\\" "\\5c")
|
||||
(str/replace "*" "\\2a")
|
||||
(str/replace "(" "\\28")
|
||||
(str/replace ")" "\\29")
|
||||
(str/replace "\u0000" "\\00")))
|
||||
|
||||
(defn- search-user
|
||||
[{:keys [::conn base-dn] :as cfg} email]
|
||||
(let [query (replace-several (:query cfg) ":username" email)
|
||||
(let [query (replace-several (:query cfg) ":username" (escape-ldap-filter-value email))
|
||||
attrs [(:attrs-username cfg)
|
||||
(:attrs-email cfg)
|
||||
(:attrs-fullname cfg)]
|
||||
@@ -49,12 +60,19 @@
|
||||
:attributes attrs}]
|
||||
(first (ldap/search conn base-dn params))))
|
||||
|
||||
(defn- get-attr
|
||||
"Retrieves an attribute from an LDAP entry. Handles multi-valued
|
||||
attributes by returning the first value."
|
||||
[entry attr-key]
|
||||
(let [v (get entry attr-key)]
|
||||
(if (coll? v) (first v) v)))
|
||||
|
||||
(defn- retrieve-user
|
||||
[{:keys [::conn] :as cfg} {:keys [email password]}]
|
||||
(when-let [{:keys [dn] :as user} (search-user cfg email)]
|
||||
(when (ldap/bind? conn dn password)
|
||||
{:fullname (get user (-> cfg :attrs-fullname keyword))
|
||||
:email email
|
||||
{:fullname (get-attr user (-> cfg :attrs-fullname keyword))
|
||||
:email (get-attr user (-> cfg :attrs-email keyword))
|
||||
:backend "ldap"})))
|
||||
|
||||
(def ^:private schema:info-data
|
||||
@@ -79,7 +97,7 @@
|
||||
(l/warn :hint "invalid response from ldap, looks like ldap is not configured correctly" :data user)
|
||||
(ex/raise :type :restriction
|
||||
:code :wrong-ldap-response
|
||||
:explain explain)))
|
||||
::sm/explain explain)))
|
||||
user)))
|
||||
|
||||
(defn- try-connectivity
|
||||
|
||||
@@ -293,6 +293,7 @@
|
||||
[:objects-storage-s3-bucket {:optional true} :string]
|
||||
[:objects-storage-s3-region {:optional true} :keyword]
|
||||
[:objects-storage-s3-endpoint {:optional true} ::sm/uri]
|
||||
[:objects-storage-s3-routes-file {:optional true} :string]
|
||||
|
||||
;; SSRF protection
|
||||
[:ssrf-allowed-hosts {:optional true} [::sm/set :string]]
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns app.graph.arrow
|
||||
"Bulk Ladybug ingest through in-memory Arrow.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns app.graph.debug
|
||||
"In-memory Ladybug sessions for the debug graph console."
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns app.graph.ingest
|
||||
"Penpot file -> Ladybug graph projection."
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns app.graph.ladybug
|
||||
"Ladybug access layer for graph-backed Penpot.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns app.graph.meta
|
||||
"`GraphMeta`: the graph's own account of who built it and from what.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns app.graph.projection.document
|
||||
"Project a Penpot file-data map into Ladybug nodes and structural edges.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns app.graph.projection.transforms
|
||||
"Derived graph links: edges a reader could compute from the projected
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns app.graph.report
|
||||
(:require
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns app.graph.schema
|
||||
"Ladybug DDL facade for the graph-backed Penpot vertical slice.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns app.graph.schema.contract
|
||||
"Deliberate choices in Penpot's graph schema, recorded as data.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns app.graph.schema.nodes
|
||||
"Single source of truth for graph node tables.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns app.graph.schema.projection
|
||||
"Derive Ladybug node column schemas from Penpot Malli sources.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns app.graph.schema.types
|
||||
"Map Malli schemas to Ladybug column types.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns app.graph.schema.values
|
||||
"Shape a Penpot value into the plain data its Ladybug column type wants.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns app.graph.stats
|
||||
(:require
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns app.graph.sync
|
||||
"Incremental Ladybug graph updates from Penpot file-change events."
|
||||
|
||||
@@ -15,6 +15,7 @@
|
||||
(:require
|
||||
[app.common.schema :as sm]
|
||||
[app.util.ssrf :as ssrf]
|
||||
[app.worker :as-alias wrk]
|
||||
[cuerdas.core :as str]
|
||||
[integrant.core :as ig]
|
||||
[java-http-clj.core :as http])
|
||||
@@ -23,6 +24,8 @@
|
||||
java.net.URI))
|
||||
|
||||
(def default-max-redirects 5)
|
||||
(def default-connect-timeout 30000)
|
||||
(def default-request-timeout 30000)
|
||||
|
||||
(defn client?
|
||||
[o]
|
||||
@@ -33,15 +36,17 @@
|
||||
:pred client?})
|
||||
|
||||
(defmethod ig/init-key ::client
|
||||
[_ _]
|
||||
(http/build-client {:connect-timeout 30000
|
||||
[_ {:keys [::wrk/executor]}]
|
||||
(http/build-client {:connect-timeout default-connect-timeout
|
||||
:executor executor
|
||||
:follow-redirects :never}))
|
||||
|
||||
(defn send!
|
||||
([client req] (send! client req {}))
|
||||
([client req {:keys [response-type] :or {response-type :string}}]
|
||||
(assert (client? client) "expected valid http client")
|
||||
(http/send req {:client client :as response-type})))
|
||||
(http/send (merge {:timeout default-request-timeout} req)
|
||||
{:client client :as response-type})))
|
||||
|
||||
(defn- resolve-client
|
||||
[params]
|
||||
|
||||
@@ -60,7 +60,13 @@
|
||||
|
||||
(defmethod handle-error :restriction
|
||||
[err request _]
|
||||
(let [{:keys [code] :as data} (ex-data err)]
|
||||
(let [data (ex-data err)
|
||||
code (get data :code)
|
||||
explain (ex/explain data)
|
||||
data (-> data
|
||||
(dissoc ::sm/explain)
|
||||
(cond-> explain (assoc :explain explain)))]
|
||||
|
||||
(if (= code :method-not-allowed)
|
||||
{::yres/status 405
|
||||
::yres/body data}
|
||||
|
||||
@@ -34,6 +34,7 @@
|
||||
[app.setup :as-alias setup]
|
||||
[app.srepl :as-alias srepl]
|
||||
[app.storage :as-alias sto]
|
||||
[app.storage.config :as sto.config]
|
||||
[app.storage.fs :as-alias sto.fs]
|
||||
[app.storage.gc-deleted :as-alias sto.gc-deleted]
|
||||
[app.storage.gc-touched :as-alias sto.gc-touched]
|
||||
@@ -148,6 +149,11 @@
|
||||
::mdef/labels []
|
||||
::mdef/type :histogram}})
|
||||
|
||||
(def ^:private storage-routing
|
||||
"Optional S3 storage targets and per semantic-bucket routing. Loaded once
|
||||
from the external routes file. Empty when the feature is not configured."
|
||||
(sto.config/load))
|
||||
|
||||
(def system-config
|
||||
{::db/pool
|
||||
{::db/uri (cf/get :database-uri)
|
||||
@@ -205,7 +211,7 @@
|
||||
::sto/storage (ig/ref ::sto/storage)}
|
||||
|
||||
::http.client/client
|
||||
{}
|
||||
{::wrk/executor (ig/ref ::wrk/executor)}
|
||||
|
||||
::session/manager
|
||||
{::db/pool (ig/ref ::db/pool)}
|
||||
@@ -522,7 +528,8 @@
|
||||
;; explicit migration because the database objects/rows will
|
||||
;; still reference the old names).
|
||||
:assets-s3 (ig/ref :app.storage.s3/backend)
|
||||
:assets-fs (ig/ref :app.storage.fs/backend)}}
|
||||
:assets-fs (ig/ref :app.storage.fs/backend)}
|
||||
::sto/bucket->target (:routes storage-routing)}
|
||||
|
||||
:app.storage.s3/backend
|
||||
{::sto.s3/region (or (cf/get :storage-assets-s3-region)
|
||||
@@ -533,6 +540,7 @@
|
||||
(cf/get :objects-storage-s3-bucket))
|
||||
::sto.s3/io-threads (or (cf/get :storage-assets-s3-io-threads)
|
||||
(cf/get :objects-storage-s3-io-threads))
|
||||
::sto.s3/targets (:targets storage-routing)
|
||||
|
||||
::wrk/netty-io-executor
|
||||
(ig/ref ::wrk/netty-io-executor)}
|
||||
|
||||
@@ -75,10 +75,10 @@
|
||||
{:method method
|
||||
:uri uri
|
||||
:body body
|
||||
:headers headers}
|
||||
:headers headers
|
||||
:timeout timeout}
|
||||
{:response-type :input-stream
|
||||
:skip-ssrf-check? true
|
||||
:timeout timeout})
|
||||
:skip-ssrf-check? true})
|
||||
status (:status resp)]
|
||||
(when (not (<= 200 status 299))
|
||||
(let [body (:body resp)]
|
||||
|
||||
@@ -390,18 +390,26 @@
|
||||
|
||||
(def ^:private sql:file-comment-users
|
||||
"WITH available_profiles AS (
|
||||
SELECT DISTINCT owner_id AS id
|
||||
FROM comment
|
||||
WHERE thread_id IN (SELECT id FROM comment_thread WHERE file_id=?)
|
||||
SELECT DISTINCT c.owner_id AS id
|
||||
FROM comment c
|
||||
JOIN comment_thread ct
|
||||
ON ct.id = c.thread_id
|
||||
WHERE ct.file_id = ?::uuid
|
||||
),
|
||||
profile_ids AS (
|
||||
SELECT id FROM available_profiles
|
||||
UNION
|
||||
SELECT ?::uuid
|
||||
)
|
||||
SELECT p.id,
|
||||
p.email,
|
||||
p.fullname AS name,
|
||||
p.fullname AS fullname,
|
||||
p.fullname,
|
||||
p.photo_id,
|
||||
p.is_active
|
||||
FROM profile AS p
|
||||
WHERE p.id IN (SELECT id FROM available_profiles) OR p.id=?")
|
||||
FROM profile p
|
||||
JOIN profile_ids AS x
|
||||
ON x.id = p.id;")
|
||||
|
||||
(defn get-file-comments-users
|
||||
[conn file-id profile-id]
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns app.rpc.commands.plugins
|
||||
(:require
|
||||
|
||||
@@ -41,6 +41,7 @@
|
||||
[app.rpc.notifications :as notifications]
|
||||
[app.storage :as sto]
|
||||
[app.util.services :as sv]
|
||||
[app.util.ssrf :as ssrf]
|
||||
[app.worker :as wrk]
|
||||
[cuerdas.core :as str]))
|
||||
|
||||
@@ -960,13 +961,18 @@ RETURNING id, deleted_at;")
|
||||
(sv/defmethod ::check-organization-sso
|
||||
"Validate an organization SSO configuration by generating a login redirect URL.
|
||||
Nitrate calls this while configuring SSO to verify client credentials and OIDC
|
||||
discovery before saving the settings."
|
||||
discovery before saving the settings. The issuer URL is nitrate-supplied
|
||||
(customer-configured), so it is checked against the SSRF blocklist before
|
||||
any outbound request is attempted."
|
||||
{::doc/added "2.18"
|
||||
::sm/params cto/schema:nitrate-sso
|
||||
::sm/result schema:check-organization-sso-result
|
||||
::rpc/auth false}
|
||||
[cfg params]
|
||||
{:valid (oidc/is-organization-sso-config-valid? cfg params)})
|
||||
(let [issuer (oidc/organization-sso-discovery-uri params)]
|
||||
{:valid (boolean (and issuer
|
||||
(ssrf/safe-url? issuer)
|
||||
(oidc/is-organization-sso-config-valid? cfg params)))}))
|
||||
|
||||
;; ---- API: notify-organization-sso-change
|
||||
(sv/defmethod ::notify-organization-sso-change
|
||||
|
||||
@@ -70,6 +70,7 @@
|
||||
[:map {:title "storage"}
|
||||
[::backends schema:backends]
|
||||
[::backend [:enum :s3 :fs]]
|
||||
[::bucket->target {:optional true} [:map-of :string :keyword]]
|
||||
::db/pool])
|
||||
|
||||
(def valid-storage?
|
||||
@@ -112,17 +113,18 @@
|
||||
params))
|
||||
|
||||
(defn- get-database-object-by-hash
|
||||
[connectable backend bucket hash]
|
||||
[connectable backend bucket target hash]
|
||||
(let [sql (str "select * from storage_object "
|
||||
" where (metadata->>'~:hash') = ? "
|
||||
" and (metadata->>'~:bucket') = ? "
|
||||
" and coalesce(metadata->>'~:storage-target', 'default') = ? "
|
||||
" and backend = ?"
|
||||
" and deleted_at is null"
|
||||
" and status = 'valid'"
|
||||
" limit 1")]
|
||||
;; NOTE: metadata is left encoded; row->storage-object is
|
||||
;; responsible for decoding it.
|
||||
(db/exec-one! connectable [sql hash bucket (name backend)])))
|
||||
(db/exec-one! connectable [sql hash bucket target (name backend)])))
|
||||
|
||||
(defn- promote-object!
|
||||
[storage object]
|
||||
@@ -185,6 +187,13 @@
|
||||
(let [ds (db/get-connectable storage)]
|
||||
(get-database-object ds id)))
|
||||
|
||||
(defn- resolve-target-id
|
||||
"Returns the storage target id for the given semantic bucket, or nil when
|
||||
the routing does not apply (non-S3 backends)."
|
||||
[storage bucket]
|
||||
(when (= :s3 (::backend storage))
|
||||
(get (::bucket->target storage) bucket :default)))
|
||||
|
||||
(defn put-object!
|
||||
"Creates a new object with the provided content."
|
||||
[{:keys [::backend ::db/pool] :as storage}
|
||||
@@ -193,9 +202,16 @@
|
||||
(assert (impl/content? content) "expected an instance of content")
|
||||
|
||||
(let [id (or (::id params) (uuid/random))
|
||||
mdata (cond-> (get-metadata params)
|
||||
base-mdata (get-metadata params)
|
||||
bucket (:bucket base-mdata)
|
||||
target (resolve-target-id storage bucket)
|
||||
target-str (or (some-> target name) "default")
|
||||
mdata (cond-> base-mdata
|
||||
(satisfies? impl/IContentHash content)
|
||||
(assoc :hash (impl/get-hash content)))
|
||||
(assoc :hash (impl/get-hash content))
|
||||
|
||||
(some? target)
|
||||
(assoc :storage-target target-str))
|
||||
|
||||
touched-at (if touch
|
||||
(or touched-at (ct/now))
|
||||
@@ -214,10 +230,11 @@
|
||||
(not= tempfile-bucket (:bucket mdata)))
|
||||
(get-database-object-by-hash pool backend
|
||||
(:bucket mdata)
|
||||
target-str
|
||||
(:hash mdata)))]
|
||||
|
||||
;; PHASE 2: an existing reference is found: reuse or repair it.
|
||||
(if (impl/exists-object? backend' hit)
|
||||
(if (impl/exists-object? backend' (row->storage-object hit))
|
||||
|
||||
;; PHASE 2a: healthy reference. Optionally refresh touched_at
|
||||
;; and reuse the object as it is.
|
||||
@@ -312,6 +329,16 @@
|
||||
(ct/is-after? (:expired-at object) (ct/now))))
|
||||
(-> (impl/get-object-url backend object nil) file-url->path))))
|
||||
|
||||
(defn target-resolvable?
|
||||
"Returns true when the backend referenced by `backend-id` can resolve
|
||||
`target` to a real destination. GC callers must refuse to delete objects
|
||||
whose target is not resolvable, so a misconfigured target never removes
|
||||
the database row while orphaning the blob."
|
||||
[storage backend-id target]
|
||||
(assert (valid-storage? storage))
|
||||
(-> (impl/resolve-backend storage backend-id)
|
||||
(impl/target-resolvable? target)))
|
||||
|
||||
(defn del-object!
|
||||
[storage object-or-id]
|
||||
(assert (valid-storage? storage))
|
||||
|
||||
@@ -0,0 +1,143 @@
|
||||
;; This Source Code Form is subject to the terms of the Mozilla Public
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns app.storage.config
|
||||
"Configuration of the optional S3 storage targets and their per
|
||||
semantic-bucket routing.
|
||||
|
||||
The routing is read from an external EDN file referenced by the
|
||||
`PENPOT_OBJECTS_STORAGE_S3_ROUTES_FILE` environment variable. The file
|
||||
declares additional S3 targets and, optionally, a map from Penpot
|
||||
semantic bucket to target id:
|
||||
|
||||
{:targets
|
||||
{:temp {:bucket \"penpot-temp\"}
|
||||
:cold {:bucket \"penpot-cold\" :region :us-east-1}}
|
||||
:routes
|
||||
{\"tempfile\" :temp
|
||||
\"file-data\" :temp}}
|
||||
|
||||
The implicit `:default` target always exists and is built from the
|
||||
existing `PENPOT_OBJECTS_STORAGE_S3_*` configuration, so the feature is
|
||||
fully backward compatible when the file is absent."
|
||||
(:refer-clojure :exclude [load])
|
||||
(:require
|
||||
[app.common.exceptions :as ex]
|
||||
[app.common.schema :as sm]
|
||||
[app.common.uri :as u]
|
||||
[app.config :as cf]
|
||||
[app.storage :as sto]
|
||||
[app.storage.s3 :as sto.s3]
|
||||
[clojure.edn :as edn]
|
||||
[clojure.java.io :as io]))
|
||||
|
||||
(def ^:private schema:target
|
||||
"The EDN declares targets with the same shape as the S3 backend, except
|
||||
the endpoint is accepted as a plain string and normalized to a URI by
|
||||
`normalize-target`."
|
||||
[:merge
|
||||
sto.s3/schema:target
|
||||
[:map
|
||||
[:endpoint {:optional true} [:or :string ::sm/uri]]]])
|
||||
|
||||
(def ^:private schema:file
|
||||
[:map {:title "storage-routes"}
|
||||
[:targets [:map-of :keyword schema:target]]
|
||||
[:routes {:optional true} [:map-of :string :keyword]]])
|
||||
|
||||
(def ^:private valid-file?
|
||||
(sm/validator schema:file))
|
||||
|
||||
(def ^:private explain-file
|
||||
(sm/explainer schema:file))
|
||||
|
||||
(defn- read-file
|
||||
[path]
|
||||
(try
|
||||
(-> (io/file path) slurp (edn/read-string))
|
||||
(catch Exception cause
|
||||
(ex/raise :type :validation
|
||||
:code :invalid-storage-routes-file
|
||||
:hint "unable to read storage routes file"
|
||||
:path (str path)
|
||||
:cause cause))))
|
||||
|
||||
(defn- assert-backend-s3
|
||||
"The routing only makes sense on top of the S3 backend."
|
||||
[]
|
||||
(let [backend (or (sto/get-legacy-backend)
|
||||
(cf/get :objects-storage-backend)
|
||||
:fs)]
|
||||
(when-not (= :s3 backend)
|
||||
(ex/raise :type :validation
|
||||
:code :invalid-storage-routes-backend
|
||||
:hint "storage routes file requires the :s3 backend"
|
||||
:backend (keyword backend)))))
|
||||
|
||||
(defn- validate
|
||||
[data path]
|
||||
(when-not (valid-file? data)
|
||||
(ex/raise :type :validation
|
||||
:code :invalid-storage-routes
|
||||
:hint "invalid storage routes file"
|
||||
:path (str path)
|
||||
:explain (explain-file data)))
|
||||
|
||||
(let [targets (:targets data)
|
||||
routes (:routes data)]
|
||||
|
||||
(when (contains? targets :default)
|
||||
(ex/raise :type :validation
|
||||
:code :reserved-storage-target
|
||||
:hint "`:default` is a reserved storage target id"
|
||||
:path (str path)))
|
||||
|
||||
(doseq [[bucket target] routes]
|
||||
(when-not (contains? sto/valid-buckets bucket)
|
||||
(ex/raise :type :validation
|
||||
:code :invalid-storage-routes-bucket
|
||||
:hint "unknown semantic bucket in storage routes"
|
||||
:bucket bucket
|
||||
:path (str path)))
|
||||
|
||||
(when-not (contains? targets target)
|
||||
(ex/raise :type :validation
|
||||
:code :unknown-storage-target
|
||||
:hint "storage route points to an undeclared target"
|
||||
:bucket bucket
|
||||
:target target
|
||||
:path (str path))))
|
||||
|
||||
data))
|
||||
|
||||
(defn- normalize-target
|
||||
[target]
|
||||
(cond-> target
|
||||
(string? (:endpoint target))
|
||||
(update :endpoint u/uri)))
|
||||
|
||||
(defn- normalize-targets
|
||||
[targets]
|
||||
(persistent!
|
||||
(reduce-kv (fn [acc id target]
|
||||
(assoc! acc id (normalize-target target)))
|
||||
(transient {})
|
||||
targets)))
|
||||
|
||||
(defn load
|
||||
"Reads and validates the optional S3 routing file.
|
||||
|
||||
Returns a map with `:targets` (id -> target definition) and `:routes`
|
||||
(semantic bucket -> target id), or `{:targets nil :routes nil}` when the
|
||||
configuration key is unset."
|
||||
[]
|
||||
(if-let [path (not-empty (cf/get :objects-storage-s3-routes-file))]
|
||||
(let [data (-> (read-file path) (validate path))]
|
||||
(assert-backend-s3)
|
||||
{:targets (normalize-targets (:targets data))
|
||||
:routes (:routes data)})
|
||||
{:targets nil
|
||||
:routes nil}))
|
||||
@@ -141,7 +141,7 @@
|
||||
(Files/deleteIfExists ^Path path)))
|
||||
|
||||
(defmethod impl/del-objects-in-bulk :fs
|
||||
[backend ids]
|
||||
[backend _target ids]
|
||||
(assert (valid-backend? backend) "expected a valid backend instance")
|
||||
(let [base (fs/path (::directory backend))]
|
||||
(reduce (fn [fail-ids id]
|
||||
@@ -153,3 +153,7 @@
|
||||
(conj fail-ids id)))))
|
||||
#{} ids)))
|
||||
|
||||
(defmethod impl/target-resolvable? :fs
|
||||
[_backend _target]
|
||||
true)
|
||||
|
||||
@@ -78,6 +78,24 @@
|
||||
(let [ids (db/create-array conn "uuid" ids)]
|
||||
(db/exec-one! conn [sql:delete-give-up ids max-attempts])))
|
||||
|
||||
(def ^:private sql:defer-unresolvable
|
||||
"UPDATE storage_object
|
||||
SET deleted_at = NOW() + INTERVAL '1 day'
|
||||
WHERE id = ANY(?::uuid[])")
|
||||
|
||||
(defn- park-unresolvable!
|
||||
"Refuses to delete rows whose target id is not configured: logs the
|
||||
misconfiguration and pushes `deleted_at` forward so the rows leave the
|
||||
selection window without being deleted and without counting a deletion
|
||||
attempt (they are therefore never subject to the give-up window)."
|
||||
[conn backend-id target ids]
|
||||
(l/err :hint "storage target is not configured, deletion refused"
|
||||
:backend (name backend-id)
|
||||
:target target
|
||||
:ids (mapv str ids))
|
||||
(let [ids (db/create-array conn "uuid" ids)]
|
||||
(db/exec-one! conn [sql:defer-unresolvable ids])))
|
||||
|
||||
(defn- process-chunk
|
||||
"Attempt to delete a chunk of storage objects from a specific backend.
|
||||
|
||||
@@ -87,11 +105,11 @@
|
||||
|
||||
Returns the number of successfully deleted objects, or 0 if no rows
|
||||
could be locked."
|
||||
[conn storage backend-id ids]
|
||||
[conn storage backend-id target ids]
|
||||
(if-let [locked-ids (lock-ids conn ids)]
|
||||
(let [fail-ids (try
|
||||
(-> (impl/resolve-backend storage backend-id)
|
||||
(impl/del-objects-in-bulk locked-ids))
|
||||
(impl/del-objects-in-bulk target locked-ids))
|
||||
(catch Throwable cause
|
||||
(l/err :hint "error on physical deletion, will retry"
|
||||
:ids locked-ids
|
||||
@@ -118,12 +136,15 @@
|
||||
(count ok-ids))
|
||||
0))
|
||||
|
||||
(defn- group-by-backend
|
||||
(defn- group-by-route
|
||||
[items]
|
||||
(d/group-by (comp keyword :backend) :id #{} items))
|
||||
(d/group-by (fn [item]
|
||||
[(keyword (:backend item)) (:target item)])
|
||||
:id #{} items))
|
||||
|
||||
(def ^:private sql:get-deleted-chunk
|
||||
"SELECT id, backend
|
||||
"SELECT id, backend,
|
||||
coalesce(metadata->>'~:storage-target', 'default') as target
|
||||
FROM storage_object
|
||||
WHERE deleted_at IS NOT NULL
|
||||
AND deleted_at <= ?
|
||||
@@ -139,19 +160,27 @@
|
||||
|
||||
(defn- clean-deleted!
|
||||
[cfg]
|
||||
(loop [total 0]
|
||||
(let [deleted (db/tx-run! cfg
|
||||
(fn [{:keys [::db/conn ::sto/storage]}]
|
||||
(let [chunk (get-deleted-chunk conn chunk-size)]
|
||||
(when (seq chunk)
|
||||
(let [by-backend (group-by-backend chunk)]
|
||||
(reduce-kv (fn [acc backend-id ids]
|
||||
(+ acc (process-chunk conn storage backend-id ids)))
|
||||
0
|
||||
by-backend))))))]
|
||||
(if deleted
|
||||
(recur (+ total deleted))
|
||||
total))))
|
||||
(loop [deleted 0
|
||||
parked 0]
|
||||
(let [result (db/tx-run! cfg
|
||||
(fn [{:keys [::db/conn ::sto/storage]}]
|
||||
(let [chunk (get-deleted-chunk conn chunk-size)]
|
||||
(when (seq chunk)
|
||||
(let [by-route (group-by-route chunk)]
|
||||
(reduce-kv
|
||||
(fn [acc [backend-id target] ids]
|
||||
(if (sto/target-resolvable? storage backend-id target)
|
||||
(update acc :deleted + (process-chunk conn storage backend-id target ids))
|
||||
(do
|
||||
(park-unresolvable! conn backend-id target ids)
|
||||
(update acc :parked + (count ids)))))
|
||||
{:deleted 0 :parked 0}
|
||||
by-route))))))]
|
||||
(if result
|
||||
(recur (+ deleted (:deleted result))
|
||||
(+ parked (:parked result)))
|
||||
{:deleted deleted
|
||||
:parked parked}))))
|
||||
|
||||
(defmethod ig/assert-key ::handler
|
||||
[_ params]
|
||||
@@ -161,6 +190,7 @@
|
||||
(defmethod ig/init-key ::handler
|
||||
[_ cfg]
|
||||
(fn [_]
|
||||
(let [total (clean-deleted! cfg)]
|
||||
(l/inf :hint "task finished" :total total)
|
||||
{:deleted total})))
|
||||
(let [{:keys [deleted parked]} (clean-deleted! cfg)]
|
||||
(l/inf :hint "task finished" :total deleted :parked parked)
|
||||
{:deleted deleted
|
||||
:parked parked})))
|
||||
@@ -72,12 +72,14 @@
|
||||
:context cfg))
|
||||
|
||||
(defmulti del-objects-in-bulk
|
||||
"Delete multiple objects in bulk. Returns #{fail-ids} — the set of ids
|
||||
whose blob deletion failed. Empty set = all succeeded."
|
||||
(fn [cfg _] (::sto/type cfg)))
|
||||
"Delete multiple objects in bulk. `target` is an optional backend-specific
|
||||
destination id (used by the S3 storage targets); backends that do not route
|
||||
ignore it. Returns #{fail-ids} — the set of ids whose blob deletion failed.
|
||||
Empty set = all succeeded."
|
||||
(fn [cfg _ _] (::sto/type cfg)))
|
||||
|
||||
(defmethod del-objects-in-bulk :default
|
||||
[cfg _]
|
||||
[cfg _ _]
|
||||
(ex/raise :type :internal
|
||||
:code :invalid-storage-backend
|
||||
:context cfg))
|
||||
@@ -90,6 +92,17 @@
|
||||
:code :invalid-storage-backend
|
||||
:context cfg))
|
||||
|
||||
(defmulti target-resolvable?
|
||||
"Returns true when the backend can resolve `target` to a real destination.
|
||||
GC callers must refuse deletion (and keep the row) when this is false."
|
||||
(fn [cfg _] (::sto/type cfg)))
|
||||
|
||||
(defmethod target-resolvable? :default
|
||||
[cfg _]
|
||||
(ex/raise :type :internal
|
||||
:code :invalid-storage-backend
|
||||
:context cfg))
|
||||
|
||||
;; --- HELPERS
|
||||
|
||||
(defn uuid->hex
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns app.storage.pending-gc
|
||||
"A maintenance task that reclaims storage objects created in 'pending'
|
||||
@@ -20,10 +20,12 @@
|
||||
[integrant.core :as ig]))
|
||||
|
||||
(def ^:private sql:get-pending-sobjects
|
||||
"SELECT id, backend
|
||||
"SELECT id, backend,
|
||||
coalesce(metadata->>'~:storage-target', 'default') as target
|
||||
FROM storage_object
|
||||
WHERE status = 'pending'
|
||||
AND created_at <= now() - interval '24 hours'
|
||||
AND (deleted_at IS NULL OR deleted_at <= now())
|
||||
ORDER BY created_at ASC
|
||||
LIMIT ?
|
||||
FOR UPDATE
|
||||
@@ -36,30 +38,71 @@
|
||||
(def ^:private sql:delete-pending-sobject
|
||||
"DELETE FROM storage_object WHERE id = ? AND status = 'pending'")
|
||||
|
||||
(def ^:private sql:park-unresolvable
|
||||
"UPDATE storage_object
|
||||
SET deleted_at = NOW() + INTERVAL '1 day'
|
||||
WHERE id = ANY(?::uuid[])
|
||||
AND status = 'pending'")
|
||||
|
||||
(defn- park-unresolvable!
|
||||
"Refuses to delete rows whose target id is not configured: logs the
|
||||
misconfiguration and pushes `deleted_at` forward so the rows are excluded
|
||||
from the next selection without being deleted."
|
||||
[conn backend-id target ids]
|
||||
(l/err :hint "storage target is not configured, deletion refused"
|
||||
:backend (name backend-id)
|
||||
:target target
|
||||
:ids (mapv str ids))
|
||||
(let [ids (db/create-array conn "uuid" ids)]
|
||||
(db/exec-one! conn [sql:park-unresolvable ids])))
|
||||
|
||||
(def ^:private chunk-size
|
||||
100)
|
||||
|
||||
(defn- group-by-route
|
||||
[rows]
|
||||
(group-by (fn [{:keys [backend target]}]
|
||||
[(keyword backend) target])
|
||||
rows))
|
||||
|
||||
(defn- delete-pending-rows!
|
||||
"Select, lock and delete a chunk of pending rows in a single transaction.
|
||||
Returns the deleted rows or nil when there is nothing left to reclaim."
|
||||
Rows whose target id is not configured are never deleted: they are parked
|
||||
(error logged, `deleted_at` pushed forward) so the loop terminates and the
|
||||
rows survive.
|
||||
|
||||
Returns a map `{:deleted rows :parked count}`, or nil when there is nothing
|
||||
left to reclaim."
|
||||
[cfg]
|
||||
(db/tx-run! cfg
|
||||
(fn [{:keys [::db/conn]}]
|
||||
(fn [{:keys [::db/conn ::sto/storage]}]
|
||||
;; NOTE: db/exec! returns an empty vector when there are no
|
||||
;; rows left; use not-empty to detect it.
|
||||
(when-let [chunk (not-empty (get-pending-chunk conn chunk-size))]
|
||||
(doseq [{:keys [id]} chunk]
|
||||
(db/exec-one! conn [sql:delete-pending-sobject id]))
|
||||
chunk))))
|
||||
(reduce-kv
|
||||
(fn [acc [backend-id target] rows]
|
||||
(if (sto/target-resolvable? storage backend-id target)
|
||||
(do
|
||||
(doseq [{:keys [id]} rows]
|
||||
(db/exec-one! conn [sql:delete-pending-sobject id]))
|
||||
(update acc :deleted into rows))
|
||||
(do
|
||||
(park-unresolvable! conn backend-id target (mapv :id rows))
|
||||
(update acc :parked + (count rows)))))
|
||||
{:deleted [] :parked 0}
|
||||
(group-by-route chunk))))))
|
||||
|
||||
(defn- delete-blobs!
|
||||
"Best-effort removal of the orphaned blobs. Runs after the pending rows
|
||||
have been committed so a failure here never blocks their reclamation."
|
||||
have been committed so a failure here never blocks their reclamation.
|
||||
|
||||
The `:storage-target` metadata is load-bearing: the S3 backend reads it
|
||||
(`s3/target-id`) to resolve the bucket/prefix/client for `del-object`."
|
||||
[storage rows]
|
||||
(doseq [{:keys [id backend]} rows]
|
||||
(doseq [{:keys [id backend target]} rows]
|
||||
(try
|
||||
(-> (impl/resolve-backend storage (keyword backend))
|
||||
(impl/del-object {:id id}))
|
||||
(impl/del-object (with-meta {:id id} {:storage-target target})))
|
||||
(catch Throwable cause
|
||||
(l/err :hint "error deleting orphaned pending blob"
|
||||
:id (str id)
|
||||
@@ -68,12 +111,16 @@
|
||||
|
||||
(defn- process!
|
||||
[{::sto/keys [storage] :as cfg}]
|
||||
(loop [total 0]
|
||||
(if-let [rows (delete-pending-rows! cfg)]
|
||||
(do
|
||||
(delete-blobs! storage rows)
|
||||
(recur (long (+ total (count rows)))))
|
||||
total)))
|
||||
(loop [total-deleted 0
|
||||
total-parked 0]
|
||||
(if-let [result (delete-pending-rows! cfg)]
|
||||
(let [removed (:deleted result)
|
||||
parked (:parked result)]
|
||||
(delete-blobs! storage removed)
|
||||
(recur (long (+ total-deleted (count removed)))
|
||||
(long (+ total-parked parked))))
|
||||
{:processed total-deleted
|
||||
:parked total-parked})))
|
||||
|
||||
(defmethod ig/assert-key ::handler
|
||||
[_ params]
|
||||
@@ -83,6 +130,7 @@
|
||||
(defmethod ig/init-key ::handler
|
||||
[_ cfg]
|
||||
(fn [_]
|
||||
(let [total (process! cfg)]
|
||||
(l/inf :hint "task finished" :total total)
|
||||
{:processed total})))
|
||||
(let [{:keys [processed parked]} (process! cfg)]
|
||||
(l/inf :hint "task finished" :total processed :parked parked)
|
||||
{:processed processed
|
||||
:parked parked})))
|
||||
@@ -88,13 +88,21 @@
|
||||
|
||||
;; --- BACKEND INIT
|
||||
|
||||
(def schema:target
|
||||
[:map {:title "s3-target"}
|
||||
[:bucket ::sm/text]
|
||||
[:region {:optional true} :keyword]
|
||||
[:endpoint {:optional true} ::sm/uri]
|
||||
[:prefix {:optional true} ::sm/text]])
|
||||
|
||||
(def ^:private schema:config
|
||||
[:map {:title "s3-backend-config"}
|
||||
::wrk/netty-io-executor
|
||||
[::region {:optional true} :keyword]
|
||||
[::bucket {:optional true} ::sm/text]
|
||||
[::prefix {:optional true} ::sm/text]
|
||||
[::endpoint {:optional true} ::sm/uri]])
|
||||
[::endpoint {:optional true} ::sm/uri]
|
||||
[::targets {:optional true} [:map-of :keyword schema:target]]])
|
||||
|
||||
(defmethod ig/expand-key ::backend
|
||||
[k v]
|
||||
@@ -104,42 +112,131 @@
|
||||
[_ params]
|
||||
(assert (sm/check schema:config params)))
|
||||
|
||||
(defn- build-client-pair
|
||||
[{:keys [::wrk/netty-io-executor]} region endpoint]
|
||||
(let [params {::region region
|
||||
::endpoint endpoint
|
||||
::wrk/netty-io-executor netty-io-executor}
|
||||
client (build-s3-client params)
|
||||
presigner (build-s3-presigner params)]
|
||||
{:client @client
|
||||
:presigner presigner
|
||||
:close-fn #(.close ^java.lang.AutoCloseable client)}))
|
||||
|
||||
(defn- build-client-pair-or-cleanup
|
||||
"Builds a client pair, closing the pairs already built when the build
|
||||
fails so a failed backend init does not leak clients."
|
||||
[acc params region endpoint]
|
||||
(try
|
||||
(build-client-pair params region endpoint)
|
||||
(catch Throwable cause
|
||||
(doseq [f (:close-fns acc)]
|
||||
(ex/ignoring (f)))
|
||||
(throw cause))))
|
||||
|
||||
(defn- build-targets
|
||||
"Resolves the implicit `:default` target plus the declared targets, sharing
|
||||
one S3 client/presigner pair per distinct `[region endpoint]`."
|
||||
[{:keys [::region ::endpoint ::bucket ::prefix ::targets] :as params}]
|
||||
(let [defs (merge {:default {:region region :endpoint endpoint
|
||||
:bucket bucket :prefix prefix}}
|
||||
(into {}
|
||||
(map (fn [[id target]]
|
||||
[id {:region (or (:region target) region)
|
||||
:endpoint (or (:endpoint target) endpoint)
|
||||
:bucket (:bucket target)
|
||||
:prefix (or (:prefix target) prefix)}]))
|
||||
targets))
|
||||
result (reduce-kv
|
||||
(fn [acc id {:keys [region endpoint bucket prefix]}]
|
||||
(let [k [region endpoint]
|
||||
pair (or (get-in acc [:pairs k])
|
||||
(build-client-pair-or-cleanup acc params region endpoint))
|
||||
acc (cond-> acc
|
||||
(nil? (get-in acc [:pairs k]))
|
||||
(-> (assoc-in [:pairs k] pair)
|
||||
(update :close-fns conj (:close-fn pair))))]
|
||||
(assoc-in acc [:targets id]
|
||||
{::client (:client pair)
|
||||
::presigner (:presigner pair)
|
||||
::bucket bucket
|
||||
::prefix prefix})))
|
||||
{:pairs {} :targets {} :close-fns []}
|
||||
defs)]
|
||||
(select-keys result [:targets :close-fns])))
|
||||
|
||||
(defmethod ig/init-key ::backend
|
||||
[_ params]
|
||||
(when (and (contains? params ::region)
|
||||
(contains? params ::bucket))
|
||||
(let [client (build-s3-client params)
|
||||
presigner (build-s3-presigner params)]
|
||||
(let [{:keys [targets close-fns]} (build-targets params)]
|
||||
(assoc params
|
||||
::sto/type :s3
|
||||
::counter (AtomicLong. 0)
|
||||
::client @client
|
||||
::presigner presigner
|
||||
::close-fn #(.close ^java.lang.AutoCloseable client)))))
|
||||
::default-target :default
|
||||
::targets targets
|
||||
::close-fns (vec close-fns)))))
|
||||
|
||||
(defmethod ig/resolve-key ::backend
|
||||
[_ params]
|
||||
(dissoc params ::close-fn))
|
||||
(dissoc params ::close-fns))
|
||||
|
||||
(defmethod ig/halt-key! ::backend
|
||||
[_ {:keys [::close-fn]}]
|
||||
(when (fn? close-fn)
|
||||
(close-fn)))
|
||||
[_ {:keys [::close-fns]}]
|
||||
(doseq [f close-fns]
|
||||
(when (fn? f)
|
||||
(f))))
|
||||
|
||||
(def ^:private schema:backend
|
||||
[:map {:title "s3-backend"}
|
||||
;; [::region :keyword]
|
||||
;; [::bucket ::sm/text]
|
||||
[::client [:fn #(instance? S3AsyncClient %)]]
|
||||
[::presigner [:fn #(instance? S3Presigner %)]]
|
||||
[::prefix {:optional true} ::sm/text]
|
||||
#_[::sto/type [:= :s3]]])
|
||||
[::default-target :keyword]
|
||||
[::targets
|
||||
[:map-of :keyword
|
||||
[:map
|
||||
[::client [:fn #(instance? S3AsyncClient %)]]
|
||||
[::presigner [:fn #(instance? S3Presigner %)]]
|
||||
[::bucket ::sm/text]
|
||||
[::prefix {:optional true} ::sm/text]]]]])
|
||||
|
||||
(sm/register! ::backend schema:backend)
|
||||
|
||||
(def ^:private valid-backend?
|
||||
(sm/validator schema:backend))
|
||||
|
||||
;; --- TARGET RESOLUTION
|
||||
|
||||
(defn- target-id
|
||||
"Returns the configured target id stored on the object, or the default
|
||||
target name when the object predates the routing feature."
|
||||
[backend object]
|
||||
(or (some-> (meta object) :storage-target)
|
||||
(name (::default-target backend))))
|
||||
|
||||
(defn- resolve-target-by-id
|
||||
[backend target-id]
|
||||
(let [tid (cond
|
||||
(nil? target-id) (::default-target backend)
|
||||
(keyword? target-id) target-id
|
||||
:else (keyword target-id))]
|
||||
(or (get (::targets backend) tid)
|
||||
(ex/raise :type :internal
|
||||
:code :invalid-storage-target
|
||||
:hint "storage target is not configured"
|
||||
:target target-id
|
||||
:available (vec (keys (::targets backend)))))))
|
||||
|
||||
(defn- resolve-target
|
||||
[backend object]
|
||||
(resolve-target-by-id backend (target-id backend object)))
|
||||
|
||||
(defmethod impl/target-resolvable? :s3
|
||||
[backend target-id]
|
||||
(let [tid (cond
|
||||
(nil? target-id) (::default-target backend)
|
||||
(keyword? target-id) target-id
|
||||
:else (keyword target-id))]
|
||||
(contains? (::targets backend) tid)))
|
||||
|
||||
;; --- API IMPL
|
||||
|
||||
(defmethod impl/put-object :s3
|
||||
@@ -210,13 +307,14 @@
|
||||
true)))
|
||||
|
||||
(defmethod impl/del-objects-in-bulk :s3
|
||||
[backend ids]
|
||||
[backend target ids]
|
||||
(assert (valid-backend? backend) "expected a valid backend instance")
|
||||
(let [key->id (into {} (map (fn [id]
|
||||
[(str (::prefix backend) (impl/id->path id)) id]))
|
||||
(let [target (resolve-target-by-id backend target)
|
||||
key->id (into {} (map (fn [id]
|
||||
[(str (::prefix target) (impl/id->path id)) id]))
|
||||
ids)
|
||||
result (try
|
||||
(p/await! (del-object-in-bulk backend ids))
|
||||
(p/await! (del-object-in-bulk target ids))
|
||||
(catch Throwable cause
|
||||
(l/err :hint "error on s3 bulk deletion"
|
||||
:ids ids
|
||||
@@ -320,8 +418,9 @@
|
||||
^Subscriber subscriber))))))
|
||||
|
||||
(defn- put-object
|
||||
[{:keys [::client ::bucket ::prefix ::counter]} {:keys [id] :as object} content]
|
||||
(let [path (dm/str prefix (impl/id->path id))
|
||||
[{:keys [::counter] :as backend} {:keys [id] :as object} content]
|
||||
(let [{:keys [::client ::bucket ::prefix]} (resolve-target backend object)
|
||||
path (dm/str prefix (impl/id->path id))
|
||||
mdata (meta object)
|
||||
mtype (:content-type mdata "application/octet-stream")
|
||||
rbody (make-request-body counter content)
|
||||
@@ -344,8 +443,9 @@
|
||||
(proxy-super close))))
|
||||
|
||||
(defn- get-object-data
|
||||
[{:keys [::client ::bucket ::prefix]} {:keys [id size]}]
|
||||
(let [gor (.. (GetObjectRequest/builder)
|
||||
[backend {:keys [id size] :as object}]
|
||||
(let [{:keys [::client ::bucket ::prefix]} (resolve-target backend object)
|
||||
gor (.. (GetObjectRequest/builder)
|
||||
(bucket bucket)
|
||||
(key (str prefix (impl/id->path id)))
|
||||
(build))]
|
||||
@@ -369,16 +469,18 @@
|
||||
(p/fmap #(.asInputStream ^ResponseBytes %)))))))
|
||||
|
||||
(defn- head-object
|
||||
[{:keys [::client ::bucket ::prefix]} {:keys [id]}]
|
||||
(let [hor (.. (HeadObjectRequest/builder)
|
||||
[backend {:keys [id] :as object}]
|
||||
(let [{:keys [::client ::bucket ::prefix]} (resolve-target backend object)
|
||||
hor (.. (HeadObjectRequest/builder)
|
||||
(bucket bucket)
|
||||
(key (str prefix (impl/id->path id)))
|
||||
(build))]
|
||||
(.headObject ^S3AsyncClient client ^HeadObjectRequest hor)))
|
||||
|
||||
(defn- get-object-bytes
|
||||
[{:keys [::client ::bucket ::prefix]} {:keys [id]}]
|
||||
(let [gor (.. (GetObjectRequest/builder)
|
||||
[backend {:keys [id] :as object}]
|
||||
(let [{:keys [::client ::bucket ::prefix]} (resolve-target backend object)
|
||||
gor (.. (GetObjectRequest/builder)
|
||||
(bucket bucket)
|
||||
(key (str prefix (impl/id->path id)))
|
||||
(build))
|
||||
@@ -392,7 +494,7 @@
|
||||
(ct/duration {:minutes 10}))
|
||||
|
||||
(defn- get-object-url
|
||||
[{:keys [::presigner ::bucket ::prefix]} {:keys [id]}
|
||||
[backend {:keys [id] :as object}
|
||||
{:keys [max-age content-disposition] :or {max-age default-max-age}}]
|
||||
(assert (ct/duration? max-age) "expected valid duration instance")
|
||||
|
||||
@@ -400,7 +502,8 @@
|
||||
;; object store sets that header on the response the client fetches after
|
||||
;; following the redirect. It is only set when asked for, so urls for
|
||||
;; objects served inline stay byte identical to before.
|
||||
(let [gorb (.. (GetObjectRequest/builder)
|
||||
(let [{:keys [::presigner ::bucket ::prefix]} (resolve-target backend object)
|
||||
gorb (.. (GetObjectRequest/builder)
|
||||
(bucket bucket)
|
||||
(key (dm/str prefix (impl/id->path id))))
|
||||
gorb (cond-> gorb
|
||||
@@ -415,8 +518,9 @@
|
||||
(u/uri (str (.url ^PresignedGetObjectRequest pgor)))))
|
||||
|
||||
(defn- del-object
|
||||
[{:keys [::bucket ::client ::prefix]} {:keys [id] :as obj}]
|
||||
(let [dor (.. (DeleteObjectRequest/builder)
|
||||
[backend {:keys [id] :as object}]
|
||||
(let [{:keys [::bucket ::client ::prefix]} (resolve-target backend object)
|
||||
dor (.. (DeleteObjectRequest/builder)
|
||||
(bucket bucket)
|
||||
(key (dm/str prefix (impl/id->path id)))
|
||||
(build))]
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns app.tasks.demo-purge
|
||||
"Task handler for delayed demo profile deletion. Submitted at demo
|
||||
|
||||
@@ -0,0 +1,76 @@
|
||||
;; This Source Code Form is subject to the terms of the Mozilla Public
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns backend-tests.auth-ldap-test
|
||||
(:require
|
||||
[app.auth.ldap :as ldap-auth]
|
||||
[clj-ldap.client :as ldap]
|
||||
[clojure.test :as t]))
|
||||
|
||||
;; --- search-user: filter must be escaped (RED: currently not escaped)
|
||||
|
||||
(t/deftest search-user-escapes-email-in-filter
|
||||
(t/testing "wildcard * is escaped before building LDAP filter"
|
||||
(let [captured-query (atom nil)
|
||||
fake-search (fn [_conn _base-dn params]
|
||||
(reset! captured-query (:filter params))
|
||||
[])]
|
||||
(with-redefs [ldap/search fake-search]
|
||||
(#'ldap-auth/search-user {:query "(mail=:username)" :sizelimit 1
|
||||
:attrs-username "uid" :attrs-email "mail"
|
||||
:attrs-fullname "cn"}
|
||||
"fry*@planetexpress.com"))
|
||||
;; After fix: * should be escaped as \2a
|
||||
(t/is (= "(mail=fry\\2a@planetexpress.com)" @captured-query)
|
||||
"filter must have * escaped per RFC 4515"))))
|
||||
|
||||
;; --- retrieve-user: email must come from directory, not client (RED)
|
||||
|
||||
(t/deftest retrieve-user-uses-directory-email
|
||||
(t/testing "returned email is from LDAP directory, not client input"
|
||||
(let [fake-search (fn [_conn _base-dn _params]
|
||||
[{:dn "cn=fry,ou=people,dc=planetexpress,dc=com"
|
||||
:mail "fry@planetexpress.com"
|
||||
:cn "Philip J. Fry"
|
||||
:uid "fry"}])
|
||||
fake-bind? (fn [_conn _dn _password] true)]
|
||||
(with-redefs [ldap/search fake-search
|
||||
ldap/bind? fake-bind?]
|
||||
(let [cfg {:query "(mail=:username)" :sizelimit 1
|
||||
:attrs-username "uid" :attrs-email "mail"
|
||||
:attrs-fullname "cn"}
|
||||
result (#'ldap-auth/retrieve-user cfg {:email "fry*@planetexpress.com" :password "fry"})]
|
||||
;; After fix: email should be from directory (fry@planetexpress.com)
|
||||
;; BUG: email is client input (fry*@planetexpress.com)
|
||||
(t/is (= "fry@planetexpress.com" (:email result))
|
||||
"email must come from LDAP directory attribute, not client input"))))))
|
||||
|
||||
;; --- authenticate: full flow with directory email (RED)
|
||||
|
||||
(t/deftest authenticate-returns-directory-email
|
||||
(t/testing "authenticate returns directory email for profile"
|
||||
(let [fake-search (fn [_conn _base-dn _params]
|
||||
[{:dn "cn=amy,ou=people,dc=planetexpress,dc=com"
|
||||
:mail "amy@planetexpress.com"
|
||||
:cn "Amy Wong"
|
||||
:uid "amy"}])
|
||||
fake-bind? (fn [_conn _dn _password] true)]
|
||||
(with-redefs [ldap/search fake-search
|
||||
ldap/bind? fake-bind?
|
||||
ldap/connect (fn [_cfg] (reify java.lang.AutoCloseable (close [_] nil)))]
|
||||
(let [cfg {:query "(mail=:username)" :sizelimit 1
|
||||
:attrs-username "uid" :attrs-email "mail"
|
||||
:attrs-fullname "cn"
|
||||
:bind-dn "cn=admin,dc=planetexpress,dc=com"
|
||||
:bind-password "GoodNewsEveryone"
|
||||
:host "localhost" :port 10389
|
||||
:ssl false :tls false
|
||||
:base-dn "ou=people,dc=planetexpress,dc=com"}
|
||||
result (ldap-auth/authenticate cfg {:email "*@planetexpress.com" :password "amy"})]
|
||||
;; After fix: email should be amy@planetexpress.com (directory)
|
||||
;; BUG: email is *@planetexpress.com (client)
|
||||
(t/is (= "amy@planetexpress.com" (:email result))
|
||||
"authenticate must return directory email, not client-supplied wildcard"))))))
|
||||
@@ -2,7 +2,7 @@
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns backend-tests.demo-test
|
||||
(:require
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns backend-tests.graph-binder-gate-test
|
||||
"Binder gate for the incremental-sync statement templates.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns backend-tests.graph-sync-parity-test
|
||||
"Cold projection and incremental sync are two implementations of one mapping,
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
;; This Source Code Form is subject to the terms of the Mozilla Public
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns backend-tests.http-client-test
|
||||
(:require
|
||||
[app.http.client :as http]
|
||||
[clojure.test :as t]
|
||||
[java-http-clj.core :as jhttp]
|
||||
[mockery.core :refer [with-mocks]]))
|
||||
|
||||
(t/deftest send-injects-default-timeout-when-absent
|
||||
(with-mocks [mock {:target 'java-http-clj.core/send
|
||||
:return {:status 200 :body ""}}]
|
||||
(let [client (jhttp/build-client {})]
|
||||
(http/send! client {:method :get :uri "https://example.com/"})
|
||||
(let [[req _opts] (:call-args @mock)]
|
||||
(t/is (= http/default-request-timeout (:timeout req)))))))
|
||||
|
||||
(t/deftest send-preserves-caller-supplied-timeout
|
||||
(with-mocks [mock {:target 'java-http-clj.core/send
|
||||
:return {:status 200 :body ""}}]
|
||||
(let [client (jhttp/build-client {})]
|
||||
(http/send! client {:method :get
|
||||
:uri "https://example.com/"
|
||||
:timeout 5000})
|
||||
(let [[req _opts] (:call-args @mock)]
|
||||
(t/is (= 5000 (:timeout req)))))))
|
||||
@@ -8,6 +8,7 @@
|
||||
(:require
|
||||
[app.common.exceptions :as ex]
|
||||
[app.config :as cf]
|
||||
[app.http.client :as http]
|
||||
[app.media.remote :as media.remote]
|
||||
[app.setup :as-alias setup]
|
||||
[app.util.json :as json]
|
||||
@@ -500,6 +501,22 @@
|
||||
:headers {}})]
|
||||
(t/is (= 200 (:status resp))))))))
|
||||
|
||||
(t/deftest service-request-puts-configured-timeout-in-request
|
||||
(t/testing "service-request puts media-processing-service-timeout on the http request"
|
||||
(let [captured (atom nil)]
|
||||
(with-redefs [cf/get (th/config-get-mock config-mock)
|
||||
http/req (fn [_client request _opts]
|
||||
(reset! captured request)
|
||||
{:status 200
|
||||
:body (json-stream {:width 100 :height 100})})]
|
||||
(media.remote/service-request
|
||||
(mk-system)
|
||||
{:method :post
|
||||
:uri "http://localhost:6065/api/image/info"
|
||||
:body nil
|
||||
:headers {}})
|
||||
(t/is (= 5000 (:timeout @captured)))))))
|
||||
|
||||
;; ---------------------------------------------------------------------------
|
||||
;; Shared key
|
||||
;; ---------------------------------------------------------------------------
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns backend-tests.passwords-test
|
||||
(:require
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns backend-tests.rpc-demo-test
|
||||
(:require
|
||||
|
||||
@@ -17,6 +17,7 @@
|
||||
[app.msgbus :as mbus]
|
||||
[app.nitrate :as nitrate]
|
||||
[app.rpc :as-alias rpc]
|
||||
[app.util.ssrf :as ssrf]
|
||||
[app.worker :as wrk]
|
||||
[backend-tests.helpers :as th]
|
||||
[clojure.set :as set]
|
||||
@@ -1806,13 +1807,14 @@
|
||||
|
||||
(t/deftest check-organization-sso-returns-valid-true
|
||||
(let [organization-id (uuid/random)
|
||||
out (with-redefs [oidc/is-organization-sso-config-valid? (constantly true)]
|
||||
(th/management-command!
|
||||
{::th/type :check-organization-sso
|
||||
:organization-id organization-id
|
||||
:client-id "test-client"
|
||||
:client-secret "test-secret"
|
||||
:issuer "https://idp.example.com"}))]
|
||||
out (with-redefs [ssrf/safe-url? (constantly true)
|
||||
oidc/is-organization-sso-config-valid? (constantly true)]
|
||||
(th/management-command!
|
||||
{::th/type :check-organization-sso
|
||||
:organization-id organization-id
|
||||
:client-id "test-client"
|
||||
:client-secret "test-secret"
|
||||
:issuer "https://idp.example.com"}))]
|
||||
(t/is (th/success? out))
|
||||
(t/is (true? (-> out :result :valid)))))
|
||||
|
||||
@@ -1827,19 +1829,36 @@
|
||||
|
||||
(t/deftest check-organization-sso-passes-issuer-to-validation
|
||||
(let [organization-id (uuid/random)
|
||||
out (with-redefs [oidc/is-organization-sso-config-valid?
|
||||
(fn [_cfg sso]
|
||||
(and (= "test-client" (:client-id sso))
|
||||
(= "https://idp.example.com/" (:issuer sso))))]
|
||||
(th/management-command!
|
||||
{::th/type :check-organization-sso
|
||||
:organization-id organization-id
|
||||
:client-id "test-client"
|
||||
:client-secret "test-secret"
|
||||
:issuer "https://idp.example.com/"}))]
|
||||
out (with-redefs [ssrf/safe-url? (constantly true)
|
||||
oidc/is-organization-sso-config-valid?
|
||||
(fn [_cfg sso]
|
||||
(and (= "test-client" (:client-id sso))
|
||||
(= "https://idp.example.com/" (:issuer sso))))]
|
||||
(th/management-command!
|
||||
{::th/type :check-organization-sso
|
||||
:organization-id organization-id
|
||||
:client-id "test-client"
|
||||
:client-secret "test-secret"
|
||||
:issuer "https://idp.example.com/"}))]
|
||||
(t/is (th/success? out))
|
||||
(t/is (true? (-> out :result :valid)))))
|
||||
|
||||
(t/deftest check-organization-sso-returns-valid-false-on-ssrf-blocked-issuer
|
||||
(t/testing "an SSRF-blocked issuer must not reach the OIDC validation flow"
|
||||
(let [called? (atom false)
|
||||
out (with-redefs [oidc/is-organization-sso-config-valid?
|
||||
(fn [_cfg _sso] (reset! called? true) true)]
|
||||
(th/management-command!
|
||||
{::th/type :check-organization-sso
|
||||
:organization-id (uuid/random)
|
||||
:client-id "test-client"
|
||||
:client-secret "test-secret"
|
||||
:issuer "http://127.0.0.1/idp"}))]
|
||||
(t/is (th/success? out))
|
||||
(t/is (false? (-> out :result :valid)))
|
||||
(t/is (false? @called?)
|
||||
"OIDC validation should not run when the issuer is SSRF-blocked"))))
|
||||
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
;; PUSH AUDIT EVENTS
|
||||
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns backend-tests.rpc-plugins-test
|
||||
(:require
|
||||
|
||||
@@ -0,0 +1,97 @@
|
||||
;; This Source Code Form is subject to the terms of the Mozilla Public
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns backend-tests.storage-config-test
|
||||
(:require
|
||||
[app.common.uri :as u]
|
||||
[app.config :as cf]
|
||||
[app.storage.config :as sto.config]
|
||||
[clojure.java.io :as io]
|
||||
[clojure.test :as t]))
|
||||
|
||||
(defn- write-routes!
|
||||
[content]
|
||||
(let [file (java.io.File/createTempFile "storage-routes" ".edn")]
|
||||
(spit file content)
|
||||
file))
|
||||
|
||||
(defn- delete!
|
||||
[file]
|
||||
(io/delete-file file true))
|
||||
|
||||
(defn- load-with
|
||||
[config]
|
||||
(binding [cf/config (merge cf/config
|
||||
{:objects-storage-backend :s3}
|
||||
config)]
|
||||
(sto.config/load)))
|
||||
|
||||
(defn- load-error
|
||||
"Runs `load` against a routes file with `content` and returns the raised
|
||||
throwable, or nil when it succeeds."
|
||||
[content config]
|
||||
(let [file (write-routes! content)]
|
||||
(try
|
||||
(try
|
||||
(load-with (assoc config :objects-storage-s3-routes-file (.getAbsolutePath file)))
|
||||
nil
|
||||
(catch Throwable cause cause))
|
||||
(finally
|
||||
(delete! file)))))
|
||||
|
||||
(t/deftest returns-empty-when-unset
|
||||
(t/is (= {:targets nil :routes nil}
|
||||
(load-with {:objects-storage-s3-routes-file nil}))))
|
||||
|
||||
(t/deftest loads-and-normalizes-routes
|
||||
(let [file (write-routes!
|
||||
"{:targets {:temp {:bucket \"penpot-temp\" :endpoint \"https://s3.example.com\"}} :routes {\"tempfile\" :temp}}")]
|
||||
(try
|
||||
(let [result (load-with {:objects-storage-s3-routes-file (.getAbsolutePath file)})]
|
||||
(t/is (= #{:temp} (set (keys (:targets result)))))
|
||||
(t/is (= "penpot-temp" (get-in result [:targets :temp :bucket])))
|
||||
(t/is (u/uri? (get-in result [:targets :temp :endpoint])))
|
||||
(t/is (= {"tempfile" :temp} (:routes result))))
|
||||
(finally
|
||||
(delete! file)))))
|
||||
|
||||
(t/deftest rejects-reserved-default-target
|
||||
(t/is (some? (load-error "{:targets {:default {:bucket \"x\"}}}"
|
||||
{}))))
|
||||
|
||||
(t/deftest rejects-unknown-route-target
|
||||
(t/is (some? (load-error
|
||||
"{:targets {:temp {:bucket \"x\"}} :routes {\"tempfile\" :missing}}"
|
||||
{}))))
|
||||
|
||||
(t/deftest rejects-invalid-semantic-bucket
|
||||
(t/is (some? (load-error
|
||||
"{:targets {:temp {:bucket \"x\"}} :routes {\"not-a-bucket\" :temp}}"
|
||||
{}))))
|
||||
|
||||
(t/deftest rejects-unreadable-file
|
||||
(t/is (some? (load-error "{:targets" {}))))
|
||||
|
||||
(t/deftest rejects-routing-when-backend-is-not-s3
|
||||
(t/is (some? (load-error
|
||||
"{:targets {:temp {:bucket \"x\"}} :routes {\"tempfile\" :temp}}"
|
||||
{:objects-storage-backend :fs}))))
|
||||
|
||||
(t/deftest rejects-route-pointing-to-default
|
||||
(let [err (load-error
|
||||
"{:targets {:temp {:bucket \"x\"}} :routes {\"tempfile\" :default}}"
|
||||
{})]
|
||||
(t/is (some? err))
|
||||
(t/is (= :unknown-storage-target (:code (ex-data err))))))
|
||||
|
||||
(t/deftest loads-empty-targets-map
|
||||
(let [file (write-routes! "{:targets {}}")]
|
||||
(try
|
||||
(let [result (load-with {:objects-storage-s3-routes-file (.getAbsolutePath file)})]
|
||||
(t/is (= {} (:targets result)))
|
||||
(t/is (nil? (:routes result))))
|
||||
(finally
|
||||
(delete! file)))))
|
||||
@@ -13,13 +13,16 @@
|
||||
[app.rpc :as-alias rpc]
|
||||
[app.storage :as sto]
|
||||
[app.storage.fs :as-alias sto.fs]
|
||||
[app.storage.gc-deleted :as sto.gc-deleted]
|
||||
[app.storage.impl :as impl]
|
||||
[app.storage.pending-gc :as sto.pending-gc]
|
||||
[app.storage.s3 :as-alias sto.s3]
|
||||
[backend-tests.helpers :as th]
|
||||
[clojure.test :as t]
|
||||
[cuerdas.core :as str]
|
||||
[datoteka.fs :as fs]
|
||||
[datoteka.io :as io]
|
||||
[integrant.core :as ig]
|
||||
[mockery.core :refer [with-mocks]]
|
||||
[promesa.core :as p])
|
||||
(:import
|
||||
@@ -41,6 +44,17 @@
|
||||
[storage]
|
||||
(assoc storage ::sto/backend :fs))
|
||||
|
||||
(declare fake-s3-backend-with-targets)
|
||||
|
||||
(defn configure-s3-storage
|
||||
"Returns a storage configured with the fake S3 backend and the given
|
||||
semantic-bucket -> target routing."
|
||||
[bucket->target]
|
||||
(-> (:app.storage/storage th/*system*)
|
||||
(assoc ::sto/backend :s3)
|
||||
(assoc-in [::sto/backends :s3] (fake-s3-backend-with-targets))
|
||||
(assoc ::sto/bucket->target bucket->target)))
|
||||
|
||||
(t/deftest put-and-retrieve-object
|
||||
(let [storage (-> (:app.storage/storage th/*system*)
|
||||
(configure-storage-backend))
|
||||
@@ -708,7 +722,7 @@
|
||||
{:id (:id object)})
|
||||
|
||||
(with-mocks [_mock {:target 'app.storage.impl/del-objects-in-bulk
|
||||
:return (fn [_ ids] (set ids))}]
|
||||
:return (fn [_ _ ids] (set ids))}]
|
||||
(let [res (th/run-task! :storage-gc-deleted {})]
|
||||
(t/is (= 0 (:deleted res)))))
|
||||
|
||||
@@ -832,9 +846,145 @@
|
||||
|
||||
(defn- fake-s3-backend
|
||||
[]
|
||||
{::sto/type :s3
|
||||
::sto.s3/client (reify S3AsyncClient)
|
||||
::sto.s3/presigner (reify S3Presigner)})
|
||||
{::sto/type :s3
|
||||
::sto.s3/default-target :default
|
||||
::sto.s3/targets
|
||||
{:default {::sto.s3/client (reify S3AsyncClient)
|
||||
::sto.s3/presigner (reify S3Presigner)
|
||||
::sto.s3/bucket "test-bucket"
|
||||
::sto.s3/prefix nil}}})
|
||||
|
||||
(defn- fake-s3-target
|
||||
[bucket prefix]
|
||||
{::sto.s3/client (reify S3AsyncClient)
|
||||
::sto.s3/presigner (reify S3Presigner)
|
||||
::sto.s3/bucket bucket
|
||||
::sto.s3/prefix prefix})
|
||||
|
||||
(defn- fake-s3-backend-with-targets
|
||||
[]
|
||||
(assoc (fake-s3-backend)
|
||||
::sto.s3/targets
|
||||
{:default (fake-s3-target "default" nil)
|
||||
:temp (fake-s3-target "temp" "tmp/")}))
|
||||
|
||||
(t/deftest s3-bulk-delete-resolves-target
|
||||
(let [backend (fake-s3-backend-with-targets)
|
||||
captured (atom nil)]
|
||||
(with-mocks [_mock {:target 'app.storage.s3/del-object-in-bulk
|
||||
:return (fn [target _ids]
|
||||
(reset! captured target)
|
||||
(p/resolved nil))}]
|
||||
(t/is (= #{} (impl/del-objects-in-bulk backend :temp #{(uuid/next)})))
|
||||
(t/is (= "temp" (::sto.s3/bucket @captured)))
|
||||
(t/is (= "tmp/" (::sto.s3/prefix @captured))))))
|
||||
|
||||
(t/deftest s3-bulk-delete-rejects-unknown-target
|
||||
(let [backend (fake-s3-backend-with-targets)
|
||||
captured (atom nil)]
|
||||
(with-mocks [_mock {:target 'app.storage.s3/del-object-in-bulk
|
||||
:return (fn [target _ids]
|
||||
(reset! captured target)
|
||||
(p/resolved nil))}]
|
||||
(let [ex (try
|
||||
(impl/del-objects-in-bulk backend :missing #{(uuid/next)})
|
||||
nil
|
||||
(catch Throwable cause cause))]
|
||||
(t/is (some? ex))
|
||||
(t/is (= :invalid-storage-target (:code (ex-data ex))))
|
||||
(t/is (nil? @captured))))))
|
||||
|
||||
(t/deftest s3-get-object-url-rejects-unknown-target
|
||||
(let [backend (fake-s3-backend-with-targets)
|
||||
ex (try
|
||||
(impl/get-object-url backend (with-meta {:id (uuid/next)}
|
||||
{:storage-target "ghost"}) {})
|
||||
nil
|
||||
(catch Throwable cause cause))]
|
||||
(t/is (some? ex))
|
||||
(t/is (= :invalid-storage-target (:code (ex-data ex))))))
|
||||
|
||||
(t/deftest s3-get-object-data-rejects-unknown-target
|
||||
(let [backend (fake-s3-backend-with-targets)
|
||||
ex (try
|
||||
(impl/get-object-data backend (with-meta {:id (uuid/next) :size 1}
|
||||
{:storage-target "ghost"}))
|
||||
nil
|
||||
(catch Throwable cause cause))]
|
||||
(t/is (some? ex))
|
||||
(t/is (= :invalid-storage-target (:code (ex-data ex))))))
|
||||
|
||||
(t/deftest s3-target-resolvable-checks-configured-targets
|
||||
(let [backend (fake-s3-backend-with-targets)]
|
||||
(t/is (true? (impl/target-resolvable? backend nil)))
|
||||
(t/is (true? (impl/target-resolvable? backend "default")))
|
||||
(t/is (true? (impl/target-resolvable? backend :temp)))
|
||||
(t/is (false? (impl/target-resolvable? backend "ghost")))))
|
||||
|
||||
(t/deftest fs-target-is-always-resolvable
|
||||
(t/is (true? (impl/target-resolvable? {::sto/type :fs} "anything"))))
|
||||
|
||||
(t/deftest s3-build-targets-shares-clients-and-closes-them
|
||||
(let [clients (atom [])
|
||||
close-count (atom 0)
|
||||
mk-client (fn [_]
|
||||
(let [c (reify
|
||||
clojure.lang.IDeref
|
||||
(deref [_] (Object.))
|
||||
java.lang.AutoCloseable
|
||||
(close [_] (swap! close-count inc)))]
|
||||
(swap! clients conj c)
|
||||
c))]
|
||||
(with-mocks [_c {:target 'app.storage.s3/build-s3-client
|
||||
:return mk-client}
|
||||
_p {:target 'app.storage.s3/build-s3-presigner
|
||||
:return (fn [_] (reify S3Presigner))}]
|
||||
(let [backend (ig/init-key :app.storage.s3/backend
|
||||
{:app.storage.s3/region :eu-central-1
|
||||
:app.storage.s3/bucket "main"
|
||||
:app.worker/netty-io-executor :executor
|
||||
:app.storage.s3/targets
|
||||
{:same {:bucket "same"}
|
||||
:other {:bucket "other" :region :us-east-1}}})]
|
||||
;; one client pair per distinct [region endpoint]
|
||||
(t/is (= 2 (count @clients)))
|
||||
;; same region reuses the default client
|
||||
(t/is (= (get-in backend [:app.storage.s3/targets :default :app.storage.s3/client])
|
||||
(get-in backend [:app.storage.s3/targets :same :app.storage.s3/client])))
|
||||
;; different region gets its own client
|
||||
(t/is (not= (get-in backend [:app.storage.s3/targets :default :app.storage.s3/client])
|
||||
(get-in backend [:app.storage.s3/targets :other :app.storage.s3/client])))
|
||||
|
||||
(ig/halt-key! :app.storage.s3/backend backend)
|
||||
(t/is (= 2 @close-count))))))
|
||||
|
||||
(t/deftest s3-build-targets-closes-clients-on-failure
|
||||
(let [close-count (atom 0)
|
||||
mk-client (fn [params]
|
||||
(when (= :us-east-1 (:app.storage.s3/region params))
|
||||
(throw (RuntimeException. "boom")))
|
||||
(reify
|
||||
clojure.lang.IDeref
|
||||
(deref [_] (Object.))
|
||||
java.lang.AutoCloseable
|
||||
(close [_] (swap! close-count inc))))]
|
||||
(with-mocks [_c {:target 'app.storage.s3/build-s3-client
|
||||
:return mk-client}
|
||||
_p {:target 'app.storage.s3/build-s3-presigner
|
||||
:return (fn [_] (reify S3Presigner))}]
|
||||
(let [ex (try
|
||||
(ig/init-key :app.storage.s3/backend
|
||||
{:app.storage.s3/region :eu-central-1
|
||||
:app.storage.s3/bucket "main"
|
||||
:app.worker/netty-io-executor :executor
|
||||
:app.storage.s3/targets
|
||||
{:same {:bucket "same"}
|
||||
:other {:bucket "other" :region :us-east-1}}})
|
||||
nil
|
||||
(catch Throwable cause cause))]
|
||||
(t/is (some? ex))
|
||||
;; the pair built before the failing one is closed
|
||||
(t/is (= 1 @close-count))))))
|
||||
|
||||
(t/deftest s3-exists-object-returns-true-on-found
|
||||
(with-mocks [mock {:target 'app.storage.s3/head-object
|
||||
@@ -874,3 +1024,311 @@
|
||||
(t/is (= "boom" (ex-message (ex-cause ex)))))
|
||||
;; one initial attempt plus max-retries
|
||||
(t/is (= 4 (:call-count @mock)))))
|
||||
|
||||
;; --- Storage target metadata / dedup
|
||||
|
||||
(t/deftest put-object-routes-bucket-to-storage-target
|
||||
(let [storage (configure-s3-storage {"tempfile" :temp})]
|
||||
(with-mocks [_mock {:target 'app.storage.impl/put-object
|
||||
:return (fn [_ object _] object)}]
|
||||
(let [object (sto/put-object! storage {::sto/content (sto/content "content")
|
||||
:bucket "tempfile"
|
||||
:content-type "text/plain"})
|
||||
row (th/db-exec-one!
|
||||
["select backend, metadata->>'~:storage-target' as target
|
||||
from storage_object where id = ?" (:id object)])]
|
||||
(t/is (= "s3" (:backend row)))
|
||||
(t/is (= "temp" (:target row)))))))
|
||||
|
||||
(t/deftest put-object-falls-back-to-default-storage-target
|
||||
(let [storage (configure-s3-storage {"tempfile" :temp})]
|
||||
(with-mocks [_mock {:target 'app.storage.impl/put-object
|
||||
:return (fn [_ object _] object)}]
|
||||
(let [object (sto/put-object! storage {::sto/content (sto/content "content")
|
||||
:bucket "file-media-object"
|
||||
:content-type "text/plain"})
|
||||
row (th/db-exec-one!
|
||||
["select metadata->>'~:storage-target' as target
|
||||
from storage_object where id = ?" (:id object)])]
|
||||
(t/is (= "default" (:target row)))))))
|
||||
|
||||
(t/deftest put-object-fs-does-not-set-storage-target
|
||||
(let [storage (configure-storage-backend (:app.storage/storage th/*system*))
|
||||
object (sto/put-object! storage {::sto/content (sto/content "content")
|
||||
:bucket "file-media-object"
|
||||
:content-type "text/plain"})
|
||||
row (th/db-exec-one!
|
||||
["select metadata->>'~:storage-target' as target
|
||||
from storage_object where id = ?" (:id object)])]
|
||||
(t/is (nil? (:target row)))))
|
||||
|
||||
(t/deftest dedup-is-isolated-per-storage-target
|
||||
(let [routed (configure-s3-storage {"file-data" :temp})
|
||||
unrouted (configure-s3-storage {})
|
||||
content (-> (sto/content "content")
|
||||
(sto/wrap-with-hash "same-hash"))]
|
||||
(with-mocks [_mock {:target 'app.storage.impl/put-object
|
||||
:return (fn [_ object _] object)}]
|
||||
(let [object1 (sto/put-object! routed {::sto/content content
|
||||
::sto/deduplicate? true
|
||||
:bucket "file-data"
|
||||
:content-type "text/plain"})
|
||||
object2 (sto/put-object! unrouted {::sto/content content
|
||||
::sto/deduplicate? true
|
||||
:bucket "file-data"
|
||||
:content-type "text/plain"})
|
||||
row (th/db-exec-one! ["select count(*) from storage_object"])]
|
||||
;; same semantic bucket, different target: no dedup hit, two rows
|
||||
(t/is (not= (:id object1) (:id object2)))
|
||||
(t/is (= 2 (:count row)))))))
|
||||
|
||||
(t/deftest dedup-reuses-object-within-same-storage-target
|
||||
(let [storage (configure-s3-storage {"file-data" :temp})
|
||||
content (-> (sto/content "content")
|
||||
(sto/wrap-with-hash "same-hash"))]
|
||||
(with-mocks [_p {:target 'app.storage.impl/put-object
|
||||
:return (fn [_ object _] object)}
|
||||
_e {:target 'app.storage.impl/exists-object?
|
||||
:return (fn [_ _] true)}]
|
||||
(let [object1 (sto/put-object! storage {::sto/content content
|
||||
::sto/deduplicate? true
|
||||
:bucket "file-data"
|
||||
:content-type "text/plain"})
|
||||
object2 (sto/put-object! storage {::sto/content content
|
||||
::sto/deduplicate? true
|
||||
:bucket "file-data"
|
||||
:content-type "text/plain"})]
|
||||
;; same semantic bucket and target: dedup reuses the object
|
||||
(t/is (= (:id object1) (:id object2)))))))
|
||||
|
||||
(t/deftest dedup-hit-carries-storage-target-metadata
|
||||
(let [storage (configure-s3-storage {"file-data" :temp})
|
||||
content (-> (sto/content "content")
|
||||
(sto/wrap-with-hash "same-hash"))
|
||||
captured (atom nil)]
|
||||
(with-mocks [_p {:target 'app.storage.impl/put-object
|
||||
:return (fn [_ object _] object)}
|
||||
_e {:target 'app.storage.impl/exists-object?
|
||||
:return (fn [_ object]
|
||||
(reset! captured object)
|
||||
true)}]
|
||||
(sto/put-object! storage {::sto/content content
|
||||
::sto/deduplicate? true
|
||||
:bucket "file-data"
|
||||
:content-type "text/plain"})
|
||||
(sto/put-object! storage {::sto/content content
|
||||
::sto/deduplicate? true
|
||||
:bucket "file-data"
|
||||
:content-type "text/plain"})
|
||||
(t/is (some? @captured))
|
||||
(t/is (= "temp" (:storage-target (meta @captured)))))))
|
||||
|
||||
(t/deftest dedup-repair-carries-storage-target-metadata
|
||||
(let [storage (configure-s3-storage {"file-data" :temp})
|
||||
content (-> (sto/content "content")
|
||||
(sto/wrap-with-hash "same-hash"))
|
||||
calls (atom [])]
|
||||
(with-mocks [_p {:target 'app.storage.impl/put-object
|
||||
:return (fn [_ object _]
|
||||
(swap! calls conj object)
|
||||
object)}
|
||||
_h {:target 'app.storage.s3/head-object
|
||||
:return (p/rejected (-> (NoSuchKeyException/builder)
|
||||
(.message "no key")
|
||||
(.build)))}]
|
||||
(let [object1 (sto/put-object! storage {::sto/content content
|
||||
::sto/deduplicate? true
|
||||
:bucket "file-data"
|
||||
:content-type "text/plain"})
|
||||
;; second put finds the row but the real exists-object? sees a
|
||||
;; missing blob and repairs it in place
|
||||
object2 (sto/put-object! storage {::sto/content content
|
||||
::sto/deduplicate? true
|
||||
:bucket "file-data"
|
||||
:content-type "text/plain"})
|
||||
row (th/db-exec-one!
|
||||
["select status from storage_object where id = ?" (:id object1)])
|
||||
count (th/db-exec-one! ["select count(*) from storage_object"])]
|
||||
(t/is (= (:id object1) (:id object2)))
|
||||
(t/is (= "valid" (:status row)))
|
||||
(t/is (= 1 (:count count)))
|
||||
(t/is (= "temp" (:storage-target (meta (last @calls)))))))))
|
||||
|
||||
;; --- GC target routing
|
||||
|
||||
(defn- storage-with-s3-targets
|
||||
[]
|
||||
(assoc (:app.storage/storage th/*system*)
|
||||
::sto/backends
|
||||
{:s3 (fake-s3-backend-with-targets)}))
|
||||
|
||||
(t/deftest gc-deleted-deletes-from-routed-target
|
||||
(let [storage (storage-with-s3-targets)
|
||||
cfg {::db/pool th/*pool* ::sto/storage storage}
|
||||
id (uuid/next)
|
||||
captured (atom nil)]
|
||||
(th/db-exec! ["insert into storage_object (id, size, backend, metadata, deleted_at, status)
|
||||
values (?, 1, 's3', ?, ?, 'valid')"
|
||||
id
|
||||
(db/tjson {:bucket "file-data" :storage-target "temp"})
|
||||
(ct/in-past {:minutes 1})])
|
||||
(with-mocks [_mock {:target 'app.storage.impl/del-objects-in-bulk
|
||||
:return (fn [_ target _ids]
|
||||
(reset! captured target)
|
||||
#{})}]
|
||||
(t/is (= 1 (:deleted (#'sto.gc-deleted/clean-deleted! cfg))))
|
||||
(t/is (= "temp" @captured)))))
|
||||
|
||||
(t/deftest gc-deleted-refuses-unknown-target-and-keeps-row
|
||||
(let [storage (storage-with-s3-targets)
|
||||
cfg {::db/pool th/*pool* ::sto/storage storage}
|
||||
id (uuid/next)]
|
||||
(th/db-exec! ["insert into storage_object (id, size, backend, metadata, deleted_at, status)
|
||||
values (?, 1, 's3', ?, ?, 'valid')"
|
||||
id
|
||||
(db/tjson {:bucket "file-data" :storage-target "ghost"})
|
||||
(ct/in-past {:minutes 1})])
|
||||
(with-mocks [mock {:target 'app.storage.impl/del-objects-in-bulk
|
||||
:return (fn [_ _ _] #{})}]
|
||||
(let [result (#'sto.gc-deleted/clean-deleted! cfg)
|
||||
row (th/db-exec-one!
|
||||
["select status, deleted_at, deletion_attempts
|
||||
from storage_object where id = ?" id])]
|
||||
(t/is (= 0 (:deleted result)))
|
||||
(t/is (= 1 (:parked result)))
|
||||
(t/is (= 0 (:call-count @mock)))
|
||||
(t/is (= "valid" (:status row)))
|
||||
(t/is (ct/is-after? (:deleted-at row) (ct/now)))
|
||||
(t/is (= 0 (:deletion-attempts row)))))))
|
||||
|
||||
(t/deftest gc-deleted-give-up-not-applied-to-unknown-target
|
||||
(let [storage (storage-with-s3-targets)
|
||||
cfg {::db/pool th/*pool* ::sto/storage storage}
|
||||
id (uuid/next)]
|
||||
(th/db-exec! ["insert into storage_object (id, size, backend, metadata, deleted_at, status, deletion_attempts)
|
||||
values (?, 1, 's3', ?, ?, 'valid', 10)"
|
||||
id
|
||||
(db/tjson {:bucket "file-data" :storage-target "ghost"})
|
||||
(ct/in-past {:minutes 1})])
|
||||
(with-mocks [_mock {:target 'app.storage.impl/del-objects-in-bulk
|
||||
:return (fn [_ _ _] #{})}]
|
||||
(#'sto.gc-deleted/clean-deleted! cfg))
|
||||
(let [row (th/db-exec-one! ["select count(*) from storage_object where id = ?" id])]
|
||||
(t/is (= 1 (:count row))))))
|
||||
|
||||
(t/deftest gc-deleted-normal-failure-defers-and-gives-up
|
||||
(let [storage (storage-with-s3-targets)
|
||||
cfg {::db/pool th/*pool* ::sto/storage storage}
|
||||
id (uuid/next)]
|
||||
(th/db-exec! ["insert into storage_object (id, size, backend, metadata, deleted_at, status)
|
||||
values (?, 1, 's3', ?, ?, 'valid')"
|
||||
id
|
||||
(db/tjson {:bucket "file-data" :storage-target "temp"})
|
||||
(ct/in-past {:minutes 1})])
|
||||
(with-mocks [_mock {:target 'app.storage.impl/del-objects-in-bulk
|
||||
:return (fn [_ _ ids] (set ids))}]
|
||||
(let [result (#'sto.gc-deleted/clean-deleted! cfg)
|
||||
row (th/db-exec-one!
|
||||
["select deleted_at, deletion_attempts
|
||||
from storage_object where id = ?" id])]
|
||||
(t/is (= 0 (:deleted result)))
|
||||
(t/is (= 0 (:parked result)))
|
||||
(t/is (ct/is-after? (:deleted-at row) (ct/now)))
|
||||
(t/is (= 1 (:deletion-attempts row))))
|
||||
|
||||
;; force the give-up threshold and let the next pass remove the row
|
||||
(th/db-update! :storage-object
|
||||
{:deletion-attempts 7
|
||||
:deleted-at (ct/in-past {:minutes 1})}
|
||||
{:id id})
|
||||
(let [result (#'sto.gc-deleted/clean-deleted! cfg)]
|
||||
(t/is (= 0 (:deleted result)))
|
||||
(let [row (th/db-exec-one! ["select count(*) from storage_object where id = ?" id])]
|
||||
(t/is (= 0 (:count row))))))))
|
||||
|
||||
(t/deftest pending-gc-deletes-resolvable-target
|
||||
(let [storage (storage-with-s3-targets)
|
||||
cfg {::db/pool th/*pool* ::sto/storage storage}
|
||||
id (uuid/next)
|
||||
captured (atom nil)]
|
||||
(th/db-exec! ["insert into storage_object (id, size, backend, metadata, created_at, status)
|
||||
values (?, 1, 's3', ?, ?, 'pending')"
|
||||
id
|
||||
(db/tjson {:storage-target "temp"})
|
||||
(ct/in-past {:days 2})])
|
||||
(with-mocks [_mock {:target 'app.storage.impl/del-object
|
||||
:return (fn [_ object]
|
||||
(reset! captured object)
|
||||
nil)}]
|
||||
(let [result (#'sto.pending-gc/process! cfg)]
|
||||
(t/is (= 1 (:processed result)))
|
||||
(t/is (= 0 (:parked result)))
|
||||
(t/is (= "temp" (:storage-target (meta @captured))))
|
||||
(let [row (th/db-exec-one! ["select count(*) from storage_object where id = ?" id])]
|
||||
(t/is (= 0 (:count row))))))))
|
||||
|
||||
(t/deftest pending-gc-refuses-unknown-target-and-keeps-row
|
||||
(let [storage (storage-with-s3-targets)
|
||||
cfg {::db/pool th/*pool* ::sto/storage storage}
|
||||
id (uuid/next)
|
||||
captured (atom nil)]
|
||||
(th/db-exec! ["insert into storage_object (id, size, backend, metadata, created_at, status)
|
||||
values (?, 1, 's3', ?, ?, 'pending')"
|
||||
id
|
||||
(db/tjson {:storage-target "ghost"})
|
||||
(ct/in-past {:days 2})])
|
||||
(with-mocks [mock {:target 'app.storage.impl/del-object
|
||||
:return (fn [_ object]
|
||||
(reset! captured object)
|
||||
nil)}]
|
||||
(let [result (#'sto.pending-gc/process! cfg)
|
||||
row (th/db-exec-one!
|
||||
["select status, deleted_at from storage_object where id = ?" id])]
|
||||
(t/is (= 0 (:processed result)))
|
||||
(t/is (= 1 (:parked result)))
|
||||
(t/is (= 0 (:call-count @mock)))
|
||||
(t/is (nil? @captured))
|
||||
(t/is (= "pending" (:status row)))
|
||||
(t/is (ct/is-after? (:deleted-at row) (ct/now)))))))
|
||||
|
||||
(t/deftest gc-deleted-legacy-rows-delete-from-default-target
|
||||
(let [storage (storage-with-s3-targets)
|
||||
cfg {::db/pool th/*pool* ::sto/storage storage}
|
||||
id (uuid/next)
|
||||
captured (atom nil)]
|
||||
(th/db-exec! ["insert into storage_object (id, size, backend, metadata, deleted_at, status)
|
||||
values (?, 1, 's3', ?, ?, 'valid')"
|
||||
id
|
||||
(db/tjson {:bucket "file-data"})
|
||||
(ct/in-past {:minutes 1})])
|
||||
(with-mocks [_mock {:target 'app.storage.impl/del-objects-in-bulk
|
||||
:return (fn [_ target _ids]
|
||||
(reset! captured target)
|
||||
#{})}]
|
||||
(let [result (#'sto.gc-deleted/clean-deleted! cfg)]
|
||||
(t/is (= 1 (:deleted result)))
|
||||
(t/is (= 0 (:parked result)))
|
||||
(t/is (= "default" @captured))
|
||||
(let [row (th/db-exec-one! ["select count(*) from storage_object where id = ?" id])]
|
||||
(t/is (= 0 (:count row))))))))
|
||||
|
||||
(t/deftest pending-gc-legacy-rows-delete-from-default-target
|
||||
(let [storage (storage-with-s3-targets)
|
||||
cfg {::db/pool th/*pool* ::sto/storage storage}
|
||||
id (uuid/next)
|
||||
captured (atom nil)]
|
||||
(th/db-exec! ["insert into storage_object (id, size, backend, metadata, created_at, status)
|
||||
values (?, 1, 's3', ?, ?, 'pending')"
|
||||
id
|
||||
(db/tjson {:bucket "file-data"})
|
||||
(ct/in-past {:days 2})])
|
||||
(with-mocks [_mock {:target 'app.storage.impl/del-object
|
||||
:return (fn [_ object]
|
||||
(reset! captured object)
|
||||
nil)}]
|
||||
(let [result (#'sto.pending-gc/process! cfg)]
|
||||
(t/is (= 1 (:processed result)))
|
||||
(t/is (= 0 (:parked result)))
|
||||
(t/is (= "default" (:storage-target (meta @captured))))
|
||||
(let [row (th/db-exec-one! ["select count(*) from storage_object where id = ?" id])]
|
||||
(t/is (= 0 (:count row))))))))
|
||||
@@ -0,0 +1,106 @@
|
||||
import { describe, it } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { rpcPost, extractCookie } from "./helpers/client.mjs";
|
||||
|
||||
async function loginWithLdap(email, password) {
|
||||
const res = await rpcPost("login-with-ldap", { email, password });
|
||||
if (res.status !== 200 || res.body.type) {
|
||||
throw new Error(
|
||||
`LDAP login failed: ${JSON.stringify(res.body)}`
|
||||
);
|
||||
}
|
||||
const cookie = extractCookie(res.setCookie);
|
||||
return { profile: res.body, cookie };
|
||||
}
|
||||
|
||||
describe("LDAP injection — T5-N1-03", () => {
|
||||
|
||||
it("normal LDAP login works with valid credentials", async () => {
|
||||
const { profile, cookie } = await loginWithLdap(
|
||||
"fry@planetexpress.com",
|
||||
"fry"
|
||||
);
|
||||
assert.equal(profile.email, "fry@planetexpress.com");
|
||||
assert.ok(profile.id, "profile should have id");
|
||||
assert.ok(cookie, "cookie should be set");
|
||||
});
|
||||
|
||||
it("wildcard injection: *@planetexpress.com must not return client literal as email", async () => {
|
||||
// ATTACK SCENARIO (from Criptored audit):
|
||||
// 1. Attacker (amy) sends email="*@planetexpress.com" with her own password
|
||||
// 2. LDAP filter becomes (mail=*@planetexpress.com) — * is a wildcard
|
||||
// 3. With sizelimit=1, LDAP returns amy's entry (first match)
|
||||
// 4. Bind succeeds: amy's DN + amy's password = valid
|
||||
//
|
||||
// EXPECTED BEHAVIOR AFTER FIX (two valid outcomes):
|
||||
// A) If * is escaped: LDAP finds no match → wrong-credentials (injection blocked)
|
||||
// B) If * matches: profile email must be "amy@planetexpress.com" (directory), not "*@planetexpress.com" (client)
|
||||
//
|
||||
// Either outcome is correct — the vulnerability is fixed.
|
||||
try {
|
||||
const { profile } = await loginWithLdap("*@planetexpress.com", "amy");
|
||||
// Outcome B: login succeeded, verify email is from directory
|
||||
assert.equal(
|
||||
profile.email,
|
||||
"amy@planetexpress.com",
|
||||
"email must come from LDAP directory, not client input"
|
||||
);
|
||||
} catch (e) {
|
||||
// Outcome A: injection blocked — * is escaped, no LDAP match
|
||||
assert.ok(
|
||||
e.message.includes("wrong-credentials"),
|
||||
"wildcard should be rejected or return directory email"
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
it("identity swap: alternate email must return primary directory email", async () => {
|
||||
// Professor has two emails in LDAP: professor@ and hubert@.
|
||||
// Login with hubert@ — the profile email should be the one
|
||||
// the LDAP directory returns as attrs-email, not what the client typed.
|
||||
//
|
||||
// EXPECTED BEHAVIOR AFTER FIX:
|
||||
// Profile email should be "professor@planetexpress.com" (primary directory email),
|
||||
// NOT "hubert@planetexpress.com" (client literal).
|
||||
//
|
||||
// CURRENT BUG: email is "hubert@planetexpress.com" (client literal) — test FAILS
|
||||
const { profile, cookie } = await loginWithLdap(
|
||||
"hubert@planetexpress.com",
|
||||
"professor"
|
||||
);
|
||||
assert.ok(profile.id, "profile should have id");
|
||||
assert.ok(cookie, "cookie should be set");
|
||||
// This assertion FAILS with current code (RED) — proves the vulnerability
|
||||
assert.equal(
|
||||
profile.email,
|
||||
"professor@planetexpress.com",
|
||||
"email must come from LDAP directory, not client input"
|
||||
);
|
||||
});
|
||||
|
||||
it("wrong password fails", async () => {
|
||||
try {
|
||||
await loginWithLdap("fry@planetexpress.com", "wrong-password");
|
||||
assert.fail("should have thrown");
|
||||
} catch (e) {
|
||||
assert.ok(
|
||||
e.message.includes("LDAP login failed") ||
|
||||
e.message.includes("wrong-credentials"),
|
||||
"should fail with wrong credentials"
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
it("non-existent user fails", async () => {
|
||||
try {
|
||||
await loginWithLdap("nobody@planetexpress.com", "password");
|
||||
assert.fail("should have thrown");
|
||||
} catch (e) {
|
||||
assert.ok(
|
||||
e.message.includes("LDAP login failed") ||
|
||||
e.message.includes("wrong-credentials"),
|
||||
"should fail for non-existent user"
|
||||
);
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -72,6 +72,13 @@
|
||||
[:map {:title "PlainColorAttrs"}
|
||||
[:color schema:hex-color]])
|
||||
|
||||
(def schema:image-transform
|
||||
[:map {:title "ImageTransform" :closed true}
|
||||
[:x {:optional true} ::sm/safe-number]
|
||||
[:y {:optional true} ::sm/safe-number]
|
||||
[:width {:optional true} ::sm/safe-number]
|
||||
[:height {:optional true} ::sm/safe-number]])
|
||||
|
||||
(def schema:image
|
||||
[:map {:title "ImageColor" :closed true}
|
||||
[:width [::sm/int {:min 0 :gen/gen sg/int}]]
|
||||
@@ -79,7 +86,8 @@
|
||||
[:mtype {:gen/gen (sg/elements cm/image-types)} ::sm/text]
|
||||
[:id ::sm/uuid]
|
||||
[:name {:optional true} ::sm/text]
|
||||
[:keep-aspect-ratio {:optional true} :boolean]])
|
||||
[:keep-aspect-ratio {:optional true} :boolean]
|
||||
[:transform {:optional true} schema:image-transform]])
|
||||
|
||||
(def image-attrs
|
||||
"A set of attrs that corresponds to image data type"
|
||||
|
||||
@@ -119,12 +119,15 @@
|
||||
|
||||
(defn write-image-fill
|
||||
[offset buffer opacity image]
|
||||
(let [image-id (get image :id)
|
||||
image-width (get image :width)
|
||||
image-height (get image :height)
|
||||
alpha (mth/floor (* opacity 0xff))
|
||||
keep-aspect-ratio (if (get image :keep-aspect-ratio false) 0x01 0x00)
|
||||
flags (bit-or keep-aspect-ratio 0x00)]
|
||||
(let [image-id (get image :id)
|
||||
image-width (get image :width)
|
||||
image-height (get image :height)
|
||||
alpha (mth/floor (* opacity 0xff))
|
||||
keep-aspect-ratio (if (get image :keep-aspect-ratio false) 0x01 0x00)
|
||||
transform (get image :transform)
|
||||
has-transform? (some? transform)
|
||||
transform-flag (if has-transform? 0x02 0x00)
|
||||
flags (bit-or keep-aspect-ratio transform-flag)]
|
||||
(buf/write-byte buffer (+ offset 0) 0x03)
|
||||
(buf/write-uuid buffer (+ offset 4) image-id)
|
||||
(buf/write-byte buffer (+ offset 20) alpha)
|
||||
@@ -132,6 +135,17 @@
|
||||
(buf/write-short buffer (+ offset 22) 0) ;; 2-byte padding (reserved for future use)
|
||||
(buf/write-int buffer (+ offset 24) image-width)
|
||||
(buf/write-int buffer (+ offset 28) image-height)
|
||||
(if has-transform?
|
||||
(do
|
||||
(buf/write-float buffer (+ offset 32) (double (get transform :x 0.0)))
|
||||
(buf/write-float buffer (+ offset 36) (double (get transform :y 0.0)))
|
||||
(buf/write-float buffer (+ offset 40) (double (get transform :width 1.0)))
|
||||
(buf/write-float buffer (+ offset 44) (double (get transform :height 1.0))))
|
||||
(do
|
||||
(buf/write-float buffer (+ offset 32) 0.0)
|
||||
(buf/write-float buffer (+ offset 36) 0.0)
|
||||
(buf/write-float buffer (+ offset 40) 1.0)
|
||||
(buf/write-float buffer (+ offset 44) 1.0)))
|
||||
(+ offset FILL-U8-SIZE)))
|
||||
|
||||
(defn- write-metadata
|
||||
@@ -208,28 +222,36 @@
|
||||
:type type}})
|
||||
|
||||
3 ;; image fill
|
||||
(let [id (buf/read-uuid dbuffer (+ doffset 4))
|
||||
alpha (buf/read-unsigned-byte dbuffer (+ doffset 20))
|
||||
opacity (mth/precision (/ alpha 0xff) 2)
|
||||
flags (buf/read-unsigned-byte dbuffer (+ doffset 21))
|
||||
ratio (boolean (bit-and flags 0x01))
|
||||
width (buf/read-int dbuffer (+ doffset 24))
|
||||
height (buf/read-int dbuffer (+ doffset 28))
|
||||
mtype (buf/read-short mbuffer (+ moffset 2))
|
||||
mtype (case mtype
|
||||
0x01 "image/jpeg"
|
||||
0x02 "image/png"
|
||||
0x03 "image/gif"
|
||||
0x04 "image/webp"
|
||||
0x05 "image/svg+xml")]
|
||||
(let [id (buf/read-uuid dbuffer (+ doffset 4))
|
||||
alpha (buf/read-unsigned-byte dbuffer (+ doffset 20))
|
||||
opacity (mth/precision (/ alpha 0xff) 2)
|
||||
flags (buf/read-unsigned-byte dbuffer (+ doffset 21))
|
||||
ratio (not (zero? (bit-and flags 0x01)))
|
||||
has-tf (not (zero? (bit-and flags 0x02)))
|
||||
width (buf/read-int dbuffer (+ doffset 24))
|
||||
height (buf/read-int dbuffer (+ doffset 28))
|
||||
transform (when has-tf
|
||||
{:x (buf/read-float dbuffer (+ doffset 32))
|
||||
:y (buf/read-float dbuffer (+ doffset 36))
|
||||
:width (buf/read-float dbuffer (+ doffset 40))
|
||||
:height (buf/read-float dbuffer (+ doffset 44))})
|
||||
mtype (buf/read-short mbuffer (+ moffset 2))
|
||||
mtype (case mtype
|
||||
0x01 "image/jpeg"
|
||||
0x02 "image/png"
|
||||
0x03 "image/gif"
|
||||
0x04 "image/webp"
|
||||
0x05 "image/svg+xml")]
|
||||
{:fill-opacity opacity
|
||||
:fill-image {:id id
|
||||
:width width
|
||||
:height height
|
||||
:mtype mtype
|
||||
:keep-aspect-ratio ratio
|
||||
;; FIXME: we are not encodign the name, looks useless
|
||||
:name "sample"}}))]
|
||||
:fill-image (cond-> {:id id
|
||||
:width width
|
||||
:height height
|
||||
:mtype mtype
|
||||
:keep-aspect-ratio ratio
|
||||
;; FIXME: we are not encodign the name, looks useless
|
||||
:name "sample"}
|
||||
(some? transform)
|
||||
(assoc :transform transform))}))]
|
||||
|
||||
(if refs?
|
||||
(let [ref-file (buf/read-uuid mbuffer (+ moffset 4))
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns app.common.types.path.fit
|
||||
"Curve fitting helpers."
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns app.common.types.path.selection
|
||||
"Transforms selected path nodes and handlers."
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns app.common.types.tokens-status
|
||||
(:require
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns common-tests.files-migrations-0026-test
|
||||
(:require
|
||||
|
||||
@@ -0,0 +1,275 @@
|
||||
;; This Source Code Form is subject to the terms of the Mozilla Public
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns common-tests.geom-image-bounds-resize-test
|
||||
(:require
|
||||
#?(:clj [clojure.test :refer [deftest is testing]]
|
||||
:cljs [cljs.test :refer-macros [deftest is testing]])
|
||||
[app.common.math :as mth]
|
||||
[app.common.schema :as sm]
|
||||
[app.common.types.color :as clr]
|
||||
[app.common.types.fills :as fills]
|
||||
[app.common.types.fills.impl :as fills.impl]
|
||||
[app.common.uuid :as uuid]))
|
||||
|
||||
(deftest test-image-transform-schema
|
||||
(testing "validates image with transform"
|
||||
(let [img {:id (uuid/custom 1)
|
||||
:width 400
|
||||
:height 300
|
||||
:mtype "image/png"
|
||||
:keep-aspect-ratio true
|
||||
:transform {:x 0.1 :y -0.2 :width 1.5 :height 2.0}}]
|
||||
(is (sm/validate clr/schema:image img))))
|
||||
|
||||
(testing "validates image without transform"
|
||||
(let [img {:id (uuid/custom 1)
|
||||
:width 400
|
||||
:height 300
|
||||
:mtype "image/png"
|
||||
:keep-aspect-ratio true}]
|
||||
(is (sm/validate clr/schema:image img))))
|
||||
|
||||
(testing "validates fill with image transform"
|
||||
(let [fill {:fill-opacity 0.8
|
||||
:fill-image {:id (uuid/custom 1)
|
||||
:width 400
|
||||
:height 300
|
||||
:mtype "image/png"
|
||||
:keep-aspect-ratio true
|
||||
:transform {:x -0.5 :y -0.5 :width 2.0 :height 2.0}}}]
|
||||
(is (sm/validate fills/schema:fill fill)))))
|
||||
|
||||
(deftest test-image-fill-buffer-roundtrip
|
||||
(testing "roundtrip image fill without transform"
|
||||
(let [fill-vec [{:fill-opacity 0.9
|
||||
:fill-image {:id (uuid/custom 1)
|
||||
:width 800
|
||||
:height 600
|
||||
:mtype "image/jpeg"
|
||||
:keep-aspect-ratio true
|
||||
:name "sample"}}]
|
||||
coerced (fills/from-plain fill-vec)
|
||||
plain (into [] coerced)]
|
||||
(is (= 1 (count plain)))
|
||||
(is (= 0.9 (:fill-opacity (first plain))))
|
||||
(is (= 800 (-> plain first :fill-image :width)))
|
||||
(is (= 600 (-> plain first :fill-image :height)))
|
||||
(is (true? (-> plain first :fill-image :keep-aspect-ratio)))
|
||||
(is (nil? (-> plain first :fill-image :transform)))))
|
||||
|
||||
(testing "roundtrip image fill with transform"
|
||||
(let [fill-vec [{:fill-opacity 0.75
|
||||
:fill-image {:id (uuid/custom 2)
|
||||
:width 1920
|
||||
:height 1080
|
||||
:mtype "image/webp"
|
||||
:keep-aspect-ratio false
|
||||
:name "sample"
|
||||
:transform {:x 0.25 :y -0.15 :width 1.5 :height 2.0}}}]
|
||||
coerced (fills/from-plain fill-vec)
|
||||
plain (into [] coerced)
|
||||
tf (-> plain first :fill-image :transform)]
|
||||
(is (= 1 (count plain)))
|
||||
(is (= 0.75 (:fill-opacity (first plain))))
|
||||
(is (= 1920 (-> plain first :fill-image :width)))
|
||||
(is (= 1080 (-> plain first :fill-image :height)))
|
||||
(is (false? (-> plain first :fill-image :keep-aspect-ratio)))
|
||||
(is (some? tf))
|
||||
(is (mth/close? 0.25 (double (:x tf))))
|
||||
(is (mth/close? -0.15 (double (:y tf))))
|
||||
(is (mth/close? 1.5 (double (:width tf))))
|
||||
(is (mth/close? 2.0 (double (:height tf)))))))
|
||||
|
||||
(defn compute-bounds-resize-transform
|
||||
"Mathematical model for independent image bounds resizing"
|
||||
[{:keys [width height handler center? sx sy transform]}]
|
||||
(let [w-new (* width sx)
|
||||
h-new (* height sy)
|
||||
[dx dy] (if ^boolean center?
|
||||
[(/ (* width (- 1.0 sx)) 2.0)
|
||||
(/ (* height (- 1.0 sy)) 2.0)]
|
||||
[(case handler
|
||||
(:left :bottom-left :top-left) (* width (- 1.0 sx))
|
||||
0.0)
|
||||
(case handler
|
||||
(:top :top-left :top-right) (* height (- 1.0 sy))
|
||||
0.0)])
|
||||
nx0 (get transform :x 0.0)
|
||||
ny0 (get transform :y 0.0)
|
||||
nw0 (get transform :width 1.0)
|
||||
nh0 (get transform :height 1.0)
|
||||
nx' (/ (- (* nx0 width) dx) w-new)
|
||||
ny' (/ (- (* ny0 height) dy) h-new)
|
||||
nw' (/ nw0 sx)
|
||||
nh' (/ nh0 sy)]
|
||||
{:transform {:x nx' :y ny' :width nw' :height nh'}
|
||||
:rendered-pixel-rect {:x (* nx' w-new)
|
||||
:y (* ny' h-new)
|
||||
:width (* nw' w-new)
|
||||
:height (* nh' h-new)}}))
|
||||
|
||||
(deftest test-handle-anchoring-mathematics
|
||||
(testing "Right handle crop (shrinking width to 50%)"
|
||||
(let [res (compute-bounds-resize-transform
|
||||
{:width 200 :height 100 :handler :right :center? false :sx 0.5 :sy 1.0})]
|
||||
(is (mth/close? 0.0 (-> res :transform :x)))
|
||||
(is (mth/close? 0.0 (-> res :transform :y)))
|
||||
(is (mth/close? 2.0 (-> res :transform :width)))
|
||||
(is (mth/close? 1.0 (-> res :transform :height)))
|
||||
;; Rendered pixel content remains 200x100 starting at (0, 0)
|
||||
(is (mth/close? 0.0 (-> res :rendered-pixel-rect :x)))
|
||||
(is (mth/close? 0.0 (-> res :rendered-pixel-rect :y)))
|
||||
(is (mth/close? 200.0 (-> res :rendered-pixel-rect :width)))
|
||||
(is (mth/close? 100.0 (-> res :rendered-pixel-rect :height)))))
|
||||
|
||||
(testing "Left handle crop (shrinking width to 50% from left)"
|
||||
(let [res (compute-bounds-resize-transform
|
||||
{:width 200 :height 100 :handler :left :center? false :sx 0.5 :sy 1.0})]
|
||||
(is (mth/close? -1.0 (-> res :transform :x)))
|
||||
(is (mth/close? 0.0 (-> res :transform :y)))
|
||||
(is (mth/close? 2.0 (-> res :transform :width)))
|
||||
(is (mth/close? 1.0 (-> res :transform :height)))
|
||||
;; Rendered pixel content has left at -100, width 200 -> right edge at +100 (matches right edge of 100px container!)
|
||||
(is (mth/close? -100.0 (-> res :rendered-pixel-rect :x)))
|
||||
(is (mth/close? 200.0 (-> res :rendered-pixel-rect :width)))))
|
||||
|
||||
(testing "Top handle crop (shrinking height to 50% from top)"
|
||||
(let [res (compute-bounds-resize-transform
|
||||
{:width 200 :height 100 :handler :top :center? false :sx 1.0 :sy 0.5})]
|
||||
(is (mth/close? 0.0 (-> res :transform :x)))
|
||||
(is (mth/close? -1.0 (-> res :transform :y)))
|
||||
(is (mth/close? 1.0 (-> res :transform :width)))
|
||||
(is (mth/close? 2.0 (-> res :transform :height)))
|
||||
;; Rendered pixel content has top at -50, height 100 -> bottom edge at +50 (matches bottom edge of 50px container!)
|
||||
(is (mth/close? -50.0 (-> res :rendered-pixel-rect :y)))
|
||||
(is (mth/close? 100.0 (-> res :rendered-pixel-rect :height)))))
|
||||
|
||||
(testing "Top-Left handle crop (shrinking both dimensions to 50%)"
|
||||
(let [res (compute-bounds-resize-transform
|
||||
{:width 200 :height 100 :handler :top-left :center? false :sx 0.5 :sy 0.5})]
|
||||
(is (mth/close? -1.0 (-> res :transform :x)))
|
||||
(is (mth/close? -1.0 (-> res :transform :y)))
|
||||
(is (mth/close? 2.0 (-> res :transform :width)))
|
||||
(is (mth/close? 2.0 (-> res :transform :height)))
|
||||
(is (mth/close? -100.0 (-> res :rendered-pixel-rect :x)))
|
||||
(is (mth/close? -50.0 (-> res :rendered-pixel-rect :y)))
|
||||
(is (mth/close? 200.0 (-> res :rendered-pixel-rect :width)))
|
||||
(is (mth/close? 100.0 (-> res :rendered-pixel-rect :height)))))
|
||||
|
||||
(testing "Center resize (Alt modifier)"
|
||||
(let [res (compute-bounds-resize-transform
|
||||
{:width 200 :height 100 :handler :right :center? true :sx 0.5 :sy 0.5})]
|
||||
(is (mth/close? -0.5 (-> res :transform :x)))
|
||||
(is (mth/close? -0.5 (-> res :transform :y)))
|
||||
(is (mth/close? 2.0 (-> res :transform :width)))
|
||||
(is (mth/close? 2.0 (-> res :transform :height)))
|
||||
(is (mth/close? -50.0 (-> res :rendered-pixel-rect :x)))
|
||||
(is (mth/close? -25.0 (-> res :rendered-pixel-rect :y)))
|
||||
(is (mth/close? 200.0 (-> res :rendered-pixel-rect :width)))
|
||||
(is (mth/close? 100.0 (-> res :rendered-pixel-rect :height)))))
|
||||
|
||||
(testing "Bottom handle crop (shrinking height to 50% from bottom)"
|
||||
(let [res (compute-bounds-resize-transform
|
||||
{:width 200 :height 100 :handler :bottom :center? false :sx 1.0 :sy 0.5})]
|
||||
(is (mth/close? 0.0 (-> res :transform :x)))
|
||||
(is (mth/close? 0.0 (-> res :transform :y)))
|
||||
(is (mth/close? 1.0 (-> res :transform :width)))
|
||||
(is (mth/close? 2.0 (-> res :transform :height)))
|
||||
(is (mth/close? 0.0 (-> res :rendered-pixel-rect :y)))
|
||||
(is (mth/close? 100.0 (-> res :rendered-pixel-rect :height)))))
|
||||
|
||||
(testing "Top-Right handle crop (shrinking both dimensions to 50%)"
|
||||
(let [res (compute-bounds-resize-transform
|
||||
{:width 200 :height 100 :handler :top-right :center? false :sx 0.5 :sy 0.5})]
|
||||
(is (mth/close? 0.0 (-> res :transform :x)))
|
||||
(is (mth/close? -1.0 (-> res :transform :y)))
|
||||
(is (mth/close? 2.0 (-> res :transform :width)))
|
||||
(is (mth/close? 2.0 (-> res :transform :height)))
|
||||
(is (mth/close? 0.0 (-> res :rendered-pixel-rect :x)))
|
||||
(is (mth/close? -50.0 (-> res :rendered-pixel-rect :y)))
|
||||
(is (mth/close? 200.0 (-> res :rendered-pixel-rect :width)))
|
||||
(is (mth/close? 100.0 (-> res :rendered-pixel-rect :height)))))
|
||||
|
||||
(testing "Bottom-Left handle crop (shrinking both dimensions to 50%)"
|
||||
(let [res (compute-bounds-resize-transform
|
||||
{:width 200 :height 100 :handler :bottom-left :center? false :sx 0.5 :sy 0.5})]
|
||||
(is (mth/close? -1.0 (-> res :transform :x)))
|
||||
(is (mth/close? 0.0 (-> res :transform :y)))
|
||||
(is (mth/close? 2.0 (-> res :transform :width)))
|
||||
(is (mth/close? 2.0 (-> res :transform :height)))
|
||||
(is (mth/close? -100.0 (-> res :rendered-pixel-rect :x)))
|
||||
(is (mth/close? 0.0 (-> res :rendered-pixel-rect :y)))
|
||||
(is (mth/close? 200.0 (-> res :rendered-pixel-rect :width)))
|
||||
(is (mth/close? 100.0 (-> res :rendered-pixel-rect :height)))))
|
||||
|
||||
(testing "Expanding bounds beyond original size (empty space exposure)"
|
||||
(let [res (compute-bounds-resize-transform
|
||||
{:width 200 :height 100 :handler :right :center? false :sx 2.0 :sy 1.0})]
|
||||
(is (mth/close? 0.0 (-> res :transform :x)))
|
||||
(is (mth/close? 0.0 (-> res :transform :y)))
|
||||
(is (mth/close? 0.5 (-> res :transform :width)))
|
||||
(is (mth/close? 1.0 (-> res :transform :height)))
|
||||
;; Rendered pixel content is 200px wide in a 400px container -> exposes 200px empty space
|
||||
(is (mth/close? 0.0 (-> res :rendered-pixel-rect :x)))
|
||||
(is (mth/close? 200.0 (-> res :rendered-pixel-rect :width))))))
|
||||
|
||||
(deftest test-sequential-resize-operations
|
||||
(testing "Sequential crops: crop right then crop left"
|
||||
;; Initial shape: 200x100, transform: {:x 0 :y 0 :width 1 :height 1}
|
||||
;; Step 1: Crop right handle from 200 to 150 (sx = 0.75)
|
||||
(let [step1 (compute-bounds-resize-transform
|
||||
{:width 200 :height 100 :handler :right :center? false :sx 0.75 :sy 1.0})
|
||||
tf1 (:transform step1)]
|
||||
(is (mth/close? 0.0 (:x tf1)))
|
||||
(is (mth/close? (/ 1.0 0.75) (:width tf1)))
|
||||
;; Step 2: Now shape is 150x100 with tf1. Crop left handle from 150 to 100 (sx = 100/150 = 2/3)
|
||||
(let [step2 (compute-bounds-resize-transform
|
||||
{:width 150 :height 100 :handler :left :center? false :sx (/ 2.0 3.0) :sy 1.0 :transform tf1})
|
||||
tf2 (:transform step2)]
|
||||
;; The final 100x100 container has bitmap with width 200px
|
||||
(is (mth/close? 200.0 (-> step2 :rendered-pixel-rect :width)))
|
||||
;; The bitmap left edge is at -50px in the 100px container, so right edge is at -50 + 200 = 150px
|
||||
(is (mth/close? -50.0 (-> step2 :rendered-pixel-rect :x))))))
|
||||
|
||||
(testing "Bounds resize followed by standard proportional scaling"
|
||||
;; Step 1: Bounds resize crops width from 200 to 100
|
||||
(let [step1 (compute-bounds-resize-transform
|
||||
{:width 200 :height 100 :handler :right :center? false :sx 0.5 :sy 1.0})
|
||||
tf1 (:transform step1)]
|
||||
(is (mth/close? 2.0 (:width tf1)))
|
||||
(is (mth/close? 1.0 (:height tf1)))
|
||||
|
||||
;; Step 2: Standard proportional scale of the 100x100 cropped shape to 200x200 (scale 2x)
|
||||
;; During standard scale, normalized transform tf1 is kept constant!
|
||||
(let [scaled-w (* 100.0 2.0)
|
||||
scaled-h (* 100.0 2.0)
|
||||
rendered-w (* (:width tf1) scaled-w)
|
||||
rendered-h (* (:height tf1) scaled-h)]
|
||||
;; The underlying bitmap scaled from 200x100 to 400x200, matching the 2x scale of the cropped frame!
|
||||
(is (mth/close? 400.0 rendered-w))
|
||||
(is (mth/close? 200.0 rendered-h))))))
|
||||
|
||||
(deftest test-proportion-lock-invariance
|
||||
(testing "Shape proportion-lock attribute remains unchanged"
|
||||
(let [shape {:id (uuid/custom 10)
|
||||
:type :rect
|
||||
:width 200
|
||||
:height 100
|
||||
:proportion-lock true
|
||||
:fills [{:fill-image {:id (uuid/custom 1)
|
||||
:width 800
|
||||
:height 600
|
||||
:keep-aspect-ratio true}}]}
|
||||
;; Simulate bounds resize interaction
|
||||
has-img? (boolean (or (some :fill-image (:fills shape)) (:fill-image shape)))
|
||||
mod-pressed? true
|
||||
bounds-resize? (and has-img? mod-pressed?)
|
||||
lock-during-drag (if bounds-resize? false (:proportion-lock shape))]
|
||||
;; During drag, lock is bypassed (unless Shift is pressed)
|
||||
(is (false? lock-during-drag))
|
||||
;; Shape's persistent setting is completely preserved
|
||||
(is (true? (:proportion-lock shape))))))
|
||||
@@ -29,6 +29,7 @@
|
||||
[common-tests.geom-flex-layout-test]
|
||||
[common-tests.geom-grid-layout-test]
|
||||
[common-tests.geom-grid-test]
|
||||
[common-tests.geom-image-bounds-resize-test]
|
||||
[common-tests.geom-line-test]
|
||||
[common-tests.geom-modif-tree-test]
|
||||
[common-tests.geom-modifiers-test]
|
||||
@@ -108,6 +109,7 @@
|
||||
'common-tests.geom-flex-layout-test
|
||||
'common-tests.geom-grid-layout-test
|
||||
'common-tests.geom-grid-test
|
||||
'common-tests.geom-image-bounds-resize-test
|
||||
'common-tests.geom-line-test
|
||||
'common-tests.geom-modif-tree-test
|
||||
'common-tests.geom-modifiers-test
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns common-tests.types.tokens-status-test
|
||||
(:require
|
||||
|
||||
@@ -157,6 +157,11 @@ services:
|
||||
# PENPOT_OBJECTS_STORAGE_S3_ENDPOINT: <ENDPOINT>
|
||||
# PENPOT_OBJECTS_STORAGE_S3_BUCKET: <BUKET_NAME>
|
||||
|
||||
## Optional: route specific internal semantic buckets (for example
|
||||
## tempfile) to additional S3 targets. The file is EDN; see
|
||||
## docs/technical-guide/configuration.md for the format.
|
||||
# PENPOT_OBJECTS_STORAGE_S3_ROUTES_FILE: /opt/data/storage-routes.edn
|
||||
|
||||
## Telemetry. When enabled, a periodical process will send anonymous data about this
|
||||
## instance. Telemetry data will enable us to learn how the application is used,
|
||||
## based on real scenarios. If you want to help us, please leave it enabled. You can
|
||||
|
||||
@@ -549,6 +549,51 @@ PENPOT_OBJECTS_STORAGE_S3_ENDPOINT: <endpoint-uri>
|
||||
These settings are equally useful if you have a Minio storage system.
|
||||
</p>
|
||||
|
||||
#### S3 targets and per-bucket routing
|
||||
|
||||
__Since version 2.19.0__
|
||||
|
||||
By default the S3 backend stores every object in the configured bucket. You can
|
||||
route specific internal semantic buckets (for example `tempfile` or `file-data`)
|
||||
to additional S3 targets. Each target has its own bucket and, optionally, its
|
||||
own key prefix, region and endpoint.
|
||||
|
||||
Targets and routing are declared in an external EDN file referenced by the
|
||||
`PENPOT_OBJECTS_STORAGE_S3_ROUTES_FILE` environment variable:
|
||||
|
||||
```clojure
|
||||
{:targets
|
||||
{:temp {:bucket "penpot-temp"}
|
||||
:cold {:bucket "penpot-cold"
|
||||
:region :us-east-1
|
||||
:endpoint "https://s3.us-east-1.amazonaws.com"
|
||||
:prefix "objects/"}}
|
||||
:routes
|
||||
{"tempfile" :temp
|
||||
"file-data" :temp}}
|
||||
```
|
||||
|
||||
- `:targets` declares the named destinations. `:bucket` is required; `:prefix`,
|
||||
`:region` and `:endpoint` are optional and inherit the values from the default
|
||||
target (`PENPOT_OBJECTS_STORAGE_S3_*`) when omitted.
|
||||
- `:routes` maps a Penpot semantic bucket to a target id. Semantic buckets that
|
||||
are not listed use the default target (the configured
|
||||
`PENPOT_OBJECTS_STORAGE_S3_BUCKET`).
|
||||
- The `:default` target id is reserved and implicit; do not declare it.
|
||||
- The file is optional and only applies to the `s3` backend. Without it, all
|
||||
objects use the default bucket and the behavior is unchanged.
|
||||
|
||||
The target id is stored in the object metadata, so it must stay stable. If you
|
||||
remove or rename a target id, objects already written with it cannot be located:
|
||||
reads and serving of those objects fail, and the garbage-collection tasks refuse
|
||||
to delete them. Those rows are parked for one day and retried on later runs,
|
||||
without counting deletion attempts and without ever reaching the give-up window,
|
||||
so the row is never removed until the routes file declares the target id again.
|
||||
|
||||
All backend replicas must mount the same routes file with the same target ids. A
|
||||
replica with a stale file misroutes new writes and fails reads and garbage
|
||||
collection for the targets it does not declare.
|
||||
|
||||
### File Data Storage
|
||||
|
||||
__Since version 2.11.0__
|
||||
|
||||
@@ -16,11 +16,12 @@ that may be used for any kind of user uploaded files. Currently:
|
||||
There is an abstract interface and several implementations (or **backends**),
|
||||
depending on where the objects are actually stored:
|
||||
|
||||
* <code class="language-clojure">:assets-fs</code> stores ojects in the file system, under a given base path.
|
||||
* <code class="language-clojure">:assets-s3</code> stores them in any cloud storage with an AWS-S3 compatible
|
||||
* <code class="language-clojure">:fs</code> stores objects in the file system, under a given base path.
|
||||
* <code class="language-clojure">:s3</code> stores them in any cloud storage with an AWS-S3 compatible
|
||||
interface.
|
||||
* <code class="language-clojure">:assets-db</code> stores them inside the PostgreSQL database, in a special table
|
||||
with a binary column.
|
||||
|
||||
Legacy rows may still reference the deprecated <code class="language-clojure">:assets-fs</code> and
|
||||
<code class="language-clojure">:assets-s3</code> names, which alias the current backends.
|
||||
|
||||
## Storage API
|
||||
|
||||
@@ -83,6 +84,17 @@ The storage module may use the bucket (hardcoded) to make special treatment to
|
||||
object, such as storing in a different path, or guessing how to know if an object
|
||||
is referenced from other place.
|
||||
|
||||
When the <code class="language-clojure">:s3</code> backend is used, a semantic bucket may also be routed to a
|
||||
named S3 **target** (a different bucket and/or key prefix). The chosen target id
|
||||
is stored in the object metadata (<code class="language-clojure">:storage-target</code>) and resolved again on
|
||||
reads, URL signing, deduplication and garbage collection. Objects without a
|
||||
stored target use the default target. See the configuration guide for the
|
||||
routing file format.
|
||||
|
||||
If an object references a target id that is not configured anymore, it cannot be
|
||||
located: reads and serving fail, and garbage collection refuses to delete the
|
||||
object, keeping the database row until the target is declared again.
|
||||
|
||||
## Sharing and deleting objects
|
||||
|
||||
To save storage space, duplicated objects wre shared. So, if for example
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns app.auth
|
||||
"Resolves the caller's session cookie to a real profile id.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns app.handlers.export
|
||||
"Handle export jobs"
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns app.handlers.jobs
|
||||
"REST surface for export jobs, under `/api/export/jobs`.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns app.jobs
|
||||
"Export job model and lifecycle.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns app.jobs.scheduler
|
||||
"Admission control for export jobs.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns app.jobs.store
|
||||
"Redis persistence for export jobs.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns app.jobs.utils
|
||||
"Temp file ownership for export jobs.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns app.router
|
||||
"Method + path dispatch.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns app.wasm.pool
|
||||
"Pool of headless render workers.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns app.wasm.render
|
||||
"Headless render pipeline: renders exports with the render-wasm Skia pipeline,
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns app.wasm.worker
|
||||
"Render worker entry point.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns exporter-tests.export-shapes-test
|
||||
"Chunking of the browser backend."
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns exporter-tests.jobs-test
|
||||
"Job state machine. Runs without redis: a store write with no connection is
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns exporter-tests.scheduler-test
|
||||
"Admission control. A headless job leases one render worker for its whole run,
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
;; License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
;;
|
||||
;; Copyright (c) KALEIDOS INC Sucursal en España SL
|
||||
;; Copyright (c) KALEIDOS SUBSIDIARY SL
|
||||
|
||||
(ns exporter-tests.wasm-pool-test
|
||||
"Worker leasing, against a stub pool: `with-worker` must give the worker back
|
||||
|
||||
@@ -274,6 +274,25 @@ test("Renders a file with different text leaves decoration", async ({
|
||||
await expect(workspace.canvas).toHaveScreenshot();
|
||||
});
|
||||
|
||||
// Both paragraphs decorate the same spans; the first one paints every span with
|
||||
// the same fill, which used to collapse the decorated spans into their
|
||||
// neighbours and drop their underline / line-through.
|
||||
test("Renders text spans decorated independently of their fill", async ({
|
||||
page,
|
||||
}) => {
|
||||
const workspace = new WasmWorkspacePage(page);
|
||||
await workspace.setupEmptyFile();
|
||||
await workspace.mockGetFile("render-wasm/get-file-text-span-decoration.json");
|
||||
|
||||
await workspace.goToWorkspace({
|
||||
id: "1d0f6a4c-0000-8000-8006-000000000001",
|
||||
pageId: "1d0f6a4c-0000-8000-8006-000000000002",
|
||||
});
|
||||
|
||||
await workspace.waitForFirstRenderWithoutUI();
|
||||
await expect(workspace.canvas).toHaveScreenshot();
|
||||
});
|
||||
|
||||
test("Renders a file with different text shadows combinations", async ({
|
||||
page,
|
||||
}) => {
|
||||
|
||||
|
Before Width: | Height: | Size: 241 KiB After Width: | Height: | Size: 299 KiB |
|
Before Width: | Height: | Size: 175 KiB After Width: | Height: | Size: 220 KiB |
|
Before Width: | Height: | Size: 168 KiB After Width: | Height: | Size: 181 KiB |
|
Before Width: | Height: | Size: 80 KiB After Width: | Height: | Size: 117 KiB |
|
Before Width: | Height: | Size: 450 KiB After Width: | Height: | Size: 471 KiB |
|
Before Width: | Height: | Size: 519 KiB After Width: | Height: | Size: 665 KiB |
|
Before Width: | Height: | Size: 143 KiB After Width: | Height: | Size: 185 KiB |
|
Before Width: | Height: | Size: 88 KiB After Width: | Height: | Size: 118 KiB |
|
Before Width: | Height: | Size: 116 KiB After Width: | Height: | Size: 152 KiB |
|
Before Width: | Height: | Size: 124 KiB After Width: | Height: | Size: 139 KiB |