📎 Test the serving path rejects unconfigured storage targets

The read/serve path (get-object-data) must fail with
:invalid-storage-target when an object references a target id that is
not configured, matching get-object-url and bulk delete. Add the
missing test so removing a target id cannot silently serve from the
wrong place.

Closes #11630

AI-assisted-by: deepseek-flash

.serena/memories/backend/storage.md

@@ -8,13 +8,30 @@
 - The backend stores the binary content.
 - Supported backends are `:fs` and `:s3`.
 - FS uses one root directory and a UUID-derived path.
-- S3 uses one configured bucket and an optional prefix.
+- S3 uses a default configured bucket and an optional prefix, plus optional named targets.
 - A Penpot bucket is metadata. It is not an S3 bucket or a filesystem directory.
 - FS and S3 use the same UUID-derived object path. The bucket does not change the path.
 - `PENPOT_OBJECTS_STORAGE_*` configures the current object backend.
 - Deprecated asset-storage config keys remain supported for migration.
 - Database rows keep the backend name. Keep the legacy `:assets-fs` and `:assets-s3` aliases.
 
+## S3 Targets and Routing
+
+- The `:s3` backend keeps `storage_object.backend = 's3'`; routing lives inside the backend.
+- Routing maps a Penpot semantic bucket to a named target (own bucket, optional prefix/region/endpoint).
+- Targets are declared in an EDN file referenced by `PENPOT_OBJECTS_STORAGE_S3_ROUTES_FILE` (`app.storage.config/load`).
+- Schema: `{:targets {<id> {:bucket ... :prefix? ... :region? ... :endpoint? ...}} :routes {"<semantic-bucket>" <id>}}`.
+- The reserved `:default` target is implicit and built from `PENPOT_OBJECTS_STORAGE_S3_*`; declared targets inherit missing region/endpoint/prefix from it.
+- Without a routes file, `::sto/bucket->target` is nil and every object uses `:default` (unchanged behavior).
+- The chosen target id is stored in object metadata as `:storage-target` (plain string) by `put-object!`.
+- `app.storage.s3/resolve-target` reads the object metadata; `nil` (legacy) and `"default"` use the default target, an unknown non-nil id raises `:invalid-storage-target` (no fallback) on reads/serving/deletes.
+- `impl/target-resolvable?` (wrapped as `sto/target-resolvable?`) reports whether a target id is configured; `:fs` is always true.
+- GC-deleted and pending-gc refuse to delete rows whose target is not resolvable: they log `:err`, park the row (`deleted_at = now()+1d`, no attempts, no give-up) and never remove it until the target is configured again.
+- `deleted_at` doubles as the pending-gc park marker; the pending selection skips rows whose `deleted_at` is in the future.
+- One S3 client/presigner is built per distinct `[region endpoint]` and shared by targets; a failed init closes the already-built pairs.
+- Target ids are stored in metadata, so they must stay stable; removing one makes its old rows unreadable and unGC-able by design.
+- `:storage-target` metadata is load-bearing: `pending-gc` passes it via `with-meta` so `del-object` resolves the right target.
+
 ## Object Lifecycle
 
 - `put-object!` creates the database row before it writes backend content.
@@ -64,7 +81,7 @@ Since `put-object!` uses backend-specific operations (`impl/resolve-backend` + `
 ## Deduplication
 
 - Deduplication requires `::sto/deduplicate?`, a content hash, and bucket metadata.
-- The lookup matches hash, bucket, backend, and `deleted_at IS NULL`.
+- The lookup matches hash, bucket, backend, storage target (`:storage-target`, coalesced to `default`), and `deleted_at IS NULL`.
 - The lookup only considers rows with `status='valid'`; pending rows are invisible.
 - A hit whose blob is missing is repaired in place: the same row/id is kept,
   and `put-object!` rewrites the blob under that id. This heals all existing
@@ -93,6 +110,8 @@ Since `put-object!` uses backend-specific operations (`impl/resolve-backend` + `
 
 - The valid bucket set lives in `app.storage/valid-buckets`.
 - `file-media-object` is the default bucket for old rows without bucket metadata.
+- Under `:s3`, any valid bucket may be routed to a named target; unrouted buckets use `:default`.
+- GC resolves the target from `metadata.:storage-target` for deleted and pending rows.
 - Do not assign a new bucket without adding its access and cleanup behavior.
 - The touched-object collector raises an internal error for an unknown bucket.
 - It supports `file-media-object`, `team-font-variant`, `file-object-thumbnail`, `file-thumbnail`, `profile`, `file-data`, `tempfile`, and `organization`.

backend/src/app/config.clj

@@ -293,6 +293,7 @@
     [:objects-storage-s3-bucket {:optional true} :string]
     [:objects-storage-s3-region {:optional true} :keyword]
     [:objects-storage-s3-endpoint {:optional true} ::sm/uri]
+    [:objects-storage-s3-routes-file {:optional true} :string]
 
     ;; SSRF protection
     [:ssrf-allowed-hosts {:optional true} [::sm/set :string]]

backend/src/app/main.clj

@@ -34,6 +34,7 @@
    [app.setup :as-alias setup]
    [app.srepl :as-alias srepl]
    [app.storage :as-alias sto]
+   [app.storage.config :as sto.config]
    [app.storage.fs :as-alias sto.fs]
    [app.storage.gc-deleted :as-alias sto.gc-deleted]
    [app.storage.gc-touched :as-alias sto.gc-touched]
@@ -148,6 +149,11 @@
     ::mdef/labels []
     ::mdef/type :histogram}})
 
+(def ^:private storage-routing
+  "Optional S3 storage targets and per semantic-bucket routing. Loaded once
+  from the external routes file. Empty when the feature is not configured."
+  (sto.config/load))
+
 (def system-config
   {::db/pool
    {::db/uri        (cf/get :database-uri)
@@ -522,7 +528,8 @@
      ;; explicit migration because the database objects/rows will
      ;; still reference the old names).
      :assets-s3 (ig/ref :app.storage.s3/backend)
-     :assets-fs (ig/ref :app.storage.fs/backend)}}
+     :assets-fs (ig/ref :app.storage.fs/backend)}
+    ::sto/bucket->target (:routes storage-routing)}
 
    :app.storage.s3/backend
    {::sto.s3/region     (or (cf/get :storage-assets-s3-region)
@@ -533,6 +540,7 @@
                             (cf/get :objects-storage-s3-bucket))
     ::sto.s3/io-threads (or (cf/get :storage-assets-s3-io-threads)
                             (cf/get :objects-storage-s3-io-threads))
+    ::sto.s3/targets    (:targets storage-routing)
 
     ::wrk/netty-io-executor
     (ig/ref ::wrk/netty-io-executor)}

backend/src/app/storage.clj

@@ -70,6 +70,7 @@
   [:map {:title "storage"}
    [::backends schema:backends]
    [::backend [:enum :s3 :fs]]
+   [::bucket->target {:optional true} [:map-of :string :keyword]]
    ::db/pool])
 
 (def valid-storage?
@@ -112,17 +113,18 @@
              params))
 
 (defn- get-database-object-by-hash
-  [connectable backend bucket hash]
+  [connectable backend bucket target hash]
   (let [sql (str "select * from storage_object "
                  " where (metadata->>'~:hash') = ? "
                  "   and (metadata->>'~:bucket') = ? "
+                 "   and coalesce(metadata->>'~:storage-target', 'default') = ? "
                  "   and backend = ?"
                  "   and deleted_at is null"
                  "   and status = 'valid'"
                  " limit 1")]
     ;; NOTE: metadata is left encoded; row->storage-object is
     ;; responsible for decoding it.
-    (db/exec-one! connectable [sql hash bucket (name backend)])))
+    (db/exec-one! connectable [sql hash bucket target (name backend)])))
 
 (defn- promote-object!
   [storage object]
@@ -185,6 +187,13 @@
   (let [ds (db/get-connectable storage)]
     (get-database-object ds id)))
 
+(defn- resolve-target-id
+  "Returns the storage target id for the given semantic bucket, or nil when
+  the routing does not apply (non-S3 backends)."
+  [storage bucket]
+  (when (= :s3 (::backend storage))
+    (get (::bucket->target storage) bucket :default)))
+
 (defn put-object!
   "Creates a new object with the provided content."
   [{:keys [::backend ::db/pool] :as storage}
@@ -193,9 +202,16 @@
   (assert (impl/content? content) "expected an instance of content")
 
   (let [id         (or (::id params) (uuid/random))
-        mdata      (cond-> (get-metadata params)
+        base-mdata (get-metadata params)
+        bucket     (:bucket base-mdata)
+        target     (resolve-target-id storage bucket)
+        target-str (or (some-> target name) "default")
+        mdata      (cond-> base-mdata
                      (satisfies? impl/IContentHash content)
-                     (assoc :hash (impl/get-hash content)))
+                     (assoc :hash (impl/get-hash content))
+
+                     (some? target)
+                     (assoc :storage-target target-str))
 
         touched-at (if touch
                      (or touched-at (ct/now))
@@ -214,10 +230,11 @@
                              (not= tempfile-bucket (:bucket mdata)))
                     (get-database-object-by-hash pool backend
                                                  (:bucket mdata)
+                                                 target-str
                                                  (:hash mdata)))]
 
       ;; PHASE 2: an existing reference is found: reuse or repair it.
-      (if (impl/exists-object? backend' hit)
+      (if (impl/exists-object? backend' (row->storage-object hit))
 
         ;; PHASE 2a: healthy reference. Optionally refresh touched_at
         ;; and reuse the object as it is.
@@ -312,6 +329,16 @@
                    (ct/is-after? (:expired-at object) (ct/now))))
       (-> (impl/get-object-url backend object nil) file-url->path))))
 
+(defn target-resolvable?
+  "Returns true when the backend referenced by `backend-id` can resolve
+  `target` to a real destination. GC callers must refuse to delete objects
+  whose target is not resolvable, so a misconfigured target never removes
+  the database row while orphaning the blob."
+  [storage backend-id target]
+  (assert (valid-storage? storage))
+  (-> (impl/resolve-backend storage backend-id)
+      (impl/target-resolvable? target)))
+
 (defn del-object!
   [storage object-or-id]
   (assert (valid-storage? storage))

backend/src/app/storage/config.clj

@@ -0,0 +1,143 @@
+;; This Source Code Form is subject to the terms of the Mozilla Public
+;; License, v. 2.0. If a copy of the MPL was not distributed with this
+;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
+;;
+;; Copyright (c) KALEIDOS SUBSIDIARY SL
+
+(ns app.storage.config
+  "Configuration of the optional S3 storage targets and their per
+  semantic-bucket routing.
+
+  The routing is read from an external EDN file referenced by the
+  `PENPOT_OBJECTS_STORAGE_S3_ROUTES_FILE` environment variable. The file
+  declares additional S3 targets and, optionally, a map from Penpot
+  semantic bucket to target id:
+
+    {:targets
+     {:temp {:bucket \"penpot-temp\"}
+      :cold {:bucket \"penpot-cold\" :region :us-east-1}}
+     :routes
+     {\"tempfile\" :temp
+      \"file-data\" :temp}}
+
+  The implicit `:default` target always exists and is built from the
+  existing `PENPOT_OBJECTS_STORAGE_S3_*` configuration, so the feature is
+  fully backward compatible when the file is absent."
+  (:refer-clojure :exclude [load])
+  (:require
+   [app.common.exceptions :as ex]
+   [app.common.schema :as sm]
+   [app.common.uri :as u]
+   [app.config :as cf]
+   [app.storage :as sto]
+   [app.storage.s3 :as sto.s3]
+   [clojure.edn :as edn]
+   [clojure.java.io :as io]))
+
+(def ^:private schema:target
+  "The EDN declares targets with the same shape as the S3 backend, except
+  the endpoint is accepted as a plain string and normalized to a URI by
+  `normalize-target`."
+  [:merge
+   sto.s3/schema:target
+   [:map
+    [:endpoint {:optional true} [:or :string ::sm/uri]]]])
+
+(def ^:private schema:file
+  [:map {:title "storage-routes"}
+   [:targets [:map-of :keyword schema:target]]
+   [:routes {:optional true} [:map-of :string :keyword]]])
+
+(def ^:private valid-file?
+  (sm/validator schema:file))
+
+(def ^:private explain-file
+  (sm/explainer schema:file))
+
+(defn- read-file
+  [path]
+  (try
+    (-> (io/file path) slurp (edn/read-string))
+    (catch Exception cause
+      (ex/raise :type :validation
+                :code :invalid-storage-routes-file
+                :hint "unable to read storage routes file"
+                :path (str path)
+                :cause cause))))
+
+(defn- assert-backend-s3!
+  "The routing only makes sense on top of the S3 backend."
+  []
+  (let [backend (or (sto/get-legacy-backend)
+                    (cf/get :objects-storage-backend)
+                    :fs)]
+    (when-not (= :s3 backend)
+      (ex/raise :type :validation
+                :code :invalid-storage-routes-backend
+                :hint "storage routes file requires the :s3 backend"
+                :backend (keyword backend)))))
+
+(defn- validate!
+  [data path]
+  (when-not (valid-file? data)
+    (ex/raise :type :validation
+              :code :invalid-storage-routes
+              :hint "invalid storage routes file"
+              :path (str path)
+              :explain (explain-file data)))
+
+  (let [targets (:targets data)
+        routes  (:routes data)]
+
+    (when (contains? targets :default)
+      (ex/raise :type :validation
+                :code :reserved-storage-target
+                :hint "`:default` is a reserved storage target id"
+                :path (str path)))
+
+    (doseq [[bucket target] routes]
+      (when-not (contains? sto/valid-buckets bucket)
+        (ex/raise :type :validation
+                  :code :invalid-storage-routes-bucket
+                  :hint "unknown semantic bucket in storage routes"
+                  :bucket bucket
+                  :path (str path)))
+
+      (when-not (contains? targets target)
+        (ex/raise :type :validation
+                  :code :unknown-storage-target
+                  :hint "storage route points to an undeclared target"
+                  :bucket bucket
+                  :target target
+                  :path (str path))))
+
+    data))
+
+(defn- normalize-target
+  [target]
+  (cond-> target
+    (string? (:endpoint target))
+    (update :endpoint u/uri)))
+
+(defn- normalize-targets
+  [targets]
+  (persistent!
+   (reduce-kv (fn [acc id target]
+                (assoc! acc id (normalize-target target)))
+              (transient {})
+              targets)))
+
+(defn load
+  "Reads and validates the optional S3 routing file.
+
+  Returns a map with `:targets` (id -> target definition) and `:routes`
+  (semantic bucket -> target id), or `{:targets nil :routes nil}` when the
+  configuration key is unset."
+  []
+  (if-let [path (not-empty (cf/get :objects-storage-s3-routes-file))]
+    (let [data (-> (read-file path) (validate! path))]
+      (assert-backend-s3!)
+      {:targets (normalize-targets (:targets data))
+       :routes  (:routes data)})
+    {:targets nil
+     :routes  nil}))

backend/src/app/storage/fs.clj

@@ -141,7 +141,7 @@
     (Files/deleteIfExists ^Path path)))
 
 (defmethod impl/del-objects-in-bulk :fs
-  [backend ids]
+  [backend _target ids]
   (assert (valid-backend? backend) "expected a valid backend instance")
   (let [base (fs/path (::directory backend))]
     (reduce (fn [fail-ids id]
@@ -153,3 +153,7 @@
                     (conj fail-ids id)))))
             #{} ids)))
 
+(defmethod impl/target-resolvable? :fs
+  [_backend _target]
+  true)
+

backend/src/app/storage/gc_deleted.clj

@@ -78,6 +78,29 @@
   (let [ids (db/create-array conn "uuid" ids)]
     (db/exec-one! conn [sql:delete-give-up ids max-attempts])))
 
+(defn- log-refusal!
+  "Indirection over the error log so tests can capture the refusal payload."
+  [backend-id target ids]
+  (l/err :hint "storage target is not configured, deletion refused"
+         :backend (name backend-id)
+         :target target
+         :ids (mapv str ids)))
+
+(def ^:private sql:defer-unresolvable
+  "UPDATE storage_object
+      SET deleted_at = NOW() + INTERVAL '1 day'
+    WHERE id = ANY(?::uuid[])")
+
+(defn- park-unresolvable!
+  "Refuses to delete rows whose target id is not configured: logs the
+  misconfiguration and pushes `deleted_at` forward so the rows leave the
+  selection window without being deleted and without counting a deletion
+  attempt (they are therefore never subject to the give-up window)."
+  [conn backend-id target ids]
+  (log-refusal! backend-id target ids)
+  (let [ids (db/create-array conn "uuid" ids)]
+    (db/exec-one! conn [sql:defer-unresolvable ids])))
+
 (defn- process-chunk
   "Attempt to delete a chunk of storage objects from a specific backend.
 
@@ -87,11 +110,11 @@
 
   Returns the number of successfully deleted objects, or 0 if no rows
   could be locked."
-  [conn storage backend-id ids]
+  [conn storage backend-id target ids]
   (if-let [locked-ids (lock-ids conn ids)]
     (let [fail-ids (try
                      (-> (impl/resolve-backend storage backend-id)
-                         (impl/del-objects-in-bulk locked-ids))
+                         (impl/del-objects-in-bulk target locked-ids))
                      (catch Throwable cause
                        (l/err :hint "error on physical deletion, will retry"
                               :ids locked-ids
@@ -118,12 +141,15 @@
       (count ok-ids))
     0))
 
-(defn- group-by-backend
+(defn- group-by-route
   [items]
-  (d/group-by (comp keyword :backend) :id #{} items))
+  (d/group-by (fn [item]
+                [(keyword (:backend item)) (:target item)])
+              :id #{} items))
 
 (def ^:private sql:get-deleted-chunk
-  "SELECT id, backend
+  "SELECT id, backend,
+          coalesce(metadata->>'~:storage-target', 'default') as target
      FROM storage_object
     WHERE deleted_at IS NOT NULL
       AND deleted_at <= ?
@@ -139,19 +165,27 @@
 
 (defn- clean-deleted!
   [cfg]
-  (loop [total 0]
-    (let [deleted (db/tx-run! cfg
-                              (fn [{:keys [::db/conn ::sto/storage]}]
-                                (let [chunk (get-deleted-chunk conn chunk-size)]
-                                  (when (seq chunk)
-                                    (let [by-backend (group-by-backend chunk)]
-                                      (reduce-kv (fn [acc backend-id ids]
-                                                   (+ acc (process-chunk conn storage backend-id ids)))
-                                                 0
-                                                 by-backend))))))]
-      (if deleted
-        (recur (+ total deleted))
-        total))))
+  (loop [deleted 0
+         parked  0]
+    (let [result (db/tx-run! cfg
+                             (fn [{:keys [::db/conn ::sto/storage]}]
+                               (let [chunk (get-deleted-chunk conn chunk-size)]
+                                 (when (seq chunk)
+                                   (let [by-route (group-by-route chunk)]
+                                     (reduce-kv
+                                      (fn [acc [backend-id target] ids]
+                                        (if (sto/target-resolvable? storage backend-id target)
+                                          (update acc :deleted + (process-chunk conn storage backend-id target ids))
+                                          (do
+                                            (park-unresolvable! conn backend-id target ids)
+                                            (update acc :parked + (count ids)))))
+                                      {:deleted 0 :parked 0}
+                                      by-route))))))]
+      (if result
+        (recur (+ deleted (:deleted result))
+               (+ parked (:parked result)))
+        {:deleted deleted
+         :parked  parked}))))
 
 (defmethod ig/assert-key ::handler
   [_ params]
@@ -161,6 +195,7 @@
 (defmethod ig/init-key ::handler
   [_ cfg]
   (fn [_]
-    (let [total (clean-deleted! cfg)]
-      (l/inf :hint "task finished" :total total)
-      {:deleted total})))
+    (let [{:keys [deleted parked]} (clean-deleted! cfg)]
+      (l/inf :hint "task finished" :total deleted :parked parked)
+      {:deleted deleted
+       :parked  parked})))

backend/src/app/storage/impl.clj

@@ -72,12 +72,14 @@
             :context cfg))
 
 (defmulti del-objects-in-bulk
-  "Delete multiple objects in bulk. Returns #{fail-ids} — the set of ids
-  whose blob deletion failed. Empty set = all succeeded."
-  (fn [cfg _] (::sto/type cfg)))
+  "Delete multiple objects in bulk. `target` is an optional backend-specific
+  destination id (used by the S3 storage targets); backends that do not route
+  ignore it. Returns #{fail-ids} — the set of ids whose blob deletion failed.
+  Empty set = all succeeded."
+  (fn [cfg _ _] (::sto/type cfg)))
 
 (defmethod del-objects-in-bulk :default
-  [cfg _]
+  [cfg _ _]
   (ex/raise :type :internal
             :code :invalid-storage-backend
             :context cfg))
@@ -90,6 +92,17 @@
             :code :invalid-storage-backend
             :context cfg))
 
+(defmulti target-resolvable?
+  "Returns true when the backend can resolve `target` to a real destination.
+  GC callers must refuse deletion (and keep the row) when this is false."
+  (fn [cfg _] (::sto/type cfg)))
+
+(defmethod target-resolvable? :default
+  [cfg _]
+  (ex/raise :type :internal
+            :code :invalid-storage-backend
+            :context cfg))
+
 ;; --- HELPERS
 
 (defn uuid->hex

backend/src/app/storage/pending_gc.clj

@@ -20,10 +20,12 @@
    [integrant.core :as ig]))
 
 (def ^:private sql:get-pending-sobjects
-  "SELECT id, backend
+  "SELECT id, backend,
+          coalesce(metadata->>'~:storage-target', 'default') as target
      FROM storage_object
     WHERE status = 'pending'
       AND created_at <= now() - interval '24 hours'
+      AND (deleted_at IS NULL OR deleted_at <= now())
     ORDER BY created_at ASC
     LIMIT ?
       FOR UPDATE
@@ -36,30 +38,76 @@
 (def ^:private sql:delete-pending-sobject
   "DELETE FROM storage_object WHERE id = ? AND status = 'pending'")
 
+(defn- log-refusal!
+  "Indirection over the error log so tests can capture the refusal payload."
+  [backend-id target ids]
+  (l/err :hint "storage target is not configured, deletion refused"
+         :backend (name backend-id)
+         :target target
+         :ids (mapv str ids)))
+
+(def ^:private sql:park-unresolvable
+  "UPDATE storage_object
+      SET deleted_at = NOW() + INTERVAL '1 day'
+    WHERE id = ANY(?::uuid[])
+      AND status = 'pending'")
+
+(defn- park-unresolvable!
+  "Refuses to delete rows whose target id is not configured: logs the
+  misconfiguration and pushes `deleted_at` forward so the rows are excluded
+  from the next selection without being deleted."
+  [conn backend-id target ids]
+  (log-refusal! backend-id target ids)
+  (let [ids (db/create-array conn "uuid" ids)]
+    (db/exec-one! conn [sql:park-unresolvable ids])))
+
 (def ^:private chunk-size
   100)
 
+(defn- group-by-route
+  [rows]
+  (group-by (fn [{:keys [backend target]}]
+              [(keyword backend) target])
+            rows))
+
 (defn- delete-pending-rows!
   "Select, lock and delete a chunk of pending rows in a single transaction.
-  Returns the deleted rows or nil when there is nothing left to reclaim."
+  Rows whose target id is not configured are never deleted: they are parked
+  (error logged, `deleted_at` pushed forward) so the loop terminates and the
+  rows survive.
+
+  Returns a map `{:deleted rows :parked count}`, or nil when there is nothing
+  left to reclaim."
   [cfg]
   (db/tx-run! cfg
-              (fn [{:keys [::db/conn]}]
+              (fn [{:keys [::db/conn ::sto/storage]}]
                 ;; NOTE: db/exec! returns an empty vector when there are no
                 ;; rows left; use not-empty to detect it.
                 (when-let [chunk (not-empty (get-pending-chunk conn chunk-size))]
-                  (doseq [{:keys [id]} chunk]
-                    (db/exec-one! conn [sql:delete-pending-sobject id]))
-                  chunk))))
+                  (reduce-kv
+                   (fn [acc [backend-id target] rows]
+                     (if (sto/target-resolvable? storage backend-id target)
+                       (do
+                         (doseq [{:keys [id]} rows]
+                           (db/exec-one! conn [sql:delete-pending-sobject id]))
+                         (update acc :deleted into rows))
+                       (do
+                         (park-unresolvable! conn backend-id target (mapv :id rows))
+                         (update acc :parked + (count rows)))))
+                   {:deleted [] :parked 0}
+                   (group-by-route chunk))))))
 
 (defn- delete-blobs!
   "Best-effort removal of the orphaned blobs. Runs after the pending rows
-  have been committed so a failure here never blocks their reclamation."
+  have been committed so a failure here never blocks their reclamation.
+
+  The `:storage-target` metadata is load-bearing: the S3 backend reads it
+  (`s3/target-id`) to resolve the bucket/prefix/client for `del-object`."
   [storage rows]
-  (doseq [{:keys [id backend]} rows]
+  (doseq [{:keys [id backend target]} rows]
     (try
       (-> (impl/resolve-backend storage (keyword backend))
-          (impl/del-object {:id id}))
+          (impl/del-object (with-meta {:id id} {:storage-target target})))
       (catch Throwable cause
         (l/err :hint "error deleting orphaned pending blob"
                :id (str id)
@@ -68,12 +116,16 @@
 
 (defn- process!
   [{::sto/keys [storage] :as cfg}]
-  (loop [total 0]
-    (if-let [rows (delete-pending-rows! cfg)]
-      (do
-        (delete-blobs! storage rows)
-        (recur (long (+ total (count rows)))))
-      total)))
+  (loop [total-deleted 0
+         total-parked  0]
+    (if-let [result (delete-pending-rows! cfg)]
+      (let [removed (:deleted result)
+            parked  (:parked result)]
+        (delete-blobs! storage removed)
+        (recur (long (+ total-deleted (count removed)))
+               (long (+ total-parked parked))))
+      {:processed total-deleted
+       :parked    total-parked})))
 
 (defmethod ig/assert-key ::handler
   [_ params]
@@ -83,6 +135,7 @@
 (defmethod ig/init-key ::handler
   [_ cfg]
   (fn [_]
-    (let [total (process! cfg)]
-      (l/inf :hint "task finished" :total total)
-      {:processed total})))
+    (let [{:keys [processed parked]} (process! cfg)]
+      (l/inf :hint "task finished" :total processed :parked parked)
+      {:processed processed
+       :parked    parked})))

backend/src/app/storage/s3.clj

@@ -88,13 +88,21 @@
 
 ;; --- BACKEND INIT
 
+(def schema:target
+  [:map {:title "s3-target"}
+   [:bucket ::sm/text]
+   [:region {:optional true} :keyword]
+   [:endpoint {:optional true} ::sm/uri]
+   [:prefix {:optional true} ::sm/text]])
+
 (def ^:private schema:config
   [:map {:title "s3-backend-config"}
    ::wrk/netty-io-executor
    [::region {:optional true} :keyword]
    [::bucket {:optional true} ::sm/text]
    [::prefix {:optional true} ::sm/text]
-   [::endpoint {:optional true} ::sm/uri]])
+   [::endpoint {:optional true} ::sm/uri]
+   [::targets {:optional true} [:map-of :keyword schema:target]]])
 
 (defmethod ig/expand-key ::backend
   [k v]
@@ -104,42 +112,132 @@
   [_ params]
   (assert (sm/check schema:config params)))
 
+(defn- build-client-pair
+  [{:keys [::wrk/netty-io-executor]} region endpoint]
+  (let [params    {::region region
+                   ::endpoint endpoint
+                   ::wrk/netty-io-executor netty-io-executor}
+        client    (build-s3-client params)
+        presigner (build-s3-presigner params)]
+    {:client    @client
+     :presigner presigner
+     :close-fn  #(.close ^java.lang.AutoCloseable client)}))
+
+(defn- build-client-pair-or-cleanup
+  "Builds a client pair, closing the pairs already built when the build
+  fails so a failed backend init does not leak clients."
+  [acc params region endpoint]
+  (try
+    (build-client-pair params region endpoint)
+    (catch Throwable cause
+      (doseq [f (:close-fns acc)]
+        (ex/ignoring (f)))
+      (throw cause))))
+
+(defn- build-targets
+  "Resolves the implicit `:default` target plus the declared targets, sharing
+  one S3 client/presigner pair per distinct `[region endpoint]`."
+  [{:keys [::region ::endpoint ::bucket ::prefix ::targets] :as params}]
+  (let [defs   (merge {:default {:region region :endpoint endpoint
+                                 :bucket bucket :prefix prefix}}
+                      (into {}
+                            (map (fn [[id target]]
+                                   [id {:region   (or (:region target) region)
+                                        :endpoint (or (:endpoint target) endpoint)
+                                        :bucket   (:bucket target)
+                                        :prefix   (or (:prefix target) prefix)}]))
+                            targets))
+        result (reduce-kv
+                (fn [acc id {:keys [region endpoint bucket prefix]}]
+                  (let [k    [region endpoint]
+                        pair (or (get-in acc [:pairs k])
+                                 (build-client-pair-or-cleanup acc params region endpoint))
+                        acc  (cond-> acc
+                               (nil? (get-in acc [:pairs k]))
+                               (-> (assoc-in [:pairs k] pair)
+                                   (update :close-fns conj (:close-fn pair))))]
+                    (assoc-in acc [:targets id]
+                              {::client    (:client pair)
+                               ::presigner (:presigner pair)
+                               ::bucket    bucket
+                               ::prefix    prefix})))
+                {:pairs {} :targets {} :close-fns []}
+                defs)]
+    (select-keys result [:targets :close-fns])))
+
 (defmethod ig/init-key ::backend
   [_ params]
   (when (and (contains? params ::region)
              (contains? params ::bucket))
-    (let [client    (build-s3-client params)
-          presigner (build-s3-presigner params)]
+    (let [{:keys [targets close-fns]} (build-targets params)]
       (assoc params
              ::sto/type :s3
              ::counter (AtomicLong. 0)
-             ::client @client
-             ::presigner presigner
-             ::close-fn #(.close ^java.lang.AutoCloseable client)))))
+             ::default-target :default
+             ::targets targets
+             ::close-fns (vec close-fns)))))
 
 (defmethod ig/resolve-key ::backend
   [_ params]
-  (dissoc params ::close-fn))
+  (dissoc params ::close-fns))
 
 (defmethod ig/halt-key! ::backend
-  [_ {:keys [::close-fn]}]
-  (when (fn? close-fn)
-    (close-fn)))
+  [_ {:keys [::close-fns]}]
+  (doseq [f close-fns]
+    (when (fn? f)
+      (f))))
 
 (def ^:private schema:backend
   [:map {:title "s3-backend"}
-   ;; [::region :keyword]
-   ;; [::bucket ::sm/text]
-   [::client [:fn #(instance? S3AsyncClient %)]]
-   [::presigner [:fn #(instance? S3Presigner %)]]
-   [::prefix {:optional true} ::sm/text]
-   #_[::sto/type [:= :s3]]])
+   [::default-target :keyword]
+   [::targets
+    [:map-of :keyword
+     [:map
+      [::client [:fn #(instance? S3AsyncClient %)]]
+      [::presigner [:fn #(instance? S3Presigner %)]]
+      [::bucket ::sm/text]
+      [::prefix {:optional true} ::sm/text]]]]])
 
 (sm/register! ::backend schema:backend)
 
 (def ^:private valid-backend?
   (sm/validator schema:backend))
 
+;; --- TARGET RESOLUTION
+
+(defn- target-id
+  "Returns the configured target id stored on the object, or the default
+  target name when the object predates the routing feature."
+  [backend object]
+  (or (:storage-target object)
+      (some-> (meta object) :storage-target)
+      (name (::default-target backend))))
+
+(defn- resolve-target-by-id
+  [backend target-id]
+  (let [tid (cond
+              (nil? target-id)     (::default-target backend)
+              (keyword? target-id) target-id
+              :else                (keyword target-id))]
+    (or (get (::targets backend) tid)
+        (ex/raise :type :internal
+                  :code :invalid-storage-target
+                  :hint "storage target is not configured"
+                  :target target-id
+                  :available (vec (keys (::targets backend)))))))
+
+(defn- resolve-target
+  [backend object]
+  (resolve-target-by-id backend (target-id backend object)))
+
+(defmethod impl/target-resolvable? :s3
+  [backend target-id]
+  (let [tid (cond
+              (nil? target-id)     (::default-target backend)
+              (keyword? target-id) target-id
+              :else                (keyword target-id))]
+    (contains? (::targets backend) tid)))
+
 ;; --- API IMPL
 
 (defmethod impl/put-object :s3
@@ -210,13 +308,14 @@
       true)))
 
 (defmethod impl/del-objects-in-bulk :s3
-  [backend ids]
+  [backend target ids]
   (assert (valid-backend? backend) "expected a valid backend instance")
-  (let [key->id (into {} (map (fn [id]
-                                [(str (::prefix backend) (impl/id->path id)) id]))
+  (let [target  (resolve-target-by-id backend target)
+        key->id (into {} (map (fn [id]
+                                [(str (::prefix target) (impl/id->path id)) id]))
                       ids)
         result  (try
-                  (p/await! (del-object-in-bulk backend ids))
+                  (p/await! (del-object-in-bulk target ids))
                   (catch Throwable cause
                     (l/err :hint "error on s3 bulk deletion"
                            :ids ids
@@ -320,8 +419,9 @@
                       ^Subscriber subscriber))))))
 
 (defn- put-object
-  [{:keys [::client ::bucket ::prefix ::counter]} {:keys [id] :as object} content]
-  (let [path    (dm/str prefix (impl/id->path id))
+  [{:keys [::counter] :as backend} {:keys [id] :as object} content]
+  (let [{:keys [::client ::bucket ::prefix]} (resolve-target backend object)
+        path    (dm/str prefix (impl/id->path id))
         mdata   (meta object)
         mtype   (:content-type mdata "application/octet-stream")
         rbody   (make-request-body counter content)
@@ -344,8 +444,9 @@
       (proxy-super close))))
 
 (defn- get-object-data
-  [{:keys [::client ::bucket ::prefix]} {:keys [id size]}]
-  (let [gor (.. (GetObjectRequest/builder)
+  [backend {:keys [id size] :as object}]
+  (let [{:keys [::client ::bucket ::prefix]} (resolve-target backend object)
+        gor (.. (GetObjectRequest/builder)
                 (bucket bucket)
                 (key (str prefix (impl/id->path id)))
                 (build))]
@@ -369,16 +470,18 @@
              (p/fmap #(.asInputStream ^ResponseBytes %)))))))
 
 (defn- head-object
-  [{:keys [::client ::bucket ::prefix]} {:keys [id]}]
-  (let [hor (.. (HeadObjectRequest/builder)
+  [backend {:keys [id] :as object}]
+  (let [{:keys [::client ::bucket ::prefix]} (resolve-target backend object)
+        hor (.. (HeadObjectRequest/builder)
                 (bucket bucket)
                 (key (str prefix (impl/id->path id)))
                 (build))]
     (.headObject ^S3AsyncClient client ^HeadObjectRequest hor)))
 
 (defn- get-object-bytes
-  [{:keys [::client ::bucket ::prefix]} {:keys [id]}]
-  (let [gor (.. (GetObjectRequest/builder)
+  [backend {:keys [id] :as object}]
+  (let [{:keys [::client ::bucket ::prefix]} (resolve-target backend object)
+        gor (.. (GetObjectRequest/builder)
                 (bucket bucket)
                 (key (str prefix (impl/id->path id)))
                 (build))
@@ -392,7 +495,7 @@
   (ct/duration {:minutes 10}))
 
 (defn- get-object-url
-  [{:keys [::presigner ::bucket ::prefix]} {:keys [id]}
+  [backend {:keys [id] :as object}
    {:keys [max-age content-disposition] :or {max-age default-max-age}}]
   (assert (ct/duration? max-age) "expected valid duration instance")
 
@@ -400,7 +503,8 @@
   ;; object store sets that header on the response the client fetches after
   ;; following the redirect. It is only set when asked for, so urls for
   ;; objects served inline stay byte identical to before.
-  (let [gorb (.. (GetObjectRequest/builder)
+  (let [{:keys [::presigner ::bucket ::prefix]} (resolve-target backend object)
+        gorb (.. (GetObjectRequest/builder)
                  (bucket bucket)
                  (key (dm/str prefix (impl/id->path id))))
         gorb (cond-> gorb
@@ -415,8 +519,9 @@
     (u/uri (str (.url ^PresignedGetObjectRequest pgor)))))
 
 (defn- del-object
-  [{:keys [::bucket ::client ::prefix]} {:keys [id] :as obj}]
-  (let [dor (.. (DeleteObjectRequest/builder)
+  [backend {:keys [id] :as object}]
+  (let [{:keys [::bucket ::client ::prefix]} (resolve-target backend object)
+        dor (.. (DeleteObjectRequest/builder)
                 (bucket bucket)
                 (key (dm/str prefix (impl/id->path id)))
                 (build))]

backend/test/backend_tests/storage_config_test.clj

@@ -0,0 +1,97 @@
+;; This Source Code Form is subject to the terms of the Mozilla Public
+;; License, v. 2.0. If a copy of the MPL was not distributed with this
+;; file, You can obtain one at http://mozilla.org/MPL/2.0/.
+;;
+;; Copyright (c) KALEIDOS SUBSIDIARY SL
+
+(ns backend-tests.storage-config-test
+  (:require
+   [app.common.uri :as u]
+   [app.config :as cf]
+   [app.storage.config :as sto.config]
+   [clojure.java.io :as io]
+   [clojure.test :as t]))
+
+(defn- write-routes!
+  [content]
+  (let [file (java.io.File/createTempFile "storage-routes" ".edn")]
+    (spit file content)
+    file))
+
+(defn- delete!
+  [file]
+  (io/delete-file file true))
+
+(defn- load-with
+  [config]
+  (binding [cf/config (merge cf/config
+                             {:objects-storage-backend :s3}
+                             config)]
+    (sto.config/load)))
+
+(defn- load-error
+  "Runs `load` against a routes file with `content` and returns the raised
+  throwable, or nil when it succeeds."
+  [content config]
+  (let [file (write-routes! content)]
+    (try
+      (try
+        (load-with (assoc config :objects-storage-s3-routes-file (.getAbsolutePath file)))
+        nil
+        (catch Throwable cause cause))
+      (finally
+        (delete! file)))))
+
+(t/deftest returns-empty-when-unset
+  (t/is (= {:targets nil :routes nil}
+           (load-with {:objects-storage-s3-routes-file nil}))))
+
+(t/deftest loads-and-normalizes-routes
+  (let [file (write-routes!
+              "{:targets {:temp {:bucket \"penpot-temp\" :endpoint \"https://s3.example.com\"}} :routes {\"tempfile\" :temp}}")]
+    (try
+      (let [result (load-with {:objects-storage-s3-routes-file (.getAbsolutePath file)})]
+        (t/is (= #{:temp} (set (keys (:targets result)))))
+        (t/is (= "penpot-temp" (get-in result [:targets :temp :bucket])))
+        (t/is (u/uri? (get-in result [:targets :temp :endpoint])))
+        (t/is (= {"tempfile" :temp} (:routes result))))
+      (finally
+        (delete! file)))))
+
+(t/deftest rejects-reserved-default-target
+  (t/is (some? (load-error "{:targets {:default {:bucket \"x\"}}}"
+                           {}))))
+
+(t/deftest rejects-unknown-route-target
+  (t/is (some? (load-error
+                "{:targets {:temp {:bucket \"x\"}} :routes {\"tempfile\" :missing}}"
+                {}))))
+
+(t/deftest rejects-invalid-semantic-bucket
+  (t/is (some? (load-error
+                "{:targets {:temp {:bucket \"x\"}} :routes {\"not-a-bucket\" :temp}}"
+                {}))))
+
+(t/deftest rejects-unreadable-file
+  (t/is (some? (load-error "{:targets" {}))))
+
+(t/deftest rejects-routing-when-backend-is-not-s3
+  (t/is (some? (load-error
+                "{:targets {:temp {:bucket \"x\"}} :routes {\"tempfile\" :temp}}"
+                {:objects-storage-backend :fs}))))
+
+(t/deftest rejects-route-pointing-to-default
+  (let [err (load-error
+             "{:targets {:temp {:bucket \"x\"}} :routes {\"tempfile\" :default}}"
+             {})]
+    (t/is (some? err))
+    (t/is (= :unknown-storage-target (:code (ex-data err))))))
+
+(t/deftest loads-empty-targets-map
+  (let [file (write-routes! "{:targets {}}")]
+    (try
+      (let [result (load-with {:objects-storage-s3-routes-file (.getAbsolutePath file)})]
+        (t/is (= {} (:targets result)))
+        (t/is (nil? (:routes result))))
+      (finally
+        (delete! file)))))

backend/test/backend_tests/storage_test.clj

@@ -13,13 +13,16 @@
    [app.rpc :as-alias rpc]
    [app.storage :as sto]
    [app.storage.fs :as-alias sto.fs]
+   [app.storage.gc-deleted :as sto.gc-deleted]
    [app.storage.impl :as impl]
+   [app.storage.pending-gc :as sto.pending-gc]
    [app.storage.s3 :as-alias sto.s3]
    [backend-tests.helpers :as th]
    [clojure.test :as t]
    [cuerdas.core :as str]
    [datoteka.fs :as fs]
    [datoteka.io :as io]
+   [integrant.core :as ig]
    [mockery.core :refer [with-mocks]]
    [promesa.core :as p])
   (:import
@@ -41,6 +44,17 @@
   [storage]
   (assoc storage ::sto/backend :fs))
 
+(declare fake-s3-backend-with-targets)
+
+(defn configure-s3-storage
+  "Returns a storage configured with the fake S3 backend and the given
+  semantic-bucket -> target routing."
+  [bucket->target]
+  (-> (:app.storage/storage th/*system*)
+      (assoc ::sto/backend :s3)
+      (assoc-in [::sto/backends :s3] (fake-s3-backend-with-targets))
+      (assoc ::sto/bucket->target bucket->target)))
+
 (t/deftest put-and-retrieve-object
   (let [storage (-> (:app.storage/storage th/*system*)
                     (configure-storage-backend))
@@ -708,7 +722,7 @@
                    {:id (:id object)})
 
     (with-mocks [_mock {:target 'app.storage.impl/del-objects-in-bulk
-                        :return (fn [_ ids] (set ids))}]
+                        :return (fn [_ _ ids] (set ids))}]
       (let [res (th/run-task! :storage-gc-deleted {})]
         (t/is (= 0 (:deleted res)))))
 
@@ -832,9 +846,146 @@
 
 (defn- fake-s3-backend
   []
-  {::sto/type         :s3
-   ::sto.s3/client    (reify S3AsyncClient)
-   ::sto.s3/presigner (reify S3Presigner)})
+  {::sto/type              :s3
+   ::sto.s3/default-target :default
+   ::sto.s3/targets
+   {:default {::sto.s3/client    (reify S3AsyncClient)
+              ::sto.s3/presigner (reify S3Presigner)
+              ::sto.s3/bucket    "test-bucket"
+              ::sto.s3/prefix    nil}}})
+
+(defn- fake-s3-target
+  [bucket prefix]
+  {::sto.s3/client    (reify S3AsyncClient)
+   ::sto.s3/presigner (reify S3Presigner)
+   ::sto.s3/bucket    bucket
+   ::sto.s3/prefix    prefix})
+
+(defn- fake-s3-backend-with-targets
+  []
+  (assoc (fake-s3-backend)
+         ::sto.s3/targets
+         {:default (fake-s3-target "default" nil)
+          :temp    (fake-s3-target "temp" "tmp/")}))
+
+(t/deftest s3-bulk-delete-resolves-target
+  (let [backend  (fake-s3-backend-with-targets)
+        captured (atom nil)]
+    (with-mocks [_mock {:target 'app.storage.s3/del-object-in-bulk
+                        :return (fn [target _ids]
+                                  (reset! captured target)
+                                  (p/resolved nil))}]
+      (t/is (= #{} (impl/del-objects-in-bulk backend :temp #{(uuid/next)})))
+      (t/is (= "temp" (::sto.s3/bucket @captured)))
+      (t/is (= "tmp/" (::sto.s3/prefix @captured))))))
+
+(t/deftest s3-bulk-delete-rejects-unknown-target
+  (let [backend  (fake-s3-backend-with-targets)
+        captured (atom nil)]
+    (with-mocks [_mock {:target 'app.storage.s3/del-object-in-bulk
+                        :return (fn [target _ids]
+                                  (reset! captured target)
+                                  (p/resolved nil))}]
+      (let [ex (try
+                 (impl/del-objects-in-bulk backend :missing #{(uuid/next)})
+                 nil
+                 (catch Throwable cause cause))]
+        (t/is (some? ex))
+        (t/is (= :invalid-storage-target (:code (ex-data ex))))
+        (t/is (nil? @captured))))))
+
+(t/deftest s3-get-object-url-rejects-unknown-target
+  (let [backend (fake-s3-backend-with-targets)
+        ex      (try
+                  (impl/get-object-url backend {:id (uuid/next)
+                                                :storage-target "ghost"} {})
+                  nil
+                  (catch Throwable cause cause))]
+    (t/is (some? ex))
+    (t/is (= :invalid-storage-target (:code (ex-data ex))))))
+
+(t/deftest s3-get-object-data-rejects-unknown-target
+  (let [backend (fake-s3-backend-with-targets)
+        ex      (try
+                  (impl/get-object-data backend {:id (uuid/next)
+                                                 :size 1
+                                                 :storage-target "ghost"})
+                  nil
+                  (catch Throwable cause cause))]
+    (t/is (some? ex))
+    (t/is (= :invalid-storage-target (:code (ex-data ex))))))
+
+(t/deftest s3-target-resolvable-checks-configured-targets
+  (let [backend (fake-s3-backend-with-targets)]
+    (t/is (true? (impl/target-resolvable? backend nil)))
+    (t/is (true? (impl/target-resolvable? backend "default")))
+    (t/is (true? (impl/target-resolvable? backend :temp)))
+    (t/is (false? (impl/target-resolvable? backend "ghost")))))
+
+(t/deftest fs-target-is-always-resolvable
+  (t/is (true? (impl/target-resolvable? {::sto/type :fs} "anything"))))
+
+(t/deftest s3-build-targets-shares-clients-and-closes-them
+  (let [clients     (atom [])
+        close-count (atom 0)
+        mk-client   (fn [_]
+                      (let [c (reify
+                                clojure.lang.IDeref
+                                (deref [_] (Object.))
+                                java.lang.AutoCloseable
+                                (close [_] (swap! close-count inc)))]
+                        (swap! clients conj c)
+                        c))]
+    (with-mocks [_c {:target 'app.storage.s3/build-s3-client
+                     :return mk-client}
+                 _p {:target 'app.storage.s3/build-s3-presigner
+                     :return (fn [_] (reify S3Presigner))}]
+      (let [backend (ig/init-key :app.storage.s3/backend
+                                 {:app.storage.s3/region :eu-central-1
+                                  :app.storage.s3/bucket "main"
+                                  :app.worker/netty-io-executor :executor
+                                  :app.storage.s3/targets
+                                  {:same  {:bucket "same"}
+                                   :other {:bucket "other" :region :us-east-1}}})]
+        ;; one client pair per distinct [region endpoint]
+        (t/is (= 2 (count @clients)))
+        ;; same region reuses the default client
+        (t/is (= (get-in backend [:app.storage.s3/targets :default :app.storage.s3/client])
+                 (get-in backend [:app.storage.s3/targets :same :app.storage.s3/client])))
+        ;; different region gets its own client
+        (t/is (not= (get-in backend [:app.storage.s3/targets :default :app.storage.s3/client])
+                    (get-in backend [:app.storage.s3/targets :other :app.storage.s3/client])))
+
+        (ig/halt-key! :app.storage.s3/backend backend)
+        (t/is (= 2 @close-count))))))
+
+(t/deftest s3-build-targets-closes-clients-on-failure
+  (let [close-count (atom 0)
+        mk-client   (fn [params]
+                      (when (= :us-east-1 (:app.storage.s3/region params))
+                        (throw (RuntimeException. "boom")))
+                      (reify
+                        clojure.lang.IDeref
+                        (deref [_] (Object.))
+                        java.lang.AutoCloseable
+                        (close [_] (swap! close-count inc))))]
+    (with-mocks [_c {:target 'app.storage.s3/build-s3-client
+                     :return mk-client}
+                 _p {:target 'app.storage.s3/build-s3-presigner
+                     :return (fn [_] (reify S3Presigner))}]
+      (let [ex (try
+                 (ig/init-key :app.storage.s3/backend
+                              {:app.storage.s3/region :eu-central-1
+                               :app.storage.s3/bucket "main"
+                               :app.worker/netty-io-executor :executor
+                               :app.storage.s3/targets
+                               {:same  {:bucket "same"}
+                                :other {:bucket "other" :region :us-east-1}}})
+                 nil
+                 (catch Throwable cause cause))]
+        (t/is (some? ex))
+        ;; the pair built before the failing one is closed
+        (t/is (= 1 @close-count))))))
 
 (t/deftest s3-exists-object-returns-true-on-found
   (with-mocks [mock {:target 'app.storage.s3/head-object
@@ -874,3 +1025,322 @@
       (t/is (= "boom" (ex-message (ex-cause ex)))))
     ;; one initial attempt plus max-retries
     (t/is (= 4 (:call-count @mock)))))
+
+;; --- Storage target metadata / dedup
+
+(t/deftest put-object-routes-bucket-to-storage-target
+  (let [storage (configure-s3-storage {"tempfile" :temp})]
+    (with-mocks [_mock {:target 'app.storage.impl/put-object
+                        :return (fn [_ object _] object)}]
+      (let [object (sto/put-object! storage {::sto/content (sto/content "content")
+                                             :bucket "tempfile"
+                                             :content-type "text/plain"})
+            row    (th/db-exec-one!
+                    ["select backend, metadata->>'~:storage-target' as target
+                        from storage_object where id = ?" (:id object)])]
+        (t/is (= "s3" (:backend row)))
+        (t/is (= "temp" (:target row)))))))
+
+(t/deftest put-object-falls-back-to-default-storage-target
+  (let [storage (configure-s3-storage {"tempfile" :temp})]
+    (with-mocks [_mock {:target 'app.storage.impl/put-object
+                        :return (fn [_ object _] object)}]
+      (let [object (sto/put-object! storage {::sto/content (sto/content "content")
+                                             :bucket "file-media-object"
+                                             :content-type "text/plain"})
+            row    (th/db-exec-one!
+                    ["select metadata->>'~:storage-target' as target
+                        from storage_object where id = ?" (:id object)])]
+        (t/is (= "default" (:target row)))))))
+
+(t/deftest put-object-fs-does-not-set-storage-target
+  (let [storage (configure-storage-backend (:app.storage/storage th/*system*))
+        object  (sto/put-object! storage {::sto/content (sto/content "content")
+                                          :bucket "file-media-object"
+                                          :content-type "text/plain"})
+        row     (th/db-exec-one!
+                 ["select metadata->>'~:storage-target' as target
+                     from storage_object where id = ?" (:id object)])]
+    (t/is (nil? (:target row)))))
+
+(t/deftest dedup-is-isolated-per-storage-target
+  (let [routed   (configure-s3-storage {"file-data" :temp})
+        unrouted (configure-s3-storage {})
+        content  (-> (sto/content "content")
+                     (sto/wrap-with-hash "same-hash"))]
+    (with-mocks [_mock {:target 'app.storage.impl/put-object
+                        :return (fn [_ object _] object)}]
+      (let [object1 (sto/put-object! routed {::sto/content content
+                                             ::sto/deduplicate? true
+                                             :bucket "file-data"
+                                             :content-type "text/plain"})
+            object2 (sto/put-object! unrouted {::sto/content content
+                                               ::sto/deduplicate? true
+                                               :bucket "file-data"
+                                               :content-type "text/plain"})
+            row     (th/db-exec-one! ["select count(*) from storage_object"])]
+        ;; same semantic bucket, different target: no dedup hit, two rows
+        (t/is (not= (:id object1) (:id object2)))
+        (t/is (= 2 (:count row)))))))
+
+(t/deftest dedup-reuses-object-within-same-storage-target
+  (let [storage (configure-s3-storage {"file-data" :temp})
+        content (-> (sto/content "content")
+                    (sto/wrap-with-hash "same-hash"))]
+    (with-mocks [_p {:target 'app.storage.impl/put-object
+                     :return (fn [_ object _] object)}
+                 _e {:target 'app.storage.impl/exists-object?
+                     :return (fn [_ _] true)}]
+      (let [object1 (sto/put-object! storage {::sto/content content
+                                              ::sto/deduplicate? true
+                                              :bucket "file-data"
+                                              :content-type "text/plain"})
+            object2 (sto/put-object! storage {::sto/content content
+                                              ::sto/deduplicate? true
+                                              :bucket "file-data"
+                                              :content-type "text/plain"})]
+        ;; same semantic bucket and target: dedup reuses the object
+        (t/is (= (:id object1) (:id object2)))))))
+
+(t/deftest dedup-hit-carries-storage-target-metadata
+  (let [storage  (configure-s3-storage {"file-data" :temp})
+        content  (-> (sto/content "content")
+                     (sto/wrap-with-hash "same-hash"))
+        captured (atom nil)]
+    (with-mocks [_p {:target 'app.storage.impl/put-object
+                     :return (fn [_ object _] object)}
+                 _e {:target 'app.storage.impl/exists-object?
+                     :return (fn [_ object]
+                               (reset! captured object)
+                               true)}]
+      (sto/put-object! storage {::sto/content content
+                                ::sto/deduplicate? true
+                                :bucket "file-data"
+                                :content-type "text/plain"})
+      (sto/put-object! storage {::sto/content content
+                                ::sto/deduplicate? true
+                                :bucket "file-data"
+                                :content-type "text/plain"})
+      (t/is (some? @captured))
+      (t/is (= "temp" (:storage-target (meta @captured)))))))
+
+(t/deftest dedup-repair-carries-storage-target-metadata
+  (let [storage  (configure-s3-storage {"file-data" :temp})
+        content  (-> (sto/content "content")
+                     (sto/wrap-with-hash "same-hash"))
+        calls    (atom [])]
+    (with-mocks [_p {:target 'app.storage.impl/put-object
+                     :return (fn [_ object _]
+                               (swap! calls conj object)
+                               object)}
+                 _h {:target 'app.storage.s3/head-object
+                     :return (p/rejected (-> (NoSuchKeyException/builder)
+                                             (.message "no key")
+                                             (.build)))}]
+      (let [object1 (sto/put-object! storage {::sto/content content
+                                              ::sto/deduplicate? true
+                                              :bucket "file-data"
+                                              :content-type "text/plain"})
+            ;; second put finds the row but the real exists-object? sees a
+            ;; missing blob and repairs it in place
+            object2 (sto/put-object! storage {::sto/content content
+                                              ::sto/deduplicate? true
+                                              :bucket "file-data"
+                                              :content-type "text/plain"})
+            row     (th/db-exec-one!
+                     ["select status from storage_object where id = ?" (:id object1)])
+            count   (th/db-exec-one! ["select count(*) from storage_object"])]
+        (t/is (= (:id object1) (:id object2)))
+        (t/is (= "valid" (:status row)))
+        (t/is (= 1 (:count count)))
+        (t/is (= "temp" (:storage-target (meta (last @calls)))))))))
+
+;; --- GC target routing
+
+(defn- storage-with-s3-targets
+  []
+  (assoc (:app.storage/storage th/*system*)
+         ::sto/backends
+         {:s3 (fake-s3-backend-with-targets)}))
+
+(t/deftest gc-deleted-deletes-from-routed-target
+  (let [storage  (storage-with-s3-targets)
+        cfg      {::db/pool th/*pool* ::sto/storage storage}
+        id       (uuid/next)
+        captured (atom nil)]
+    (th/db-exec! ["insert into storage_object (id, size, backend, metadata, deleted_at, status)
+                   values (?, 1, 's3', ?, ?, 'valid')"
+                  id
+                  (db/tjson {:bucket "file-data" :storage-target "temp"})
+                  (ct/in-past {:minutes 1})])
+    (with-mocks [_mock {:target 'app.storage.impl/del-objects-in-bulk
+                        :return (fn [_ target _ids]
+                                  (reset! captured target)
+                                  #{})}]
+      (t/is (= 1 (:deleted (#'sto.gc-deleted/clean-deleted! cfg))))
+      (t/is (= "temp" @captured)))))
+
+(t/deftest gc-deleted-refuses-unknown-target-and-keeps-row
+  (let [storage  (storage-with-s3-targets)
+        cfg      {::db/pool th/*pool* ::sto/storage storage}
+        id       (uuid/next)
+        logged   (atom nil)]
+    (th/db-exec! ["insert into storage_object (id, size, backend, metadata, deleted_at, status)
+                   values (?, 1, 's3', ?, ?, 'valid')"
+                  id
+                  (db/tjson {:bucket "file-data" :storage-target "ghost"})
+                  (ct/in-past {:minutes 1})])
+    (with-mocks [mock {:target 'app.storage.impl/del-objects-in-bulk
+                       :return (fn [_ _ _] #{})}]
+      (with-redefs [app.storage.gc-deleted/log-refusal!
+                    (fn [backend-id target ids]
+                      (reset! logged [backend-id target (vec ids)]))]
+        (let [result (#'sto.gc-deleted/clean-deleted! cfg)
+              row    (th/db-exec-one!
+                      ["select status, deleted_at, deletion_attempts
+                          from storage_object where id = ?" id])]
+          (t/is (= 0 (:deleted result)))
+          (t/is (= 1 (:parked result)))
+          (t/is (= 0 (:call-count @mock)))
+          (t/is (= "valid" (:status row)))
+          (t/is (ct/is-after? (:deleted-at row) (ct/now)))
+          (t/is (= 0 (:deletion-attempts row)))
+          (t/is (= [:s3 "ghost" [id]] @logged)))))))
+
+(t/deftest gc-deleted-give-up-not-applied-to-unknown-target
+  (let [storage (storage-with-s3-targets)
+        cfg     {::db/pool th/*pool* ::sto/storage storage}
+        id      (uuid/next)]
+    (th/db-exec! ["insert into storage_object (id, size, backend, metadata, deleted_at, status, deletion_attempts)
+                   values (?, 1, 's3', ?, ?, 'valid', 10)"
+                  id
+                  (db/tjson {:bucket "file-data" :storage-target "ghost"})
+                  (ct/in-past {:minutes 1})])
+    (with-mocks [_mock {:target 'app.storage.impl/del-objects-in-bulk
+                        :return (fn [_ _ _] #{})}]
+      (with-redefs [app.storage.gc-deleted/log-refusal! (fn [& _] nil)]
+        (#'sto.gc-deleted/clean-deleted! cfg)))
+    (let [row (th/db-exec-one! ["select count(*) from storage_object where id = ?" id])]
+      (t/is (= 1 (:count row))))))
+
+(t/deftest gc-deleted-normal-failure-defers-and-gives-up
+  (let [storage (storage-with-s3-targets)
+        cfg     {::db/pool th/*pool* ::sto/storage storage}
+        id      (uuid/next)]
+    (th/db-exec! ["insert into storage_object (id, size, backend, metadata, deleted_at, status)
+                   values (?, 1, 's3', ?, ?, 'valid')"
+                  id
+                  (db/tjson {:bucket "file-data" :storage-target "temp"})
+                  (ct/in-past {:minutes 1})])
+    (with-mocks [_mock {:target 'app.storage.impl/del-objects-in-bulk
+                        :return (fn [_ _ ids] (set ids))}]
+      (let [result (#'sto.gc-deleted/clean-deleted! cfg)
+            row    (th/db-exec-one!
+                    ["select deleted_at, deletion_attempts
+                        from storage_object where id = ?" id])]
+        (t/is (= 0 (:deleted result)))
+        (t/is (= 0 (:parked result)))
+        (t/is (ct/is-after? (:deleted-at row) (ct/now)))
+        (t/is (= 1 (:deletion-attempts row))))
+
+      ;; force the give-up threshold and let the next pass remove the row
+      (th/db-update! :storage-object
+                     {:deletion-attempts 7
+                      :deleted-at (ct/in-past {:minutes 1})}
+                     {:id id})
+      (let [result (#'sto.gc-deleted/clean-deleted! cfg)]
+        (t/is (= 0 (:deleted result)))
+        (let [row (th/db-exec-one! ["select count(*) from storage_object where id = ?" id])]
+          (t/is (= 0 (:count row))))))))
+
+(t/deftest pending-gc-deletes-resolvable-target
+  (let [storage  (storage-with-s3-targets)
+        cfg      {::db/pool th/*pool* ::sto/storage storage}
+        id       (uuid/next)
+        captured (atom nil)]
+    (th/db-exec! ["insert into storage_object (id, size, backend, metadata, created_at, status)
+                   values (?, 1, 's3', ?, ?, 'pending')"
+                  id
+                  (db/tjson {:storage-target "temp"})
+                  (ct/in-past {:days 2})])
+    (with-mocks [_mock {:target 'app.storage.impl/del-object
+                        :return (fn [_ object]
+                                  (reset! captured object)
+                                  nil)}]
+      (let [result (#'sto.pending-gc/process! cfg)]
+        (t/is (= 1 (:processed result)))
+        (t/is (= 0 (:parked result)))
+        (t/is (= "temp" (:storage-target (meta @captured))))
+        (let [row (th/db-exec-one! ["select count(*) from storage_object where id = ?" id])]
+          (t/is (= 0 (:count row))))))))
+
+(t/deftest pending-gc-refuses-unknown-target-and-keeps-row
+  (let [storage  (storage-with-s3-targets)
+        cfg      {::db/pool th/*pool* ::sto/storage storage}
+        id       (uuid/next)
+        logged   (atom nil)
+        captured (atom nil)]
+    (th/db-exec! ["insert into storage_object (id, size, backend, metadata, created_at, status)
+                   values (?, 1, 's3', ?, ?, 'pending')"
+                  id
+                  (db/tjson {:storage-target "ghost"})
+                  (ct/in-past {:days 2})])
+    (with-mocks [mock {:target 'app.storage.impl/del-object
+                       :return (fn [_ object]
+                                 (reset! captured object)
+                                 nil)}]
+      (with-redefs [app.storage.pending-gc/log-refusal!
+                    (fn [backend-id target ids]
+                      (reset! logged [backend-id target (vec ids)]))]
+        (let [result (#'sto.pending-gc/process! cfg)
+              row    (th/db-exec-one!
+                      ["select status, deleted_at from storage_object where id = ?" id])]
+          (t/is (= 0 (:processed result)))
+          (t/is (= 1 (:parked result)))
+          (t/is (= 0 (:call-count @mock)))
+          (t/is (nil? @captured))
+          (t/is (= "pending" (:status row)))
+          (t/is (ct/is-after? (:deleted-at row) (ct/now)))
+          (t/is (= [:s3 "ghost" [id]] @logged)))))))
+
+(t/deftest gc-deleted-legacy-rows-delete-from-default-target
+  (let [storage  (storage-with-s3-targets)
+        cfg      {::db/pool th/*pool* ::sto/storage storage}
+        id       (uuid/next)
+        captured (atom nil)]
+    (th/db-exec! ["insert into storage_object (id, size, backend, metadata, deleted_at, status)
+                   values (?, 1, 's3', ?, ?, 'valid')"
+                  id
+                  (db/tjson {:bucket "file-data"})
+                  (ct/in-past {:minutes 1})])
+    (with-mocks [_mock {:target 'app.storage.impl/del-objects-in-bulk
+                        :return (fn [_ target _ids]
+                                  (reset! captured target)
+                                  #{})}]
+      (let [result (#'sto.gc-deleted/clean-deleted! cfg)]
+        (t/is (= 1 (:deleted result)))
+        (t/is (= 0 (:parked result)))
+        (t/is (= "default" @captured))
+        (let [row (th/db-exec-one! ["select count(*) from storage_object where id = ?" id])]
+          (t/is (= 0 (:count row))))))))
+
+(t/deftest pending-gc-legacy-rows-delete-from-default-target
+  (let [storage  (storage-with-s3-targets)
+        cfg      {::db/pool th/*pool* ::sto/storage storage}
+        id       (uuid/next)
+        captured (atom nil)]
+    (th/db-exec! ["insert into storage_object (id, size, backend, metadata, created_at, status)
+                   values (?, 1, 's3', ?, ?, 'pending')"
+                  id
+                  (db/tjson {:bucket "file-data"})
+                  (ct/in-past {:days 2})])
+    (with-mocks [_mock {:target 'app.storage.impl/del-object
+                        :return (fn [_ object]
+                                  (reset! captured object)
+                                  nil)}]
+      (let [result (#'sto.pending-gc/process! cfg)]
+        (t/is (= 1 (:processed result)))
+        (t/is (= 0 (:parked result)))
+        (t/is (= "default" (:storage-target (meta @captured))))
+        (let [row (th/db-exec-one! ["select count(*) from storage_object where id = ?" id])]
+          (t/is (= 0 (:count row))))))))

docker/images/docker-compose.yaml

@@ -157,6 +157,11 @@ services:
       # PENPOT_OBJECTS_STORAGE_S3_ENDPOINT: <ENDPOINT>
       # PENPOT_OBJECTS_STORAGE_S3_BUCKET: <BUKET_NAME>
 
+      ## Optional: route specific internal semantic buckets (for example
+      ## tempfile) to additional S3 targets. The file is EDN; see
+      ## docs/technical-guide/configuration.md for the format.
+      # PENPOT_OBJECTS_STORAGE_S3_ROUTES_FILE: /opt/data/storage-routes.edn
+
       ## Telemetry. When enabled, a periodical process will send anonymous data about this
       ## instance. Telemetry data will enable us to learn how the application is used,
       ## based on real scenarios. If you want to help us, please leave it enabled. You can

docs/technical-guide/configuration.md

@@ -549,6 +549,51 @@ PENPOT_OBJECTS_STORAGE_S3_ENDPOINT: <endpoint-uri>
 These settings are equally useful if you have a Minio storage system.
 </p>
 
+#### S3 targets and per-bucket routing
+
+__Since version 2.19.0__
+
+By default the S3 backend stores every object in the configured bucket. You can
+route specific internal semantic buckets (for example `tempfile` or `file-data`)
+to additional S3 targets. Each target has its own bucket and, optionally, its
+own key prefix, region and endpoint.
+
+Targets and routing are declared in an external EDN file referenced by the
+`PENPOT_OBJECTS_STORAGE_S3_ROUTES_FILE` environment variable:
+
+```clojure
+{:targets
+ {:temp {:bucket "penpot-temp"}
+  :cold {:bucket "penpot-cold"
+         :region :us-east-1
+         :endpoint "https://s3.us-east-1.amazonaws.com"
+         :prefix "objects/"}}
+ :routes
+ {"tempfile" :temp
+  "file-data" :temp}}
+```
+
+- `:targets` declares the named destinations. `:bucket` is required; `:prefix`,
+  `:region` and `:endpoint` are optional and inherit the values from the default
+  target (`PENPOT_OBJECTS_STORAGE_S3_*`) when omitted.
+- `:routes` maps a Penpot semantic bucket to a target id. Semantic buckets that
+  are not listed use the default target (the configured
+  `PENPOT_OBJECTS_STORAGE_S3_BUCKET`).
+- The `:default` target id is reserved and implicit; do not declare it.
+- The file is optional and only applies to the `s3` backend. Without it, all
+  objects use the default bucket and the behavior is unchanged.
+
+The target id is stored in the object metadata, so it must stay stable. If you
+remove or rename a target id, objects already written with it cannot be located:
+reads and serving of those objects fail, and the garbage-collection tasks refuse
+to delete them. Those rows are parked for one day and retried on later runs,
+without counting deletion attempts and without ever reaching the give-up window,
+so the row is never removed until the routes file declares the target id again.
+
+All backend replicas must mount the same routes file with the same target ids. A
+replica with a stale file misroutes new writes and fails reads and garbage
+collection for the targets it does not declare.
+
 ### File Data Storage
 
 __Since version 2.11.0__

docs/technical-guide/developer/subsystems/assets-storage.md

@@ -16,11 +16,12 @@ that may be used for any kind of user uploaded files. Currently:
 There is an abstract interface and several implementations (or **backends**),
 depending on where the objects are actually stored:
 
- * <code class="language-clojure">:assets-fs</code> stores ojects in the file system, under a given base path.
- * <code class="language-clojure">:assets-s3</code> stores them in any cloud storage with an AWS-S3 compatible
+ * <code class="language-clojure">:fs</code> stores objects in the file system, under a given base path.
+ * <code class="language-clojure">:s3</code> stores them in any cloud storage with an AWS-S3 compatible
    interface.
- * <code class="language-clojure">:assets-db</code> stores them inside the PostgreSQL database, in a special table
-   with a binary column.
+
+Legacy rows may still reference the deprecated <code class="language-clojure">:assets-fs</code> and
+<code class="language-clojure">:assets-s3</code> names, which alias the current backends.
 
 ## Storage API
 
@@ -83,6 +84,17 @@ The storage module may use the bucket (hardcoded) to make special treatment to
 object, such as storing in a different path, or guessing how to know if an object
 is referenced from other place.
 
+When the <code class="language-clojure">:s3</code> backend is used, a semantic bucket may also be routed to a
+named S3 **target** (a different bucket and/or key prefix). The chosen target id
+is stored in the object metadata (<code class="language-clojure">:storage-target</code>) and resolved again on
+reads, URL signing, deduplication and garbage collection. Objects without a
+stored target use the default target. See the configuration guide for the
+routing file format.
+
+If an object references a target id that is not configured anymore, it cannot be
+located: reads and serving fail, and garbage collection refuses to delete the
+object, keeping the database row until the target is declared again.
+
 ## Sharing and deleting objects
 
 To save storage space, duplicated objects wre shared. So, if for example