Paul Rastoin eb69e56441 [Upgrade] Add a detached-run wrapper for the upgrade command (#23497) ...

Long upgrades are started over `kubectl exec` into the command-runner
pod, where the process is a child of the exec'd shell: a dropped SSM
tunnel, a closed laptop or a dead VPN kills the run mid-way. There is no
tmux in the image (Alpine, `sh: tmux: not found`).

`scripts/upgrade-background.sh` puts the run in its own session with no
controlling terminal and streams its output from a log file, so losing
the connection detaches the stream instead of killing the upgrade.

Builds on #23481. The cooperative shutdown path is untouched, this is
tooling around it. No TypeScript changed.

## Commands

```bash
yarn upgrade:background [args]   # start detached, then stream the log
yarn upgrade:background:logs     # re-attach from another shell
yarn upgrade:background:stop     # graceful stop; --now immediate, --force SIGKILL
```

`[args]` is forwarded verbatim to `upgrade`, so it takes that command's
options and no others:

| Option | Effect |
| --- | --- |
| `-d`, `--dry-run` | simulate without making changes |
| `-v`, `--verbose` | verbose output |
| `-w`, `--workspace-id <id>` | restrict to a workspace, repeatable; all
provisioned workspaces if omitted |
| `--start-from-workspace-id <id>` | resume from a workspace, ascending
id order |
| `--workspace-count-limit <n>` | process at most n workspaces,
ascending id order |

`-w` and `--start-from-workspace-id` are mutually exclusive, `upgrade`
rejects the combination.

## Example

```ts
➜  twenty-server git:(claude/upgrade-detached-run-wrapper-5nkb0v) ✗ yarn upgrade:background
Running (pid 15779), logging to /tmp/twenty-upgrade.log
Ctrl+C detaches the stream only. Use 'yarn upgrade:background:stop' to stop the run.
^C%
➜  twenty-server git:(claude/upgrade-detached-run-wrapper-5nkb0v) ✗ yarn upgrade:background:stop
SIGTERM sent to 15779, it finishes the step in progress then stops (exit 143)
Tail of /tmp/twenty-upgrade.log:
[Nest] 15779  - 07/30/2026, 10:47:50 AM     LOG [UpgradeAwareEntityMetadataAdapter] [upgrade-metadata] hidden columns on FieldMetadataEntity: standardOverrides,isCustom
[Nest] 15779  - 07/30/2026, 10:47:50 AM     LOG [UpgradeAwareEntityMetadataAdapter] [upgrade-metadata] applied cursor=217 renamed=0 unavailable=0 hiddenColumns=5
[Nest] 15779  - 07/30/2026, 10:47:50 AM     LOG [UpgradeAwareEntityMetadataAdapter] [upgrade-metadata] hidden columns on RolePermissionFlagEntity: flag
[Nest] 15779  - 07/30/2026, 10:47:50 AM     LOG [UpgradeAwareEntityMetadataAdapter] [upgrade-metadata] hidden columns on ObjectMetadataEntity: standardOverrides,isCustom
[Nest] 15779  - 07/30/2026, 10:47:50 AM     LOG [UpgradeAwareEntityMetadataAdapter] [upgrade-metadata] hidden columns on FieldMetadataEntity: standardOverrides,isCustom
[Nest] 15779  - 07/30/2026, 10:47:50 AM     LOG [UpgradeAwareEntityMetadataAdapter] [upgrade-metadata] applied cursor=207 renamed=0 unavailable=0 hiddenColumns=5
[Nest] 15779  - 07/30/2026, 10:47:50 AM     LOG [DatabaseConfigDriver] [INIT] Loading initial config variables from database
[Nest] 15779  - 07/30/2026, 10:47:50 AM     LOG [DatabaseConfigDriver] [INIT] Config variables loaded: 1 values found in DB, 104 falling to env vars/defaults
[Nest] 15779  - 07/30/2026, 10:47:50 AM     LOG [UpgradeCommand] Initialized upgrade sequence: 217 step(s)
[upgrade] event=sequence.initialized stepCount=217 dryRun=false
[Nest] 15779  - 07/30/2026, 10:47:50 AM     LOG [WorkspaceIteratorService] Running on workspace 20202020-1c25-4d02-bf25-6aeccf7ea419 1/2
[Nest] 15779  - 07/30/2026, 10:47:50 AM     LOG [WorkspaceCommandRunnerService] Upgrading workspace 20202020-1c25-4d02-bf25-6aeccf7ea419 1/2
[upgrade] event=workspace.start workspaceId=20202020-1c25-4d02-bf25-6aeccf7ea419 index=1 total=2 dryRun=false
[Nest] 15779  - 07/30/2026, 10:47:50 AM     LOG [WorkspaceCommandRunnerService] Upgrade for workspace 20202020-1c25-4d02-bf25-6aeccf7ea419 completed.
[upgrade] event=workspace.success workspaceId=20202020-1c25-4d02-bf25-6aeccf7ea419 executedByVersion=unknown dryRun=false
[Nest] 15779  - 07/30/2026, 10:47:50 AM     LOG [WorkspaceIteratorService] Running on workspace 3b8e6458-5fc1-4e63-8563-008ccddaa6db 2/2
[Nest] 15779  - 07/30/2026, 10:47:50 AM     LOG [WorkspaceCommandRunnerService] Upgrading workspace 3b8e6458-5fc1-4e63-8563-008ccddaa6db 2/2
[upgrade] event=workspace.start workspaceId=3b8e6458-5fc1-4e63-8563-008ccddaa6db index=2 total=2 dryRun=false
[Nest] 15779  - 07/30/2026, 10:47:50 AM     LOG [DummySleepCommand] Sleeping for 30000ms on workspace 3b8e6458-5fc1-4e63-8563-008ccddaa6db
[Nest] 15779  - 07/30/2026, 10:47:53 AM    WARN [CommandShutdownService] Received SIGTERM, finishing the step in progress then stopping. Send SIGTERM again to exit immediately.
Follow with 'yarn upgrade:background:logs'. Still stuck: 'stop --now'. Last resort: 'stop --force'.
➜  twenty-server git:(claude/upgrade-detached-run-wrapper-5nkb0v) ✗ yarn upgrade:background:stop --now
Second SIGTERM sent to 15779, immediate exit with the step in progress left unfinished
Tail of /tmp/twenty-upgrade.log:
[Nest] 15779  - 07/30/2026, 10:47:50 AM     LOG [UpgradeAwareEntityMetadataAdapter] [upgrade-metadata] hidden columns on RolePermissionFlagEntity: flag
[Nest] 15779  - 07/30/2026, 10:47:50 AM     LOG [UpgradeAwareEntityMetadataAdapter] [upgrade-metadata] hidden columns on ObjectMetadataEntity: standardOverrides,isCustom
[Nest] 15779  - 07/30/2026, 10:47:50 AM     LOG [UpgradeAwareEntityMetadataAdapter] [upgrade-metadata] hidden columns on FieldMetadataEntity: standardOverrides,isCustom
[Nest] 15779  - 07/30/2026, 10:47:50 AM     LOG [UpgradeAwareEntityMetadataAdapter] [upgrade-metadata] applied cursor=207 renamed=0 unavailable=0 hiddenColumns=5
[Nest] 15779  - 07/30/2026, 10:47:50 AM     LOG [DatabaseConfigDriver] [INIT] Loading initial config variables from database
[Nest] 15779  - 07/30/2026, 10:47:50 AM     LOG [DatabaseConfigDriver] [INIT] Config variables loaded: 1 values found in DB, 104 falling to env vars/defaults
[Nest] 15779  - 07/30/2026, 10:47:50 AM     LOG [UpgradeCommand] Initialized upgrade sequence: 217 step(s)
[upgrade] event=sequence.initialized stepCount=217 dryRun=false
[Nest] 15779  - 07/30/2026, 10:47:50 AM     LOG [WorkspaceIteratorService] Running on workspace 20202020-1c25-4d02-bf25-6aeccf7ea419 1/2
[Nest] 15779  - 07/30/2026, 10:47:50 AM     LOG [WorkspaceCommandRunnerService] Upgrading workspace 20202020-1c25-4d02-bf25-6aeccf7ea419 1/2
[upgrade] event=workspace.start workspaceId=20202020-1c25-4d02-bf25-6aeccf7ea419 index=1 total=2 dryRun=false
[Nest] 15779  - 07/30/2026, 10:47:50 AM     LOG [WorkspaceCommandRunnerService] Upgrade for workspace 20202020-1c25-4d02-bf25-6aeccf7ea419 completed.
[upgrade] event=workspace.success workspaceId=20202020-1c25-4d02-bf25-6aeccf7ea419 executedByVersion=unknown dryRun=false
[Nest] 15779  - 07/30/2026, 10:47:50 AM     LOG [WorkspaceIteratorService] Running on workspace 3b8e6458-5fc1-4e63-8563-008ccddaa6db 2/2
[Nest] 15779  - 07/30/2026, 10:47:50 AM     LOG [WorkspaceCommandRunnerService] Upgrading workspace 3b8e6458-5fc1-4e63-8563-008ccddaa6db 2/2
[upgrade] event=workspace.start workspaceId=3b8e6458-5fc1-4e63-8563-008ccddaa6db index=2 total=2 dryRun=false
[Nest] 15779  - 07/30/2026, 10:47:50 AM     LOG [DummySleepCommand] Sleeping for 30000ms on workspace 3b8e6458-5fc1-4e63-8563-008ccddaa6db
[Nest] 15779  - 07/30/2026, 10:47:53 AM    WARN [CommandShutdownService] Received SIGTERM, finishing the step in progress then stopping. Send SIGTERM again to exit immediately.
[Nest] 15779  - 07/30/2026, 10:48:00 AM    WARN [CommandShutdownService] Received SIGTERM again, exiting immediately. The step in progress is left unfinished, rerun the command to resume from the last recorded step.
EXIT=143
```

## Not a concurrency guard

`start` refuses when it can see a live run, but that only keeps this
wrapper's own bookkeeping straight, one PID file and one log per run.
The PID file is in the container's `/tmp`, so a second pod or a laptop
pointed at the same database sees none of it.

Nothing in `upgrade` prevents two sequences either: `upgradeMigration`
records only `completed` and `failed`, so it has no in-progress state to
lock against, and the sequence runner takes no advisory lock. Out of
scope here, and worth a follow-up if we want it enforced rather than
operational.

## Dockerfile

`scripts/` was not copied into the server image, so all three commands
would have failed with ENOENT in the pod. Added the COPY, plus a `chmod
+x` matching the one already on `entrypoint.sh`.

Rest is documented in `docs/UPGRADE_COMMANDS.md`, including the
exit-code table and why a graceful stop is always safe to rerun.

2026-07-30 12:24:07 +00:00

..

UPGRADE_COMMANDS.md

[Upgrade] Add a detached-run wrapper for the upgrade command (#23497)

2026-07-30 12:24:07 +00:00