Matthieu B 2283c4de68 fix: prevent startup race condition during migrations ...

This fixes a critical issue where Gunicorn workers would fail to start
after upgrading to v2025.11.0, causing containers to show as unhealthy
with only the uv wrapper process running and no actual workers.

Root Cause:
-----------
In v2025.11.0, library scanning and session recovery were added to the
create_app() function, which runs during EVERY app creation including:
1. During 'flask db upgrade' (migrations)
2. During Gunicorn master when_ready() hook
3. During each Gunicorn worker spawn

The migration 20251103_properly_fix_foreign_keys recreates 4 database
tables with CASCADE foreign keys using raw SQL. This holds exclusive
database locks during table recreation.

When library scanning and session recovery try to query these tables
during migration, they hit database locks, creating a race condition
that causes workers to timeout and crash during startup.

Fix:
----
- Skip library scanning during migrations (FLASK_SKIP_SCHEDULER=true)
- Skip activity monitoring/session recovery during migrations
- Make Gunicorn log level configurable (GUNICORN_LOG_LEVEL env var)
- Add worker lifecycle hooks for better crash debugging
- Increase healthcheck start period from 10s to 60s
- Increase Gunicorn worker timeout from 30s to 120s

Testing:
--------
- Verified app starts successfully with FLASK_SKIP_SCHEDULER=true
- Verified library scanning runs normally without the flag
- Confirmed 0.38s startup during migrations vs 1.61s normal startup

Closes #976

2025-11-03 20:41:52 +01:00

5.0 KiB

Raw Permalink Blame History

View Raw