From a9be1d377992d99ecb807f42bc587bc251fdbc10 Mon Sep 17 00:00:00 2001 From: Viktor Petersson Date: Mon, 27 Apr 2026 17:06:04 +0000 Subject: [PATCH] =?UTF-8?q?chore:=20drop=20pyzmq=20+=20libzmq,=20finalize?= =?UTF-8?q?=20ZMQ=E2=86=92Redis=20migration?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit With both legs of the viewer signalling path on Redis (PR1: command bus, PR2: reply bus), the pyzmq runtime dependency and the libzmq* build deps are no longer used. - pyproject.toml: remove pyzmq==23.2.1 from server, viewer, wifi-connect, and mypy dep groups (4 places). - uv.lock: regenerated; pyzmq + transitive py drop out. - tools/image_builder/{__main__,utils}.py: remove libzmq3-dev / libzmq5-dev / libzmq5 from the base apt list and from the viewer context's apt list. docker/uv-builder.j2 likewise drops libzmq3-dev from both the prebuilt-uv branch and the pip-fallback branch (32-bit ARM). The rendered docker/Dockerfile.* artifacts are gitignored, so no committed Dockerfile churn here — they regenerate cleanly via `python -m tools.image_builder --dockerfiles-only`. - send_zmq_message.py → send_viewer_message.py. The script already publishes via Redis (fixed in the PR1 follow-up); rename + update callers (bin/start_wifi_connect.sh, docker/Dockerfile.wifi-connect.j2) now that the ZMQ name is misleading. - bin/start_server.sh: drop the stale "single-worker because ZmqPublisher binds 10001" comment. The publisher is now a Redis client — no port bind, multi-worker is safe whenever the operator wants to opt in (not changed in this PR). - CLAUDE.md: update the architecture description (ZMQ ports 10001 / 5558 are gone, Redis carries the viewer signalling traffic now). Co-Authored-By: Claude Opus 4.7 (1M context) --- CLAUDE.md | 10 ++--- bin/start_server.sh | 4 -- bin/start_wifi_connect.sh | 4 +- docker/Dockerfile.wifi-connect.j2 | 2 +- docker/uv-builder.j2 | 2 - pyproject.toml | 4 -- send_zmq_message.py => send_viewer_message.py | 0 tools/image_builder/__main__.py | 3 -- tools/image_builder/utils.py | 3 -- uv.lock | 41 ------------------- 10 files changed, 8 insertions(+), 65 deletions(-) rename send_zmq_message.py => send_viewer_message.py (100%) diff --git a/CLAUDE.md b/CLAUDE.md index fa12fc42..07a8945c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -12,11 +12,11 @@ Anthias runs as a set of Docker containers: - **anthias-server** (port 80 in prod, 8000 in dev) — uvicorn (ASGI) serving the Django web app, REST API, the React frontend's static assets (via WhiteNoise), uploaded media at `/anthias_assets/`, and the WebSocket endpoint at `/ws` (Django Channels with a Redis-backed channel layer). Always plain HTTP — TLS is opt-in and handled by the **anthias-caddy** sidecar that `bin/enable_ssl.sh` installs as a compose override (Caddy local CA by default, or auto Let's Encrypt with `--domain`, or BYO cert with `--cert`/`--key`). - **anthias-celery** — Async task queue (asset downloads, cleanup). Publishes asset-update events back to the WebSocket consumers via the Channels Redis layer. -- **anthias-viewer** — Drives the display, receives instructions via ZMQ, talks to anthias-server over HTTP. -- **redis** (port 6379) — Celery broker + result backend, and Channels channel layer. +- **anthias-viewer** — Drives the display, receives instructions over the Redis pub/sub `anthias.viewer` channel, talks to anthias-server over HTTP. +- **redis** (port 6379) — Celery broker + result backend, Channels channel layer, and the viewer signalling bus (pub/sub channel + per-correlation-ID reply lists). - **webview** — Qt-based browser for rendering content on the display; fetches `/anthias_assets/` from anthias-server. -Inter-service communication uses ZMQ (port 10001 publisher, 5558 collector) for the viewer signalling path; WebSocket fan-out from Celery to browsers goes via Channels/Redis. The primary database is SQLite stored at `~/.anthias/anthias.db`, with configuration in `~/.anthias/anthias.conf`. (Pre-rebrand installations have these at `~/.screenly/screenly.db` and `~/.screenly/screenly.conf`; `bin/migrate_legacy_paths.sh` migrates them on upgrade and leaves back-compat symlinks.) +Inter-service messaging is all Redis: WebSocket fan-out from Celery to browsers goes via Channels/Redis, and server↔viewer commands/replies use Redis pub/sub on `anthias.viewer` with BLPOP on `anthias.reply.` for the few request-reply paths. The primary database is SQLite stored at `~/.anthias/anthias.db`, with configuration in `~/.anthias/anthias.conf`. (Pre-rebrand installations have these at `~/.screenly/screenly.db` and `~/.screenly/screenly.conf`; `bin/migrate_legacy_paths.sh` migrates them on upgrade and leaves back-compat symlinks.) ### Key Directories @@ -24,9 +24,9 @@ Inter-service communication uses ZMQ (port 10001 publisher, 5558 collector) for - `anthias_django/` — Django project settings, URLs, ASGI/WSGI - `api/` — REST API (views, serializers, URLs for v1, v1.1, v1.2, v2) - `static/src/` — TypeScript/React frontend (components, Redux store, hooks, tests) -- `viewer/` — Viewer service (scheduling, media player, ZMQ communication) +- `viewer/` — Viewer service (scheduling, media player, Redis pub/sub messaging) - `webview/` — C++ Qt-based WebView (Qt5 for Pi 1-4, Qt6 for Pi 5/x86) -- `lib/` — Shared Python utilities (auth, device helpers, diagnostics, ZMQ) +- `lib/` — Shared Python utilities (auth, device helpers, diagnostics) - `docker/` — Dockerfile Jinja2 templates for each service - `tests/` — Python unit/integration tests - `bin/` — Shell scripts for install, dev setup, testing, upgrades diff --git a/bin/start_server.sh b/bin/start_server.sh index 8099d8e0..617350e5 100755 --- a/bin/start_server.sh +++ b/bin/start_server.sh @@ -30,10 +30,6 @@ else ./manage.py dbbackup --noinput --clean fi -# Single-worker on purpose: ZmqPublisher.get_instance() in api/views/* -# binds tcp://0.0.0.0:10001, which would EADDRINUSE across multiple -# workers. Hoist the publisher into a sidecar (or move it to the -# Channels layer) before adding `--workers N` here. UVICORN_BIND_HOST="${LISTEN:-0.0.0.0}" UVICORN_BIND_PORT="${PORT:-8080}" diff --git a/bin/start_wifi_connect.sh b/bin/start_wifi_connect.sh index 9fcb5804..fda75604 100644 --- a/bin/start_wifi_connect.sh +++ b/bin/start_wifi_connect.sh @@ -31,7 +31,7 @@ while [[ true ]]; do echo "Skipping setting up Wifi-Connect Access Point." if [[ "$IS_CONNECTED" = 'false' ]]; then - python send_zmq_message.py --action='show_splash' + python send_viewer_message.py --action='show_splash' fi exit 0 @@ -41,7 +41,7 @@ while [[ true ]]; do echo "Connect to the Access Point and configure the SSID and Passphrase for the network to connect to." if [[ "$IS_CONNECTED" = '' ]]; then - python send_zmq_message.py --action='setup_wifi' + python send_viewer_message.py --action='setup_wifi' fi IS_CONNECTED='false' diff --git a/docker/Dockerfile.wifi-connect.j2 b/docker/Dockerfile.wifi-connect.j2 index 252590bb..077e35f2 100644 --- a/docker/Dockerfile.wifi-connect.j2 +++ b/docker/Dockerfile.wifi-connect.j2 @@ -30,7 +30,7 @@ COPY --from=uv-builder /venv /venv ENV PATH="/venv/bin:$PATH" ENV VIRTUAL_ENV="/venv" -COPY send_zmq_message.py ./ +COPY send_viewer_message.py ./ RUN touch /var/lib/misc/dnsmasq.leases diff --git a/docker/uv-builder.j2 b/docker/uv-builder.j2 index cbf11ee4..07074852 100644 --- a/docker/uv-builder.j2 +++ b/docker/uv-builder.j2 @@ -21,7 +21,6 @@ RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ build-essential \ libffi-dev \ libssl-dev \ - libzmq3-dev \ python3 \ python3-dev \ python3-pip \ @@ -42,7 +41,6 @@ RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ build-essential \ libffi-dev \ libssl-dev \ - libzmq3-dev \ python3 \ python3-dev \ {% for dep in builder_extra_apt | default([]) %} diff --git a/pyproject.toml b/pyproject.toml index 7587c4ab..0cc7715b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -59,7 +59,6 @@ server = [ "python-dateutil==2.9.0.post0", "pytz==2025.2", "PyYAML==6.0.2", - "pyzmq==23.2.1", "redis==7.4.0", "requests[security]==2.33.1", "setuptools==79.0.1", @@ -90,7 +89,6 @@ viewer = [ "python-dateutil==2.9.0.post0", "python-vlc==3.0.21203", "pytz==2025.2", - "pyzmq==23.2.1", "redis==7.4.0", "requests[security]==2.33.1", "tenacity==9.1.2", @@ -102,7 +100,6 @@ wifi-connect = [ "Cython==3.2.4", "future==1.0.0", "netifaces==0.11.0", - "pyzmq==23.2.1", "redis==7.4.0", ] dev = [ @@ -148,7 +145,6 @@ mypy = [ "djangorestframework==3.16.1", "drf-spectacular==0.29.0", "pytz==2025.2", - "pyzmq==23.2.1", "whitenoise==6.8.2", ] diff --git a/send_zmq_message.py b/send_viewer_message.py similarity index 100% rename from send_zmq_message.py rename to send_viewer_message.py diff --git a/tools/image_builder/__main__.py b/tools/image_builder/__main__.py index 488b034f..0acbafbc 100644 --- a/tools/image_builder/__main__.py +++ b/tools/image_builder/__main__.py @@ -65,9 +65,6 @@ def build_image( 'libcec-dev ', 'libffi-dev', 'libssl-dev', - 'libzmq3-dev', - 'libzmq5-dev', - 'libzmq5', 'lsb-release', 'mplayer', 'net-tools', diff --git a/tools/image_builder/utils.py b/tools/image_builder/utils.py index dde54942..90f014f2 100644 --- a/tools/image_builder/utils.py +++ b/tools/image_builder/utils.py @@ -195,7 +195,6 @@ def get_viewer_context(board: str) -> dict[str, Any]: 'libsqlite3-dev', 'libsrtp2-dev', 'libssl-dev', - 'libzmq3-dev', 'libswscale-dev', 'libsystemd-dev', 'libts-dev', @@ -232,8 +231,6 @@ def get_viewer_context(board: str) -> dict[str, Any]: 'libxslt1-dev', 'libxss-dev', 'libxtst-dev', - 'libzmq5-dev', - 'libzmq5', 'net-tools', 'procps', 'psmisc', diff --git a/uv.lock b/uv.lock index 7436c2a1..c54db9ce 100644 --- a/uv.lock +++ b/uv.lock @@ -157,7 +157,6 @@ mypy = [ { name = "pygit2" }, { name = "python-on-whales" }, { name = "pytz" }, - { name = "pyzmq" }, { name = "requests" }, { name = "ruff" }, { name = "tenacity" }, @@ -199,7 +198,6 @@ server = [ { name = "python-dateutil" }, { name = "pytz" }, { name = "pyyaml" }, - { name = "pyzmq" }, { name = "redis" }, { name = "requests" }, { name = "setuptools" }, @@ -245,7 +243,6 @@ test = [ { name = "python-vlc" }, { name = "pytz" }, { name = "pyyaml" }, - { name = "pyzmq" }, { name = "redis" }, { name = "requests" }, { name = "selenium" }, @@ -281,7 +278,6 @@ viewer = [ { name = "python-dateutil" }, { name = "python-vlc" }, { name = "pytz" }, - { name = "pyzmq" }, { name = "redis" }, { name = "requests" }, { name = "sh" }, @@ -293,7 +289,6 @@ wifi-connect = [ { name = "cython" }, { name = "future" }, { name = "netifaces" }, - { name = "pyzmq" }, { name = "redis" }, ] @@ -363,7 +358,6 @@ mypy = [ { name = "pygit2", specifier = "==1.19.1" }, { name = "python-on-whales", specifier = "==0.79.0" }, { name = "pytz", specifier = "==2025.2" }, - { name = "pyzmq", specifier = "==23.2.1" }, { name = "requests", specifier = "==2.33.1" }, { name = "ruff", specifier = "==0.14.10" }, { name = "tenacity", specifier = "==9.1.2" }, @@ -405,7 +399,6 @@ server = [ { name = "python-dateutil", specifier = "==2.9.0.post0" }, { name = "pytz", specifier = "==2025.2" }, { name = "pyyaml", specifier = "==6.0.2" }, - { name = "pyzmq", specifier = "==23.2.1" }, { name = "redis", specifier = "==7.4.0" }, { name = "requests", extras = ["security"], specifier = "==2.33.1" }, { name = "setuptools", specifier = "==79.0.1" }, @@ -451,7 +444,6 @@ test = [ { name = "python-vlc", specifier = "==3.0.21203" }, { name = "pytz", specifier = "==2025.2" }, { name = "pyyaml", specifier = "==6.0.2" }, - { name = "pyzmq", specifier = "==23.2.1" }, { name = "redis", specifier = "==7.4.0" }, { name = "requests", extras = ["security"], specifier = "==2.33.1" }, { name = "selenium", specifier = "==4.36.0" }, @@ -487,7 +479,6 @@ viewer = [ { name = "python-dateutil", specifier = "==2.9.0.post0" }, { name = "python-vlc", specifier = "==3.0.21203" }, { name = "pytz", specifier = "==2025.2" }, - { name = "pyzmq", specifier = "==23.2.1" }, { name = "redis", specifier = "==7.4.0" }, { name = "requests", extras = ["security"], specifier = "==2.33.1" }, { name = "sh", specifier = "==2.2.2" }, @@ -499,7 +490,6 @@ wifi-connect = [ { name = "cython", specifier = "==3.2.4" }, { name = "future", specifier = "==1.0.0" }, { name = "netifaces", specifier = "==0.11.0" }, - { name = "pyzmq", specifier = "==23.2.1" }, { name = "redis", specifier = "==7.4.0" }, ] @@ -1546,15 +1536,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/3e/73/2ce007f4198c80fcf2cb24c169884f833fe93fbc03d55d302627b094ee91/psutil-7.2.1-cp37-abi3-win_arm64.whl", hash = "sha256:0d67c1822c355aa6f7314d92018fb4268a76668a536f133599b91edd48759442", size = 133836, upload-time = "2025-12-29T08:26:43.086Z" }, ] -[[package]] -name = "py" -version = "1.11.0" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/98/ff/fec109ceb715d2a6b4c4a85a61af3b40c723a961e8828319fbcb15b868dc/py-1.11.0.tar.gz", hash = "sha256:51c75c4126074b472f746a24399ad32f6053d1b34b68d2fa41e558e6f4a98719", size = 207796, upload-time = "2021-11-04T17:17:01.377Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/f6/f0/10642828a8dfb741e5f3fbaac830550a518a775c7fff6f04a007259b0548/py-1.11.0-py2.py3-none-any.whl", hash = "sha256:607c53218732647dff4acdfcd50cb62615cedf612e72d1724fb1a0cc6405b378", size = 98708, upload-time = "2021-11-04T17:17:00.152Z" }, -] - [[package]] name = "pyasn1" version = "0.6.3" @@ -1903,28 +1884,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/fa/de/02b54f42487e3d3c6efb3f89428677074ca7bf43aae402517bc7cca949f3/PyYAML-6.0.2-cp313-cp313-win_amd64.whl", hash = "sha256:8388ee1976c416731879ac16da0aff3f63b286ffdd57cdeb95f3f2e085687563", size = 156446, upload-time = "2024-08-06T20:33:04.33Z" }, ] -[[package]] -name = "pyzmq" -version = "23.2.1" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "cffi", marker = "implementation_name == 'pypy'" }, - { name = "py", marker = "implementation_name == 'pypy'" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/72/37/d5603f352522e249e44ee767a8a59b3fe7cf7f708a94fd40a637c6890add/pyzmq-23.2.1.tar.gz", hash = "sha256:2b381aa867ece7d0a82f30a0c7f3d4387b7cf2e0697e33efaa5bed6c5784abcd", size = 1218264, upload-time = "2022-08-12T09:01:12.157Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/b4/e3/081b1a02af85b60e19191ea5d41b422ea7a0126043b7c54b5f3c2ce8e065/pyzmq-23.2.1-cp311-cp311-macosx_10_15_universal2.whl", hash = "sha256:e753eee6d3b93c5354e8ba0a1d62956ee49355f0a36e00570823ef64e66183f5", size = 1829830, upload-time = "2022-08-12T09:17:58.12Z" }, - { url = "https://files.pythonhosted.org/packages/38/e7/3ff3068074316da9990e889382b7a5f08728ebca893a2b8e018c1531633f/pyzmq-23.2.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:f07016e3cf088dbfc6e7c5a7b3f540db5c23b0190d539e4fd3e2b5e6beffa4b5", size = 1244536, upload-time = "2022-08-12T09:18:12.296Z" }, - { url = "https://files.pythonhosted.org/packages/dc/24/3ac8d73b09a6db9d26d4c6fb454101dbc9e1ea1e6df50cd50041decbd895/pyzmq-23.2.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:4805af9614b0b41b7e57d17673459facf85604dac502a5a9244f6e8c9a4de658", size = 880172, upload-time = "2022-08-12T09:12:29.324Z" }, - { url = "https://files.pythonhosted.org/packages/a7/bb/54986ab979a4f964c6b76817b6fdd7403a9cf76f368a0c67ec6bb65a783d/pyzmq-23.2.1-cp311-cp311-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:39dd252b683816935702825e5bf775df16090619ced9bb4ba68c2d0b6f0c9b18", size = 1130741, upload-time = "2022-08-12T09:11:08.068Z" }, - { url = "https://files.pythonhosted.org/packages/4d/b4/804256cc2a3668b152c5ddf65bf319ae8d922ef6bfadee24e0ef23f096d6/pyzmq-23.2.1-cp311-cp311-manylinux_2_28_x86_64.whl", hash = "sha256:84678153432241bcdca2210cf4ff83560b200556867aea913ffbb960f5d5f340", size = 1075742, upload-time = "2022-08-12T09:18:41.418Z" }, - { url = "https://files.pythonhosted.org/packages/6b/36/0546d92a57ca60a456f801f4e1a55932c8c6e324e1b77f6faca660728b4e/pyzmq-23.2.1-cp311-cp311-musllinux_1_1_aarch64.whl", hash = "sha256:90d88f9d9a2ae6cfb1dc4ea2d1710cdf6456bc1b9a06dd1bb485c5d298f2517e", size = 1407475, upload-time = "2022-08-12T09:12:32.562Z" }, - { url = "https://files.pythonhosted.org/packages/a7/2c/9ff05957a40dc87889a584457a6c700e77ad3f623e4f5e33a29dfcc2ea59/pyzmq-23.2.1-cp311-cp311-musllinux_1_1_i686.whl", hash = "sha256:794871988c34727c7f79bdfe2546e6854ae1fa2e1feb382784f23a9c6c63ecb3", size = 1737821, upload-time = "2022-08-12T09:13:42.82Z" }, - { url = "https://files.pythonhosted.org/packages/b6/a0/79ae7abdc28eecc261806a9ebbfb03f05709a501350009c5925e0ed73a29/pyzmq-23.2.1-cp311-cp311-musllinux_1_1_x86_64.whl", hash = "sha256:c56b1a62a1fb87565343c57b6743fd5da6e138b8c6562361d7d9b5ce4acf399a", size = 1626767, upload-time = "2022-08-12T09:13:44.173Z" }, - { url = "https://files.pythonhosted.org/packages/cd/88/dcbe0bfc6b744badda43322e2538ab269451e0f298d3e4616da92a25dfa3/pyzmq-23.2.1-cp311-cp311-win32.whl", hash = "sha256:c3ebf1668664d20c8f7d468955f18379b7d1f7bc8946b13243d050fa3888c7ff", size = 887707, upload-time = "2022-08-12T09:09:56.207Z" }, - { url = "https://files.pythonhosted.org/packages/b7/90/da3761e269f7d5a7916e98364f55745ca1a979a6316a265fc4b4ac26e144/pyzmq-23.2.1-cp311-cp311-win_amd64.whl", hash = "sha256:ec9803aca9491fd6f0d853d2a6147f19f8deaaa23b1b713d05c5d09e56ea7142", size = 1139705, upload-time = "2022-08-12T09:08:42.647Z" }, -] - [[package]] name = "redis" version = "7.4.0"