mirror of
https://github.com/Screenly/Anthias.git
synced 2026-08-01 02:05:59 -04:00
dependabot/uv/python-819b434b42
8 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Guillaume Jobin
|
6bbe55bf52 |
fix: bias yt-dlp format_sort toward board's HW-decoded codec (Pi 5 HEVC) (#3093)
* fix: bias yt-dlp format_sort toward board's HW-decoded codec (GH #3092) On Pi 5 (HEVC-only hardware decode), download_youtube_asset was requesting H.264 via a hardcoded format_sort preference. The download succeeded, but normalize_video_asset then rejected the asset with "Video codec 'h264' is not hardware-decoded on this device. Supported: hevc." The capability detection already existed — it just wasn't consulted before the download. Fix: call _hw_decoded_codecs() before building ydl_opts and set the leading format_sort entry to the board's preferred codec. HEVC-only boards (Pi 5) now request vcodec:hevc; all others keep vcodec:h264. Unknown boards fall back to h264 so the download still proceeds. Adds four tests covering Pi 5 (HEVC), Pi 4 (H.264 preferred when both supported), Pi 3 (H.264 only), and unrecognised board (h264 fallback). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: accept H.264 software decode on Pi 5 so YouTube downloads work Pi 5's BCM2712 has no H.264 hardware decoder; the Cortex-A76 cores software-decode 1080p H.264 without frame drops. However YouTube rarely serves HEVC, so the previous HEVC-only codec gate on Pi 5 rejected every YouTube download regardless of format_sort preference. - processing.py: add h264 to pi5's accepted codec set alongside hevc - celery_tasks.py: simplify format_sort to prefer hevc on any board that has it (yt-dlp falls back to h264 on YouTube anyway) - test_processing.py: replace pi5+h264-rejection tests with pi3+hevc-rejection tests (mocked ffprobe, no ffmpeg needed); add new test asserting h264 IS accepted on pi5 via software decode - test_celery_tasks.py: update pi4 test to expect hevc preference Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * tests: add Pi 5 unsupported-codec rejection coverage; restore recipe doc Add test_video_unsupported_codec_still_rejected_on_pi5 to verify that codecs outside Pi 5's accepted set (VP9, AV1, …) still raise UnsupportedVideoCodecError even after h264 was added as a software- decode fallback. Restores the removed-behavior coverage flagged by the code review for that path. Also restore _ffmpeg_reencode_recipe / _handbrake_steps to prefer h264 when it's in the accepted set (faster to encode; both codecs are hardware-decoded on Pi 4 where the set is {h264, hevc}). Update the docstrings to reflect the current logic. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix: update stale comment — Pi 5 accepts H.264 via software decode The previous comment said "HEVC-only boards (Pi 5) must not receive an H.264 stream", which was true before this PR but is now wrong: Pi 5 accepts H.264 via Cortex-A76 software decode since YouTube rarely serves HEVC. Update the comment to accurately describe the current behaviour. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * style: apply ruff format to fix CI linter failure Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * refactor: move yt-dlp vcodec preference into _PREFERRED_DOWNLOAD_VCODEC Add _PREFERRED_DOWNLOAD_VCODEC dict and preferred_download_vcodec() in processing.py alongside _HW_DECODE_VIDEO_CODECS, so per-board download preferences live with the rest of the codec definitions rather than as ad-hoc logic in celery_tasks.py. Pi 5 keeps its HEVC preference (hardware decode path). Pi 4, x86, Rock Pi 4 and all other boards revert to H.264 — it is their primary hardware path and more widely available on YouTube. x86 HEVC via VAAPI is known-broken (black screen), so narrowing the HEVC bias to Pi 5 only closes that risk. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
Viktor Petersson
|
8c6cfaf26a |
feat(server): offer HandBrake GUI steps for rejected video uploads (#3040)
* feat(server): offer HandBrake GUI steps for rejected video uploads - Add _handbrake_steps mirroring the ffmpeg recipe's codec/1080p choices - Persist the steps to metadata.error_handbrake on rejection - Render them as a numbered list with a handbrake.fr link in the modal Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(server): make HandBrake steps a real preset-based walkthrough - Lead with the stock "Fast 1080p30" preset (H.264 MP4, 1080p cap) - HEVC boards just flip Video Encoder to "H.265 (x265)" - Spell out source pick, Save As/Browse, and Start Encode - Drop the cap arg: the 1080p preset is the low-RAM fix too Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: assert on HANDBRAKE_URL constant, not the literal Reference processing.HANDBRAKE_URL instead of a duplicated URL literal so CodeQL stops flagging the assertion as incomplete-URL-substring sanitization (false positive in a test containment check). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: address Copilot review on HandBrake steps - Reword final step: upload as a new asset (the Edit modal has no upload control) - Rename stale-clear test to mention error_handbrake too Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Viktor Petersson
|
5d21924080 |
fix(celery): don't report the by-design codec rejection to Sentry (#3041)
The upload codec/resolution gate raises ``UnsupportedVideoCodecError`` as a deliberate, operator-facing rejection — it's surfaced in the UI as a "Failed" pill plus a copy-pasteable ffmpeg re-encode recipe via ``_NormalizeAssetTask.on_failure``. It's an expected outcome (e.g. Pi 5 has no H.264 HW decode block, an unknown arm64 board can't certify any codec), not a fault, so it shouldn't reach Sentry — but every rejection was landing there as an unhandled task error (Sentry ANTHIAS-1J, ANTHIAS-20). List it in the video task's ``throws``: Celery then logs it at INFO without a traceback, and sentry-sdk's CeleryIntegration skips ``task.throws`` exceptions (``_capture_exception`` returns early on ``isinstance(exc, task.throws)``), so the gate stops flooding Sentry. ``on_failure`` still runs, so the operator-facing error pill and recipe are unchanged. Regression test asserts the video task declares it in ``throws`` and the image task (which never raises it) does not. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Viktor Petersson
|
10c68b26cc |
feat(viewer,build,balena): add arm64/Qt6 pi3-64 board and the Rock Pi 4 fleet; keep 32-bit pi3 as legacy (#2985)
* feat(viewer,build): add arm64/Qt6 pi3-64 board; keep 32-bit pi3 as legacy Revises issue #2906 Phase 2. The original plan (delete the Qt 5 toolchain, force Pi 2/Pi 3 onto Qt 6) is abandoned: Qt 5 was fixed up on master and stays. Instead, add a NEW board target `pi3-64` — a 64-bit (arm64) Qt 6 viewer image for Raspberry Pi 3 hardware on a 64-bit OS — as its own image stream, disk image, and balena fleet. The legacy 32-bit armhf/Qt5 `pi3` board is left untouched and flagged as legacy/maintenance. pi3-64 mirrors the existing `pi4-64` path (Qt 6, eglfs_kms; video played in-process by AnthiasViewer's QtMultimedia pipeline — QMediaPlayer + the ffmpeg/libavcodec backend with V4L2 HW decode, no external player). VideoCore IV is H.264-only HW decode. Board selection is by `uname -m`: a Pi 3 on a 64-bit OS gets `pi3-64`, a 32-bit OS keeps `pi3` (the model string is identical on both arches). - image_builder: pi3-64 build params (arm64) + is_qt6; constants. - Dockerfile.viewer.j2 + start_viewer.sh: pi3-64 shares the pi4-64 eglfs KMS path; renamed board-agnostic eglfs-kms-pi4.json -> eglfs-kms.json. - Detection: install.sh / upgrade_containers.sh (aarch64 Pi 3 -> pi3-64). - Runtime: media_player force_mpv set (selects MPVMediaPlayer, the QtMultimedia D-Bus shim); processing codec grid {'h264'}. - CI: docker-build matrix + mirror-latest-tags. - Balena (fleet screenly_ose/anthias-pi3-64, device type raspberrypi3-64): disk-image + manual-deploy workflows, balena_ota_deploy.sh, balena_fleet_maintenance.py, balena_unpin_devices.py, deploy_to_balena.sh, balena-host-config.json. - Pi Imager: SUPPORTED_BOARDS += pi3-64 (non-maintenance); pi3 stays legacy. - Docs + tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(website): link the Pi 3 (64-bit) bullet like its siblings Copilot review: the list is introduced as 'links to the images', so the new pi3-64 entry should be navigable like the surrounding bullets. Link the label to the release-images section. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(balena): add the Rock Pi 4 fleet (screenly_ose/anthias-rockpi4) Wires the anthias-rockpi4 balena fleet (device type rockpi-4b-rk3399) into the OTA deploy + disk-image pipeline. The fleet has no board-specific image build: it runs the generic arm64 containers, so bin/balena_ota_deploy.sh / bin/deploy_to_balena.sh map the rockpi4 board to the <short-hash>-arm64 image tags (and strip the /dev/vchiq mount — no VideoCore on RK3399), and the disk-image preflight verifies the arm64 images exist. Root-cause fix for the fleet's codec gate: balena ships no anthias_host_agent service, so host:board_subtype was never published and resolve_device_key() stayed 'arm64' — whose HW-decode set is empty, rejecting every video upload. The model-string → subtype table moves to the dependency-free anthias_common.device_helper.detect_board_subtype (single source, imported by host_agent), and anthias_common.board.get_board_subtype now falls back to reading /proc/device-tree/model in-container when Redis has no value. The device tree is kernel-global — the same mechanism get_device_type has always used for Pi detection — so the rockpi4 fleet resolves its {h264, hevc} envelope without a host-side daemon, and compose installs whose host_agent died self-heal too. - build-balena-disk-image.yaml: rockpi4 in both matrices, fleet + rockpi-4b-rk3399 image cases, arm64 images in the preflight check. - deploy-balena-manual.yaml: rockpi4 board option. - balena-host-config.json: rockpi4 declared {} (config.txt is RPi-only; the reconcile hard-fails on a missing key). - balena_fleet_maintenance.py / balena_unpin_devices.py: fleet added. - tests: get_board_subtype Redis-first + device-tree-fallback order; detect_board_subtype patch targets follow the move. - docs: board-enablement, balena-fleet-host-config, installation-options. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Viktor Petersson
|
b57c1a16a7 |
feat(viewer,server): 1 GB SBC enablement — low-RAM degradation gates (#2915)
Boards with < 1.5 GiB MemTotal (Pi 2/Pi 3 1GB, Pi 4 1GB, Rock Pi 4 1GB, generic-arm64 1GB SKUs) OOM-cycled when QtMultimedia loaded a 4K HEVC asset alongside the two QtWebEngine renderers introduced by #2905. On-device repro on a 1 GB Rock Pi 4 confirmed `global_oom` on the container's bash process followed by a restart loop, plus the kernel keeping sshd in banner-exchange until power-cycle. This patch puts the device into a graceful degraded mode before the OOM cascade fires. * bin/upgrade_containers.sh — exports ANTHIAS_LOW_RAM=1 when TOTAL_MEMORY_KB < 1572864 (1.5 GiB), 0 otherwise. Threshold cleanly splits the 1 GB SKUs from the 2 GB+ SKUs in the supported fleet. * docker-compose.yml.tmpl — forwards ANTHIAS_LOW_RAM to the viewer container. * anthias_host_agent — publishes host:total_mem_kb to Redis alongside the existing host:board_subtype so server-side gates can read MemTotal without re-opening /proc/meminfo themselves. * anthias_common.board — adds LOW_RAM_THRESHOLD_KB, get_total_mem_kb(), is_low_ram_device() helpers. * anthias_webview (view.cpp) — when ANTHIAS_LOW_RAM=1, aliases webView2 onto webView1 so the rest of the dual-buffer logic still runs but never spawns a second Chromium renderer (~100 MB physical RAM saved per device). UX: page swap is in-place with a brief blank during load, no preloaded crossfade. * anthias_server.processing — extends the codec gate with a low-RAM 1080p resolution cap. Above-1080p uploads on low-RAM boards are rejected at upload with the existing recipe machinery, extended to inject `-vf scale=1920:1080:force_original_aspect_ratio=decrease` so the operator's re-encode lands within the envelope. If an upload fails BOTH codec and resolution gates, the codec message wins but the recipe folds in the downscale so a single re-encode satisfies both. * Diagnostics page — Memory card surfaces a "Low-RAM mode" badge with the threshold MiB so operators can see why the device degraded. /api/v2/info's `memory` field gains a `low_ram: bool` for API clients. * docs/board-enablement.md — rewrote the stale `--hwdec=drm-copy` / per-codec dispatch text (removed in #2905); documented the known rkvdec mainline limitation on Armbian 6.18 (HEVC stateless engages via `-hwaccel drm` but produces decode errors; H.264 has no v4l2_request binding in +rpt1 7.1.3) and the new low-RAM mode. Tests cover the four matrix cells (low/high-RAM × in/over-cap), the recipe shape with and without `cap_to_1080p`, the cap defence against unknown / zero dimensions, host_agent's MemTotal parser, and the API endpoint's new low_ram field. Out of scope (separate work): HEVC HW-decode on arm64 — depends on an upstream rkvdec driver fix landing in Debian-shipped Armbian kernels; Anthias does not maintain its own kernel/distro. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Viktor Petersson
|
57b4f25c77 |
feat(viewer,server): per-board HW decode dispatch + codec gate on upload (#2885)
* perf(viewer): pi4-64/pi5 use mpv --vo=gpu --gpu-context=drm On Pi the connector's preferred mode is usually 4K (most modern TVs report 3840x2160 in their EDID), and the previous --vo=drm path ran a CPU zimg upscale from 1080p source to that 4K output. On a 4-core A72 that's the bottleneck — mpv VO drops 59-75 frames per 30s on a stock 1080p H.264 signage clip. Pi5's A76 is faster but the same upscale path is still the limit. Switching the VO to GL with the DRM context (mpv --vo=gpu --gpu-context=drm) hands the upscale to the V3D and leaves everything else identical — mpv still owns DRM master, still reads --drm-mode=1920x1080@60 (kept), still runs in --vd-lavc-threads=4 software decode (mpv 0.40 in Debian Trixie has v4l2m2m-copy but not v4l2request, so --hwdec=auto-safe falls back to software on this asset; that hasn't changed). Measured on a 4K-connected Pi4-64 Rev 1.5, same clip, same 30 s window: --vo=drm : 59-75 vo drops / 30 s --vo=gpu --gpu-context=drm (this patch) : 3-6 vo drops / 30 s `decoder-frame-drop-count` is 0 in both — the regression was purely on the VO side, and shifting scaling off the CPU is what buys the headroom. x86 (cage + --gpu-context=wayland) is unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * perf(viewer): drop --drm-mode pin on Pi4-64/Pi5 under --gpu-context=drm The previous commit moved Pi4-64/Pi5 to `mpv --vo=gpu --gpu-context=drm` but kept the `--drm-mode=1920x1080@60` pin from the old --vo=drm path. On-device testing showed the pin *hurts* throughput under GBM: 294 vo drops/30s with the pin, 3-6 without, on the same 4K-connected Pi4 and the same H.264 clip. The pin existed in the first place to dodge CPU zimg upscale to 4K, which the A72 couldn't keep up with on the legacy --vo=drm path. Under --gpu-context=drm the V3D does the scaling for free at the connector's preferred mode, so the workaround is no longer needed and is in fact harmful. `--vd-lavc-threads=4` stays — software decode under --hwdec=auto-safe (mpv 0.40 has v4l2m2m-copy but not v4l2request) still benefits from explicit threading. Verified on a 4K-connected Pi4-64 across H.264 (30/24 fps) and HEVC clips: 2-6 vo drops/30s in every case. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(viewer): consolidate Qt6 boards onto cage + Wayland, pin Pi 4 to 1080p Folds in PR #2883: Pi 4-64 / Pi 5 now run under cage with mpv on --vo=gpu --gpu-context=wayland, joining x86 and arm64 on a single Wayland-based display stack. Drops the --vo=drm legacy path entirely from MPVMediaPlayer. Qt 5 boards (pi2 / pi3) stay on linuxfb via VLCMediaPlayer — out of scope here. Replaces the perf branch's `--vo=gpu --gpu-context=drm` standalone fix with the consolidated cage path. The previous standalone finding (3-6 vo drops / 30 s on Pi 4 at 4K) was a Pi-without-cage optimization; once Pi runs under cage like every other Qt6 board, the same trick applies via wayland but cage's composite step adds its own pass and the V3D on Pi 4 can't keep up at 4K (738 vo drops / 30 s measured at native 4K under cage). Fix: move the 1080p mode pin one layer up from app code to host config — the new ansible/.../cmdline.txt.j2 conditional appends `video=HDMI-A-1:1920x1080@60 video=HDMI-A-2:1920x1080@60` when `device_type == 'pi4-64'`. With output pinned to 1080p there's no upscale anywhere in the pipeline, matching the bandwidth profile of today's --vo=drm production setup. Pi 5 / x86 / arm64 keep the connector's preferred mode (typically 4K). Pi 5's V3D 7.1 has roughly 2× Pi 4's throughput; x86 iGPUs handle 4K via VAAPI; arm64 SBC perf varies by SoC. Other notable changes folded in from #2883: * tools/image_builder/utils.py — `cage` + `qt6-wayland` move out of the per-board branch into the shared is_qt6 block. `wlr-randr` (was x86-only) goes in the shared block too since rotation now happens via wlr-randr on every Qt6 board. `va-driver-all` stays x86-only (no VAAPI on Pi / ARM SoCs). * docker/Dockerfile.viewer.j2 — QT_QPA_PLATFORM=wayland gated on is_qt6 instead of board in ('x86', 'arm64'). * bin/start_viewer.sh — case on DEVICE_TYPE: every Qt6 board takes the cage + sudo path. Pi2 / Pi3 stay on the legacy direct-sudo path. * src/anthias_viewer/media_player.py — single --vo=gpu --gpu-context=wayland for all reachable device types. The per-board rotate_args block is gone: every Qt6 device inherits the transform from cage via wlr-randr, so mpv would double-rotate if it set --video-rotate. * tests/test_media_player.py — parametrised tests for all four Qt6 boards (x86, arm64, pi4-64, pi5) hitting the same VO path; rotation tests assert mpv *never* sets --video-rotate under cage. * website/data/faq.yaml — rotation entry points at Settings page / wlr-randr; resolution entry calls out the Pi 4 1080p pin. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(ansible): propagate tags into boot.yml include_tasks The `Configure boot partition` task in system/tasks/main.yml was tagged `touches-boot-partition` / `raspberry-pi` but those tags weren't propagated to the tasks inside boot.yml — Ansible's default include_tasks behaviour matches the include against --tags but leaves the included tasks tag-less, so they get filtered back out. Running `ansible-playbook ... --tags touches-boot-partition` therefore did nothing. Use the explicit `apply: tags:` form so the include's tags are copied onto each task in boot.yml. With this, the standalone "re-render boot config" workflow actually works, which matters on Pi 4 now that the 1080p HDMI mode pin in cmdline.txt.j2 needs to land without re-running the whole playbook. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(viewer): keep Pi 4 on linuxfb; only Pi 5 / x86 / arm64 go cage On-device testing on a Pi 4 Model B Rev 1.5 with a 4K HDMI display showed cage+wayland is fundamentally too heavy for the V3D 6.0: --vo=drm (existing, no cage) : 59-75 drops/30s --vo=gpu --gpu-context=drm (no cage, GPU scale): 3-6 drops/30s --vo=gpu --gpu-context=wayland (cage, even at : 730+ drops/30s, 1080p HDMI cmdline pin to avoid 4K scale) mpv at 99% CPU running ~1/4× real time The 1080p HDMI pin doesn't recover Pi 4 — cage's composite pass costs more than the V3D 6.0 has spare bandwidth for, regardless of output resolution, with the webview running in the background or not. Pi 5's V3D 7.1 has roughly 2× the throughput and is expected to keep up; x86 / arm64 already shipped on cage and remain unchanged. Net result: * Pi 4-64 stays on Qt linuxfb (no compositor) with mpv on --vo=gpu --gpu-context=drm. mpv writes straight to KMS via libgbm and lets the V3D do video scaling — keeping the standalone perf-branch finding that drops from 59-75 → 3-6 on the same clip. * Pi 5 / x86 / arm64 stay (or move) onto cage + qt6-wayland + wlr-randr with mpv on --vo=gpu --gpu-context=wayland. * Pi 2 / Pi 3 stay on the Qt5 + VLC + linuxfb track they were already on. * The Pi 4 1080p HDMI cmdline pin added in the previous commit is reverted (no longer needed without cage). * Rotation handling: mpv emits --video-rotate=N on Pi 4 (no compositor to apply the transform) and skips it on the cage boards (wlr-randr handles it there). Goal-wise this is the partial-consolidation we agreed to as last resort: three of four Qt6 boards share one Wayland stack, Pi 4 keeps the framebuffer path for as long as the V3D 6.0 + mpv 0.40 combo lacks the headroom. Pi 4 remains in scope for revisiting once mpv ships the v4l2request hwdec. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(viewer): mirror host render-GID for all Qt 6 boards, not just cage mpv uses /dev/dri/renderD128 for --vo=gpu on every Qt 6 board now — wayland (cage path on x86 / arm64 / pi5) and drm (linuxfb path on Pi 4) both go through Mesa GL. The render-GID mirror was inside the cage branch of start_viewer.sh, so Pi 4's mpv ran as viewer user, hit the render node owned by GID 992, got "Permission denied", and bailed with "Failed initializing any suitable GPU context!". Hoist the render-GID setup above the per-board case so it runs for every Qt 6 board. cage / linuxfb branching stays as-is. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(viewer): Pi 4 stays on --vo=drm (Qt linuxfb DRM master contention) Earlier commits switched Pi 4 to mpv --vo=gpu --gpu-context=drm based on a 3-6 vo-drop/30 s measurement. That test was run as root in a fresh container — no Qt linuxfb in the picture. In the production viewer where AnthiasWebview holds the framebuffer via Qt linuxfb, --vo=gpu fails: failed to open /dev/dri/renderD128: Permission denied [vo/gpu/drm] Failed to acquire DRM master: Permission denied [vo/gpu] Failed initializing any suitable GPU context! Error opening/initializing the selected video_out (--vo) device. Video: no video Mesa GBM holds DRM master persistently and contends with Qt linuxfb's framebuffer use. mpv's classic --vo=drm has its own master juggling (briefly grab → render → drop) that coexists fine with linuxfb — that's why master's existing Pi 4 config works. Revert Pi 4 mpv flags to the production master config: --vo=drm --drm-mode=1920x1080@60 --vd-lavc-threads=4 The standalone perf-finding from this branch's earlier history turns out not to apply in production; retracted from the roll-up. Pi 5 / x86 / arm64 unchanged (they're on cage + --vo=gpu --gpu-context=wayland, which has its own DRM master flow via cage). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(viewer): cage opens on the first connected connector, not HDMI-A-1 Without `-o`, cage uses whatever output the DRM backend enumerates first — typically HDMI-A-1 on Pi 5 (closer to USB-C) and the on-board panel / first HDMI on x86 / arm64. If the operator plugs into the *other* port (Pi 5 HDMI-A-2, or any DP connector on x86), cage renders to a disconnected connector and the screen stays black. start_viewer.sh now iterates /sys/class/drm/card*-*, picks the first connector whose status reads "connected", strips the cardN- prefix to get the bare name cage expects (HDMI-A-1, HDMI-A-2, DP-1, eDP-1, …), and passes it via `-o`. Falls back to letting cage pick if nothing is connected yet — the display may come up via HPD after cage starts, or this is a build/CI host with no display at all. Caught while end-to-end testing on the rig: Pi 5 cable on HDMI-A-2 went to a black screen even though `cat /sys/class/drm/card1-HDMI-A-2/status` reported "connected" and cage / the viewer were running. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(viewer): mpv from apt.raspberrypi.com on Pi 4 / Pi 5, hwdec auto-copy Stock Debian Trixie's mpv 0.40 is compiled without `v4l2request` hwdec, so Pi 5's Hantro stateless decoder is invisible to it and mpv falls back to software decode for every H.264 / H.265 source. Pi 4's V4L2 M2M decoder is reachable via `v4l2m2m-copy` but mpv's `--hwdec=auto-safe` whitelist explicitly excludes that method, so auto-detect picked software there too. Two changes, applied together because they only make sense together: * Pi 4 / Pi 5 viewer images now pull mpv (and the FFmpeg library family it depends on) from `archive.raspberrypi.com/debian trixie main`. The Pi-tuned build ships `v4l2request` hwdec (Pi 5) and a maintained `v4l2m2m-copy` (Pi 4). An apt-pin restricts the Pi repo to the mpv + libav* packages only, so curl / ca-certificates / etc. continue to come from stock Debian and the rest of the image stays on the same baseline. * `MPVMediaPlayer.play()` switches `--hwdec=auto-safe` → `--hwdec=auto-copy`. auto-copy is the same family but with a broader whitelist that *includes* the v4l2-family copy hwdecs. Net effect: x86 still picks vaapi-copy (unchanged), Pi 4 picks v4l2m2m-copy, Pi 5 picks v4l2request, arm64 falls through to software (no v4l2request in stock Debian mpv, no vendor-tuned Rockchip plugin in stock either — Tier-2 follow-up). Plus an `ANTHIAS_DEBUG_DROPS=1` env knob: when set on the viewer container, mpv's stdout/stderr go to `/data/.anthias/mpv.log` (host-bound) instead of `/dev/null`, and `--no-terminal` is dropped so the status line ("AV: ... Dropped: N") is emitted. Lets us read per-asset frame-drop counts straight from the production viewer pipeline (no custom harness, no rebuild) during the test-grid runs. Default (unset) preserves the silent behaviour. Also: drops the `cage -o <connector>` autodetect attempt — cage 0.1.x in Trixie doesn't accept `-o`, just `-m last`. Use that instead so cage opens on the most-recently-connected output regardless of HDMI-A-N enumeration order. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(viewer): use deb-packaged Pi keyring for archive.raspberrypi.com apt update against http://archive.raspberrypi.com/debian trixie was failing in the Pi 4 / Pi 5 viewer image builds: Sub-process /usr/bin/sqv returned an error code (1): Signing key on CF8A1AF502A2AA2D763BAE7E82B129927FA3303E is not bound: No binding signature at time … Policy rejected non-revocation signature (PositiveCertification) requiring second pre-image resistance SHA1 is not considered secure since 2026-02-01 Pi's bare `raspberrypi.gpg.key` URL still serves the original 2012-vintage RSA 2048 key with SHA1 binding signatures that Trixie's sqv refuses to certify under the post-2026-02-01 crypto policy. The deb-packaged keyring inside `raspberrypi-archive-keyring_2025.1+rpt1_all.deb` ships the *same* key fingerprint but with rebuilt binding signatures that sqv accepts — that's the keyring Pi OS Trixie itself installs, which is why `apt update` against this exact repo works on a real Pi 5 device today. Fetch the deb directly with curl, extract its bundled `.pgp` keyring, and point `signed-by=` at the installed copy. The pin block restricts what packages the Pi repo can supply (mpv + libav* + ffmpeg + libpostproc — the FFmpeg family), so the rest of the image keeps its stock-Debian baseline. Also extend the pin to cover libpostproc* and ffmpeg, since mpv's apt deps drag those into the Pi-tagged version on install; without the pin extension, apt rejected the resolve with "broken packages". Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(viewer): per-codec hwdec on Pi via Lua hook mpv 0.40's `--hwdec` accepts a single value at startup, so we can't ask it to try v4l2m2m-copy for H.264 *and* drm-copy for HEVC out of the box. The Pi-tuned mpv from archive.raspberrypi.com supports both hwdec methods but each covers a different codec subset: * v4l2m2m-copy — Pi 4's V3D V4L2 M2M decoder. H.264 works; Pi 5's Hantro G2 is V4L2-stateless-only so this no-ops there. * drm-copy — FFmpeg's `v4l2_request_hevc` hwaccel. HEVC only, works on both Pi 4 and Pi 5. Add a small `on_load` Lua hook (inlined as `_PI_HWDEC_LUA`, written to /tmp on first play(), loaded with `--script=`) that checks `video-codec-name` and picks the right hwdec at file open. Net effect: Pi 4 H.264 → v4l2m2m-copy (HW) Pi 4 HEVC → drm-copy (HW) Pi 5 H.264 → v4l2m2m-copy (no device, falls back to SW — only path until mpv re-adds v4l2_request_h264 hwdec) Pi 5 HEVC → drm-copy (HW) The base `--hwdec=auto-copy` startup value still applies on x86 / arm64 (vaapi-copy on Intel/AMD; software fall-back on Rockchip), where the hook isn't loaded. Verified on real hardware: $ mpv ... --script=/tmp/anthias-pi-hwdec.lua test_hevc.mp4 [pi-hwdec] codec=hevc -> hwdec=drm-copy Using hardware decoding (drm-copy). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(viewer,server): HW-decode everywhere on Pi 4 / Pi 5 / x86 The previous per-codec Lua hook in media_player.py was a silent no-op: mpv's video-codec-name property is empty at every script event before hwdec init (on_load, on_preloaded), so --hwdec=auto-copy leaked through. auto-copy's upstream whitelist excludes v4l2m2m-copy, so H.264 on Pi 4 fell back to software despite the V3D V4L2 M2M decoder being available. Viewer (src/anthias_viewer/media_player.py) - Replace the Lua hook with ffprobe-driven dispatch from Python at launch time. ffprobe is in the viewer image; the call is ~50 ms. - Per-board mapping: Pi 4 → {h264: v4l2m2m-copy, hevc: drm-copy}; Pi 5 → {hevc: drm-copy}. Pi 5 H.264 falls back to auto-copy because mpv has no v4l2-request H.264 hwdec for the Hantro G1, and passing v4l2m2m-copy there just logs "Could not find a valid device" before SW-falling-back. - Live-verified on Pi 4: "Using hardware decoding (v4l2m2m-copy)" for 1080p H.264 and "Using hardware decoding (drm-copy)" for HEVC at 1080p and 4K. Asset processor (src/anthias_server/processing.py) - Pi 5 profile drops H.264 from passthrough_video_codecs — Pi 5 has no mpv H.264 HW path, so H.264 uploads must transcode to HEVC at upload time to keep the HW-decode-everywhere contract. - Pi 4 profile adds passthrough_video_max_pixels for H.264, capped at 1080p (1920*1080). 4K H.264 clears the codec gate but the V3D H.264 envelope tops at 1080p60, so the cap forces it through a libx265 re-encode at upload time. HEVC keeps no cap (the dedicated HEVC block handles 4Kp60). - _ffprobe_summary now returns video_pixels alongside codec / container / audio_codec; _video_can_passthrough enforces the per-codec pixel cap when the profile declares one. Tests - test_media_player.py: new per-board hwdec tests (Pi 4 H.264 → v4l2m2m-copy; Pi 5 H.264 → auto-copy; both → drm-copy for HEVC; auto-copy fallback when ffprobe fails; no probe on x86 / arm64). - test_processing.py: matrix tests updated to include video_pixels; parametrised rows now exercise Pi 5 H.264-no-passthrough and the Pi 4 4K H.264 cap. New end-to-end tests prove _run_video_normalisation transcodes Pi 5 H.264 → HEVC and Pi 4 4K H.264 → HEVC. Docs (docs/board-enablement.md, new) - Goal + per-board HW-decode capability table. - Asset processor codec policy spelled out as a contract. - BBB test bed recipe (source clips, libx265 transcode commands, ANTHIAS_DEBUG_DROPS=1, mpv.log slicing). Follow-up: Pi 5 4K HEVC HW The Hantro G2 decoder can't allocate 4K dst buffers from Pi 5's default 64 MB CMA ("v4l2_request_hevc_start_frame: Failed to get dst buffer") and SW-falls-back. Adding cma=512M to the kernel cmdline does NOT work — the kernel takes the cmdline value over the device-tree linux,cma node, orphaning rpi-hevc-dec ("Failed to probe hardware -517") and unpopulating /dev/video*, which kills HEVC HW at every resolution. The right fix is a dtparam/dtoverlay in /boot/firmware/config.txt that resizes the existing DT-declared region without orphaning the codec's reserved-mem reference. Until that lands, the pi5 profile should downscale 4K → 1080p HEVC. Documented in cmdline.txt.j2 and docs/board-enablement.md. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test(viewer,server): mock _probe_video_codec; fix mypy on Popen IO types CI failures on the previous commit ( |
||
|
Viktor Petersson
|
d2ebdd5450 |
fix(api): v1/v1.1 normalize dispatch + stuck-row reconciler (#2870) (#2873)
* fix(api): dispatch normalize for v1/v1.1 video uploads + reconcile stuck rows (#2870) * Wire normalize-task dispatch into v1 and v1.1 ``AssetListView.post`` via a shared ``dispatch_pending_normalize`` helper; before this, v1/v1.1 video uploads landed in whatever codec the operator sent and the per-board passthrough set silently dropped non-h264/hevc files from rotation forever. * Set ``_pending_normalize='video'`` / ``is_processing=1`` on local video uploads in ``CreateAssetSerializerV1_1``. Image normalisation stays opt-in via v1.2 / v2 — promoting it onto the legacy endpoints would shift synchronous availability semantics for older clients. * Add ``reconcile_stuck_processing`` celery-beat task (10-min cadence, 60-min threshold) to recover rows whose normalize task was lost (worker SIGKILL, OOM, backup restore that bypassed dispatch). Ages rows via ``metadata.processing_started_at`` — stamped by each dispatch helper, granted on first sight for legacy / restored rows. * Drop ``hurry.filesize`` for ``django.template.defaultfilters.filesizeformat``. ``free_space`` on ``/api/v1/info``, ``/api/v1.1/info``, ``/api/v2/info`` changes from ``"15G"`` → ``"15.0 GB"`` (NBSP separator, decimal, full unit). Same change on home-page disk fields. * Bump version to 2026.05.1 in pyproject.toml, package.json, uv.lock. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * test: patch v1.1 validate_uri + pass ext on local-upload fixtures CI failures on the new dispatch tests: * v1 / v1.1 video and image tests hit ``validate_uri('/data/…')`` → 500 ``Invalid file path``. The mixin patch only covered ``serializers.mixins.validate_uri``; v1 / v1.1 import the same helper as ``serializers.v1_1.validate_uri``. Patch both binding sites. * v1.2 / v2 image test expected ``dispatch_normalize_image`` once but got zero. The mixin's rename writes ``<asset_id><ext>``; without ``ext`` in the payload the file lands extensionless and ``needs_image_normalisation`` returns False. Pass ``'ext': '.heic'``. * Also pass ``'ext': '.mp4'`` on the video payload — mixin uses it for the rename destination too; v1 / v1.1 ignore it. * Re-run ``ruff format`` on tests/test_processing.py — pre-push run used ``ruff check`` only. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix(reconciler): handle naive timestamps, single-flight via Redis lock Addresses Copilot review on #2873: * ``_parse_processing_started_at`` now returns ``None`` for naive (no-tzinfo) ISO-8601 strings. The dispatch helpers stamp via ``timezone.now()`` (tz-aware under ``USE_TZ=True``), so a naive value is by definition a hand-edit of metadata — comparing it to the tz-aware ``cutoff`` would raise ``TypeError`` and abort the sweep for every other stuck row. Route it through the stamp-on-first-sight branch instead. * Wrap the reconciler body in a SETNX + Lua-compare-and-delete lock on ``RECONCILE_STUCK_LOCK_KEY`` — same pattern as ``revalidate_asset_urls``. Default Anthias runs one worker with embedded beat (so single-flighted today), but the lock bounds blast radius if a future deploy ever runs two workers with ``-B``. * Rewrite the ``dispatch_download`` docstring: stamping ``processing_started_at`` does *not* let the reconciler re-download — it routes via mimetype='video' through ``normalize_video_asset``, whose ``path.isfile`` guard fails fast and ``on_failure`` writes ``metadata.error_message`` so the operator sees a "Failed" pill instead of a row stuck on "Processing". Tests: naive-timestamp recovery + lock-prevents-overlap + lock-released-after-clean-run. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(processing): make stamp helper public, tighten reconciler types Addresses second Copilot pass on #2873: * Rename ``_stamp_processing_start`` → ``stamp_processing_start``. The single underscore marked it as module-private but it's already called cross-package (anthias_common.youtube.dispatch_download, anthias_server.celery_tasks.reconcile_stuck_processing). Promoting to a public symbol matches its actual API surface; docstring now states the cross-module contract explicitly. * Wrap the read-modify-write in ``transaction.atomic()``. SQLite is single-writer at the file level so the wrap is mostly documenting intent today, but on Postgres / MySQL it produces an actual transaction boundary against a near-simultaneous operator PATCH or worker landing ``error_message``. * Fix the docstring claim of "no-op via the filter().update() pattern" — the code is a SELECT-then-UPDATE; the no-op semantics come from the ``filter().first() is None`` guard, not from the update query shape. * Tighten ``_parse_processing_started_at`` return type from ``Any`` to ``datetime | None``. Pull ``datetime`` into the module-level ``from datetime import …`` so the annotation resolves; drop the redundant local imports inside the parser and reconciler bodies. * Fix the reconciler docstring's "10-min grace window" — the actual grace is ``RECONCILE_STUCK_THRESHOLD_S`` (60 min) from the first-sweep stamp, with the 10-min cadence just dictating how often the row is re-checked. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: honest race acknowledgement on stamp helper + fix OpenAPI example Round-3 Copilot review on #2873: * ``stamp_processing_start`` docstring: replace the implied "transaction.atomic() makes this safe" framing with a clear acknowledgement that this is a read-modify-write race against arbitrary other metadata writers (operator PATCH, normalize task landing error_message). Document why we accept it: stamps fire at *dispatch* time, before the worker picks the task up, so the practical window is microscopic — matches the same race the wider codebase already accepts in ``_set_processing_error`` et al. ``select_for_update()`` would close it on Postgres / MySQL but is a no-op on SQLite (Anthias's only deployed backend), so we'd pay complexity for zero practical safety today. The ``transaction.atomic()`` wrap stays — cheap savepoint that survives a future multi-writer-backend switch. * ``InfoViewMixin`` OpenAPI ``example`` field: update ``free_space`` from ``'10G'`` to ``'10.0 GB'`` so the generated docs match the new ``filesizeformat`` shape (NBSP separator, decimal, full unit). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: tighten reconciler constants' cost + threshold rationale Round-4 Copilot review on #2873: * ``RECONCILE_STUCK_INTERVAL_S`` comment: the previous "one SELECT, one UPDATE per stuck row" undercounted — each per-row branch itself does SELECT + UPDATE inside ``stamp_processing_start``, and the re-dispatch branch may do more on top. Rewrite to reflect the actual cost shape: the initial SELECT plus per-row stamp / dispatch work, with the observation that a healthy fleet has an empty filtered set so the tick costs one SELECT total. * ``RECONCILE_STUCK_THRESHOLD_S`` comment: removed the claim that a row past the threshold "has had its worker time-limit expire" — rows can be stuck for reasons that never involved a worker picking the task up (Redis flake during ``.delay()``, container restart between enqueue and accept). Rewrite to call out both classes of failure so the threshold's role is honest: "if a row has carried is_processing=True for over an hour, something went wrong — recover it." Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Viktor Petersson
|
2d0132b6a4 |
feat(processing): normalise HEIC/HEIF/TIFF images and exotic-codec videos at upload time (#2832)
* feat(processing): add upload-time normalisation for images and exotic-codec videos
Two new Celery tasks run on every fresh upload, mirroring the
``download_youtube_asset`` async pattern:
* ``normalize_image_asset`` converts HEIC / HEIF / TIFF to lossless
WebP via Pillow + pillow-heif, preserving alpha. Other image
formats short-circuit out as a no-op.
* ``normalize_video_asset`` ffprobes the upload, passes through if
it's already H.264/HEVC in an accepted container with a viewer-
friendly audio codec, otherwise transcodes to H.264 + AAC MP4 with
``-threads 2 -preset medium -crf 23`` so two cores stay free for
the on-device viewer.
Both tasks land their output via a staging-file rename, write
``Asset.metadata`` flags (``original_ext``, ``transcoded`` /
``converted``, ``error_message``), and clear ``is_processing`` on
success — or via a custom ``Task.on_failure`` on permanent failure
so a row never stays stuck on the "Processing" pill.
Schema:
* New ``Asset.metadata`` JSONField (default dict) plus migration.
Exposed read+write on ``AssetSerializerV2`` (read-only on v1.x).
Wiring:
* ``CreateAssetSerializerMixin.prepare_asset`` flags ``is_processing``
and stashes ``_pending_normalize`` (``image``/``video``/``None``);
``AssetListViewV2`` and ``AssetListViewV1_2`` dispatch the matching
task after persistence.
* The HTMX ``assets_upload`` view now persists the source extension
on disk so the task can identify the format, replaces the
``probe_video_duration`` hop with ``normalize_video_asset``
(whose passthrough branch is the same probe + duration path),
and dispatches ``normalize_image_asset`` for HEIC/HEIF/TIFF.
* Add-asset modal accepts the wider extension list.
Resource control:
* ``anthias-celery`` worker command wrapped with
``nice -n 19 ionice -c 3`` in compose templates so transcodes
never starve the on-device viewer.
* ``ffmpeg`` invocation pins ``-threads 2`` for the same reason.
Dependencies:
* New: Pillow, pillow-heif (Python); libheif1 in
``base_apt_dependencies`` (~1 MB extracted).
* No changes to ffmpeg/ffprobe — already runtime deps.
Tests:
* ``tests/test_processing.py`` covers both tasks: HEIC/HEIF/TIFF
conversion (incl. uppercase ext, RGBA handling), JPEG no-op,
corrupt-input failure path, six-row passthrough decision table,
exotic-codec → H.264 transcode (mpeg2, mjpeg), MP4-with-non-H264
in-place transcode, ffmpeg timeout/failure/zero-byte cleanup,
ffprobe missing-stream parsing, on_failure metadata write,
prepare_asset routing for HEIC / video / remote URL / JPEG.
* PDF support is deferred to a follow-up — out of scope here per
the issue's "image/video first" framing.
* ci(mypy): include Pillow in mypy group so processing.py type-checks
* fix(processing): address Copilot review comments
* assets_upload now falls back to UploadedFile.content_type and
finally an extension-based classification so HEIC/HEIF/TIFF
uploads still classify on hosts whose mimetypes DB doesn't ship
``image/heic`` mappings.
* _ffprobe_summary derives the container from ffprobe's
``format.format_name`` (a comma-joined synonym list — pick the
first token in the passthrough set) instead of trusting the
filename extension. A ``.bin`` file containing MP4 bytes now
classifies correctly; a ``.mp4`` file containing avi-only bytes
no longer slips into the passthrough branch.
* Zero-byte ffmpeg output now removes the staging file before
raising, mirroring the timeout/error branches above. All three
failure paths share a small _drop_staging() helper so cleanup
stays consistent.
New tests:
* ffprobe summary prefers format_name over filename, with a
deterministic fallback to the extension when format_name is
absent.
* zero-byte transcode output cleans up its staging file (asserts
no leftover ``staging`` files in assetdir).
* assets_upload classifies HEIC via Content-Type when guess_type
returns None.
* test(processing): drop /tmp/ paths from ffprobe summary tests
SonarCloud's python:S5443 flags hardcoded ``/tmp/`` paths as
"publicly writable directory" usage. The flagged lines pass these
strings as labels to ``_ffprobe_summary`` (whose internals are
mocked away in those tests) — only the extension is consumed by
the real code path. Switching to ``fixture.<ext>`` keeps the test
intent clear and silences the security hotspot.
* feat(processing): pick transcode target per board (H.264 vs HEVC)
The previous pipeline always emitted H.264, which is wasteful on
boards whose player can hardware- or software-decode HEVC: a typical
clip re-encodes ~30-50% smaller at perceptual parity. Introduce a
board-profile grid keyed on ``DEVICE_TYPE`` (set at image-build time
in the Dockerfile) so each device gets the codec its on-device player
actually decodes well:
┌──────────┬─────────────────┬──────────────┬──────────────┐
│ Board │ Player │ HEVC OK? │ Target codec │
├──────────┼─────────────────┼──────────────┼──────────────┤
│ pi2/pi3 │ VLC + mmal-vc4 │ no HW, slow CPU │ H.264 │
│ pi4-64 │ mpv + V4L2 HEVC │ HW-decoded │ HEVC │
│ pi5 │ mpv + SW decode │ A76 SW @ 1080p │ HEVC │
│ x86 │ mpv + va/nv/qsv │ HW-decoded │ HEVC │
│ unset │ (dev / unknown) │ assume no │ H.264 │
└──────────┴─────────────────┴──────────────┴──────────────┘
Per-board passthrough also tightens: an HEVC upload to a pi3 device
no longer slips through unplayable — it gets transcoded to H.264.
Conversely, an H.264 upload on pi5 still passes through unchanged
(no point re-encoding to HEVC on a row that already plays).
The ``Asset.metadata['transcode_target']`` field now records the
codec the device wanted, written on both passthrough and transcode
paths so the operator can see "this device wanted hevc, the upload
already was hevc, no work needed" without inferring.
* ``_BOARD_PROFILES`` maps each ``DEVICE_TYPE`` value the image
builder emits to ``{transcode_target, passthrough_video_codecs,
video_args}``. ``_resolve_board_profile`` reads the env var.
* ``_video_can_passthrough`` and ``_transcode_to_target`` accept an
optional profile arg; tests pin the profile per case rather than
mutating env (and one parametrised test still uses env so the
resolve path is exercised end-to-end).
* HEVC encode args include ``-tag:v hvc1`` for broader player compat
(mpv/VLC don't care, but iOS / browsers prefer hvc1 over hev1).
* libx265 CRF 28 chosen as the rough perceptual equivalent of
libx264 CRF 23 — matches the heuristic in libx265's own docs.
Tests:
* New parametrised tests for the codec grid: per-board target codec
resolution, per-board passthrough decision, per-board ffmpeg argv
(including ``-tag:v hvc1`` only on HEVC boards), pi3 + HEVC source
→ libx264 transcode, pi5 passthrough records target codec.
* Updated existing passthrough test to pin DEVICE_TYPE=pi5 since
the default profile is now H.264-only.
* feat(youtube,ui): chain YouTube into the same processing pipeline + error pill
Two unifications driven by the same goal — every "row processing"
state and every "row failed" state should look identical to the
operator regardless of which celery task handled the row.
YouTube → normalize_video_asset chain
-------------------------------------
``download_youtube_asset`` no longer terminates the row's
in-flight state on its own. After yt-dlp lands the .mp4 it:
* writes ``metadata['source']='youtube'`` and
``metadata['source_url']`` so an operator can recover the
original URL after ``name`` is overwritten with the resolved
title,
* leaves ``is_processing=True``,
* dispatches ``normalize_video_asset`` to take over.
The chained pass runs ffprobe and decides per-board passthrough vs.
transcode using the codec grid landed in this PR. That matters
because yt-dlp's ``format_sort: vcodec:h264`` is a *preference*, not
a guarantee — when no H.264 rendition is available yt-dlp falls
back to whatever it can get (vp9 webm, av1, ...). Without the
chain, those downloads would land on a pi3 device unplayable. With
the chain, the same codec grid that protects file uploads protects
YouTube downloads too, and the row carries the same metadata shape
(``original_ext``, ``transcoded``, ``transcode_target``).
Failure-path unification
------------------------
``_DownloadYoutubeTask.on_failure`` now reuses
``processing._set_processing_error`` + ``processing._notify`` —
single source of truth for the error_message contract instead of
two near-duplicate blocks. A failed YouTube download now writes
``metadata.error_message`` (``DownloadError: 404 Not Found`` etc.)
exactly like a failed normalisation does.
UI: error pill
--------------
The asset table row template renders a warn-coloured "Failed" pill
(in the column previously occupied by the active toggle) when
``metadata.error_message`` is populated and ``is_processing`` is
clear. The full message rides along on the title/aria-label so the
operator can hover for context — no extra modal needed. Same shape
as the existing ``processing-pill`` so the column layout stays
stable across in-progress / failed / done states.
Tests
-----
* ``test_download_youtube_asset_success_chains_into_normalize_video``
— happy path now asserts ``is_processing=True`` post-task and
``dispatch_normalize_video`` was called with the asset_id.
* ``test_download_youtube_asset_on_failure_writes_error_metadata``
— replaces the old "clears processing" test; asserts both
``is_processing=False`` and the ExceptionType+message in
``metadata.error_message``.
* Three other YouTube tests updated to mock
``dispatch_normalize_video`` so they don't hit a real broker.
* ``test_asset_row_renders_error_pill_when_processing_failed`` and
``test_asset_row_no_error_pill_when_metadata_clean`` lock in the
template's pill rendering.
* feat(processing): normalise BMP, ICO, TGA, JPEG 2000, and AVIF to WebP
Extends the image-normalisation pipeline to cover the realistic set
of "operator drags an unusual image format into the upload modal"
cases, all handled by Pillow's built-in decoders without a new apt
or wheel dependency:
┌──────────┬────────────────────────────────────────────────────┐
│ Format │ Why we want it converted │
├──────────┼────────────────────────────────────────────────────┤
│ BMP │ Uncompressed; a 4K BMP is ~30 MB vs ~1 MB as WebP. │
│ ICO │ Multi-frame Windows icon; pick the largest, flatten│
│ TGA │ Screenshot tools / game asset exports; no browser │
│ │ support. │
│ JPEG2000 │ .jp2/.j2k/.jpx/.jpc/.jpf — scanner output; no │
│ │ browser support. │
│ AVIF │ Modern phone exports. Chromium 85+ renders AVIF, │
│ │ but the legacy Pi 2/3 Qt5 WebEngine predates it, │
│ │ so converting on upload means one playback path │
│ │ across the fleet. │
└──────────┴────────────────────────────────────────────────────┘
JPEG / PNG / WebP / GIF / SVG remain untouched — already
viewer-friendly *and* well-compressed.
Implementation:
* Extend ``NORMALIZE_IMAGE_EXTS``; the rest of the pipeline already
accepts any extension in this set (RGBA conversion happens inside
``_convert_image_to_webp`` regardless of source format).
* Replace the duplicate extension set in ``assets_upload`` with a
call to ``processing.needs_image_normalisation`` so the source of
truth lives in one place.
* Widen the upload modal's <input accept> attribute.
Tests:
* ``test_image_normalises_to_lossless_webp_across_formats`` is a
parametrised matrix that round-trips each new format end-to-end:
source synthesised via Pillow, runs through
``_run_image_normalisation``, asserts the WebP output decodes
cleanly back to a 16x16 image. Catches both decoder-side
regressions (Pillow drops a format) and writer-side regressions
(RGBA convert mode breaks one source).
* ``test_needs_image_normalisation`` extended to cover every entry
in the new set plus negative cases (.jpg/.png/.webp/.gif/.svg
stay False). Total: 109 image-format assertions.
* fix(processing): address Copilot review on commit
|