SteveGilvarry 7e15324385 fix: corrupted scaled/converted output for rotated monitors — SWScale guessed buffer alignment from width ...

SWScale::Convert chose av_image_fill_arrays alignment by heuristic
(width % 32 ? 1 : 32) for both buffers. Image buffers are always laid
out align-32, so any width not divisible by 32 made Convert read luma
rows 16 bytes short and chroma planes from packed offsets: diagonal
shear plus garbage chroma. Rotating a monitor is what produces such
widths (1280x720 ROTATE_270 -> 720 wide, 3840x2160 ROTATE_90 -> 2160
wide, both % 32 == 16), so every scaled view and every re-encode of a
rotated monitor was corrupted while unrotated monitors (1280/2688/3840
all % 32 == 0) were untouched. The rotate/flip segfault fix exposed
this: before it, rotated planar frames crashed zmc before reaching
Scale.

Alignment is a fact about how a buffer was laid out, not something
derivable from dimensions. Convert now takes explicit in/out alignment:
Image::Scale passes 32/32, the videostore encode path mirrors
get_out_frame's allocation choice, libvnc passes 1 (packed VNC
framebuffer) and 32 (Image WriteBuffer). Remove the unused
SetDefaults/ConvertDefaults API rather than threading alignments
through dead code.

Tests: new Scale regression case on a 720x1280 YUV420P image
(column-banded luma, uniform chroma) fails before the fix exactly as
observed live (sheared rows, V plane reading 0) and passes after.
Full suite 84/84.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

2026-06-13 10:19:09 +10:00

8.7 KiB

Raw Blame History

View Raw