Repository navigation
Conversation
… size VP9 predicts motion vectors from the previous frame only if that frame has the same size (UsePrevFrameMvs in compute_image_size(), libvpx's use_prev_frame_mvs). avd_vp9_uses_prev_mv() does not compare sizes, so after a size change the decoder reads the previous frame's MV buffer, which is laid out for the other size. Frames of different sizes already pass validate_dec_params() when they share the 64x16 aligned capture format, e.g. 1920x1080 and 1900x1080. Accepting frames smaller than the capture format (later in this series) makes this common: with that change alone, every VP9 stream that changes size on an inter frame without error resilience fails at the first frame of the new size with "open firmware error 0x8", a timeout and a decoder reset. With this patch no frame fails in the firmware. Record the size of each frame and require the previous one to match. The host test decodes a 1900x1080 frame after a 1920x1088 one and checks that the prior bank's MV buffer is not used. Signed-off-by: Thomas Glanzmann <thomas@glanzmann.de>
set_refs() packs the horizontal reference scale into the upper 16 bits and the vertical one into the lower 16 bits. The decoder reads them the other way round. A reference scaled the same in both directions (2:1, 4:3) decodes either way, so only references whose aspect ratio differs from the frame's were wrong. libvpx's vp90-2-05-resize.ivf (352x288 to 282x173) and vp90-2-18-resize.ivf (310x230 to 248x138) predict from such references. With frames smaller than the capture format accepted (later in this series), they now decode bit-exact against libvpx's MD5s: 10 of 10 and 100 of 100 frames, up from 3 and 94. Pack the vertical scale with AVD_HDR_HEIGHT() and the horizontal one with AVD_HDR_WIDTH(), as the reference size above it is. The host test checks the word for a 1440x1088 reference of a 1920x1088 frame. Signed-off-by: Thomas Glanzmann <thomas@glanzmann.de>
… changes
VP9 clears the segmentation map when the frame size changes
(compute_image_size(); libvpx zeroes last_frame_seg_map). The driver sets
VP9_SEG_RESET only for key frames and after an intra frame without
segmentation, and the decoder keeps part of the map in the seg buffer
across frames. After a size change, a frame that predicts its segment IDs
from the previous frame or keeps them (update_map 0) used the old map,
laid out for the other size.
Set VP9_SEG_RESET for the first frame of a new size, and clear the bank's
seg buffer before it runs. Test streams from libvpx with internal scaling
(VP8E_SET_SCALEMODE: full, half, full size on inter frames) and
segmentation AQ, no error resilience, against libvpx-equivalent software
decoding, with frames smaller than the capture format accepted (later in
this series):
640x360 cyclic refresh (temporal map) 25/40 -> 40/40
variance, complexity (kept map) 10/40, 25/40 -> 40/40
1280x720 cyclic refresh 40/40
variance, complexity 26/40
1920x1080 cyclic refresh 40/40
variance, complexity 28/40, 29/40
At 1280x720 and up, a kept map still goes wrong one or two frames after
the size grows back; where the decoder keeps the rest of the map is not
known. Tested with the legacy transport, which has one VP9 bank. The host
test checks that only a frame size change clears the seg buffer.
Signed-off-by: Thomas Glanzmann <thomas@glanzmann.de>
|
No blockers.
I checked the rest. HW row: a Mac on the queued two-bank transport (J815, or J813 with the open-queue backend), rebased on #2195: the 26 new resize vectors plus your synthetic aq-mode 1/2/3 streams at 640x360 and 1280x720 against libvpx MD5s, and 0 AVC, 0 failed units, 0 new kernel errors. Assisted-by: LLM (Claude:Opus 5.5) |
VP9 inter frames can change the frame size and keep predicting from references of the old size. validate_dec_params() refuses every frame whose aligned size differs from the capture format, so userspace has to reallocate the capture buffers and loses those references. FFmpeg's V4L2 request VP9 hwaccel can instead keep the buffers and decode a frame that fits in place (FFmpeg PR 20847, fix 5), and the decoder already handles what this needs: each reference carries its own size and scale (set_refs()), the frame is written with the format's stride, and the scratch buffers are sized for the format. Accept frames whose aligned size fits the capture format and keep refusing larger ones. The host test runs a 960x540 frame in a 1920x1088 format and refuses a 1984x1088 one. On a J813 with FFmpeg PR 20847 v2 (decoding in place) and the earlier patches of this series, the libvpx VP9 conformance vectors match libvpx's MD5s in 241 of 304 4:2:0 vectors, up from 215. The 26 are the 24 vp90-2-21-resize_inter vectors, vp90-2-05-resize and vp90-2-18-resize. No vector stopped matching, and no frame fails in the firmware. Userspace that renegotiates the format on every size change, such as FFmpeg's current code, decodes as before (215). H.264, HEVC, VP9 and AV1 test clips stay bit-exact. This is not complete support yet. At 1280x720 and up, a stream that keeps its segmentation map without coding it (update_map 0) still decodes wrong pixels, with no error, one or two frames after its size grows back. Before this patch the driver refused those frames. Signed-off-by: Thomas Glanzmann <thomas@glanzmann.de>
1309f5c to
b43dd04
Compare
|
@RyanTheTide thanks.
HW row: after the rebase on #2195, on the J813 with the open-queue backend:
Assisted-by: LLM (Claude:Opus 5.5) |
|
Thanks for b43dd04. The commit message of patch 4 now states the 1280x720 kept-map gap. Thanks also for the plan for the One new thing, from #2773, which touches the same lines of
HW row unchanged: the 26 resize vectors and the aq-mode streams on the open-queue transport, after the #2195 rebase. That row is also what lets the scope come off. Assisted-by: LLM (Claude:Opus 5.5) |
|
@RyanTheTide agreed on both points.
Assisted-by: LLM (Claude:Opus 5.5) |
The AVD now decodes VP9 frames that are smaller than the capture format, so a VP9 stream can change size on an inter frame and keep its references. Today the driver refuses such a frame ("unexpected bitstream resolution") and stops the stream. Accepting it surfaced three bugs in the VP9 path that a size change exposes, and each is fixed in its own commit.
Why
VP9 inter frames can change size and keep predicting from references of the old size: WebRTC and SVC streams do this, as do the libvpx
resize_interconformance vectors. FFmpeg's V4L2 request VP9 hwaccel handles it in FFmpeg PR 20847 (fix 5 of SomeComputerGuy's series): it keeps the capture buffers and decodes a frame that fits in place. On the AVD that path failed, becausevalidate_dec_params()requires the aligned frame size to equal the capture format. mainline rkvdec has the same check.The commits
avd_vp9_uses_prev_mv()didn't check the size. With only the size check relaxed, every non-error-resilient stream crashed the firmware at the first frame of the new size:open firmware error 0x8, a timeout and a decoder reset.set_refs()put the horizontal scale in the upper 16 bits. The decoder reads the vertical one there. This only showed with references whose aspect ratio differs from the frame's:vp90-2-05-resize(352x288 to 282x173) andvp90-2-18-resize.VP9_SEG_RESETfor that frame and clears the bank's seg buffer, which carries part of the map over.Testing
On a J813 (M5 MacBook Air), with this series built as a module against the 7.3 J813 tree. AVD there uses the legacy transport, one VP9 bank. Test frames are compared with libvpx's MD5s, or with FFmpeg's software decode, which is bit-exact with libvpx.
libvpx VP9 conformance vectors, with FFmpeg PR 20847 v2 (decoding in place): 241 of 304 4:2:0 vectors match, up from 215. The new matches are the 24
vp90-2-21-resize_intervectors,05-resizeand18-resize. No vector stopped matching, and no frame fails in the firmware. With FFmpeg's current code, which renegotiates on every size change, the result is unchanged (215).No regressions: H.264, HEVC Main/Main10, VP9 and AV1 test clips stay bit-exact. The 36
resize-fp-tilesvectors match on both FFmpeg paths. 720p streams with segmentation that keep their size stay bit-exact.Synthetic streams: the conformance vectors that change size on inter frames are all error resilient, so they never reach commits 1 and 3. To cover those paths I encoded streams with libvpx's internal scaling (
VP8E_SET_SCALEMODE: full, half, full on inter frames), segmentation AQ and no error resilience. The encoder is below.vp9-scalemode-enc.c (libvpx test stream encoder)
Usage:
vp9-scalemode-enc in.yuv 640 360 40 <aq> 10 25 out.ivfcodes frames 10 to 24 at half size.tests/vp9-test.pyhas a new case for each commit, and each case fails on the commit before it.tools/asahi/run-host-tests.py: 394/394. checkpatch is clean.Known gaps
update_map0) still goes wrong one or two frames after the size grows back. Clearing the seg buffer fixes this at 640x360, but where the decoder keeps the rest of the map isn't known yet. Without this series those frames are refused outright.AV1_REF_SCALE_X). I haven't changed or tested it.cc @Acelogic @DjDeveloperr @RyanTheTide @LiamRay10
Assisted-by: LLM (Claude:Opus 5.5)