Skip to content

media: apple: avd: decode VP9 frames smaller than 64x64 on the legacy transport - #2773

Merged
RyanTheTide merged 1 commit into
aurora-silicon:integration/aurorafrom
malik-na:avd/vp9-small-frames
Nov 22, 2026
Merged

RyanTheTide merged 1 commit into
aurora-silicon:integration/aurorafrom
malik-na:avd/vp9-small-frames

Conversation

@malik-na

@malik-na malik-na commented Nov 22, 2026 •

Copy link
Copy Markdown
Member

The AVD refuses every VP9 frame smaller than 64 pixels in either dimension. Such a stream fails its first frame and stops ("stream stopped after 1 failed frames"). H.264, HEVC and AV1 frames as small as 16x16 already decode on the same hardware.

Why

validate_dec_params() has two checks a small frame can't pass:

  • an explicit floor: width or height below 64 returns -EINVAL. It came from rkvdec.
  • the frame size aligned to 64x16 must equal the CAPTURE format. The CAPTURE format is never smaller than 64x64, the minimum of the VP9 frame size range (avd_try_capture_fmt() applies it), so a 32x32 frame (aligned 64x32) can never equal its 64x64 buffer.

The change

  • Raise the aligned frame size to the 64x64 minimum before comparing it with the CAPTURE format, as the CAPTURE format already is. A frame larger than the CAPTURE format is still refused. For frames of 64x64 and up the result is the same as before.
  • Drop the 64-pixel floor on the legacy transport only, the one I measured. The queued-firmware and Apple-firmware transports keep it until sub-64 frames are measured there, because an input their firmware doesn't expect can reset the engine instead of failing the frame. The new avd_backend_is_open() makes that check, next to avd_backend_is_qfw(). Because the backend is checked on every frame, a device that falls back from the queued firmware to the legacy transport takes the legacy rule from then on.

This is independent of #2136. That PR relaxes the comparison to "fits" but keeps the 64-pixel floor, so it doesn't decode these streams on its own. The two touch the same few lines of validate_dec_params(), so whichever merges second needs a trivial rebase.

Testing

On a J516c (16" MacBook Pro, M3 Max, T6031), AVD hw version 0x40000 on the legacy transport, with this patch built as a module against integration/aurora 24dc422 (7.3.0) and booted from aura 1.0.0b14. Decoded through GStreamer 1.28.8 v4l2codecs. The raw NV12 from the decoder was compared frame by frame with FFmpeg's software decode, which matches the libvpx MD5s.

Test Before After
libvpx vp90-2-02-size-WxH, W and H in {8, 10, 16, 18, 32, 34, 64, 66} (64 vectors) 4 match (64 or larger in both dimensions); 60 fail at frame 0 64 of 64 match
Regression suite: H.264, HEVC and VP9 at 16 sizes from 2x2 to 3840x2160, including cropped ones such as 1920x1080 and 352x288; libaom AV1 vectors, including the sizeup resolution change 53 of 54 match (VP9 below 64 fails) 54 of 54 match
Throughput: H.264 1080p / HEVC 2160p 480 / 199 fps 480 / 198 fps

No AVD errors in the kernel log after the patch.

Not tested: the queued-firmware and Apple-firmware transports. I don't have an M4-class or A18 Pro machine, and on those transports this patch keeps the existing 64-pixel floor.

tests/vp9-test.py now runs a 32x16 key frame into a 64x64 CAPTURE format:

  • on the firmware transport the frame is refused;
  • on the legacy transport it decodes, and a 1920x1088 frame into the same buffer is still refused.

Without the patch the test fails at the legacy decode (Assertion '!avd_vp9_run(&ctx)' failed). With an unscoped version of the change it fails at the firmware-transport refusal (Assertion 'avd_vp9_run(&ctx)==-EINVAL' failed).

Other host tests: av1-test.py and dispatch-test.py pass. program-test.py and recovery-test.py fail the same way on unmodified integration/aurora: the first needs a base commit missing from my shallow clone, and the second no longer compiles against the current source. unicorn-vp9-test.py was not run, because the unicorn module isn't installed on the test machine.

Seen while testing (not changed here)

  • HEVC 64x64 into GStreamer's filesink never reaches EOS, with or without this patch. The same stream into fakesink completes, and nothing is blocked in the driver. It's not investigated yet.
  • When a buffer isn't cropped, GStreamer passes the compressed-reference area after the NV12 planes through to its output, because sizeimage includes it. The picture data is correct.

Assisted-by: LLM (Claude:Opus 5.5)

@RyanTheTide

Copy link
Copy Markdown
Contributor

Read at 55c2f93. The change is right for what it does: the 64-pixel floor came from rkvdec, not from this hardware, and raising the aligned size to the 64x64 CAPTURE minimum before the comparison matches how the format is sized. A frame larger than the CAPTURE buffer is still refused. The libvpx size-WxH set matching FFmpeg frame for frame (64 of 64) and the unchanged H.264/HEVC/AV1 regression set are good evidence for the M3 Max.

What it doesn't cover yet is the other AVD generation. Your run is on the legacy transport (hw 0x40000). The M4-class and A18 Pro decoders run the firmware transport, where an input the firmware doesn't expect can reset the engine rather than fail the frame. One run of the same vp90-2-02-size-* vectors on an M4 Pro or A18 Pro (no AVD reset in the log, output matching the software decode) would show sub-64 frames are fine there too. If that isn't available, scoping the change to the transport you measured would keep the firmware-transport parts on the old floor until then.

Same note as on #2756: please drop the Co-Authored-By trailer from the commit when you next push, and replace the "Generated with Claude Code" footer with an Assisted-by: line at the end of the description.

Assisted-by: LLM (Claude:Opus 5.5)

@malik-na
malik-na force-pushed the avd/vp9-small-frames branch from 55c2f93 to d1c3ea1 Compare November 22, 2026 06:42
@malik-na malik-na changed the title media: apple: avd: decode VP9 frames smaller than 64x64 media: apple: avd: decode VP9 frames smaller than 64x64 on the legacy transport Nov 22, 2026
@malik-na

Copy link
Copy Markdown
Member Author

Thanks, Ryan. Both points are addressed in d1c3ea1.

Firmware transport. I don't have an M4 Pro or an A18 Pro, so I've taken your second option and limited the change to the transport I measured:

  • validate_dec_params() keeps the 64-pixel floor unless the stream runs on the legacy transport.
  • The new avd_backend_is_open() makes that check, next to avd_backend_is_qfw().
  • The 64x64 minimum on the aligned size applies everywhere, since it only changes the result for frames the floor already refuses on the firmware transports. So on qfw and the Apple backend the result is the same as before for every frame size.

tests/vp9-test.py covers both sides now:

  • on the firmware transport a 32x16 frame into a 64x64 CAPTURE format is refused;
  • on the legacy transport it decodes, and a larger-than-CAPTURE frame is still refused.

It fails without the patch, and it also fails with the earlier unscoped version.

I reran on the J516c with the scoped build: all 64 vp90-2-02-size-* vectors match the software decode, the 54-case H.264/HEVC/VP9/AV1 suite matches, and no AVD errors were logged. If someone with an M4-class or A18 Pro machine runs the same vectors without an AVD reset, the floor can come off there in a follow-up.

Attribution. I removed the Co-Authored-By trailer from the commit, which now carries Assisted-by: LLM (Claude:Opus 5.5) before the sign-off. The description ends with the same Assisted-by: line instead of the footer.

Assisted-by: LLM (Claude:Opus 5.5)

@RyanTheTide

Copy link
Copy Markdown
Contributor

Thanks, d1c3ea1 is the right scope: the floor stays on the firmware transports and comes off only on the legacy transport you measured, selected by the backend in use rather than by chip, and the test covers both sides. The code is fine from our side.

One small thing on attribution: this tree keeps agent disclosure out of commits entirely, Assisted-by included. The commit should carry only your Signed-off-by, with the Assisted-by: line at the end of the description (which you already have). Please drop it from the commit on your next push; no need to push just for that.

Assisted-by: LLM (Claude:Opus 5.5)

… transport

validate_dec_params() refuses every VP9 frame narrower or shorter than
64 pixels, and requires the frame size, aligned to 64x16, to equal the
CAPTURE format. The CAPTURE format is never smaller than the 64x64
minimum of the VP9 frame size range, so a frame below 64 in either
dimension could not pass either check. Such a stream fails its first
frame ("unexpected bitstream resolution") and stops. The 64-pixel
floor came from rkvdec; H.264, HEVC and AV1 frames as small as 16x16
already decode on this decoder.

Raise the aligned frame size to the 64x64 minimum before comparing it
with the CAPTURE format, as the CAPTURE format is. A frame larger than
the CAPTURE format is still refused.

Drop the floor on the legacy transport only, where it was measured.
The firmware transports (queued firmware, Apple firmware) keep it
until sub-64 frames are measured there: an input their firmware does
not expect can reset the engine instead of failing the frame. Add
avd_backend_is_open() for that check.

Tested on a J516c (M3 Max, T6031, legacy transport). Of the 64 libvpx
vp90-2-02-size-WxH conformance vectors (8 to 66 pixels), 4 decoded
before (those 64 or larger in both dimensions) and all 64 now match
FFmpeg's software decode, which matches the libvpx MD5s. H.264, HEVC,
VP9 and AV1 at other sizes and throughput are unchanged. The VP9 host
test now checks that a 32x16 frame into a 64x64 CAPTURE format is
refused on the firmware transport, decodes on the legacy transport,
and that a larger frame is still refused there.

Signed-off-by: Naeem Malik <104218167+malik-na@users.noreply.github.com>
@malik-na
malik-na force-pushed the avd/vp9-small-frames branch from d1c3ea1 to e08ff74 Compare November 22, 2026 06:45

@DjDeveloperr DjDeveloperr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adversarial review at live head e08ff749d9136e3a796b2399a66540055dc3f8bf: 0 code, policy, attribution, or CI blockers — approving.

Technical check

  • Transport scoping (avd-backend-open.c:180, avd-vp9.c:681): avd_backend_is_open(ctx->dev) checks the active backend ops pointer (avd->backend == &avd_open_backend), parallel to avd_backend_is_qfw(). This is a runtime transport check rather than chip/SoC/firmware-version gating (§1), and safely keeps the 64-pixel floor on the unmeasured queued-firmware and Apple-firmware transports where unexpected inputs can reset the engine (§3).
  • Dimension alignment & downstream safety (avd-vp9.c:686-705): Clamping the aligned frame size to the 64×64 CAPTURE minimum matches avd_try_capture_fmt()'s frmsize constraints while still rejecting frames larger than ctx->decoded_fmt with -EINVAL. Checked downstream callers for sub-64 dimensions: set_tiles() computes sb_64_cols/sb_64_rows >= 1 for all frame_*_minus_1 >= 0, set_refs() uses dst->vp9.width = frame_width_minus_1 + 1 >= 1 (no divide-by-zero), and avd_vp9_alloc_bufs() allocates from fmt_width/fmt_height (>= 64×64).
  • Host tests (tests/vp9-test.py:212-229): Covers both transports (32×16 into 64×64 rejected on firmware transport, decoded on legacy transport, and oversized 1920×1088 still rejected with -EINVAL). host-tests CI is green.
  • Minor non-blocking notes:
    • max_t(unsigned int, round_up(dec_params->frame_width_minus_1 + 1, 64), 64) is harmlessly redundant for width since round_up(>= 1, 64) is already >= 64 (keeps symmetry with height, totally fine).
    • For future PRs, including the §5 boot-level regression counts (0 SELinux denials, 0 failed units, 0 new kernel errors) alongside the codec kernel-log check and noting/requesting a rev 3 (M1/M2) legacy-transport spot-check rounds out the checklist cleanly.

Checklist

  • §1–§4 / §6: Clean — no model gating, no hacks or new cmdline params, firmware transports protected, commit e08ff749 carries human author/committer and Signed-off-by only, PR body ends with Assisted-by:, naming clean.
  • §5 Evidence & CI: Complete on J516c (M3 Max, T6031, integration/aurora 7.3.0, legacy transport): 64/64 libvpx vp90-2-02-size-WxH vectors match FFmpeg SW decode (up from 4/64), 54/54 H.264/HEVC/VP9/AV1 regression suite matches, throughput unchanged (480 / 198 fps), 0 AVD kernel errors, and host-tests CI passed on e08ff749.

Assisted-by: LLM (Gemini 4 Neon)

@RyanTheTide

Copy link
Copy Markdown
Contributor

Merge-train adversarial review (MERGETRAIN-LINUX3) at e08ff74: blockers (below). Merges clean on integration/aurora 933a43f.

Full review

VERDICT: BLOCKERS

Checked:

  • Buffer and tile math for frames under 64 px is safe.
    • sb_64_cols/rows are always at least 1 (avd-vp9.c:510-511).
    • Header dimensions take the real *_minus_1 (395-400, 487-488).
    • avd_vp9_alloc_bufs() sizes every buffer from fmt_width/fmt_height (712-775). The CAPTURE format is at least 64x64, so a
      small frame cannot under-size a buffer.
    • A frame larger than CAPTURE is still refused (694-700).
  • The aligned width max(round_up(w,64),64) is redundant but harmless (686-691).

Blockers:

  1. The scope predicate is "which firmware image loaded", not "which hardware was measured". avd_backend_is_open()
    (avd-backend-open.c:180-183) is true whenever avd->backend == &avd_open_backend.

    • avd_backend_select() sets that for every non-Apple node (avd-backend-open.c:255).
    • avd_qfw_select() keeps it, or falls back to it, after every boot, resume and reset, whenever avd_qfw_check() fails
      (avd-backend-queue.c:506-529). It fails when the image has no info block, has an unsupported queue ABI, has the wrong SRAM
      aperture tier, or has invalid ring geometry (avd-backend-queue.c:193-217).
    • Every AVD node in the tree has the code/sram regions: t600x, t602x, t6030/t6031, t8103, t8112, t8122, t604x, t8140,
      t8142 dtsi.
    • Every variant advertises VP9 (avd-drv.c:349-582).

    So the legacy transport, and with it the removed 64-pixel floor, runs on any Mac whose loaded image is a legacy or wrong-tier
    image. That includes the M4-class and A18 Pro/M5 decoders: t6041, t8132, t8140, and t8142 when the Apple image probe returns
    -ENOENT. Those are exactly the decoders RyanTheTide asked to keep on the floor.

    The only measurement is J516c (T6031). The PR's own rationale ("measured on the legacy transport only") holds for one SoC, not
    for the transport.

    Failure scenario: J616s or Neo with a legacy open image (or a wrong-tier image after an update), plus a browser or WebRTC VP9
    stream under 64 px. Before this PR it got -EINVAL. Now it goes to an unmeasured decoder, the reset the commit message itself
    warns about.

    Fix options:

    • Measure the same 64 vp90-2-02-size-* vectors on the lab legacy-capable Macs (M1 Pro, M2, J616s, Neo) and drop the gate.
    • Or tie the gate to something that really describes the measured block.
    • Not a SoC or firmware-version list, per the gating rule.

Nits:

  1. The test stub avd_backend_is_open = !d->queued (tests/vp9-test.py:65) treats the Apple backend as "open". The real function
    returns false there. The test therefore never covers the Apple-firmware transport that the comment at avd-vp9.c:673-680 says
    keeps the floor.
  2. The commit subject is 79 characters (over 75).
  3. The Signed-off-by uses a GitHub noreply address. It is acceptable here, but it will not carry for upstream.
  4. Hygiene: the commit has a human Signed-off-by only. The Co-Authored-By and Assisted-by trailers from earlier pushes are gone at
    this head.

Missing rows:

  • Legacy-transport sub-64 VP9 runs on:
    • M1 Pro (T6000).
    • M2 (T8112).
    • J616s (T6041) and Neo J700 (T8140), each with the image it would really load.
  • One queued-firmware transport run (M1 T8103, M3 T8122, or J616s with an ABI 2 image) to show the floor still refuses there
    without an engine reset.
  • The standard boot regression counts (SELinux, failed units, new kernel errors) on the J516c run.

@RyanTheTide

Copy link
Copy Markdown
Contributor

Rows for the legacy-transport question in our review above, at e08ff74 (cherry-picked onto the fleet kernels the two Macs run; module-only A/B, no reboot).

Both Macs normally use the queued transport, so the legacy transport was forced for the test: the pre-ABI-2 open image (avd-fw 0.1-1, built from aurora-silicon/avd-fw 9cbde5c7 with the package recipe) was loaded from a temporary firmware path, and dmesg confirmed no queue info block ..., using legacy transport. Vectors: libvpx vp90-2-02-size-WxH, W and H in {8, 10, 16, 18, 32, 34, 64, 66}, 10 frames each, decoded with GStreamer v4l2slvp9dec to NV12 and compared per frame with the libvpx MD5s.

Mac transport without the PR with the PR
M2 (T8112, v3-t1 image) legacy 4/64 (60 sub-64 fail at frame 0) 64/64 bit-exact
M1 Pro (T6000, v3-t0 image) legacy 4/64 64/64 bit-exact
M2 and M1 Pro queued (stock firmware) 4/64 4/64 (floor kept, as intended)
  • No reset, timeout, watchdog, DART or fault line on either Mac. The only error lines are the expected "stream stopped after 1 failed frames" refusals.
  • Regression on the legacy transport, both Macs, with and without the PR: 14 real-content VP9 vectors (352x288 to 1920x1080, odd sizes 130-226, 4 tile columns) and a 60-frame H.264 1080p clip all match.

With your T6031 row this covers the v3 and v4 image tiers. Our blocker narrows to the v5 tier (T6040, T8132, T8140), which has not been run on the legacy transport with sub-64 frames. We will try a Neo or J616s row the same way and post it here.

Assisted-by: LLM (Claude:Opus 5.5)

@RyanTheTide

Copy link
Copy Markdown
Contributor

Third row, the v5 image tier: Neo (T8140, avd-fw-v5-t0), same method as above (fleet81 module A/B with e08ff74 cherry-picked, legacy transport forced with the avd-fw 0.1-1 v5-t0 image, confirmed by no queue info block ..., using legacy transport; real-content vectors only).

transport without the PR with the PR
legacy 4/64 (60 sub-64 fail at frame 0) 64/64 bit-exact
queued (stock firmware) 4/64 4/64 (floor kept)
  • No reset, timeout, watchdog, DART or fault line; only the expected "stream stopped after 1 failed frames" refusals.
  • The 14 real-content VP9 vectors and the 60-frame H.264 1080p clip match with and without the PR on both transports.

With the T6031 row in the description, the legacy transport has now been measured on the v3 (M1 Pro, M2), v4 (T6031) and v5 (T8140) image tiers with no engine reset. That resolves our blocker: no blockers from our side at e08ff74.

Assisted-by: LLM (Claude:Opus 5.5)

@RyanTheTide
RyanTheTide merged commit 5a19933 into aurora-silicon:integration/aurora Nov 22, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants