Skip to content

media: apple: avd: decode VP9 frames smaller than the capture format - #2136

Open
sithglan wants to merge 4 commits into
aurora-silicon:integration/aurorafrom
sithglan:thomas/avd-vp9-smaller-frames
Open

sithglan wants to merge 4 commits into
aurora-silicon:integration/aurorafrom
sithglan:thomas/avd-vp9-smaller-frames

Conversation

@sithglan

@sithglan sithglan commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

The AVD now decodes VP9 frames that are smaller than the capture format, so a VP9 stream can change size on an inter frame and keep its references. Today the driver refuses such a frame ("unexpected bitstream resolution") and stops the stream. Accepting it surfaced three bugs in the VP9 path that a size change exposes, and each is fixed in its own commit.

Why

VP9 inter frames can change size and keep predicting from references of the old size: WebRTC and SVC streams do this, as do the libvpx resize_inter conformance vectors. FFmpeg's V4L2 request VP9 hwaccel handles it in FFmpeg PR 20847 (fix 5 of SomeComputerGuy's series): it keeps the capture buffers and decodes a frame that fits in place. On the AVD that path failed, because validate_dec_params() requires the aligned frame size to equal the capture format. mainline rkvdec has the same check.

The commits

  1. Temporal MVs only from a frame of the same size. VP9 forbids using the previous frame's MVs after a size change, and avd_vp9_uses_prev_mv() didn't check the size. With only the size check relaxed, every non-error-resilient stream crashed the firmware at the first frame of the new size: open firmware error 0x8, a timeout and a decoder reset.
  2. Reference scale halves. set_refs() put the horizontal scale in the upper 16 bits. The decoder reads the vertical one there. This only showed with references whose aspect ratio differs from the frame's: vp90-2-05-resize (352x288 to 282x173) and vp90-2-18-resize.
  3. Segmentation map reset on a size change. VP9 clears the map when the size changes. The patch sets VP9_SEG_RESET for that frame and clears the bank's seg buffer, which carries part of the map over.
  4. Accept frames that fit. A smaller frame is decoded into the current buffers, with their stride; a larger one is still refused.

Testing

On a J813 (M5 MacBook Air), with this series built as a module against the 7.3 J813 tree. AVD there uses the legacy transport, one VP9 bank. Test frames are compared with libvpx's MD5s, or with FFmpeg's software decode, which is bit-exact with libvpx.

  • libvpx VP9 conformance vectors, with FFmpeg PR 20847 v2 (decoding in place): 241 of 304 4:2:0 vectors match, up from 215. The new matches are the 24 vp90-2-21-resize_inter vectors, 05-resize and 18-resize. No vector stopped matching, and no frame fails in the firmware. With FFmpeg's current code, which renegotiates on every size change, the result is unchanged (215).

  • No regressions: H.264, HEVC Main/Main10, VP9 and AV1 test clips stay bit-exact. The 36 resize-fp-tiles vectors match on both FFmpeg paths. 720p streams with segmentation that keep their size stay bit-exact.

  • Synthetic streams: the conformance vectors that change size on inter frames are all error resilient, so they never reach commits 1 and 3. To cover those paths I encoded streams with libvpx's internal scaling (VP8E_SET_SCALEMODE: full, half, full on inter frames), segmentation AQ and no error resilience. The encoder is below.

    size AQ / segmentation relaxed check only this series
    640x360 none firmware error 40/40
    640x360 cyclic refresh (temporal) firmware error 40/40
    640x360 variance, complexity (kept map) firmware error 40/40
    1280x720 cyclic refresh not run 40/40
    1280x720 variance, complexity not run 26/40 each
    1920x1080 cyclic refresh not run 40/40
    1920x1080 variance, complexity not run 28/40, 29/40
vp9-scalemode-enc.c (libvpx test stream encoder)
// SPDX-License-Identifier: MIT
/*
 * vp9-scalemode-enc: encode raw I420 frames to VP9 IVF with inter-frame size
 * changes (libvpx internal scaling, VP8E_SET_SCALEMODE), no error resilience,
 * one key frame, and the given AQ mode (1 variance, 2 complexity, 3 cyclic
 * refresh: all use segmentation). Frames [a, b) are coded at half size.
 *
 *   vp9-scalemode-enc in.yuv W H frames aq a b out.ivf
 *
 * Build: cc -O2 vp9-scalemode-enc.c -lvpx -o vp9-scalemode-enc
 */
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <vpx/vp8cx.h>
#include <vpx/vpx_encoder.h>

static void le16(unsigned char *p, unsigned v) { p[0] = v; p[1] = v >> 8; }
static void le32(unsigned char *p, unsigned v) { le16(p, v); le16(p + 2, v >> 16); }

int main(int argc, char **argv)
{
	if (argc != 9) {
		fprintf(stderr, "usage: %s in.yuv W H frames aq a b out.ivf\n", argv[0]);
		return 1;
	}
	unsigned w = atoi(argv[2]), h = atoi(argv[3]), n = atoi(argv[4]);
	int aq = atoi(argv[5]);
	unsigned a = atoi(argv[6]), b = atoi(argv[7]);
	FILE *in = fopen(argv[1], "rb"), *out = fopen(argv[8], "wb");
	vpx_codec_enc_cfg_t cfg;
	vpx_codec_ctx_t c;
	vpx_image_t img;
	unsigned char hdr[32] = "DKIF";
	unsigned count = 0;

	if (!in || !out || vpx_codec_enc_config_default(vpx_codec_vp9_cx(), &cfg, 0))
		return 1;
	cfg.g_w = w;
	cfg.g_h = h;
	cfg.g_timebase.num = 1;
	cfg.g_timebase.den = 30;
	cfg.rc_target_bitrate = 800;
	cfg.g_error_resilient = 0;
	cfg.g_lag_in_frames = 0;
	cfg.kf_mode = VPX_KF_DISABLED;
	cfg.rc_end_usage = VPX_VBR;
	if (vpx_codec_enc_init(&c, vpx_codec_vp9_cx(), &cfg, 0))
		return 1;
	vpx_codec_control(&c, VP9E_SET_AQ_MODE, aq);
	vpx_codec_control(&c, VP8E_SET_CPUUSED, 6);
	vpx_codec_control(&c, VP9E_SET_FRAME_PARALLEL_DECODING, 0);
	if (!vpx_img_alloc(&img, VPX_IMG_FMT_I420, w, h, 1))
		return 1;

	le16(hdr + 6, 32);
	memcpy(hdr + 8, "VP90", 4);
	le16(hdr + 12, w);
	le16(hdr + 14, h);
	le32(hdr + 16, 30);
	le32(hdr + 20, 1);
	fwrite(hdr, 1, 32, out);

	for (unsigned i = 0; i <= n; i++) {
		vpx_image_t *pic = NULL;
		if (i < n) {
			size_t sz = (size_t)w * h * 3 / 2;
			if (fread(img.img_data, 1, sz, in) != sz)
				return 1;
			pic = &img;
			if (i == a || i == b) {
				struct vpx_scaling_mode m = i == a ?
					(struct vpx_scaling_mode){ VP8E_ONETWO, VP8E_ONETWO } :
					(struct vpx_scaling_mode){ VP8E_NORMAL, VP8E_NORMAL };
				vpx_codec_control(&c, VP8E_SET_SCALEMODE, &m);
			}
		}
		if (vpx_codec_encode(&c, pic, i, 1, 0, VPX_DL_GOOD_QUALITY))
			return 1;
		const vpx_codec_cx_pkt_t *p;
		vpx_codec_iter_t it = NULL;
		while ((p = vpx_codec_get_cx_data(&c, &it))) {
			unsigned char fh[12];
			if (p->kind != VPX_CODEC_CX_FRAME_PKT)
				continue;
			le32(fh, p->data.frame.sz);
			le32(fh + 4, p->data.frame.pts);
			le32(fh + 8, 0);
			fwrite(fh, 1, 12, out);
			fwrite(p->data.frame.buf, 1, p->data.frame.sz, out);
			count++;
		}
	}
	le32(hdr + 24, count);
	fseek(out, 24, SEEK_SET);
	fwrite(hdr + 24, 1, 4, out);
	fclose(out);
	fprintf(stderr, "%u frames\n", count);
	return 0;
}

Usage: vp9-scalemode-enc in.yuv 640 360 40 <aq> 10 25 out.ivf codes frames 10 to 24 at half size.

  • Host tests: tests/vp9-test.py has a new case for each commit, and each case fails on the commit before it. tools/asahi/run-host-tests.py: 394/394. checkpatch is clean.

Known gaps

  • At 1280x720 and up, a stream that keeps its segmentation map without coding it (update_map 0) still goes wrong one or two frames after the size grows back. Clearing the seg buffer fixes this at 640x360, but where the decoder keeps the rest of the map isn't known yet. Without this series those frames are refused outright.
  • Only the legacy transport (one bank) was tested. With the queued firmware's two banks, commit 3 clears only the bank of the frame being prepared.
  • The AV1 path packs its reference scale the same way (X in the upper half, AV1_REF_SCALE_X). I haven't changed or tested it.

cc @Acelogic @DjDeveloperr @RyanTheTide @LiamRay10

Assisted-by: LLM (Claude:Opus 5.5)

… size

VP9 predicts motion vectors from the previous frame only if that frame
has the same size (UsePrevFrameMvs in compute_image_size(), libvpx's
use_prev_frame_mvs). avd_vp9_uses_prev_mv() does not compare sizes, so
after a size change the decoder reads the previous frame's MV buffer,
which is laid out for the other size. Frames of different sizes already
pass validate_dec_params() when they share the 64x16 aligned capture
format, e.g. 1920x1080 and 1900x1080.

Accepting frames smaller than the capture format (later in this series)
makes this common: with that change alone, every VP9 stream that changes
size on an inter frame without error resilience fails at the first frame
of the new size with "open firmware error 0x8", a timeout and a decoder
reset. With this patch no frame fails in the firmware.

Record the size of each frame and require the previous one to match.
The host test decodes a 1900x1080 frame after a 1920x1088 one and checks
that the prior bank's MV buffer is not used.

Signed-off-by: Thomas Glanzmann <thomas@glanzmann.de>
set_refs() packs the horizontal reference scale into the upper 16 bits
and the vertical one into the lower 16 bits. The decoder reads them the
other way round. A reference scaled the same in both directions (2:1,
4:3) decodes either way, so only references whose aspect ratio differs
from the frame's were wrong.

libvpx's vp90-2-05-resize.ivf (352x288 to 282x173) and
vp90-2-18-resize.ivf (310x230 to 248x138) predict from such references.
With frames smaller than the capture format accepted (later in this
series), they now decode bit-exact against libvpx's MD5s: 10 of 10 and
100 of 100 frames, up from 3 and 94.

Pack the vertical scale with AVD_HDR_HEIGHT() and the horizontal one with
AVD_HDR_WIDTH(), as the reference size above it is. The host test checks
the word for a 1440x1088 reference of a 1920x1088 frame.

Signed-off-by: Thomas Glanzmann <thomas@glanzmann.de>
… changes

VP9 clears the segmentation map when the frame size changes
(compute_image_size(); libvpx zeroes last_frame_seg_map). The driver sets
VP9_SEG_RESET only for key frames and after an intra frame without
segmentation, and the decoder keeps part of the map in the seg buffer
across frames. After a size change, a frame that predicts its segment IDs
from the previous frame or keeps them (update_map 0) used the old map,
laid out for the other size.

Set VP9_SEG_RESET for the first frame of a new size, and clear the bank's
seg buffer before it runs. Test streams from libvpx with internal scaling
(VP8E_SET_SCALEMODE: full, half, full size on inter frames) and
segmentation AQ, no error resilience, against libvpx-equivalent software
decoding, with frames smaller than the capture format accepted (later in
this series):

  640x360    cyclic refresh (temporal map)    25/40 -> 40/40
             variance, complexity (kept map)  10/40, 25/40 -> 40/40
  1280x720   cyclic refresh                   40/40
             variance, complexity             26/40
  1920x1080  cyclic refresh                   40/40
             variance, complexity             28/40, 29/40

At 1280x720 and up, a kept map still goes wrong one or two frames after
the size grows back; where the decoder keeps the rest of the map is not
known. Tested with the legacy transport, which has one VP9 bank. The host
test checks that only a frame size change clears the seg buffer.

Signed-off-by: Thomas Glanzmann <thomas@glanzmann.de>
@RyanTheTide

Copy link
Copy Markdown
Contributor

No blockers.

  1. drivers/media/platform/apple/avd/avd-vp9.c:865-866: media: apple: avd: VP9: one segmentation map per stream, shared by the banks #2195 (open) replaces the per-bank seg buffer with one per stream (vp9_ctx->seg), because the queued transport otherwise reads the map of the frame before last. Whichever of the two lands second has to rebase this memset onto the shared buffer. On the queued transport the other bank's frame can still be in flight at prepare time: avd_vp9_run_preamble() takes any bank that is not busy (:846-853). A CPU memset of the shared map would then race that frame. With media: apple: avd: VP9: one segmentation map per stream, shared by the banks #2195 the clear must either wait until no frame is in flight or be left to VP9_SEG_RESET (:349-350). Rebasing on media: apple: avd: VP9: one segmentation map per stream, shared by the banks #2195 also closes your "clears only the bank being prepared" gap.
  2. The known 1280x720+ kept-map gap means that, with FFmpeg's in-place path, a stream the driver used to refuse now decodes into wrong pixels with no error. If the cause isn't found before merge, please say so in the commit message of patch 4 so it is not read as complete support.

I checked the rest. avd_vp9_same_size() compares the previous decoded frame. last is updated for every submitted frame on the queued path (:923) and on completion on the legacy path (:1088), which matches the spec's "previous time compute_image_size was invoked". The callers in make_flags1() and the preamble check last.valid first. The ref-scale swap matches AVD_HDR_HEIGHT/AVD_HDR_WIDTH as used for the reference size. The scratch buffers stay sized from the coded format, so a smaller frame fits. The commits carry a human Signed-off-by and no agent trailers.

HW row: a Mac on the queued two-bank transport (J815, or J813 with the open-queue backend), rebased on #2195: the 26 new resize vectors plus your synthetic aq-mode 1/2/3 streams at 640x360 and 1280x720 against libvpx MD5s, and 0 AVC, 0 failed units, 0 new kernel errors.

Assisted-by: LLM (Claude:Opus 5.5)

VP9 inter frames can change the frame size and keep predicting from
references of the old size. validate_dec_params() refuses every frame
whose aligned size differs from the capture format, so userspace has to
reallocate the capture buffers and loses those references. FFmpeg's
V4L2 request VP9 hwaccel can instead keep the buffers and decode a
frame that fits in place (FFmpeg PR 20847, fix 5), and the decoder
already handles what this needs: each reference carries its own
size and scale (set_refs()), the frame is written with the format's
stride, and the scratch buffers are sized for the format.

Accept frames whose aligned size fits the capture format and keep
refusing larger ones. The host test runs a 960x540 frame in a 1920x1088
format and refuses a 1984x1088 one.

On a J813 with FFmpeg PR 20847 v2 (decoding in place) and the earlier
patches of this series, the libvpx VP9 conformance vectors match
libvpx's MD5s in 241 of 304 4:2:0 vectors, up from 215. The 26 are the
24 vp90-2-21-resize_inter vectors, vp90-2-05-resize and
vp90-2-18-resize. No vector stopped matching, and no frame fails in the
firmware. Userspace that renegotiates the format on every size change,
such as FFmpeg's current code, decodes as before (215). H.264, HEVC,
VP9 and AV1 test clips stay bit-exact.

This is not complete support yet. At 1280x720 and up, a stream that
keeps its segmentation map without coding it (update_map 0) still
decodes wrong pixels, with no error, one or two frames after its size
grows back. Before this patch the driver refused those frames.

Signed-off-by: Thomas Glanzmann <thomas@glanzmann.de>
@sithglan
sithglan force-pushed the thomas/avd-vp9-smaller-frames branch from 1309f5c to b43dd04 Compare November 21, 2026 13:38
@sithglan

Copy link
Copy Markdown
Contributor Author

@RyanTheTide thanks.

  1. Shared seg buffer (media: apple: avd: VP9: one segmentation map per stream, shared by the banks #2195):
  2. 1280x720 gap: patch 4 (b43dd04645d5) now says in its commit message that this isn't complete support yet. At 1280x720 and up, a stream that keeps its segmentation map without coding it (update_map 0) still decodes wrong pixels, with no error, one or two frames after its size grows back, and before the patch the driver refused those frames. The cause isn't found yet.

HW row: after the rebase on #2195, on the J813 with the open-queue backend:

  • the 26 resize vectors and the synthetic aq-mode 1/2/3 streams at 640x360 and 1280x720, against the libvpx MD5s;
  • 0 AVC, 0 failed units and 0 new kernel errors.

Assisted-by: LLM (Claude:Opus 5.5)

@RyanTheTide

Copy link
Copy Markdown
Contributor

Thanks for b43dd04. The commit message of patch 4 now states the 1280x720 kept-map gap. Thanks also for the plan for the #2195 memset, which settles point 1 as a merge-order item.

One new thing, from #2773, which touches the same lines of validate_dec_params():

  • media: apple: avd: decode VP9 frames smaller than 64x64 on the legacy transport #2773 keeps the old rule on the firmware transports (open-queue and the Apple backend). On those, an input the firmware does not expect can reset the engine instead of failing the frame, so the 64-pixel floor comes off only where avd_backend_is_open() is true (avd-vp9.c:681 at e08ff74). This PR relaxes the comparison to "fits" on every transport (avd-vp9.c:704). Per the description it was tested only on the legacy transport, and the description itself shows how such inputs can fail there ("open firmware error 0x8", a timeout and a decoder reset before patch 1). Until the open-queue row is in, please scope the relaxation the same way: > with avd_backend_is_open(), != otherwise. The scope can then come off in a small follow-up with that row.
  • Suggested order: media: apple: avd: decode VP9 frames smaller than 64x64 on the legacy transport #2773 first, since it is small and independent, then rebase this onto it and reuse its avd_backend_is_open(). avd-vp9.c merges cleanly into a consistent function: floor only on the legacy transport, aligned size raised to 64, larger frames refused. tests/vp9-test.py conflicts, though, because both add a block at the same place with a different decoded_fmt, so the order of the two blocks needs care when resolving.

HW row unchanged: the 26 resize vectors and the aq-mode streams on the open-queue transport, after the #2195 rebase. That row is also what lets the scope come off.

Assisted-by: LLM (Claude:Opus 5.5)

@sithglan

Copy link
Copy Markdown
Contributor Author

@RyanTheTide agreed on both points.

Assisted-by: LLM (Claude:Opus 5.5)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants