Skip to content

media: apple: prores: hardware ProRes decode on every APR engine - #2396

Closed
RyanTheTide wants to merge 5 commits into
integration/aurorafrom
media/prores-decode-v3-20261120
Closed

RyanTheTide wants to merge 5 commits into
integration/aurorafrom
media/prores-decode-v3-20261120

Conversation

@RyanTheTide

@RyanTheTide RyanTheTide commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Every Apple ProRes engine we drive is a ProRes encode and decode engine, but the driver only exposed the encoder. Decoding ProRes on Linux fell back to software. #1259 added a decoder node last week but never ran on hardware, and it no longer applied to the current driver (T6041/T6050 windows, built-in rows, video10 numbering).

Change

  • The media: apple: add ProRes decode-v0 node #1259 decode-v0 node, rebased onto current integration/aurora. It adds a stateful V4L2 decoder (apple-prores-decode) next to the encoder. Decode jobs share the encoder's ring, and the ring accounting handles variable-size entries and the generation-bit wrap.

  • The decoder node is no longer limited to T6000 and T8140. Every compatible the driver binds (T6000, T6041, T6050, T8140) gets it, with no per-variant flag.

  • Input bounds (9d612f2). The decode descriptor has no input length, so the engine reads as far as the frame's own sizes say. The parser already held the frame size, frame and picture header sizes, picture sizes, slice table and slice sizes to the payload. It now also checks:

    • every slice header: its size and its coded Y/Cb (and, with alpha, Cr) sizes must fit the slice;
    • the picture header's slice count: 0, as our encoder writes, or the count the geometry gives.

    Each field is read once, so the parse itself stays in bounds even if the buffer changes under it. KUnit for frames that point past their buffer is in 00c08be.

Testing

Each Mac ran the branch as an out-of-tree module, swapped in for the in-tree one on its production kernel, with no reboot. Clips came from ffmpeg prores_ks, plus frames from our own hardware encoder. Output was compared against ffmpeg's software decoder.

  • MacBook Neo (T8140, 7.3.0-40): 18 rows, 0 decode errors, 0 DART faults. 1080p 422 HQ decodes at 2,514-2,528 fps to P216 and 3,043-3,134 fps to NV16; 4444 at 1,727 fps; 4K HQ at 665 fps; 4K 4444 at 447 fps.

  • MacBook Pro M1 Pro (T6000, fleet74): the same 18 rows, 0 errors, 0 faults. Every decoded output is byte-identical to the Neo's (16/16 hashes match).

  • Bit-exact with Apple's decoder: on the same M1 Pro, macOS VideoToolbox's 10-bit output equals this driver's 16-bit output rounded to 10 bits for 100 % of samples (422 Proxy and HQ, 4444, 4444 XQ). Our P216/P416 keep the extra precision that VideoToolbox drops.

  • Output is deterministic. It stays byte-identical after 800 jobs (ring wrap) and while an encode stream shares the ring.

  • Against ffmpeg: 8-bit NV16/NV24 output is 99.2-99.5 % exact with a max diff of 1. P216 keeps more than 10 bits, so it is 93-97 % exact with a max diff of 4 at 10-bit. P416 4444 is 92-94 % exact with a max diff of 19 at 12-bit (about 5 at 10-bit).

  • Encoder regression in the same module: 1080p HQ at 1,632 fps on the Neo and 1,532-1,569 fps on the M1 Pro. Same output size as before.

  • KUnit on arm64 QEMU: apple-prores 20/20 pass (the APRI cases plus the decode descriptor, frame parser, interlaced parser and ring accounting cases).

  • MacBook Pro 16-inch M4 Pro (T6041 engine, fleet75): the decoder node now registers on T6041 too, and the same 18 rows pass with 0 errors and 0 faults. Outputs are again byte-identical to the Neo's. This is the fastest engine of the three: 1080p HQ at about 3.1k fps, 4K HQ at 1.0-1.5k fps, 4K 4444 at about 620 fps.

  • Input bounds, J616s (T6040, fleet75), module swap. 20 corrupted frames per clip × 3 clips: picture size, slice table, slice sizes, slice headers and slice count past the buffer, frame size beyond bytesused, bytesused short, frame header past the frame, interlaced flag with no second field.

    • With the bounds commit, 60/60 are rejected before the engine runs: CAPTURE comes back ERROR, the poisoned CAPTURE buffer is untouched, and the next valid frame decodes to the same bytes.
    • Without it, 48/60 are rejected. The 12 that reach the engine show what it does with those fields: it ignores a wrong in-band slice count and decodes normally, and it raises ERROR on a slice header that runs past its slice.
    • Same rows, same 6 clips: byte-identical output and equal fps (within noise) with and without the bounds commit.
  • Input bounds, M1 Pro (T6000, fleet74):

    • The same 60/60 corrupt frames are rejected, with 0 DART/prores errors.
    • The per-slice header check costs 0.1 % at 1080p HQ (2918 -> 2916 fps) and 0.4 % at 2160p (774.6 -> 771.7). This is where uncached source reads cost the most.
    • The 6 clips stay byte-identical.
  • Input bounds, Neo (T8140, 7.3.0-40): the same 60/60 corrupt frames are rejected with no prores errors in dmesg; the 6 clips stay byte-identical with equal fps.

  • KUnit (arm64 QEMU): 28/28 apple-prores cases, including the 8 out-of-bounds cases. Each frame is parsed from an allocation exactly as long as the payload. Also clean under KASAN and UBSAN.

Still to run

  • What the check cannot cover: userspace may rewrite the in-band sizes in its MMAP/DMA-BUF buffer after QBUF. On the J616s, a thread toggling the picture size made the engine hang on the first such job; no output data came back in any of 100 jobs. The watchdog's engine reset then left the engine's DART stream with an invalid translation table ("TTBR invalid" faults at the ring address on every later job, even after a module reload) until a reboot. apr0's resets is the parent SCODEC power domain, which also covers the DART, so the DART must be reprogrammed after that reset (or the reset narrowed to the engine). Bouncing the payload into a driver-owned buffer would close the rewrite window itself. Both are follow-ups.

  • Throughput: macOS VideoToolbox decodes about 2.6x faster on the same M1 Pro engine (1080p HQ 5.4k fps vs 2.1k here). Engine time per job is 444 us at 1080p HQ, so the likely cause is that this driver keeps one job in flight. Keeping several decode descriptors in the ring is the follow-up.

  • Interlaced frames: the parser accepts them, but submission returns EOPNOTSUPP until the second field's geometry is defined (unchanged from media: apple: add ProRes decode-v0 node #1259).

Supersedes #1259.

Rows and commands

Kit: prores-dec-trial.sh (rmmod in-tree apple_prores, insmod the branch module, decode matrix via V4L2 MPLANE m2m, compare, ring wrap, fps, encoder regression, interleaved encode + decode, restore). Clips: testsrc2 Proxy/LT/422/HQ 1080p, 4444/4444XQ 1080p, 4444 with alpha, noisy HQ, mandelbrot XQ, 4K HQ and 4444, 1280x721 odd height, and 8 frames from the hardware encoder.

clip out exact vs ffmpeg max diff Neo = M1 Pro = M4 Pro sha
1080p Proxy..HQ P216 95.2-95.7 % 4 (10-bit) yes
1080p HQ NV16 99.47 % 1 yes
1080p 4444 / XQ / +alpha P416 92.2-92.4 % 19 (12-bit) yes
1080p 4444 NV24 99.25 % 1 yes
noisy HQ P216 50.8 % 4 yes
mandelbrot XQ P416 29.7 % 3 yes
4K HQ / 4444 P216 / P416 96.6 / 93.9 % 4 / 19 yes
1280x721 P216 92.9 % 4 yes
our HW encode, HQ P216 95.9 % 4 yes

Most differences are off by 1 (the hardware rounds from more than 10 bits). The large-diff samples are in 4444 at 12-bit.

Input bounds: audit and kit
  • Max engine read relative to the payload after a successful parse: header + table + sum of slices = picture size; pictures sum to the frame size = bytesused - data_offset. Interlaced frames are rejected before a descriptor is built. Width and height in the descriptor are the parsed values, which must equal the CAPTURE format, so the engine's own slice count from the geometry equals the parser's.
  • Before the bounds commit, a last slice whose header claimed coded sizes beyond the slice let the engine read up to about 192 KiB past the payload.
  • Kit: 20 corruptions per clip, plus post-QBUF rewrite probes; every result line, dmesg before/after and the toctou probes are kept with the rows.

Assisted-by: LLM (Claude:Opus 5.5)

RyanTheTide and others added 2 commits November 20, 2026 00:49
Add a compatible-gated stateful decoder alongside the existing encoder.

Validate complete ProRes frames and pack the 0x100-byte non-RAW descriptor.

Share the APR ring with generation-aware variable-size accounting.

Snapshot completion diagnostics before the W1C/tail handoff.

Preserve reset-based timeout recovery.

Cover descriptor layout, parser bounds, and mixed ring wrap with KUnit.

Signed-off-by: Ryan Murray <81541646+RyanTheTide@users.noreply.github.com>
Apple lists each ProRes engine in its SoCs as a ProRes encode and
decode engine, and the decode descriptor runs on the same ring as the
encoder's. The decoder node was registered only on the T6000 and T8140
variants. Register it for every compatible the driver binds (T6041 and
T6050 as well) and drop the per-variant flag.

Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
@RyanTheTide

Copy link
Copy Markdown
Contributor Author

The decoder's coded format needs V4L2_FMT_FLAG_DYN_RESOLUTION as well as V4L2_FMT_FLAG_COMPRESSED. The driver sends V4L2_EVENT_SOURCE_CHANGE when the size or chroma changes, and the uAPI defines this flag for exactly that kind of stateful decoder:

	if (ctx->decode) {
		if (f->index)
			return -EINVAL;
		f->pixelformat = V4L2_PIX_FMT_PRORES;
		/* Size and chroma changes are reported with SOURCE_CHANGE. */
		f->flags = V4L2_FMT_FLAG_COMPRESSED | V4L2_FMT_FLAG_DYN_RESOLUTION;
		return 0;
	}

GStreamer's v4l2 decoder waits for the initial source change only when this flag is set (gstv4l2videodec.c, handle_frame). Without it, it sets up the CAPTURE queue early and treats the first event as a mid-stream resolution change. It then takes the real end-of-stream LAST buffer as the end of that change and waits forever. Every v4l2proresdec pipeline hangs at EOS, even though its output is complete and bit-exact. FFmpeg's prores_v4l2m2m doesn't look at the flag, so it isn't affected.

Tested with a module built from 7c55de4 plus this one line, on a MacBook Neo (7.3.0-40, GStreamer 1.28.9 with the v4l2 ProRes patches from fedora-aurora#444):

  • The pipeline now exits 0 at EOS. The 8-frame clip takes 25 ms including start-up.
  • The NV16 and NV24 output is byte-identical to the raw V4L2 tool.
  • 1080p HQ decodes at 2987–3029 fps in GStreamer vs 3005–3059 fps through raw V4L2, so the app path costs about 1 %.

Before the change, on an M1 Pro, every run hung until the 120 s timeout killed it. The GST_DEBUG log shows the LAST flag on frame 7, then "Setup the capture queue" and a wait that never ends.

Assisted-by: LLM (Claude:Opus 5.5)

The decode descriptor has no input length, so the engine reads as far
as the frame's own sizes take it. apr_parse_frame() already holds the
frame size, frame and picture header sizes, picture sizes, slice table
and slice sizes to the payload, but two in-band fields the engine may
also follow were never checked:

- the picture header's slice count, which could ask for more slice
  table entries than the picture geometry gives;
- each slice header, whose coded Y and Cb sizes (and Cr size, with
  alpha) could run past the end of the slice and so, for the last
  slice, past the buffer.

Accept a slice count of 0, which our encoder writes, or the count the
geometry gives. Require every slice header to fit its slice (8 bytes
with alpha, where the Cr size is explicit, 6 otherwise) and its coded
component sizes to fit the rest of the slice. Every ffmpeg-made test
clip and every frame from the hardware encoder passes these checks.

Read each slice size once so that the parse itself stays inside the
payload even if userspace rewrites the buffer while it runs. Give the
KUnit fixture valid 6-byte slice headers.

Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
The decode descriptor carries no input length, so a frame must not get
past apr_parse_frame() if any size the engine follows points beyond the
queued payload. Add KUnit cases for frames that do: a picture size 1
byte, 64 KiB and 4 GiB past the buffer; a frame size larger than
bytesused; a payload truncated by one byte; a frame header past the
frame; a slice table ending past the picture; a slice count asking for
more table entries; slice sizes summing one byte past the picture; slice
headers whose coded sizes run past their slice; and a bottom-field
picture or slice past the buffer. The exact-fit frame and one that ends
on the boundary byte are accepted.

Each frame is parsed from an allocation exactly as long as the payload,
so with KASAN an over-read in the parser is reported as well.

Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
@m0d16l14n1

Copy link
Copy Markdown
Contributor

J815 row (MacBook Air 15" M5, T8142: the engine this PR's testing doesn't cover yet).

Setup

Method: one frame per clip through v4l2-ctl --stream-out-mmap --stream-mmap, with CAPTURE taken from the source-change event. The output is compared with ffmpeg's software decoder (PSNR), not with VideoToolbox. Clips are ffmpeg prores_ks.

clip CAPTURE PSNR vs ffmpeg
640x360 422 P216 74.0 dB
1080p 422 Proxy P216 77.1 dB
1080p 422 LT P216 75.1 dB
1080p 422 P216 74.5 dB
1080p 422 HQ P216 74.1 dB
1080p 4444 P416 79.6 dB
1080p 4444 XQ P416 79.3 dB
4K 422 HQ P216 74.1 dB

Results

  • 8/8 decode. The node registers next to the T8142 encoder; formats are NV16 P216 NV24 P416 CAPTURE and PRES OUTPUT.
  • 4444 streams switch CAPTURE to P416 at the source change, as on the other engines.
  • No DART faults, timeouts or decoder errors in dmesg over the run.
  • I didn't measure throughput or compare hashes with your Neo/M1 Pro outputs this time. I can do either if useful.

The decoder sends V4L2_EVENT_SOURCE_CHANGE when the stream's size or
chroma changes, but its OUTPUT format description only carried
V4L2_FMT_FLAG_COMPRESSED. The uAPI defines V4L2_FMT_FLAG_DYN_RESOLUTION
for a stateful decoder that does this.

Without the flag, GStreamer's v4l2 decoder sets up the CAPTURE queue
before the first event and then takes that event as a mid-stream
resolution change. It mistakes the real end-of-stream LAST buffer for the
end of that change and waits forever, so every v4l2proresdec pipeline
hangs at EOS even though its output is complete. FFmpeg does not look at
the flag.

Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
@RyanTheTide

Copy link
Copy Markdown
Contributor Author

d409342 adds the one-line V4L2_FMT_FLAG_DYN_RESOLUTION change proposed above, as its own commit on top of 00c08be (no rebase). The run above covered the same line on 7c55de4 (MacBook Neo, GStreamer 1.28.9): v4l2proresdec exits 0 at EOS, the output is byte-identical to raw V4L2, and the app path costs about 1 % in fps. Without the line, every run on the M1 Pro hung until the 120 s timeout. At d409342: W=1 LLVM arm64 module build has no warnings, and checkpatch --strict is clean.

@RyanTheTide

Copy link
Copy Markdown
Contributor Author

Carried unchanged into #2620 (all commits keep their SHAs; #2620 adds a merge of #2396's head and a merge of integration/aurora, conflicts from #1881 only, explained there). Closing this one in favour of it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants