Repository navigation
media: apple: prores: hardware ProRes decode on every APR engine - #2396
RyanTheTide wants to merge 5 commits into
Conversation
Add a compatible-gated stateful decoder alongside the existing encoder. Validate complete ProRes frames and pack the 0x100-byte non-RAW descriptor. Share the APR ring with generation-aware variable-size accounting. Snapshot completion diagnostics before the W1C/tail handoff. Preserve reset-based timeout recovery. Cover descriptor layout, parser bounds, and mixed ring wrap with KUnit. Signed-off-by: Ryan Murray <81541646+RyanTheTide@users.noreply.github.com>
Apple lists each ProRes engine in its SoCs as a ProRes encode and decode engine, and the decode descriptor runs on the same ring as the encoder's. The decoder node was registered only on the T6000 and T8140 variants. Register it for every compatible the driver binds (T6041 and T6050 as well) and drop the per-variant flag. Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
|
The decoder's coded format needs if (ctx->decode) {
if (f->index)
return -EINVAL;
f->pixelformat = V4L2_PIX_FMT_PRORES;
/* Size and chroma changes are reported with SOURCE_CHANGE. */
f->flags = V4L2_FMT_FLAG_COMPRESSED | V4L2_FMT_FLAG_DYN_RESOLUTION;
return 0;
}GStreamer's v4l2 decoder waits for the initial source change only when this flag is set ( Tested with a module built from 7c55de4 plus this one line, on a MacBook Neo (7.3.0-40, GStreamer 1.28.9 with the v4l2 ProRes patches from fedora-aurora#444):
Before the change, on an M1 Pro, every run hung until the 120 s timeout killed it. The GST_DEBUG log shows the LAST flag on frame 7, then "Setup the capture queue" and a wait that never ends. Assisted-by: LLM (Claude:Opus 5.5) |
The decode descriptor has no input length, so the engine reads as far as the frame's own sizes take it. apr_parse_frame() already holds the frame size, frame and picture header sizes, picture sizes, slice table and slice sizes to the payload, but two in-band fields the engine may also follow were never checked: - the picture header's slice count, which could ask for more slice table entries than the picture geometry gives; - each slice header, whose coded Y and Cb sizes (and Cr size, with alpha) could run past the end of the slice and so, for the last slice, past the buffer. Accept a slice count of 0, which our encoder writes, or the count the geometry gives. Require every slice header to fit its slice (8 bytes with alpha, where the Cr size is explicit, 6 otherwise) and its coded component sizes to fit the rest of the slice. Every ffmpeg-made test clip and every frame from the hardware encoder passes these checks. Read each slice size once so that the parse itself stays inside the payload even if userspace rewrites the buffer while it runs. Give the KUnit fixture valid 6-byte slice headers. Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
The decode descriptor carries no input length, so a frame must not get past apr_parse_frame() if any size the engine follows points beyond the queued payload. Add KUnit cases for frames that do: a picture size 1 byte, 64 KiB and 4 GiB past the buffer; a frame size larger than bytesused; a payload truncated by one byte; a frame header past the frame; a slice table ending past the picture; a slice count asking for more table entries; slice sizes summing one byte past the picture; slice headers whose coded sizes run past their slice; and a bottom-field picture or slice past the buffer. The exact-fit frame and one that ends on the boundary byte are accepted. Each frame is parsed from an allocation exactly as long as the payload, so with KASAN an over-read in the parser is reported as well. Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
|
J815 row (MacBook Air 15" M5, T8142: the engine this PR's testing doesn't cover yet). Setup
Method: one frame per clip through
Results
|
The decoder sends V4L2_EVENT_SOURCE_CHANGE when the stream's size or chroma changes, but its OUTPUT format description only carried V4L2_FMT_FLAG_COMPRESSED. The uAPI defines V4L2_FMT_FLAG_DYN_RESOLUTION for a stateful decoder that does this. Without the flag, GStreamer's v4l2 decoder sets up the CAPTURE queue before the first event and then takes that event as a mid-stream resolution change. It mistakes the real end-of-stream LAST buffer for the end of that change and waits forever, so every v4l2proresdec pipeline hangs at EOS even though its output is complete. FFmpeg does not look at the flag. Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
|
d409342 adds the one-line |
Problem
Every Apple ProRes engine we drive is a ProRes encode and decode engine, but the driver only exposed the encoder. Decoding ProRes on Linux fell back to software. #1259 added a decoder node last week but never ran on hardware, and it no longer applied to the current driver (T6041/T6050 windows, built-in rows, video10 numbering).
Change
The media: apple: add ProRes decode-v0 node #1259 decode-v0 node, rebased onto current integration/aurora. It adds a stateful V4L2 decoder (
apple-prores-decode) next to the encoder. Decode jobs share the encoder's ring, and the ring accounting handles variable-size entries and the generation-bit wrap.The decoder node is no longer limited to T6000 and T8140. Every compatible the driver binds (T6000, T6041, T6050, T8140) gets it, with no per-variant flag.
Input bounds (9d612f2). The decode descriptor has no input length, so the engine reads as far as the frame's own sizes say. The parser already held the frame size, frame and picture header sizes, picture sizes, slice table and slice sizes to the payload. It now also checks:
Each field is read once, so the parse itself stays in bounds even if the buffer changes under it. KUnit for frames that point past their buffer is in 00c08be.
Testing
Each Mac ran the branch as an out-of-tree module, swapped in for the in-tree one on its production kernel, with no reboot. Clips came from ffmpeg prores_ks, plus frames from our own hardware encoder. Output was compared against ffmpeg's software decoder.
MacBook Neo (T8140, 7.3.0-40): 18 rows, 0 decode errors, 0 DART faults. 1080p 422 HQ decodes at 2,514-2,528 fps to P216 and 3,043-3,134 fps to NV16; 4444 at 1,727 fps; 4K HQ at 665 fps; 4K 4444 at 447 fps.
MacBook Pro M1 Pro (T6000, fleet74): the same 18 rows, 0 errors, 0 faults. Every decoded output is byte-identical to the Neo's (16/16 hashes match).
Bit-exact with Apple's decoder: on the same M1 Pro, macOS VideoToolbox's 10-bit output equals this driver's 16-bit output rounded to 10 bits for 100 % of samples (422 Proxy and HQ, 4444, 4444 XQ). Our P216/P416 keep the extra precision that VideoToolbox drops.
Output is deterministic. It stays byte-identical after 800 jobs (ring wrap) and while an encode stream shares the ring.
Against ffmpeg: 8-bit NV16/NV24 output is 99.2-99.5 % exact with a max diff of 1. P216 keeps more than 10 bits, so it is 93-97 % exact with a max diff of 4 at 10-bit. P416 4444 is 92-94 % exact with a max diff of 19 at 12-bit (about 5 at 10-bit).
Encoder regression in the same module: 1080p HQ at 1,632 fps on the Neo and 1,532-1,569 fps on the M1 Pro. Same output size as before.
KUnit on arm64 QEMU: apple-prores 20/20 pass (the APRI cases plus the decode descriptor, frame parser, interlaced parser and ring accounting cases).
MacBook Pro 16-inch M4 Pro (T6041 engine, fleet75): the decoder node now registers on T6041 too, and the same 18 rows pass with 0 errors and 0 faults. Outputs are again byte-identical to the Neo's. This is the fastest engine of the three: 1080p HQ at about 3.1k fps, 4K HQ at 1.0-1.5k fps, 4K 4444 at about 620 fps.
Input bounds, J616s (T6040, fleet75), module swap. 20 corrupted frames per clip × 3 clips: picture size, slice table, slice sizes, slice headers and slice count past the buffer, frame size beyond bytesused, bytesused short, frame header past the frame, interlaced flag with no second field.
Input bounds, M1 Pro (T6000, fleet74):
Input bounds, Neo (T8140, 7.3.0-40): the same 60/60 corrupt frames are rejected with no prores errors in dmesg; the 6 clips stay byte-identical with equal fps.
KUnit (arm64 QEMU): 28/28 apple-prores cases, including the 8 out-of-bounds cases. Each frame is parsed from an allocation exactly as long as the payload. Also clean under KASAN and UBSAN.
Still to run
What the check cannot cover: userspace may rewrite the in-band sizes in its MMAP/DMA-BUF buffer after QBUF. On the J616s, a thread toggling the picture size made the engine hang on the first such job; no output data came back in any of 100 jobs. The watchdog's engine reset then left the engine's DART stream with an invalid translation table ("TTBR invalid" faults at the ring address on every later job, even after a module reload) until a reboot. apr0's
resetsis the parent SCODEC power domain, which also covers the DART, so the DART must be reprogrammed after that reset (or the reset narrowed to the engine). Bouncing the payload into a driver-owned buffer would close the rewrite window itself. Both are follow-ups.Throughput: macOS VideoToolbox decodes about 2.6x faster on the same M1 Pro engine (1080p HQ 5.4k fps vs 2.1k here). Engine time per job is 444 us at 1080p HQ, so the likely cause is that this driver keeps one job in flight. Keeping several decode descriptors in the ring is the follow-up.
Interlaced frames: the parser accepts them, but submission returns EOPNOTSUPP until the second field's geometry is defined (unchanged from media: apple: add ProRes decode-v0 node #1259).
Supersedes #1259.
Rows and commands
Kit:
prores-dec-trial.sh(rmmod in-tree apple_prores, insmod the branch module, decode matrix via V4L2 MPLANE m2m, compare, ring wrap, fps, encoder regression, interleaved encode + decode, restore). Clips: testsrc2 Proxy/LT/422/HQ 1080p, 4444/4444XQ 1080p, 4444 with alpha, noisy HQ, mandelbrot XQ, 4K HQ and 4444, 1280x721 odd height, and 8 frames from the hardware encoder.Most differences are off by 1 (the hardware rounds from more than 10 bits). The large-diff samples are in 4444 at 12-bit.
Input bounds: audit and kit
Assisted-by: LLM (Claude:Opus 5.5)