Skip to content

media: apple: prores: parse each decoder source frame once - #2462

Closed
RyanTheTide wants to merge 2 commits into
media/prores-decode-v3-20261120from
media/prores-dec-parse-20261120
Closed

RyanTheTide wants to merge 2 commits into
media/prores-decode-v3-20261120from
media/prores-dec-parse-20261120

Conversation

@RyanTheTide

@RyanTheTide RyanTheTide commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #2396 (ProRes decoder node). Base branch: media/prores-decode-v3-20261120.

Problem

On the M1 Pro, VideoToolbox decodes ProRes about 2.6x faster than our V4L2 decoder. Part of that gap was on the CPU. The decoder parsed every source frame three times per job: in buf_queue, in job_ready (atomic context, called from v4l2_m2m_job_finish() in the IRQ handler) and in device_run. Each parse walks the whole slice-size table (1020 entries at 1080p, 4080 at 2160p) with 16-bit loads from the uncached vb2 mapping. On T6000 that is 76-88 us per 1080p parse and ~300 us at 2160p. With one job in the ring, the engine sat idle for that time between frames: 47-100 us at 1080p and 331-357 us at 2160p.

Change

  • Parse a decoder source frame once, in buf_prepare, and keep the result with the buffer (struct apr_buf). A frame that does not parse still fails its own job, not QBUF, as before.
  • Copy the slice table in 256-byte blocks and check the copy, so the CPU reads it with wide loads.
  • KUnit: a 1080p table that spans several blocks, with bad entries in the first full block, the last partial block and an overrun.

No hardware programming changes. The descriptor, ring handling and output are untouched.

Testing

  • Build: W=1 clean, out of tree, for 7.3.0-aurora-fleet74, -fleet75 and 7.3.0-40 (Neo).

  • KUnit (arm64 QEMU, drivers/media/platform/apple/prores/.kunitconfig): 22/22 pass, including the new case.

  • M1 Pro (J316s, T6000), fleet74, base = media: apple: prores: hardware ProRes decode on every APR engine #2396 module vs this branch, 3 interleaved reps. Decoded output is byte-identical on all 6 rows (Proxy, 422 HQ P216 and NV16, 4444 P416, 2160p HQ and 4444). No DART or ProRes errors.

    row media: apple: prores: hardware ProRes decode on every APR engine #2396 this PR
    1080p 422 HQ P216 1859-2003 fps 2755-2769 fps
    1080p 422 HQ NV16 1869-1960 2730-2746
    1080p Proxy 2035-2692 3089-3134
    1080p 4444 P416 1397-1535 1855-1862
    2160p 422 HQ 493-554 761
    2160p 4444 362-388 493
  • J616s (T6041), fleet75: byte-identical output on all 6 rows, fps unchanged (3055-3075 -> 3084-3104 at 1080p HQ). The parse was already ~1 us there, so the uncached-read cost is a T6000 effect. No regression.

Still to run

  • Neo (J700, T8140) row: same A/B kit, queued.
  • The remaining ~1.9x on the M1 Pro is not addressed here. Measured on the same Mac in macOS 26.2, VideoToolbox decoding one frame at a time is no faster than this driver (1080p HQ 2090-2110 fps vs our 2760). Its async throughput comes from overlapping several frames: either several jobs in the engine at once or VideoToolbox adding CPU decode alongside the engine. That is being measured in a follow-up lane, which will also capture how macOS submits decode jobs. See details.
Measurement details, rejected approaches and evidence paths

Evidence (our own measurements, not tainted): traces/codec/20261120-proresdec-perf/ (m1a/m1c/m1d M1 Pro, j6a J616s, m1pro-macos-vt3 macOS), kit kits/test/20261120-proresdec-perf (ab3.sh is this A/B).

ftrace on M1 Pro, 1080p 422 HQ / 2160p 422 HQ:

Ruled out on the M1 Pro:

  • DRAM/fabric state: decode with the GPU busy (which raises the T6000 memory state) is within +-1 % of idle, and engine time is identical. NV16 (half the output bytes) runs at the same rate as P216, and Proxy runs about as fast as HQ. The engine is per-pixel bound, not bandwidth bound.
  • Descriptor CRC word 3 vs 0: no difference.
  • Several descriptors in one ring (the earlier in-flight experiment, branch media/prores-inflight-20261120, not proposed): only ~5 % faster, and unsafe. The ring tail advances when a descriptor is fetched, not when its frame is written. Retiring jobs on the tail returned buffers early: output differed, the DART logged NO PTE FOR IOVA faults after the buffers were freed, and the ring timed out. Decode stays at one job in the ring.

macOS 26.2 VideoToolbox on the same M1 Pro (muted one-shot):

row VT async VT one job at a time this PR
1080p 422 HQ 5305-5409 2087-2106 2755-2769
1080p 4444 3673-3682 1619-1660 1855-1862
2160p 422 HQ 1397-1403 819-824 761

Two concurrent VT sessions: ~2865 fps each, so the aggregate saturates at ~5.4-5.7k. Whether that rate is the engine alone or engine plus CPU decode is not yet known. VT ProRes encode at 2160p HQ: 421 fps vs 401 on Linux, so the engine clock is at parity.

Assisted-by: LLM (Claude:Opus 5.5)

The decoder parsed a source frame three times per job: in buf_queue for
the source-change check, in job_ready and in device_run. Each parse walks
the whole slice-size table (1020 entries for a 1080p frame, 4080 at
2160p) with 16-bit loads from the vb2 dma-contig mapping, which is
uncached because the engine is not DMA-coherent, so every entry is a
round trip to memory. job_ready also runs in atomic context, from
v4l2_m2m_job_finish() in the completion interrupt.

With one job in the ring this time sits between the interrupt and the
next job. On an M1 Pro (T6000), function_graph shows 76 us per parse of a
1080p ProRes 422 HQ frame and 304 us at 2160p, three times per frame,
and the engine idles 100 us (1080p) and 331 us (2160p) between jobs.

Parse the frame once, in buf_prepare, and keep the result with the
buffer. A frame that does not parse still fails its own job rather than
QBUF, as before. Copy the slice table in 256-byte blocks and check the
copy, so the CPU reads it with wide loads: one parse now takes 9 us at
1080p and 61 us at 2160p, and the gap between jobs is 3-5 us.

Decode throughput on the M1 Pro, depth 1, same clips, output unchanged
byte for byte: 1080p 422 HQ 1870 -> 2780 fps, 2160p 422 HQ 495 -> 765 fps,
1080p 4444 1470 -> 1880 fps.

Add a KUnit case with a 1080p table that spans several blocks, with a
bad entry in the first full and the last partial block.

Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>

@DjDeveloperr DjDeveloperr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Co-review for Ryan. Approving.

I checked the caching against vb2 semantics:

  • Every QBUF of a dequeued buffer runs buf_prepare again (DQBUF and queue cancel clear vb->prepared), and the decode branch rewrites parse_err/info unconditionally (apple-prores-drv.c:1727), so nothing carries over from an earlier use.
  • PREPARE_BUF then QBUF: the v4l2 layer does not refill the planes of a prepared buffer, so bytesused and data_offset stay at what the parse saw.
  • DMABUF: __prepare_dmabuf attaches and maps before calling buf_prepare, so vb2_plane_vaddr() works there as it did in buf_queue. USERPTR is not offered.
  • MMAP buffers are coherent allocations (no allow_cache_hints), Normal-NC on this non-coherent device, so the CPU read at prepare time needs no cache maintenance.
  • The cached info reaches job_ready on another CPU through the m2m rdy_spinlock, and a buffer on the ready list is never re-prepared.
  • Error semantics are unchanged: a bad frame still fails its own job, and QBUF only fails on the same payload checks as before.
  • job_ready runs from v4l2_m2m_job_finish() in the hard IRQ handler. Removing the uncached slice-table walk from there (up to ~300 us at 2160p) is worth it on its own.

Stale content after QBUF (userspace writing through its mmap, or a DMA-BUF peer): the only parsed values that reach the descriptor are width, height and chroma, and they are checked against dst_fmt (apple-prores-drv.c:1100), so output writes stay bounded by kernel state. The decode descriptor has no input length (apple-prores-desc.c:535), so the engine's input reads follow the in-band sizes of whatever is in the buffer at fetch time. That was already true before this PR: device_run parsed memory userspace could still write, and the engine fetches over the next 0.35 to 1.6 ms. This moves the start of the window from device_run to QBUF without removing a guarantee, so it is not a blocker here. For #2396: the parse is a sanity check, and the real bound on engine reads is the DART mapping (one domain shared by all contexts). Please add one test with a frame whose in-band picture size or slice table points past its buffer, to confirm it ends as a DART fault and a job error, and check the macOS decode trace for an input-length field.

Block copy: I ran the old and new apr_parse_frame() side by side in userspace under ASan and UBSan, on 600k random and mutated frames up to about 2100x1200 (tables up to ~10k entries, both picture counts, truncated payloads). Return codes and info were identical, with no out-of-bounds reads. header + 2 * slices <= size <= bytes is checked before the loop, so every block copy stays in bounds.

Nits, none blocking:

  1. apr_parse_src() no longer parses; apr_src_info() or returning a const pointer would read better.
  2. The struct apr_buf comment says "when it is queued"; the parse happens at prepare time (QBUF or PREPARE_BUF). The comment on the first member could say why it must be first, or add static_assert(offsetof(struct apr_buf, m2m) == 0).
  3. The commit message says the KUnit case has a bad entry in "the first full" block; the test uses index 128, the first entry of the second block. Index 127 would also cover the end of a full block.

Neo (T8140) row: fine as a follow-up. This is CPU-side only, and the T8140 descriptor path is unchanged.

…parse-20261120

# Conflicts:
#	drivers/media/platform/apple/prores/apple-prores-desc.c
#	drivers/media/platform/apple/prores/apple-prores-test.c
@RyanTheTide

Copy link
Copy Markdown
Contributor Author

40bb82e merges #2396's two new commits: 9d612f2, which bounds the slice headers and the in-band slice count, and its KUnit. In apr_parse_picture() the merge keeps this PR's block copy of the slice table and adds the per-slice header check inside the block loop. Each slice header is copied into a local once. The stack's final tree is the one tested on the J616s (fleet75): 6/6 clips byte-identical, and fps unchanged within noise with and without the bounds commit. KUnit is 30/30 at this head. The M1 Pro fps row with the bounds commit is still to run; that is where the uncached reads cost the most.

Assisted-by: LLM (Claude:Opus 5.5)

@RyanTheTide

Copy link
Copy Markdown
Contributor Author

Neo (T8140, 7.3.0-40) row, follow-up from the review, at the top of the stack (this PR's parse plus #2396's bounds check plus #2491): 6/6 clips byte-identical to the J616s and M1 Pro, with equal fps (1080p HQ 2692-2697).

Assisted-by: LLM (Claude:Opus 5.5)

@RyanTheTide

Copy link
Copy Markdown
Contributor Author

Carried unchanged into #2620 (all commits keep their SHAs; #2620 adds a merge of #2396's head and a merge of integration/aurora, conflicts from #1881 only, explained there). Closing this one in favour of it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants