Skip to content

media: apple: prores: hardware ProRes decode on every APR engine, several jobs in the ring - #2620

Open
RyanTheTide wants to merge 18 commits into
integration/aurorafrom
media/prores-decode-v4-20261121
Open

RyanTheTide wants to merge 18 commits into
integration/aurorafrom
media/prores-decode-v4-20261121

Conversation

@RyanTheTide

Copy link
Copy Markdown
Contributor

Problem

The ProRes decode stack is three stacked PRs: #2396 (decoder node on every APR engine, input bounds, DYN_RESOLUTION), #2462 (parse each source frame once) and #2491 (several decode jobs in the ring, retire on the engine's finished-job id). #2396 no longer merges into integration/aurora since #1881 (T8142 ProRes) landed, and #2491's head doesn't carry #2396's last commit (d409342, DYN_RESOLUTION).

Change

This branch is #2491's head a1ad1a8 with two merges on top, so every reviewed commit keeps its SHA:

The merge applies exactly #1881's prores lines onto the stack: the sorted +/- lines of git diff <merge-base> integration -- drivers/media/platform/apple/prores and of git diff 64b404a4^1 64b404a4 -- drivers/media/platform/apple/prores are identical.

Testing

Still to run

Supersedes #2396, #2462 and #2491 (closed with links here).

Assisted-by: LLM (Claude:Opus 5.5)

RyanTheTide and others added 13 commits November 20, 2026 00:49
Add a compatible-gated stateful decoder alongside the existing encoder.

Validate complete ProRes frames and pack the 0x100-byte non-RAW descriptor.

Share the APR ring with generation-aware variable-size accounting.

Snapshot completion diagnostics before the W1C/tail handoff.

Preserve reset-based timeout recovery.

Cover descriptor layout, parser bounds, and mixed ring wrap with KUnit.

Signed-off-by: Ryan Murray <81541646+RyanTheTide@users.noreply.github.com>
Apple lists each ProRes engine in its SoCs as a ProRes encode and
decode engine, and the decode descriptor runs on the same ring as the
encoder's. The decoder node was registered only on the T6000 and T8140
variants. Register it for every compatible the driver binds (T6041 and
T6050 as well) and drop the per-variant flag.

Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
The decoder parsed a source frame three times per job: in buf_queue for
the source-change check, in job_ready and in device_run. Each parse walks
the whole slice-size table (1020 entries for a 1080p frame, 4080 at
2160p) with 16-bit loads from the vb2 dma-contig mapping, which is
uncached because the engine is not DMA-coherent, so every entry is a
round trip to memory. job_ready also runs in atomic context, from
v4l2_m2m_job_finish() in the completion interrupt.

With one job in the ring this time sits between the interrupt and the
next job. On an M1 Pro (T6000), function_graph shows 76 us per parse of a
1080p ProRes 422 HQ frame and 304 us at 2160p, three times per frame,
and the engine idles 100 us (1080p) and 331 us (2160p) between jobs.

Parse the frame once, in buf_prepare, and keep the result with the
buffer. A frame that does not parse still fails its own job rather than
QBUF, as before. Copy the slice table in 256-byte blocks and check the
copy, so the CPU reads it with wide loads: one parse now takes 9 us at
1080p and 61 us at 2160p, and the gap between jobs is 3-5 us.

Decode throughput on the M1 Pro, depth 1, same clips, output unchanged
byte for byte: 1080p 422 HQ 1870 -> 2780 fps, 2160p 422 HQ 495 -> 765 fps,
1080p 4444 1470 -> 1880 fps.

Add a KUnit case with a 1080p table that spans several blocks, with a
bad entry in the first full and the last partial block.

Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
The engine works through a producer/consumer descriptor ring, but the
driver kept one job in it: v4l2-m2m runs one job at a time and the job
only finished at its completion interrupt. On an M1 Pro, VideoToolbox
decodes the same clips about 2.6 times faster on the same engine, and
between two of our jobs the engine sits idle for the interrupt and the
workqueue round trip to the next device_run (100 us per 1080p frame).

Publish a job's descriptor, move its buffers off the m2m queues into a
per-device FIFO tagged with the ring pointer past the descriptor, and
finish the m2m job at once so the next one is prepared while the engine
works. Up to APR_MAX_INFLIGHT (8) jobs sit in the ring; when it is full
the m2m job stays running until a completion makes room, so device_run
is only entered with room for one more descriptor. The interrupt reads
the tail and completes, in ring order, every job the tail has passed.

DONE and ERROR are latched per engine, so an ERROR fails every job in
the ring at that moment. A ring that stops moving for a second fails all
of its jobs and resets the engine as before; device_run waits for that
reset. stop_streaming, suspend and a STOP command with an empty OUTPUT
queue wait for the context's jobs in the ring; job_abort marks only that
context's jobs cancelled, since encoder and decoder share the ring.

Encode stays at one job in the ring: every encode job uses the one
shared statistics area and encode already runs at macOS speed.

Add KUnit cases for the ring distance, free space, tail validation and
in-order retirement across the ring wrap, and for the FIFO limits.

Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
With several decode jobs in the ring, the driver gave a job's buffers
back once the consumer tail had passed its descriptor. The tail moves
when the engine fetches a descriptor, not when the frame is written, so
buffers went back to vb2 while the engine still wrote them: wrong output
on 4:4:4 clips and DART faults at the end of a run.

The engine reports the id of the last decode job it has finished in the
upper half of ST1; the decode command word already carries a 16-bit id.
Give decode jobs device-wide ids that skip 0 (ST1 reads 0 after a
reset), read ST1 after the status clear, and retire decode jobs in ring
order once their id is reported finished. An encode descriptor carries
no id, so an encode only enters an empty ring and completes on its
interrupt, as before.

On an M1 Pro (T6000) with eight jobs in the ring every clip decodes to
the same bytes as with one job, with no DART faults; 1080p 422 HQ goes
from about 2730 to 2920 fps, 4K 422 HQ from 761 to 775 fps.

Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Several paths could still hand buffers back while the engine owned them
or would use them:

- A DONE or ERROR latched after the last job was retired by id stayed set.
  An encode published into the then empty ring was taken for finished on
  it, because the tail passes an encode as soon as it is fetched. The
  engine is idle while the ring is empty, so clear both status bits
  before publishing into an empty ring.

- ST1 = 0 counted as "finished" for every id in 0x8001..0xffff, and an id
  not issued yet retired the whole ring. 0 now means no progress, and an
  id at or after the next one to issue marks the ring hung, so the
  watchdog resets the engine before any buffer goes back.

- A job that fails before it reaches the ring (a frame that does not
  parse, an interlaced frame) completed at once, ahead of up to seven
  earlier frames of the same context. As the drain's last source buffer
  it sent LAST and EOS first and the frames still in the ring were lost.
  It now waits until the context's jobs in the ring are given back.

- apr_finish_job() removed the source and destination buffers outside
  hw_lock, so a STOP could see the destination half moved. Move both
  under the lock, as device_run does.

Also correct the watchdog comments (it fires when no job finishes, not
when the tail stops) and the ring test comment.

Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
The decode descriptor has no input length, so the engine reads as far
as the frame's own sizes take it. apr_parse_frame() already holds the
frame size, frame and picture header sizes, picture sizes, slice table
and slice sizes to the payload, but two in-band fields the engine may
also follow were never checked:

- the picture header's slice count, which could ask for more slice
  table entries than the picture geometry gives;
- each slice header, whose coded Y and Cb sizes (and Cr size, with
  alpha) could run past the end of the slice and so, for the last
  slice, past the buffer.

Accept a slice count of 0, which our encoder writes, or the count the
geometry gives. Require every slice header to fit its slice (8 bytes
with alpha, where the Cr size is explicit, 6 otherwise) and its coded
component sizes to fit the rest of the slice. Every ffmpeg-made test
clip and every frame from the hardware encoder passes these checks.

Read each slice size once so that the parse itself stays inside the
payload even if userspace rewrites the buffer while it runs. Give the
KUnit fixture valid 6-byte slice headers.

Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
The decode descriptor carries no input length, so a frame must not get
past apr_parse_frame() if any size the engine follows points beyond the
queued payload. Add KUnit cases for frames that do: a picture size 1
byte, 64 KiB and 4 GiB past the buffer; a frame size larger than
bytesused; a payload truncated by one byte; a frame header past the
frame; a slice table ending past the picture; a slice count asking for
more table entries; slice sizes summing one byte past the picture; slice
headers whose coded sizes run past their slice; and a bottom-field
picture or slice past the buffer. The exact-fit frame and one that ends
on the boundary byte are accepted.

Each frame is parsed from an allocation exactly as long as the payload,
so with KASAN an over-read in the parser is reported as well.

Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
…parse-20261120

# Conflicts:
#	drivers/media/platform/apple/prores/apple-prores-desc.c
#	drivers/media/platform/apple/prores/apple-prores-test.c
The decoder sends V4L2_EVENT_SOURCE_CHANGE when the stream's size or
chroma changes, but its OUTPUT format description only carried
V4L2_FMT_FLAG_COMPRESSED. The uAPI defines V4L2_FMT_FLAG_DYN_RESOLUTION
for a stateful decoder that does this.

Without the flag, GStreamer's v4l2 decoder sets up the CAPTURE queue
before the first event and then takes that event as a mid-stream
resolution change. It mistakes the real end-of-stream LAST buffer for the
end of that change and waits forever, so every v4l2proresdec pipeline
hangs at EOS even though its output is complete. FFmpeg does not look at
the flag.

Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
# Conflicts:
#	drivers/media/platform/apple/prores/Kconfig
#	drivers/media/platform/apple/prores/apple-prores-drv.c
@RyanTheTide

Copy link
Copy Markdown
Contributor Author

Hardware row for this head (64b404a) against the stack before the integration merge (66099fc = a1ad1a8 + d409342). The merge changes nothing in decode: every clip decodes to the same SHA as before (and as the 11-20 rows), throughput is unchanged, and the drain, kill, two-context and out-of-bounds rows behave the same.

M1 Pro (T6000), 7.3.0-aurora-fleet76; apple-prores.ko built W=1 out of tree for both revisions against the fleet76 tree and swapped in user space (rmmod/insmod), in-tree module restored after.

row 66099fc (before merge) 64b404a (this head)
SHA, 6 clips 0b67a79a 1ec84fa0 42572d48 192fc850 15288276 134b35db identical
1080p Proxy p216 fps 3330-3337 3336-3337
1080p HQ p216 / NV16 fps 2915 / 2918 2915 / 2908-2918
1080p 4444 p416 fps 1937 1937
2160p HQ p216 fps 771.5-771.8 771.5-771.7
2160p 4444 p416 fps 495.5-496.4 496.3-496.4
encode while decoding identical output identical output
DRAIN (8 in flight, one corrupt, STOP) 6/6 pass 6/6 pass
KILL x20 mid-flight, then decode same SHA same SHA
two decode contexts each == solo each == solo
OOB frames (20 corruptions x 3 clips) 60/60 rejected 60/60 rejected
bad dmesg lines 0 0

fps are N=3 interleaved runs. W=1 build: no warnings in drivers/media/platform/apple/prores.

Assisted-by: LLM (Claude:Opus 5.5)

@DjDeveloperr

Copy link
Copy Markdown
Contributor

Hardware row: M5 Pro MacBook Pro (J714s, T6050, ProRes codec 0x40018), head 64b404a494cd vs 722d76615fa8

Tested on M5 Pro (apple,j714s, apple,t6050, 48d000000.prores, kernel 7.3.0-m5-sship7 with MTE enabled on all 18 CPUs). apple-prores.ko was built out of tree (LLVM=1, 0 warnings) for both 722d76615fa8 (before the buffer-lifetime, slice-bounds, and DYN_RESOLUTION fixes) and 64b404a494cd (current head, run in both pr and full positions of the A/B harness) against the matching kernel tree, swapped in user space (rmmod/insmod), and restored to the in-tree module afterwards.

Row 722d76615fa8 (before fixes) 64b404a494cd (this head)
SHA, 6 clips 0b67a79a 1ec84fa0 42572d48 192fc850 15288276 134b35db identical (6/6, also matches M1 Pro)
1080p Proxy p216 fps (N=3 med / range) 3245.4 (2972–3914) 3952.7 (2985–4036) / 2981.9 (2904–4249)
1080p HQ p216 fps (N=3 med / range) 2967.2 (2964–3918) 2971.5 (2909–2989) / 2971.8 (2910–2983)
1080p HQ NV16 fps (N=3 med / range) 3396.9 (3396–3397) 3396.3 (3305–3397) / 3396.0 (3307–3397)
1080p 4444 p416 fps (N=3 med / range) 1985.9 (1983–2558) 1981.5 (1955–1994) / 1984.5 (1944–1988)
2160p HQ p216 fps (N=3 med / range) 966.3 (964–968) 1021.7 (996–1305) / 1004.4 (990–1016)
2160p 4444 p416 fps (N=3 med / range) 634.5 (634–649) 674.2 (646–713) / 687.8 (627–849)
Encode while decoding (MIXED) identical encode (7f1feb4f598a7c52, 1178.8 fps) + 3200 bg decode frames (0 err) identical encode (7f1feb4f598a7c52, 1173.4 / 1224.2 fps) + 3200 bg decode frames (0 err)
DRAIN (8 in flight, bad=-1/7/3, 1080p & 2160p, STOP) 2/6 pass, 4/6 FAIL (bad=7/3 returned out of order as BUF 1) 6/6 PASS (strict sequence order, ERROR on bad slot, LAST at 9/9)
KILL x20 mid-flight, then decode same SHA (0b67a79a) same SHA (0b67a79a)
Two decode contexts (1080p + 2160p) each == solo (1180 / 428 fps) each == solo (1349 / 553 and 1401 / 609 fps)
OOB frames (20 corruptions × 3 clips = 60) 48/60 pass, 12 fail (4 cases × 3 clips overwrote poison, 6 job failed dmesg lines) 60/60 PASS, 0 fail, poison intact, valid-after match, 0 bad dmesg lines
Bad dmesg lines 6 (job failed on OOB) 0

Note: apple-prores 48d000000.prores: completion found without an interrupt appeared occasionally under high-rate decode on both 722d7661 and 64b404a4 with zero frame errors or timeouts.

@RyanTheTide

Copy link
Copy Markdown
Contributor Author

Merge-train adversarial review (MERGETRAIN-LINUX2) at 64b404a: BLOCKERS.

Not merging until the blockers below are fixed or answered; the details give file:line and a failure case.

Full review

#2620 media: apple: prores: hardware ProRes decode on every APR engine, several jobs in the ring @64b404a4

VERDICT: BLOCKERS

Blockers

  1. Source-buffer TOCTOU lets the engine DMA-read past the OUTPUT buffer. Code: apple-prores-desc.c:562-566, apple-prores-drv.c:2270-2290 (buf_prepare parse), drv.c:1374-1406 (device_run uses the cached apr_buf.info), desc.c:526-552 (descriptor). The driver's own comment says the decode descriptor carries no input length, and the descriptor holds only the DD_INPUT IOVA, so the engine follows the frame's in-band sizes. Since a93c473, those sizes are checked once, at QBUF/buf_prepare. The buffer stays writable by userspace: it is MMAP'd, or a DMABUF another process holds. The engine fetches it much later, up to 8 jobs deep. Scenario: process A queues a frame that parses, then rewrites the picture size, slice table or slice sizes in its mapping. The engine then reads up to several GiB past the buffer's IOVA (32-bit picture size, up to 131072 x 16-bit slices). Every context of both nodes shares the APR DART. The read lands either in another context's mapped buffers, such as a different user's encode source frames, which then decode into A's CAPTURE buffer (a cross-process leak), or on an unmapped IOVA (DART fault, then ERROR and a reset that fails every other context's in-flight jobs). The comment "the parse itself stays in bounds even if the buffer changes under it" covers the CPU parse only, not the engine. The OOB rows (60/60) corrupt the buffer before QBUF, so they never test this. Possible fixes: decode from a driver-owned bounce copy of the validated frame, or give the engine a length or bounds if one exists. At minimum, document it as a known hardening gap and get Ryan's explicit acceptance.

Nits (should fix)

  • drv.c:1522-1576 and drv.c:1624: apr->hung is not checked when apr_take_done() starts. After a bad tail (1566) or a FUTURE id (1575) sets hung, the watchdog is queued with mod_delayed_work(system_wq, 0). A second interrupt can arrive before that work runs. It then retires jobs by done_id and apr_retire() hands their buffers back to vb2 from the IRQ, while the ring is known to be inconsistent. That breaks the guarantee in the comment at 1514-1517 ("the watchdog resets the engine before any buffer goes back"). Return early from apr_take_done() when apr->hung is set.
  • An ERROR marks every job in the ring -EIO (drv.c, the take_done error loop), including good frames behind the bad one and frames already finished by id but not yet retired. This is by design, but up to 7 valid frames are lost per corrupt frame. Mention it in the commit or in the uAPI behaviour notes.
  • drv.c:2300-2338: mid-stream SOURCE_CHANGE is queued at QBUF time, not when decode reaches the frame. No CAPTURE buffer is marked LAST for the old resolution, and dst_fmt is left stale while CAPTURE is busy; job_ready (1836) just stalls the new-size buffer. This is not the stateful-decoder dynamic-resolution sequence, so GStreamer may hang on a resolution switch. Initial-resolution handling is fine.
  • Ring wrap changed for the encoder on every variant. Base reset the ring to 0 whenever a descriptor did not fit. Now drv.c:1777 runs apr_hw_init only when !ring_ready, and apr_ring_write (drv.c:1802, desc.c) splits a descriptor across the ring end. 0x20000 is not a multiple of the 0x180 encode descriptor, so any encode stream longer than 341 frames now depends on the engine reading a split descriptor and on the BIT(24) generation semantics. The MIXED rows (M1 Pro, J714s) imply this works; no row checks it explicitly.
  • drv.c:59-60 and 1906-1913, desc.c:337: encoder limits were loosened on every SoC: max width/height 8192 to 65535, odd widths for 4:4:4, odd heights with a 128-byte stride. No encode row at more than 8192 px or with odd geometry.
  • desc.c:664 accepts frames with alpha (alpha 1/2 with 4:4:4), but desc.c:551 sets DD_ALPHA to 0 with no alpha stride. Unverified whether the engine skips the alpha plane or writes it to IOVA 0. A 4444 clip with alpha (yuva444p10) needs a row.
  • Stale text: Kconfig:17-18 and desc.c:477 still say decode exists only on T6000/T8140. 7c55de4 registers the decoder on every variant (drv.c probe).
  • 4946927's Signed-off-by is the GitHub noreply address; the other commits use ryan@aurorasilicon.org. Human, but inconsistent. The merge commits have no SoB, which is normal.
  • desc.c:676: the second-field slice cap (16384 vs 262144) is unexplained. It is dead today because drv.c:1389 refuses pictures != 1.

Missing rows

  • At this head, only M1 Pro (T6000) and J714s (T6050) have rows. Missing at 64b404a: J616s (T6041, in the lab), Neo J700 (T8140, in the lab), T8142 J815, J775c. The earlier rows are on pre-merge code.
  • Encode-only run of more than 341 frames, which forces a split-descriptor wrap, on each variant, especially the APRI-firmware ones (T6041/T8142/T6050).
  • System s2idle with decode jobs in flight. apr_suspend (drv.c:2563) waits on outstanding; that path is untested.
  • 4444-with-alpha clip, mid-stream resolution change, a modify-after-QBUF row (blocker 1), and odd-geometry or more-than-8192 encode.
  • T6050 logged "completion found without an interrupt" on both revisions. That is a lost or late IRQ, not root-caused.
  • The merge with media: apple: prores: bind T6041's second engine (apr1) without an aux window #2509 (T6041 apr1) is still pending.

@satchlj

satchlj commented Nov 22, 2026

Copy link
Copy Markdown
Member

Claude Opus 5.5 on @satchlj's behalf. J775c (Mac Studio M5 Max, T6050) rows at head 64b404a494cd. These cover the J775c entry in the MERGETRAIN-LINUX2 "missing rows" list and four of its other rows.

Setup

Rows (all errors 0; no DART, fault, timeout or ring-state lines)

Row Result
SHA, 6 clips (1080/2160, profiles 0/3/4, p216/nv16/p416) 6/6 match the 2026-11-20 J775c #2491 hashes, at boot and after a 12 s s2idle
FPS, single run 1080p p216 2994, nv16 3362, p416 2598; 2160p p216 1013, p416 669. Same as the #2491 J775c row
MIXED, KILL ×20, TWO contexts all SAME
Encode ring wrap: 800 frames, two split-descriptor wraps (0x20000 / 0x180 = 341) last-loop frames byte-identical to the first loop, 8/8, for p216 profile 3 and p416 profile 4
s2idle with jobs in flight: a 2160p decode stream and a 1080p encode stream, 32000 frames each, 10 s rtcwake -m freeze 3 s in rc 0; both streams finished with errors 0; decode last-loop sha 192fc8508fdc58e1 = solo; encode last loop 8/8 identical
4444 with alpha (prores_ks yuva444p, 8 frames) decodes; p416 output vs ffmpeg's software decode: PSNR 91.9 dB (the no-alpha clip gives 91.8 dB). No DART fault, so nothing was written to IOVA 0
Odd geometry encode: 1921x1081 p416 profile 4 encode then engine decode; PSNR vs source 80.7 dB
Wider than 8192: 8200x64 p416 profile 4 encode then engine decode; PSNR vs source 78.1 dB

Notes

  • One completion found without an interrupt (dev_warn_once) on the first boot, at +43 s, during the KILL/TWO rows. It did not appear on the second boot, which ran the wrap and in-flight s2idle rows. Same symptom as J714s; not root-caused here.
  • Not covered: modify-after-QBUF (blocker 1), mid-stream resolution change, and the media: apple: prores: bind T6041's second engine (apr1) without an aux window #2509 merge.
  • Failed units 0 on every boot. The kernel error lines are the known DCP/speaker lines, nothing from ProRes.

Assisted-by: LLM (Anthropic:Claude Opus 5.5)

@DjDeveloperr DjDeveloperr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pre-PR checklist review at live head 64b404a494cd559efad3184d2ef01cd7cf278eb3 (CI: 7/7 green; §1 no model gating, §2 default cmdline, §4 human commit attribution/SoB, §6 naming all clean).

Blockers

  1. Source-buffer DMA-read TOCTOU across the shared APR DART (drivers/media/platform/apple/prores/apple-prores-drv.c:2278-2289, drivers/media/platform/apple/prores/apple-prores-desc.c:526-554, :562-566)

    • apr_buf_prepare() parses and validates the frame's in-band sizes once at QBUF time and caches buf->info, while apr_build_decode_desc() passes DD_INPUT with no input length limit (the hardware decode descriptor carries none).
    • Because OUTPUT uses VB2_MMAP | VB2_DMABUF, userspace retains write access to the buffer after QBUF and while up to 8 jobs sit in the ring. Modifying frame_size, picture_size, slice table entries, or slice header coded sizes after QBUF lets the engine DMA-read past the OUTPUT buffer into adjacent mappings in the shared APR DART (cross-context data leak from another user's encode/decode buffers) or into an unmapped IOVA (DART fault / engine reset DoS failing other contexts' in-flight jobs).
    • The comment at apple-prores-desc.c:564-566 protects only the CPU parser, not the DMA engine.
    • Fix: Copy the validated bitstream into a driver-owned DMA bounce buffer (or bounded pool of APR_MAX_INFLIGHT private buffers) and parse that private copy before programming DD_INPUT, or enforce a hardware input-length bound if one exists.
  2. apr->hung race breaks the reset-before-buffer-return invariant and allows submitting into a hung ring (drivers/media/platform/apple/prores/apple-prores-drv.c:1522-1577, :1623-1635, :1653-1671, :1766-1784)

    • When apr_take_done() latches apr->hung = true (invalid tail at :1566 or APR_DONE_ID_FUTURE at :1575), APR_REG_IRQ_ENABLE is not masked and apr_take_done() does not check apr->hung on entry.
    • Even without a second interrupt, apr_watchdog() unconditionally calls apr_take_done() at :1657 when apr->hung is already true. A second apr_irq() or apr_watchdog() itself can retire jobs and hand buffers back to vb2 before the engine is reset; if apr_irq() retires the remaining jobs (njobs == 0), apr_watchdog() returns early at :1653, skipping the reset and leaving apr->hung = true latched for the next job.
    • Furthermore, apr_device_run() (:1766) waits only on apr->resetting, not apr->hung. If held == NULL (fewer than 8 jobs in flight), another apr_device_run() can acquire hw_lock before apr_watchdog runs and publish a new descriptor into the hung ring.
    • Fix: Mask APR_REG_IRQ_ENABLE immediately when setting apr->hung = true; return early from apr_take_done() (and skip calling it in apr_watchdog()) when apr->hung is set; have apr_device_run() wait on apr->hung || apr->resetting; ensure apr_watchdog() still performs the reset if apr->hung is set.
  3. Broken mid-stream dynamic resolution change (drivers/media/platform/apple/prores/apple-prores-drv.c:2004, :2052-2056, :2300-2339)

    • The decoder advertises V4L2_FMT_FLAG_DYN_RESOLUTION (:2004), but apr_decode_source_change() fires immediately at QBUF time without marking the last pre-change CAPTURE buffer V4L2_BUF_FLAG_LAST (so userspace cannot drain old-resolution frames before reconfiguring).
    • Worse, when vb2_is_busy(dst_vq) is true (always the case mid-stream), apr_decode_source_change() (:2319-2330) skips updating ctx->dst_fmt, and apr_stop_streaming() never refreshes it either. When userspace reacts to V4L2_EVENT_SOURCE_CHANGE with STREAMOFF(CAPTURE) -> G_FMT(CAPTURE), apr_g_fmt_cap() (:2054) still returns the stale pre-change resolution, breaking standard V4L2 stateful DRC clients.

Non-blocking notes / §5 remaining rows

  • Stale docs/comments: drivers/media/platform/apple/prores/Kconfig:25-26 and apple-prores-desc.c:477 still state decode is T6000/T8140-only, though 7c55de42 enables it on every APR variant.
  • Missing §5 rows at live head 64b404a4: T6000 (M1 Pro) and T6050 (J714s, J775c — including wrap, alpha, odd/wide encode, and in-flight s2idle) are covered; still missing live-head rows on T6041 (J616s), T8140 (Neo J700), and T8142 (J815), plus root-causing the occasional completion found without an interrupt warning on T6050.

Assisted-by: LLM (Gemini 4 Neon)

When apr_take_done() finds a tail outside the ring or a finished id that
was never issued, it marks the ring hung and queues the watchdog, which
fails every job and resets the engine before their buffers go back. Four
paths broke that order:

- the engine interrupt stayed enabled, and apr_take_done() did not check
  the flag, so a second interrupt before the watchdog ran retired jobs by
  the untrusted ST1 id and handed their buffers back to vb2 while the
  engine might still use them;
- the watchdog itself called apr_take_done() on a hung ring;
- if that left the ring empty, the watchdog returned early, skipping the
  reset and leaving the flag set for the next job;
- device_run only waited for a reset in progress, so a job could be
  published into the hung ring before the watchdog took the lock.

Mask the engine interrupt when the ring is declared hung, take no
completions while it is, and have the watchdog reset a hung ring even
when no job is left in it. device_run now waits until the ring is
neither hung nor being reset. A runtime suspend powers the engine off,
which resets it, so it clears the flag too.

Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
The decode descriptor has no input length: the engine follows the frame's
in-band sizes (frame and picture size, the slice table and each slice
header). Those sizes were checked once, in buf_prepare, on the OUTPUT
buffer itself, which userspace can still write while the buffer is queued
and while the job sits in the ring, up to eight jobs deep: it is an MMAP
buffer or a DMABUF another process may hold. Rewriting a size after QBUF
let the engine read up to several GiB past the buffer's IOVA. Every
context of both nodes shares the engine's DART, so that read could land
in another context's buffers, such as another user's encode source
frames, and decode them into the attacker's CAPTURE buffer, or fault on
an unmapped IOVA and reset the engine under every other context's jobs.

Give each slot of the job FIFO a driver-owned source copy. device_run
copies the frame into the copy of the slot its job will take, parses that
copy and points DD_INPUT at it; the engine never reads the OUTPUT buffer.
Only device_run, which v4l2-m2m runs one at a time and only with room in
the FIFO, fills a copy, and the engine is done with a slot's copy once its
job is taken off the ring or the engine is reset. The copies are cached
non-contiguous DMA allocations, so the parse runs on cached memory, with a
quarter of headroom so a stream rarely reallocates them; they are freed
when the engine runtime-suspends. At most APR_MAX_INFLIGHT copies exist
per engine.

The parse at QBUF stays, for the source-change check and to fail a
malformed frame early; a frame changed after QBUF so that it no longer
parses fails its own job. device_run also checks that the CAPTURE buffer
is large enough for the current format.

Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
The decoder advertises V4L2_FMT_FLAG_DYN_RESOLUTION, but a frame of a new
size queued while CAPTURE was set up raised SOURCE_CHANGE at QBUF time,
before decode reached it, left the CAPTURE format at the old size while
CAPTURE was busy, and never marked the last old-size CAPTURE buffer LAST.
job_ready just stalled on the new frame. A client following the stateful
decoder sequence (drain to LAST, STREAMOFF and G_FMT on CAPTURE) waited
for a LAST that never came, or read the old format back.

Handle a change where Documentation/userspace-api/media/v4l/dev-decoder.rst
puts it, in device_run when decode reaches the frame: give back every
earlier frame of the context, switch the OUTPUT and CAPTURE formats to the
new geometry, queue SOURCE_CHANGE, and return the next CAPTURE buffer
empty with LAST. The new frame stays first on OUTPUT and no job runs until
CAPTURE is restarted or a START command resumes decoding; START now also
clears vb2's LAST-dequeued state, so DQBUF works again after it. The
formats change under hw_lock, which G_FMT and TRY_FMT take, as device_run
does not hold vdev_lock.

Before CAPTURE streams, the next frame to decode still sets the formats
at QBUF, now including the CAPTURE format while CAPTURE buffers exist:
device_run refuses a CAPTURE buffer smaller than the format needs.

Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
Kconfig and the decode descriptor comment still said decode exists only
on T6000 and T8140, but every APR variant has had a decoder node since
"media: apple: prores: give every APR engine a decoder node".

The frame parser bounded the second field of an interlaced frame to
16384 slices and the first to 262144, with no reason given: both fields
have the same geometry. Use one named bound for both. Interlaced frames
are still refused before a descriptor is built, so nothing decodes
differently.

Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
@RyanTheTide

Copy link
Copy Markdown
Contributor Author

Fixes for #2620 (review) and the overlapping items of #2620 (comment), as four commits on top of 64b404a (no rewrite).

Blocker 1, source-buffer TOCTOU (3e0c7ee): the engine no longer reads the OUTPUT buffer. Each slot of the job FIFO has a driver-owned copy (at most APR_MAX_INFLIGHT per engine, cached non-contiguous DMA memory, grown with a quarter of headroom, freed at runtime suspend). device_run copies the frame into the copy of the slot its job takes, parses the copy, and points DD_INPUT at it. A frame rewritten after QBUF either decodes as it was when copied or fails its own job; the engine only ever follows sizes checked on memory userspace cannot reach. I found no public input-length field in the decode descriptor, so a copy it is. The QBUF parse stays for the source-change decision and early failure.

Blocker 2, hung ring (9c55a39): declaring the ring hung masks the engine interrupt; apr_take_done() takes nothing while it is hung; the watchdog skips it and resets a hung ring even when no job is left (no early return); device_run waits on hung || resetting. Runtime suspend (a power-off, so a reset) clears the flag.

Blocker 3, mid-stream DRC (938b9ee): handled in device_run when decode reaches the new frame, per dev-decoder.rst: earlier frames go back first, formats switch (OUTPUT and CAPTURE) under hw_lock, SOURCE_CHANGE is queued, then the next CAPTURE buffer goes back empty with LAST. Nothing runs until CAPTURE STREAMOFF/STREAMON or DECODER_CMD START (which now also clears vb2's LAST-dequeued state). G_FMT after the event returns the new size. device_run refuses a CAPTURE buffer smaller than the current format.

Cheap items from the MERGETRAIN review (0f162d2): Kconfig and the descriptor comment no longer say decode is T6000/T8140-only; one named slice bound for both fields (the 16384 had no reason). Not changed: an ERROR still fails every job in the ring at that moment (the status does not say which descriptor failed; documented in apr_take_done()); 4946927's noreply SoB would need a history rewrite.

Testing
  • arm64 LLVM W=1 build of drivers/media/platform/apple/prores/ (fleet79 config, clang 22): 0 warnings. Same for an out-of-tree build against 7.3.0-aurora-fleet79.
  • KUnit (arm64 qemu, the driver's .kunitconfig): 35/35 pass.
  • checkpatch --strict: clean on all four commits.
  • Hardware: not run yet (below).
Still to run (hardware)

Kit with both arms built for fleet79 (old = 64b404a, new = this head), the st1fix rows, OOB with the post-QBUF rewrite probe on, and a new drc tool for the resolution change. Rows per Mac (M1 Pro first, then J616s, Neo, a T6050 and J815):

  • SHA 6/6 identical to the earlier rows; FPS old vs new (the frame copy costs CPU time per job; 2160p 4444 is the worst case);
  • MIXED, DRAIN 6/6, KILL x20, TWO, OOB 60/60 unchanged;
  • TOCTOU probe (psize/lastslice/slicehdr x after/flip): only clean or error, other 0, VALID-AFTER match, no DART fault or job failed;
  • DRC 1080p->2160p (STREAMOFF), 2160p->1080p (START and STREAMOFF), 422->4444 and 4444->422 at 1080p: LAST exactly between the clips, new G_FMT after the event, output == the two solo decodes.

Assisted-by: LLM (Claude:Opus 5.5)

…hange

At a resolution change device_run switches the CAPTURE format before the
client has seen SOURCE_CHANGE; the client is still dequeuing the last
old-size frames and queues their buffers again. buf_prepare checked every
CAPTURE buffer against the current format, so on a change to a larger
size that QBUF failed with -EINVAL. dev-decoder.rst has the client queue
those buffers as usual; one of them carries the change's LAST flag. On a
Neo (T8140) a 1080p to 2160p and a 4:2:2 to 4:4:4 change failed this way;
the downsizing changes passed only because the old buffers were large
enough.

Do not check a decoder's CAPTURE buffers against the current format at
QBUF: queue_setup sized each for the format of its time. device_run gives
back a buffer too small for the current format with ERROR and no payload,
after the context's frames in the ring, and keeps the source frame queued
for one that fits, instead of failing the frame.

Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
@RyanTheTide

Copy link
Copy Markdown
Contributor Author

Neo (J700, T8140) rows on 7.3.0-aurora-fleet81, and one more fix (cc64e59).

Old arm 64b404a confirms blocker 1 on hardware: the post-QBUF rewrite probe (picture size flipped from a thread) gave a job timeout and repeated DART translation faults (reads far past the buffer) for about 40 s, then the machine reset.

0f162d2: SHA 6/6 match the reference, MIXED / DRAIN 6/6 / KILL x20 / TWO all as before, OOB 60/60 including every post-QBUF rewrite probe, 0 bad dmesg lines, no reset. DRC passed 3/5 (2160p to 1080p with START and with STREAMOFF, 4444 to 422). It failed both changes to a larger format (1080p to 2160p, 422 to 4444 at 1080p): a CAPTURE QBUF of an old buffer returned EINVAL while the last old frames were still draining.

Fix, cc64e59: device_run switches the CAPTURE format before the client sees SOURCE_CHANGE, and buf_prepare then checked the old buffers against the new, larger format. The decoder's CAPTURE buffers are no longer checked at QBUF; queue_setup sized them for their format. device_run gives back a buffer too small for the current format with ERROR and no payload, after the context's frames in the ring, and keeps the source frame queued.

Testing
  • arm64 LLVM W=1 build of drivers/media/platform/apple/prores/: 0 warnings; out-of-tree W=1 for fleet79 and fleet81: 0 warnings.
  • KUnit (arm64 qemu): 35/35 pass. checkpatch --strict: clean.
  • Still to run: the DRC and OOB rows again on the Neo with cc64e59 (queued), then the other Macs.

Assisted-by: LLM (Claude:Opus 5.5)

@RyanTheTide

RyanTheTide commented Nov 22, 2026 •

Copy link
Copy Markdown
Contributor Author

Neo (J700, T8140, ProRes id 0x40010) row at cc64e59, fleet81 kernel, this driver swapped in at runtime, default command line:

row result
SHA, 6 clips identical to the earlier rows (0b67a79a 1ec84fa0 42572d48 192fc850 15288276 134b35db)
MIXED encode + decode SAME
DRAIN (bad frame last / middle / none, 1080p and 2160p) 6/6 PASS
KILL x20, TWO contexts SAME
OOB frames + TOCTOU rewrite after QBUF and from a flipping thread 60/60 pass, 0 DART faults, no timeouts
DRC: 1080p to 2160p, 2160p to 1080p (START and STREAMOFF), 422 to 4444, 4444 to 422 5/5 PASS, output equal to the two solo decodes
new kernel log lines matching faults/timeouts/WARN 0

The two upsizing DRC cases that failed at 0f162d2 pass now. For comparison, the same TOCTOU flip case on 64b404a produced repeating DART translation faults and reset the Neo.

FPS, same Neo and kernel, interleaved N=3 medians, 64b404a vs 0f162d2 (the driver-owned source copy; cc64e59 doesn't touch this path):

clip 64b404a 0f162d2
1080p 422 HQ p216 2693.1 2708.1
1080p 422 HQ nv16 3293.6 3308.5
1080p 4444 p416 1804.1 1816.3
1080p Proxy p216 2693.0 2692.6
2160p 422 HQ p216 680.3 681.0
2160p 4444 p416 453.7 453.6

The copy costs nothing measurable here, including the 2160p 4444 worst case.

Still to run: the same rows on M1 Pro, J616s, a T6050 and J815.

Assisted-by: LLM (Claude:Opus 5.5)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants