Repository navigation
drm/asahi: m3: let queued compute submissions share a firmware batch - #2849
RyanTheTide wants to merge 1 commit into
Conversation
The M3 runtime publishes one compute batch at a time and publishes the next one only after the previous one fully retired. Later pending packets could join render batches only, so every queued compute submission paid its own firmware round trip. On a J615 (M3, 27.2 firmware) that round trip is about 260 us for a 182 ns shader, so a light compute job costs ~390 us per vkQueueSubmit even at queue depth 8. Let later packets of the same VM and urgency join a compute batch from their first command, as render batches already do, and bound compute batches by the same GPU-time budget as render batches, using a per-VM measured GPU time per compute command. The compute batch size now defaults to the compute storage's slot count; the budget and the pending queue bound the batch in practice. The runtime knob m3_compute_batch_override still wins when set. On the J615 (one-shot kernel, in-boot ABAB of the batch size 1/8/1/8): compute.light at depth 8 goes from 394 to 174 us per submit, a light render clear from 363 to 173 us, heavy ALU compute from 1180 to 857 us, with 0 firmware error events. gpubench-stress-vk passes every test that does not inject a GPU fault. Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
|
Review at a700462. Thanks, the cross-packet join is small and the J615 data is convincing. A few problems before this becomes the default:
Rows owed at this head: J613 (see 1). The other admitted T8122 boards, if any beyond J613/J615, need a boot plus the gpubench compute row. |
DjDeveloperr
left a comment
There was a problem hiding this comment.
Pre-PR Checklist Review (§1–§6) — a700462ffa129c54018dcd7798bed7ccb056faa1
The cross-packet compute batch join is concise and the J615 A/B improvement (~56% latency reduction on queued compute.light) is solid, with clean §4 attribution (human author/committer + human Signed-off-by:, zero AI trailers in commit) and green host-tests CI. However, there are concrete safety/failure-mode blockers and missing §5 hardware coverage that need to be resolved before making multi-slot compute batching the default.
Blockers
-
Fixed 2 s per-batch completion timeout vs. predictive
compute_costcan fatally fault valid workloads (§2 / §3):poll_active()(drivers/gpu/drm/asahi/m3/m3_runtime.rs:737, :813) enforces a fixed 2 s deadline per batch (batch.start.elapsed() >= Delta::from_secs(2)), and expiration triggersfail_active(ETIMEDOUT)->capture_fault(), taking the GPU down unrecoverably for the rest of the boot.next_batch()(m3_runtime.rs:261-274) bounds the batch usingcompute_cost(m3_runtime.rs:912), which is a smoothed historical per-VM mean. Previously (M3_COMPUTE_BATCH_SIZE = 1), every compute dispatch had its own 2 s timeout. With the default raised toSLOTS(16), a VM whose prior light dispatches establish a lowcompute_cost(e.g., ~100 µs) and that then queues heavier dispatches (e.g., 16 × 200 ms) passeswithin_budget()(16 × 100 µs = 1.6 ms <= 4 ms), lands all 16 commands in one batch, trips the fixed 2 s timeout, and permanently kills the GPU.- Scale the batch completion deadline with
entries.len(), or track/extend the deadline on per-slot progress, so a stale cost estimate cannot turn valid compute submissions into a fatal GPU fault.
-
Pre-publication failure misattribution and leaked VM quarantine (
m3_runtime.rs:413, :431-452, :380) (§2 / §3):publish()marks every packet inentrieswithp.mark_published()atm3_runtime.rs:413before encoding any compute command (Compute::new/replay/append/prepare_completion/set_user_timestamps/set_attachments, lines418-446).- If a joined packet (
slot > 0) fails during encoding,publish()rolls back viaj.abort_batch()and returnsErr((error, false)), andpublish_next()(m3_runtime.rs:380) callsself.fail_front(error). That failspending[0](a valid packet) while leaving the failing joined packet inpending, and leaves all joined packets permanently markedpublished = true(Packet::publishedis never cleared). - If a later unpublished failure hits
fail_pending(),policy::release_unpublished(packet.may_be_published())returnsfalsefor those never-published packets, permanently quarantining their VM job guards even though firmware never saw them. - Defer
mark_published()on non-overlapping batches until after encoding succeeds (right before firmware-visible publication), and attribute pre-publication encoding failures to the failing packet rather than unconditionally failingpending[0].
-
Missing J613 (
Profile::Extended) hardware evidence before changing its default (§1 / §5):- Raising
M3_COMPUTE_BATCH_SIZE(drivers/gpu/drm/asahi/m3/m3_params.rs:213) from1toSLOTS(16) also changes the default on J613, which runsProfile::Extended(t8122_admission.rs:60). - Until now J613 only ever exercised compute slot 0 by default; this change immediately puts J613 onto slots
1..15,Compute::append()(m3_compute.rs:252), and the Extended slot offsets (m3_compute.rs:303, :307, :314;m3_compute_storage.rs:106-110) with zero hardware runs onProfile::Extended. - Per §1 and §5, please provide a J613 hardware row at the updated head (
gpubenchcompute.light/heavydepth 8 A/B,gpubench-stress-vk,dEQP-VK.compute.*, 0 AVC / 0 failed units / 0 new kernel errors) before enabling multi-slot compute batching by default there.
- Raising
Non-blocking / Cleanup Items
- Commit message numbers vs. PR measurements: The commit message still cites the earlier dev-kernel numbers (
in-boot ABAB of the batch size 1/8/1/8,light render clear from 363) rather than this commit's actual0/1/0/1measurements reported in the PR body. Also, the PR body's "Still to run" still lists a Vulkan CTS compute subset even though the Testing section reportsdEQP-VK.compute.*(17,583 cases, 3,745 Pass, 0 Fail). - Stale doc clause (
m3_params.rs:489-490):m3_render_batch_budget_usstill saysdisabled on Pro, even though the M3 runtime only probes on T8122 (driver.rs:363-364). Please drop that clause rather than extending it to compute. - Host test coverage (
tools/asahi/m3-batching-tests.rs): Onlymay_joinwas updated; please add a test covering the compute budget path (unmeasuredcompute_costadmits 1 command; measured cost caps the batch count).
Assisted-by: LLM (Gemini 4 Neon)
Problem
On the M3 runtime (J613/J615), queued compute submissions never share a firmware batch. Later pending packets may join render batches only, so each
vkQueueSubmitof compute work pays a full firmware round trip. The kernel then publishes the next batch only after full retirement. On a J615 (15-inch M3 Air, 27.2 firmware) that round trip is about 260 µs for a 182 ns shader, so a light compute job costs ~390 µs per submit even at queue depth 8. The Neo (A18 Pro) needs about 17 µs, on Linux and on macOS.Change
may_join()no longer requires a render batch. A later packet of the same VM and urgency may extend a compute batch from its first command, as render batches already allow.m3_render_batch_budget_us, doc updated). It uses a per-VM measured GPU time per compute command (compute_cost, recorded at retirement likepass_cost). With no measurement yet, a batch takes one command.m3_compute_batch_sizenow defaults to the compute storage's slot count (m3_compute_storage::SLOTS) instead of 1. In practice the budget and the pending queue bound the batch.m3_compute_batch_overridestill wins when it is set.No new parameters, no chip or board gate. It works on the default command line.
Testing
J615, one-shot kernel = fleet81 (
1b940c26, the J615 default, which carries #2781) + this commit, default command line, gpubenchae90a40, system Mesa, headless, silent.In one boot,
m3_compute_batch_overridewas flipped 0 / 1 / 0 / 1. At 0 the runtime uses this PR's default; 1 is the old behaviour, one command per batch. Microseconds per submit, lower is better:A queued light compute submit costs 56% less, and heavy ALU compute 28% less. At depth 1 there is nothing to join, so nothing changes. render.light tri doesn't move: fleet81 lacks drm/asahi: m3: default to render batches of two #2090, and that row depends on the render batch size.
With the PR default the runtime retired 1.91 compute commands per firmware round trip, against 1.00 with the old behaviour. 0 firmware error events in all four legs.
gpubench-stress-vk --kernel-logat the default: all 10 tests that don't inject a GPU fault pass, about 940k checks, 0 firmware error lines.Vulkan CTS 1.4.7.1, every
dEQP-VK.compute.*case of our 636k-case lab sample (17,583 cases): 3,745 Pass, 13,838 NotSupported, 0 Fail. That matches the same sample on the previous J615 kernel exactly.0 AVC, 0 failed units. The J615 was back on its default kernel afterwards.
run-host-tests.py --only m3: 9/9 pass (m3-batching-tests 12 tests,may_joincases updated).checkpatch: only line-length warnings, in the style of the surrounding file (m3_runtime.rs has 126 lines over 100 columns before this change).
Still to run
compute.longandlife.faultin gpubench-stress-vk take the GPU down on the J615 with or without this change (firmware event 4, no recovery after a fault). That is the open fault-recovery item on drm/asahi: m3: run the M3 (T8122 G15G) GPU on the macOS 27.2 firmware #2781, not part of this PR.Method and raw data
7.3.0-aurora-pub4cjp= fleet811b940c26+ this commit (tree identical to a700462), a J615-only module set, default command line,grub2-rebootone-shot.completion_wait1000 / 25 / 1000 / 25: compute.light at depth 8 was 396 / 368 / 372 / 369 µs. Noise.apple,idleoff-standby-timer = <2500>to the GPU node, run as A/B/A. compute.light at depth 8 was 396.8 / 396.8 / 388.8 µs. No effect.compute.longandlife.faulttake the GPU down on this J615 at the old setting too: firmware event 4 or a lost device, then no recovery for the rest of the boot. With the old behaviour selected they fail identically.traces/gpu/20261123-j615-gpu-rtt(cjoin/, cjoin-pr/, cwait/, idleoff/).Assisted-by: LLM (Claude:Opus 5.5)