Skip to content

drm/asahi: m3: let queued compute submissions share a firmware batch - #2849

Open
RyanTheTide wants to merge 1 commit into
integration/aurorafrom
j615gpu/m3-compute-join
Open

RyanTheTide wants to merge 1 commit into
integration/aurorafrom
j615gpu/m3-compute-join

Conversation

@RyanTheTide

Copy link
Copy Markdown
Contributor

Problem

On the M3 runtime (J613/J615), queued compute submissions never share a firmware batch. Later pending packets may join render batches only, so each vkQueueSubmit of compute work pays a full firmware round trip. The kernel then publishes the next batch only after full retirement. On a J615 (15-inch M3 Air, 27.2 firmware) that round trip is about 260 µs for a 182 ns shader, so a light compute job costs ~390 µs per submit even at queue depth 8. The Neo (A18 Pro) needs about 17 µs, on Linux and on macOS.

Change

  • may_join() no longer requires a render batch. A later packet of the same VM and urgency may extend a compute batch from its first command, as render batches already allow.
  • Compute batches get the same GPU-time budget as render batches (m3_render_batch_budget_us, doc updated). It uses a per-VM measured GPU time per compute command (compute_cost, recorded at retirement like pass_cost). With no measurement yet, a batch takes one command.
  • m3_compute_batch_size now defaults to the compute storage's slot count (m3_compute_storage::SLOTS) instead of 1. In practice the budget and the pending queue bound the batch. m3_compute_batch_override still wins when it is set.

No new parameters, no chip or board gate. It works on the default command line.

Testing

J615, one-shot kernel = fleet81 (1b940c26, the J615 default, which carries #2781) + this commit, default command line, gpubench ae90a40, system Mesa, headless, silent.

In one boot, m3_compute_batch_override was flipped 0 / 1 / 0 / 1. At 0 the runtime uses this PR's default; 1 is the old behaviour, one command per batch. Microseconds per submit, lower is better:

gpubench row this PR old this PR old
compute.light, depth 8 178.2 394.9 173.6 396.8
render.light clear, depth 8 174.8 396.8 173.6 397.5
compute.heavy alu, depth 8 856.2 1207.7 844.6 1184.8
compute.heavy bw, depth 8 1644.7 1968.5 1644.7 1984.1
compute.light, depth 1 468.2 465.5 463.0 465.5
render.light tri, depth 8 500.0 500.0 500.0 500.0
  • A queued light compute submit costs 56% less, and heavy ALU compute 28% less. At depth 1 there is nothing to join, so nothing changes. render.light tri doesn't move: fleet81 lacks drm/asahi: m3: default to render batches of two #2090, and that row depends on the render batch size.

  • With the PR default the runtime retired 1.91 compute commands per firmware round trip, against 1.00 with the old behaviour. 0 firmware error events in all four legs.

  • gpubench-stress-vk --kernel-log at the default: all 10 tests that don't inject a GPU fault pass, about 940k checks, 0 firmware error lines.

  • Vulkan CTS 1.4.7.1, every dEQP-VK.compute.* case of our 636k-case lab sample (17,583 cases): 3,745 Pass, 13,838 NotSupported, 0 Fail. That matches the same sample on the previous J615 kernel exactly.

  • 0 AVC, 0 failed units. The J615 was back on its default kernel afterwards.

  • run-host-tests.py --only m3: 9/9 pass (m3-batching-tests 12 tests, may_join cases updated).

  • checkpatch: only line-length warnings, in the style of the surrounding file (m3_runtime.rs has 126 lines over 100 columns before this change).

Still to run

  • J613 (13-inch M3 Air): same runtime, same default. The lab has no J613.
  • A Vulkan CTS compute subset at the new default.
  • compute.long and life.fault in gpubench-stress-vk take the GPU down on the J615 with or without this change (firmware event 4, no recovery after a fault). That is the open fault-recovery item on drm/asahi: m3: run the M3 (T8122 G15G) GPU on the macOS 27.2 firmware #2781, not part of this PR.
Method and raw data
  • Kernel: 7.3.0-aurora-pub4cjp = fleet81 1b940c26 + this commit (tree identical to a700462), a J615-only module set, default command line, grub2-reboot one-shot.
  • Before the PR form, a dev kernel let compute join without the budget and without the new default. In-boot ABAB of the override 1 / 8 / 1 / 8: compute.light at depth 8 was 393.7 / 173.6 / 395.6 / 173.6 µs and compute.heavy alu at depth 8 was 1190 / 868 / 1168 / 845 µs. So the budget costs nothing on these rows.
  • Levers ruled out first, on the same J615 and the fleet81 default:
    • Completion-poll backoff, debugfs completion_wait 1000 / 25 / 1000 / 25: compute.light at depth 8 was 396 / 368 / 372 / 369 µs. Noise.
    • Firmware idle-off standby: a one-shot Stage 2 that only adds apple,idleoff-standby-timer = <2500> to the GPU node, run as A/B/A. compute.light at depth 8 was 396.8 / 396.8 / 388.8 µs. No effect.
    • The round trip is fixed time per batch, and this change spreads it across the commands in a batch.
  • Fairness: only packets of the same VM and urgency join, and the GPU-time budget (4000 µs on T8122) bounds each compute batch like a render batch. A VM's first compute batch has no measured cost yet, so it holds one command. Other clients wait at most one budgeted batch.
  • gpubench-stress-vk compute.long and life.fault take the GPU down on this J615 at the old setting too: firmware event 4 or a lost device, then no recovery for the rest of the boot. With the old behaviour selected they fail identically.
  • Evidence: our lab traces traces/gpu/20261123-j615-gpu-rtt (cjoin/, cjoin-pr/, cwait/, idleoff/).

Assisted-by: LLM (Claude:Opus 5.5)

The M3 runtime publishes one compute batch at a time and publishes the
next one only after the previous one fully retired. Later pending
packets could join render batches only, so every queued compute
submission paid its own firmware round trip. On a J615 (M3, 27.2
firmware) that round trip is about 260 us for a 182 ns shader, so a
light compute job costs ~390 us per vkQueueSubmit even at queue depth 8.

Let later packets of the same VM and urgency join a compute batch from
their first command, as render batches already do, and bound compute
batches by the same GPU-time budget as render batches, using a per-VM
measured GPU time per compute command. The compute batch size now
defaults to the compute storage's slot count; the budget and the
pending queue bound the batch in practice. The runtime knob
m3_compute_batch_override still wins when set.

On the J615 (one-shot kernel, in-boot ABAB of the batch size 1/8/1/8):
compute.light at depth 8 goes from 394 to 174 us per submit, a light
render clear from 363 to 173 us, heavy ALU compute from 1180 to 857 us,
with 0 firmware error events. gpubench-stress-vk passes every test that
does not inject a GPU fault.

Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
@RyanTheTide

Copy link
Copy Markdown
Contributor Author

Review at a700462.

Thanks, the cross-packet join is small and the J615 data is convincing. A few problems before this becomes the default:

  1. m3_params.rs:213 also changes the default for J613. J613 runs the Extended compute profile (t8122_admission.rs:60), and until now it only ever ran slot 0 of the compute storage by default. With this change every queued compute submit on J613 uses slots 1..15, the Extended offsets (m3_compute.rs:303/307) and append() (m3_compute.rs:252), and none of that has run on hardware. The J615 rows only cover the Legacy profile. This needs a J613 row (gpubench compute.light/heavy at depth 8, stress-vk, dEQP-VK.compute.*) before merge.
  2. The 2 s completion timeout is per batch (m3_runtime.rs:737, :813), but the budget that sizes the batch is a prediction: a smoothed per-VM mean (compute_cost, m3_runtime.rs:912). Take a VM that measured ~100 us per command and then queues long dispatches, e.g. 16 x 200 ms. All 16 still pass within_budget (16 x 100 us < 4 ms) and land in one batch. The batch hits ETIMEDOUT and fail_active takes down the GPU with no recovery. Before this change each 200 ms dispatch had its own 2 s. Scale the deadline with entries.len(), or time out on per-slot progress, so that a wrong cost estimate cannot turn valid work into a fatal fault.
  3. Pre-publication failure attribution, m3_runtime.rs:419 and :434-449. mark_published() runs on every joined packet before append/set_attachments. If a joined packet's command fails there, publish returns (err,false) and fail_front (m3_runtime.rs:380) fails the front packet, which was valid. The joined packets stay pending with published=true, so a later unpublished failure keeps their VM guard (release_unpublished). Mark each packet only once its commands are encoded, or fail the packet whose entry failed.
  4. The m3_render_batch_budget_us doc (m3_params.rs:487-490) says "disabled on Pro", but this runtime only probes on T8122 (driver.rs:363-364). Drop that clause rather than extend it to compute.
  5. The commit message gives different numbers from the PR: "light render clear from 363", and an "ABAB of the batch size 1/8/1/8", which was the earlier dev kernel. Please quote the measurements of this commit.
  6. m3-batching-tests.rs: add a case for the compute budget path, i.e. unknown cost gives 1 command and a measured cost caps the count. Right now only may_join is covered.

Rows owed at this head: J613 (see 1). The other admitted T8122 boards, if any beyond J613/J615, need a boot plus the gpubench compute row.

@DjDeveloperr DjDeveloperr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pre-PR Checklist Review (§1–§6) — a700462ffa129c54018dcd7798bed7ccb056faa1

The cross-packet compute batch join is concise and the J615 A/B improvement (~56% latency reduction on queued compute.light) is solid, with clean §4 attribution (human author/committer + human Signed-off-by:, zero AI trailers in commit) and green host-tests CI. However, there are concrete safety/failure-mode blockers and missing §5 hardware coverage that need to be resolved before making multi-slot compute batching the default.

Blockers

  1. Fixed 2 s per-batch completion timeout vs. predictive compute_cost can fatally fault valid workloads (§2 / §3):

    • poll_active() (drivers/gpu/drm/asahi/m3/m3_runtime.rs:737, :813) enforces a fixed 2 s deadline per batch (batch.start.elapsed() >= Delta::from_secs(2)), and expiration triggers fail_active(ETIMEDOUT) -> capture_fault(), taking the GPU down unrecoverably for the rest of the boot.
    • next_batch() (m3_runtime.rs:261-274) bounds the batch using compute_cost (m3_runtime.rs:912), which is a smoothed historical per-VM mean. Previously (M3_COMPUTE_BATCH_SIZE = 1), every compute dispatch had its own 2 s timeout. With the default raised to SLOTS (16), a VM whose prior light dispatches establish a low compute_cost (e.g., ~100 µs) and that then queues heavier dispatches (e.g., 16 × 200 ms) passes within_budget() (16 × 100 µs = 1.6 ms <= 4 ms), lands all 16 commands in one batch, trips the fixed 2 s timeout, and permanently kills the GPU.
    • Scale the batch completion deadline with entries.len(), or track/extend the deadline on per-slot progress, so a stale cost estimate cannot turn valid compute submissions into a fatal GPU fault.
  2. Pre-publication failure misattribution and leaked VM quarantine (m3_runtime.rs:413, :431-452, :380) (§2 / §3):

    • publish() marks every packet in entries with p.mark_published() at m3_runtime.rs:413 before encoding any compute command (Compute::new / replay / append / prepare_completion / set_user_timestamps / set_attachments, lines 418-446).
    • If a joined packet (slot > 0) fails during encoding, publish() rolls back via j.abort_batch() and returns Err((error, false)), and publish_next() (m3_runtime.rs:380) calls self.fail_front(error). That fails pending[0] (a valid packet) while leaving the failing joined packet in pending, and leaves all joined packets permanently marked published = true (Packet::published is never cleared).
    • If a later unpublished failure hits fail_pending(), policy::release_unpublished(packet.may_be_published()) returns false for those never-published packets, permanently quarantining their VM job guards even though firmware never saw them.
    • Defer mark_published() on non-overlapping batches until after encoding succeeds (right before firmware-visible publication), and attribute pre-publication encoding failures to the failing packet rather than unconditionally failing pending[0].
  3. Missing J613 (Profile::Extended) hardware evidence before changing its default (§1 / §5):

    • Raising M3_COMPUTE_BATCH_SIZE (drivers/gpu/drm/asahi/m3/m3_params.rs:213) from 1 to SLOTS (16) also changes the default on J613, which runs Profile::Extended (t8122_admission.rs:60).
    • Until now J613 only ever exercised compute slot 0 by default; this change immediately puts J613 onto slots 1..15, Compute::append() (m3_compute.rs:252), and the Extended slot offsets (m3_compute.rs:303, :307, :314; m3_compute_storage.rs:106-110) with zero hardware runs on Profile::Extended.
    • Per §1 and §5, please provide a J613 hardware row at the updated head (gpubench compute.light/heavy depth 8 A/B, gpubench-stress-vk, dEQP-VK.compute.*, 0 AVC / 0 failed units / 0 new kernel errors) before enabling multi-slot compute batching by default there.

Non-blocking / Cleanup Items

  • Commit message numbers vs. PR measurements: The commit message still cites the earlier dev-kernel numbers (in-boot ABAB of the batch size 1/8/1/8, light render clear from 363) rather than this commit's actual 0/1/0/1 measurements reported in the PR body. Also, the PR body's "Still to run" still lists a Vulkan CTS compute subset even though the Testing section reports dEQP-VK.compute.* (17,583 cases, 3,745 Pass, 0 Fail).
  • Stale doc clause (m3_params.rs:489-490): m3_render_batch_budget_us still says disabled on Pro, even though the M3 runtime only probes on T8122 (driver.rs:363-364). Please drop that clause rather than extending it to compute.
  • Host test coverage (tools/asahi/m3-batching-tests.rs): Only may_join was updated; please add a test covering the compute budget path (unmeasured compute_cost admits 1 command; measured cost caps the batch count).

Assisted-by: LLM (Gemini 4 Neon)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants