Repository navigation
drm/asahi: g16 + shared SUBMIT: Wave 6 R2 — demand-grown 32 render slots / 18 compute units, fast SUBMIT path, stamp-only barriers, and batched render release - #2826
Conversation
|
Review at ac4f4cc.
|
|
Three additions to the review above (same head, ac4f4cc), not covered there:
Assisted-by: LLM (Claude:Opus 5.5) |
…ooth receipt-tail backoff Reduce G16 completion-poll overhead while preserving per-CPU completion locality and the full post-event receipt-tail window: 1. Skip redundant `publish_checked()` walks in `Firmware::poll_step()` when flights are active and nothing changed (no retirement, no ring headroom wait, no recovery unfreeze, no off-lock scrub wakeup, no external `publish_wake`, and no 1 ms housekeeping tick). Add `Shared::publish_wake` / `Shared::wake_publish()` so external state transitions (VM close, unstarted queue cancellation) re-evaluate `engine.pending` immediately without waiting for the 1 ms tick; `Prepared::drop` only releases standby PB reservations for `Cache::prepare` waiters, whose woken `SUBMIT` enters `submit_locked()` directly. 2. Keep `aurora_g16_events` per-CPU (`new_percpu_reclaim`) and `EVENT_RETIRE_POLLS = 11`, smoothing the trailing receipt-poll backoff from `1/2/5 us` (~48 us) to `1/2/3/4/5 us` (45 us total) so post-EP 0x20 retirement finishes inline without falling back to the repoll timer. 3. Batch `Events::drain()` and `Events::drain_statistics()` ring advances: read only the active words of common type-1 stage events (3 WC reads vs 18) and non-recovery statistics records (1 WC read vs 18), and write `STATE_READ` once per drained burst (flushing partial progress on mid-burst errors). 4. Cache `recovery_pending()` once per `submit_locked()` / `poll_step()` pass and short-circuit `retire_once()` when `flights` is empty. Signed-off-by: Diljit Singh <hey@djdev.me>
… workqueue Move off-lock write-combining zeroing of displaced render pass slots out of `Shared::try_submit_inline()`, `Shared::submit()`, and `run_poll()` onto a dedicated unbound reclaim workqueue (`aurora_g16_scrub`): - Add a zero-allocation slot-indexed `ScrubQueue` protected by `Shared::scrub_queue` (`SpinLock`) and a `SCRUB_WORK` work item on `Shared::scrub_wq` so SUBMIT ioctls and completion polls never stall on WC fills. - Preserve strict slot-reuse exclusion (`scrubbing` bitmask) and wake waiting render submitters via `poll_now()` immediately when a waited slot finishes scrubbing. - Guard async `clean_mask` publication with a monotonic `generation` counter incremented by `invalidate_scrub()` so a scrub staged before firmware recovery can never leave a stale clean bit afterward. - Drain `scrub_queue` and `scrub_wq` cleanly in `sleep.rs::join_workers()` after `sleeping` is set and `events` is drained so S2R and teardown never overlap workqueue draining with concurrent enqueues. Signed-off-by: Diljit Singh <hey@djdev.me>
…and compute units to 18
Eliminate the 15-flight displacement cap at queue depth 16 and 32 for
both render and compute on G16 without imposing a 107.8 MiB per-VM
memory jump on compositors or shallow desktop VMs:
- Pack lazy extension render pass slots (`2..32`) at an 8 MiB VA stride
(`EXT_PASS_STRIDE = 0x00800000`, tilemap at `+0x600000`) starting at
`0x7024000000`, while keeping base slots `0..2` at their full 32 MiB
intervals (`16 MiB` TPC / `8 MiB` tilemap). All 32 pass slots occupy
`0x7020000000..0x7033000000` (208 MiB spare below `0x7040000000`),
raising `render_flights()` from 15 to 31.
- Grow extension pass slots in three demand-grown `RenderExtArena` tiers
(`EXT_RENDER_TIERS = [(2, 8), (8, 18), (18, 32)]`) sized against live
render pass count (`active_render_passes + render_count + 1`). Each
base slot backs 24.59 MiB (25,184 KiB: 16 MiB TPC + 8 MiB tilemap +
608 KiB metadata/scratch/barriers) and each extension slot backs
3.59 MiB (3,680 KiB: 2 MiB TPC + 1 MiB tilemap + 608 KiB):
- Base only (`0..2`, steady 60/120 Hz single-pass): 49.19 MiB pass
backing (0 MiB ext).
- Tier 0 (`2..8`, +6 slots, compositors / 2-pass / shallow depth):
+21.56 MiB ext (80% less than eager 30-slot allocation).
- Tier 1 (`8..18`, +10 slots, moderate multi-pass bursts):
+35.94 MiB (+57.50 MiB cum ext).
- Tier 2 (`18..32`, +14 slots, deep multi-pass bursts):
+50.31 MiB (+107.81 MiB cum ext).
- Fix `PASS_MASK` in `g16/slots.rs` to `u32::MAX >> (32 - RENDER_SLOTS)`
so `RENDER_SLOTS = 32` does not overflow `1u32 << 32`, and raise
`MAX_CLIENT` from 320 to 512 for headroom.
- Expand `COMPUTE_UNITS` from 16 to 18 (17 concurrent compute flights,
covering `d=16` without displacement stalls), fitting directly inside
the existing 64 MiB VA layout (`compute_base(18) == PB_HEAP`) and UMA
slot range (`COMPUTE_UMA_SLOT + 18 <= RENDER_UMA_SLOT`), and simplify
`params::compute_units()` with a compile-time `COMPUTE_UNITS >= 2`
assertion in `g16/hw.rs`.
Signed-off-by: Diljit Singh <hey@djdev.me>
…ing, and synchronize resnap Eliminate repeated SUBMIT-path CPU work across stream parsing, CDM stream gating, and completion timer management: - Replace the three-pass SUBMIT stream walk (check_stream -> prepare -> validate) with single-pass parse_stream() -> ParsedStream::prepare() -> Prepared::validate(). Structural validation and attachment parsing run once before device-state checks, preserving exact EINVAL/EOPNOTSUPP precedence over crashed/vm.status EIO. Attachment KVecs move directly into the first consuming command with zero copy, timestamp destinations resolve in place via timestamp::Destinations::empty() (G16-only), and validate() operates on the already-decoded commands. Remove the now-dead check_stream, cover, and Resolved helpers from g16/command.rs. - Expose VmReadGuard::covers_range() to G16 in mmu.rs and add command::cover_locked() so gate::cdm() holds a single vm.read_lock() across stream window reads, USC program checks, and indirect parameter checks, dropping the lock and invalidating cached window/program proofs before might_sleep() every CDM_RESCHED_RECORDS records. - Double-check resnap_deadline_ns and sleeping under resnap_handle.lock() in rearm(), and unconditionally acquire resnap_handle.lock() in cancel_resnap() before clearing resnap_deadline_ns and dropping the handle. This avoids redundant cancel+start cycles under racing callers while strictly ordering sleep_down() (sleeping = true then cancel_resnap()) against concurrent rearm() so no hrtimer handle can leak into suspend. Signed-off-by: Diljit Singh <hey@djdev.me>
…InitBM handoffs Use BarrierType::Stamp (type 0) instead of BarrierType::Hold (type 1) for same-VM non-InitBM cross-engine barriers (fs2c in prefix_compute() and c2ta/c2fs in prefix_render()), eliminating software-job hold bubbles on compute <-> render handoffs (such as postfx and deferred shading). Hardware memory-visibility invariant: each G16 GPU stage executes its FINALIZE micro-sequence (flushing stage caches to the shared coherent fabric) before firmware advances fw_stamp. Within a single VM (same ASID and page tables), waiting on fw_stamp via BarrierType::Stamp establishes full producer-to-consumer memory visibility without a software-job hold. When either the waited producer or the consumer carries InitBM, keep BarrierType::Hold via cross_engine_barrier_type(producer_initbm, consumer_initbm) across fs2c, c2ta, c2fs, and pipelined render->render barriers (with WaitTarget propagating producer InitBM state from wait_stamp_for_with() and cross_engine_wait()). Cross-VM transitions, recovery paths, and query_access CPU-timestamp waits drain flights completely before publishing. Audited waits_for_timestamps in can_queue_behind(): because G16 copies timestamp results into user query-pool BOs on CPU upon flight retirement (publish_timestamps), keeping the retirement wait when query_access is true (userspace did not pass DRM_ASAHI_COMPUTE_NO_QUERY_ACCESS) remains required for query-pool correctness and is documented inline. Signed-off-by: Diljit Singh <hey@djdev.me>
…er validation Vm::read_lock() previously called lock_vm(&self.gpuvm, None, false), which allocates a KBox<drm_gpuvm_exec>, runs drm_exec_init() (allocating a page-sized object array), and locks every external BO reservation on the VM (O(external BOs)) on every G16/G15 SUBMIT. Because every mapping mutation in mmu.rs holds the VM's shared reservation, read-only page-table coverage walks and user-memory copies need only self.gpuvm.lock_inner(), matching covers_ranges_batch(): 1. Back VmReadGuard<'a> with GpuVmInnerGuard<'a, VmInner> via self.gpuvm.lock_inner(), eliminating the per-SUBMIT KBox, drm_exec object-array allocation, and O(external BOs) lock walk. 2. In G16 Prepared::validate(), validate render VM spans and VDM root barriers under one Vm::read_lock() hold via SpanBatch::flush_locked() and gate::vdm_batch_locked(). 3. Minimize PoC cache maintenance (dc civac + dsb sy) in G16 stream gates: read only 4 bytes (1 cache line) for single-pass VDM roots and clamp CDM root-stream window reads to the validated root end. Signed-off-by: Diljit Singh <hey@djdev.me>
Avoid per-SUBMIT heap allocations without placing a large command buffer on the kernel stack across queue.lock().submit(): store a reusable KVec<u8> staging buffer on TableQueue (retained up to 4096 bytes between submissions, releasing oversized buffers on completion or error) and read drm_asahi_sync entries in 8-entry (128-byte) stack chunks rather than issuing a separate copy_from_user per sync. On guarded (G16/G17) paths, acquire the queue mutex once to snapshot the command buffer before resolving output syncobjs, preserving the pre-syncobj failure boundary; on unguarded paths, keep parsing syncs before locking the queue. Signed-off-by: Diljit Singh <hey@djdev.me>
When multiple render commands are batched into a single G16 flight, engine publication previously called commit_render() once, leaving (f.renders - 1) standby PB reservations held until Prepared::drop on the deferred cleanup worker. Pass f.renders to commit_render() so all published render commands release their standby reservations immediately upon firmware visibility. Signed-off-by: Diljit Singh <hey@djdev.me>
ac4f4cc to
f631683
Compare
|
@RyanTheTide Thanks for the detailed review — pushed Wave 6 R2 (
(Note: I also saw your three addendum items in Assisted-by: LLM (Gemini:4 Neon) |
|
Re-checked at f631683 against both reviews at ac4f4cc. Fixed:
Still open:
New in this revision, minor:
Assisted-by: LLM (Claude:Opus 5.5) |
Summary
Eight-commit Wave 6 R2 (
f63168347300c) series stacked cleanly on#2787(djdev/g16-perf-wave2-5at5610a84b974e5):882a01ec1cd14drm/asahi: g16: skip no-op publishes, batch event ring drains, and smooth receipt-tail backoff3a970ce516736drm/asahi: g16: offload displaced render pass scrubbing to an unbound workqueue594a723798f0cdrm/asahi: g16: expand render pass slots to 32 in demand-grown tiers and compute units to 185d6bef0371307drm/asahi: g16: parse SUBMIT streams in one pass, batch CDM gate locking, and synchronize resnap086fda9437006drm/asahi: g16: use stamp-only cross-engine barriers for same-VM non-InitBM handoffs11fb7fab68da5drm/asahi: use VM reservation lock for VmReadGuard and merge G16 render validation(sharedmmu.rs+ G15/G16)b60cdb650c6f6drm/asahi: reuse per-queue SUBMIT cmdbuf staging and bulk-read syncs(sharedfile.rsacross generations)f63168347300cdrm/asahi: g16: release full batched render count in commit_render()Verification (exact head
f63168347300c)Host checks
scripts/checkpatch.pl --strict --ignore FILE_PATH_CHANGES:0 errors, 0 warnings, 0 checksacross all 8 commits.run-host-tests.py:25 / 25PASS.Live hardware (
Apple M4 (G16G B0),J773g/T8132,7.3.0-ARCH-m4prod17,26B123firmware)900Pass /1,687NotSupported /0Fail /0Crash /0Timeout /0GPU errors):vk-mustpass-short:87 / 87PASSdEQP-VK.synchronization.*(1,500strided of64,872):664Pass /836NS /0FaildEQP-VK.memory_model.*(1,000strided of18,530):149Pass /851NS /0Failtype0: fs2c 13873 c2ta 20506(0faults).gpubench-vk(0CRC/status fails across 44 rows,pb: bad 0): 24-row 4K@120Hz + 4overlap.passes+ 6render.postfx+ 2compress.bw array+ 8render.deferred.rc=0across all concurrent and fresh post-hang workloads):wait_rc=-4,555.9 ms,recoveries 2 -> 3,compute_owners: retired 16 quarantined 2 reusable 15): concurrentrender.heavy d=4rc=0(33,606submits,252.02 usvs249.58 usclean), immediate freshd=1compute.heavy/compute.light/render.heavyallrc=0,0ECANCELED.wait_rc=-4,120.8 ms,recoveries 3 -> 4,render_owners: retired 30 quarantined 2 reusable 29,render_failed=0,degraded=0,pb: bad 0): concurrentcompute.heavy d=4rc=0(16,175submits,550.05 usvs547.45 usclean), immediate freshd=1render.heavy/render.postfx(CRC 668d6f0a)/compute.heavyallrc=0,0ECANCELED.vk-mustpass-short:87 / 87PASS,0kernelBUG/WARN/Oops/wedged.rtcwake -d /dev/rtc1 -m mem -s 15):apple-sleep: back from sleep, boot object resumes 0 -> 1,pcie-appleport 0 & 2 link restored,ZERO_S2R_ERRORS, post-wakevk-mustpass-short87 / 87PASS.Assisted-by: LLM (Gemini:4 Neon)