Repository navigation
drm/asahi: g16: standby PB pool, WC scrub/write trims, inline submit, multi-flight render pipelining, and Wave 3 submit/retire wins - #2741
DjDeveloperr wants to merge 10 commits into
Conversation
Previously Cache::prepare() popped a spare PbEpoch (cap 4) or allocated and UAT-mapped a fresh 80 KiB PbEpoch for every queued render job, only for almost every render to reuse the slot's existing epoch and discard the unused epoch on drop once submission depth exceeded 4. Keep a shared per-VM standby pool in Cache::standby guarded by SpinLockIrq with the invariant epochs.len() >= reserved for all accepted unpublished render commands. Cache::prepare() tops up only the deficit in SUBMIT ioctl context before any fence exists, waiting interruptibly on a CondVar for standby headroom during bursts instead of failing with E2BIG (matching Budget::reserve's 10-second administrative bound) and retrying if a concurrent InitBM pops an epoch before commit. Steady-state queued renders at any depth allocate zero PbEpochs and perform zero UAT mappings or window syncs. Fresh InitBM publications draw from standby under the device lock without allocating, unpublished rollbacks return the unexposed epoch, and Cache::close() drops surplus epochs immediately while retaining only reserved epochs until accepted jobs drain. Signed-off-by: Diljit Singh <hey@djdev.me>
…C reads Previously every compute dispatch zeroed 288.8 KiB of write-combined memory (256 KiB context_store, 16 KiB aux, 16 KiB preempt, unused seq, and 16 B its) and read back the 160-byte UMA descriptor byte by byte with read_volatile(). Every render pass zeroed ~530 KiB to >1 MiB of WC memory (including dead-code ta_scratch/fs_scratch, work/seq images immediately overwritten by prefix_render(), aux, scene_user, full 48 KiB preemption buffers, and tpc_dirty) synchronously under Shared::state. Track per-slot full-scrub necessity in Slots: - First use, cross-VM switch, cancelled preparation, affected error receipts, and firmware recovery (invalidate_scrub) perform the full scrub so no client or faulted state can ever leak. - Clean same-VM compute reuse zeroes only the 16-byte internal timestamp pair (its). - Clean same-VM render reuse zeroes only its (32 B), active heap header (4 KiB), tilemap_dirty, and active preemption record headers (2 KiB), preserving the tpc_dirty high-water mark until the next full scrub. - Upgrade FwObject::read() to 64-bit volatile word loads for aligned spans, reducing compute descriptor WC bus reads by 8x. Signed-off-by: Diljit Singh <hey@djdev.me>
…it ring can_queue_behind() previously refused to pipeline behind any in-flight compute command carrying timestamp_outputs, forcing depth-4+ timed compute workloads down to depth-1 serialization. Each compute flight already owns its own private ComputeSlot (its/work), and retire_oldest() publishes timestamps in FIFO publication order before signalling the flight's fence, so timed same-VM compute commands can pipeline freely. Expand hw::sizes::COMPUTE_UNITS and params::compute_units() from 8/6 to 16 (placing extension units at 0x7080000000..0x70b8000000, clear of the 0x7040000000..0x7080000000 dynamic PB page-list aperture and below PB_HEAP at 0x70c0000000, using hardware UMA slots 1..=16). With displacement lifetime tracking, 16 units sustain up to 15 in-flight compute commands per bursting VM instead of 5. Signed-off-by: Diljit Singh <hey@djdev.me>
…en poll cadence Eliminate four layers of host-side latency on the G16 submit/completion round trip while preserving sleep-transition safety: 1. Direct inline submit from SUBMIT ioctl: when the previous job on the logical queue has already started, all input fences are signalled, and Shared::state is uncontended (try_lock), Scheduler::push() publishes immediately on the submitting CPU before pushing to drm_sched (whose subsequent run() observes started == true as a no-op while preserving credits, fence ownership, and QUEUE_DESTROY drain semantics). Contended or dependency-blocked jobs fall back to drm_sched in strict queue order. 2. Submitter-pinned dedicated completion worker: replace WORK_CPU_UNBOUND on system_highpri_wq for EP 0x20 / hrtimer wakeups with a per-CPU high-priority reclaim OwnedQueue (aurora_g16_events) enqueued on the last submitting CPU, keeping WC receipt reads and fence wakeups hot on the submitting core; join cleanly in sleep::join_workers(). 3. Tighten post-event retirement tail: extend post-EP-0x20 trailing 5 us polls from 4 (20 us) to 10 (50 us) to absorb firmware's WC receipt tail without falling into a 100 us resnap hrtimer sleep, and rearm at repoll_us (15 us) if an evented flight is still settling. 4. Split fast completion drain from 1 ms housekeeping and stop hrtimer / delayed-work thrash: throttle power-governor updates, host-window health checks, full Session::check(), and administrative hang checks to 1 ms cadence while keeping transport checks, ring drains, and recovery/fault checks immediate; avoid cancelling/reprogramming the hrtimer or re-arming the 1-jiffy backup when a sufficient timer is already armed. Guard run_poll(), submit_locked(), and try_submit_inline() with lock-held sleeping.load(Ordering::Acquire) checks so suspend never races inline or polled publication. Signed-off-by: Diljit Singh <hey@djdev.me>
|
Big wins on the J773g, and the 0 recoveries over 228k jobs is reassuring. A targeted adversarial read of fd755f1 (not every line of the 2k):
Things I'd like to see before this leaves draft:
Assisted-by: LLM (Claude:Opus 5.5) |
…d decouple PB/UMA slots Compact the per-pass render VA layout from 256 MiB to 64 MiB stride within 0x7020000000..0x7040000000 (verified by compile-time overlap assertions), expanding RENDER_SLOTS from 2 to 8, and move scene_list (0x4000 bytes) from the shared per-allocator Allocator into per-pass PassSlot (client VA client::SCENE_LIST + i * PASS_STRIDE, 0x7020180000). Giving each PassSlot its own scene_list matches G15's per-pass storage and prevents TA(N)/FS(N-1) overlap corruption when render pipelining is enabled. This commit is bisect-safe on its own with render overlap still off: Order::ready() and Firmware::can_queue_behind() still serialize renders at depth 1 until the following commit, so the compacted 64 MiB pass stride and per-PassSlot scene_list run cleanly under single-flight render execution. Keep base RenderArena at 2 pass slots for single-pass or light VMs and lazily allocate extension pass slots 2..8 in RenderExtArena on render bursts (mirroring compute_ext). Box RenderOwners in Slots alongside ComputeOwners and switch its displacement ledger to pipelined mode (Lifetime::<8>::new(true)), keeping up to 7 concurrent passes in flight. Decouple the two device-wide PB/UMA allocator slots (parameter_id 0/1, hardware UMA slot 33) from PassSlot index 0..8: same-VM renders stay on their active reusable allocator slot with cumulative uma_submitted tracking (PoolSet::prepare_in_flight / retired_at), while cross-VM switches and post-recovery/error renders drain and switch to the displaced alternate slot. Aborted receipts permanently quarantine the affected allocator slot and retain its VM and arena leases. Check Pools::exhausted(), per-VM Pools::exhausted_for(), and Slots::render_exhausted_within() in publish_checked() so blocked render VMs cancel immediately with ECANCELED instead of spinning on EAGAIN and stalling unrelated compute. Signed-off-by: Diljit Singh <hey@djdev.me>
…nt barriers, and deeper queues Lift depth-1 render serialization so same-VM renders pipeline across PassSlot 0..RENDER_SLOTS while sharing the active exposed PB/UMA allocator slot, and remove remaining host-side queue-depth and wakeup-placement caps: - Allow same-engine render orders in Order::ready() to follow the queue's published render frontier while keeping cross-engine dependencies gated on full completion. - Extend Firmware::can_queue_behind() to admit healthy same-VM renders behind in-flight non-InitBM renders of the same VM (up to params::render_flights() = RENDER_SLOTS - 1), while still draining across VM switches, cross-engine transitions, InitBM priming renders, and error/recovery paths. - Emit explicit TA/FS hardware barriers in prefix_render() for queued renders: fragment-only or barrier-free renders overlap TA(N) with FS(N-1) (advertising DRM_ASAHI_FEATURE_FRAGMENT_BARRIERS); full render barriers hold TA(N) on FS(N-1) in hardware. - Remove the render_busy skip in Shared::submit_locked() so queued renders publish immediately on submit, and extend queue_passed() and retired_snapshot() for in-order passed receipts across Channel::Ta, Channel::Fs, shared notifier stamps, and cumulative PoolSet::retired_at() UMA completion. - Bound clean same-VM tilemap scrubbing in Slots::claim_pass_in() to tilemap_dirty.min(current_tilemap_bytes) while retaining the high-water mark for larger subsequent passes and cross-VM/recovery full scrubs. - Raise FLIGHTS (16 -> 32), SCHEDULER_CREDITS (16 -> 32), AHEAD_RECORDS (64 -> 128), and params::compute_flights() (16 -> 32), give each logical queue's drm_sched a creator-cluster-restricted ordered reclaimable workqueue, and implement gpu::Gpu::syncobj_wait_hint() with a lock-free active-burst uclamp hint (matching G17). Signed-off-by: Diljit Singh <hey@djdev.me>
…l fences early Firmware::retire() previously ran up to EVENT_RETIRE_POLLS (10) x 5 us udelay() loops while holding Shared::state whenever flights remained non-empty after an EP 0x20 interrupt. At queue depth d > 1, retiring the oldest flight left subsequent flights in self.engine.flights, causing every EP 0x20 interrupt to spin for 50 us under Shared::state (bouncing concurrent try_submit_inline() submissions onto drm_sched) and to arm a 15 us short_repoll hrtimer even though retirement had succeeded. Periodic 1 ms housekeeping also ran before retiring and signalling completed fences. Split polling into a non-sleeping Firmware::poll_step() and Firmware::retire_once() that report whether any flight retired: - Stop the EP 0x20 trailing retry loop immediately as soon as any flight retires (or flights drain, a receipt fault occurs, or the engine freezes/fails), and set short_repoll only when an event with active flights still retired nothing after the trailing retries. - Release Shared::state and complete any ready jobs before each 5 us udelay() retry so concurrent inline submissions can acquire the lock without contention. - Retire and publish ready work before periodic 1 ms housekeeping, deferring housekeeping until after job::complete_all() wakes userspace whenever jobs retired on that pass. Signed-off-by: Diljit Singh <hey@djdev.me>
Advertise DRM_ASAHI_FEATURE_COMPUTE_NO_QUERY_ACCESS in G16 GET_PARAMS and accept DRM_ASAHI_COMPUTE_NO_QUERY_ACCESS in compute stream/command validation. G16 publishes command timestamp outputs from the CPU during retire_oldest(). Record on each Flight whether it carries timestamp outputs and, in Firmware::can_queue_behind(), hold a same-queue compute command with a compute barrier behind an in-flight timed predecessor unless DRM_ASAHI_COMPUTE_NO_QUERY_ACCESS is set (clearing ComputeCommand::query_access). Ordinary compute dispatches that pass NO_QUERY_ACCESS continue to pipeline freely behind timed compute work, while query copy/reset compute shaders wait for host timestamp publication. Extend test-command-stream.py and test-publication.py with host coverage for the compute flag and query-access pipelining rules. Signed-off-by: Diljit Singh <hey@djdev.me>
fd755f1 to
4bae960
Compare
|
Thanks for the targeted adversarial read! Updated Review Points Addressed
Wave 3
|
|
Re-checked at 4bae960. Commits 1-4 are unchanged from fd755f1. 5 and 6 differ only in the commit-message note and the job.rs comment. Earlier points:
New commits:
Every row so far is from the J773g on 26B123. A functional run (vk-mustpass-short) on a J713 at this head would cover the Air's side of the same G16 code. Assisted-by: LLM (Claude:Opus 5.5) |
…cation batches When publish_checked() publishes multiple commands in a single pass (multi-command SUBMITs or queued back-to-back submissions), each command still posts its own SubmitRecord and advances qstate::WRITE individually per the M4 firmware contract, but previously called session.ring(channel) on every iteration. Defer doorbell ringing into a per-batch channel bitmask and ring each touched channel once via Firmware::flush_doorbells() after the batch loop finishes (preserving FS-before-TA ordering for render batches and fail-stopping on any post-visibility doorbell error). Count commands that share a coalesced doorbell pass in Engine::stats.grouped. Add host tests in test-publication.py covering coalesced compute/render doorbells, deferred-follower flushing, and doorbell error fail-stop. Signed-off-by: Diljit Singh <hey@djdev.me>
…scene Every G16 compute dispatch previously copied the full 8 KiB ComputeWork object (0x2000 bytes) into write-combined firmware memory, even though encode_compute_for() and the firmware only touch 0x000..0x150 (header plus up to 25 register records) and 0x720..0xf08 (metadata, ABI fields, embedded sequence, notifier, UMA descriptor, tail) — leaving 5,832 bytes (71%) perpetually zero. Similarly, every render pass zeroed and copied the full 16 KiB scene object (0x4000 bytes) to WC memory even though only 0x000..0x780 (rounded up to 0x800, 2 KiB) is touched by host or firmware. - Track on ComputeOwners whether each unit's latest claim required a full scrub (first use, cross-VM switch, cancelled prep, error, or firmware recovery). On clean same-VM reuse, prefix_compute() writes only the two active spans (0..0x150 and 0x720..0xf08: 2,360 bytes instead of 8,192 bytes); full-scrub claims still write all 0x2000 bytes after preserving any existing UMA descriptor. - Zero the full 16 KiB PassSlot::scene in PassSlot::scrub(full = true), size Scratch::scene to scene::ACTIVE_SIZE (0x800), and allow fill_scene() to zero and populate that active 2 KiB prefix, cutting 14,336 bytes (87.5%) of CPU memset and WC writes per render pass. - Extend test-resources.py to verify compute_needs_full_work() state transitions and PassSlot::scene full-scrub coverage. Signed-off-by: Diljit Singh <hey@djdev.me>
4bae960 to
65a87c4
Compare
|
Updated
Host verification: Remaining open items tracked for next runs:
Assisted-by: LLM (Gemini:4 Neon) |
|
Re-checked at 65a87c4. Commits 1-8 are unchanged from 4bae960.
No new issues. Assisted-by: LLM (Claude:Opus 5.5) |
Summary
Ten-commit G16 (M4 / T8132, 26B123 firmware) GPU performance series stacked on
djdev/g16-272(#2676), with zero new knobs or debugfs/module parameters:Waves 1–2 (commits 1–6)
PbEpochpool: decouple standby PB epochs from per-jobPreparedso consecutive renders reuse epochs without re-allocating.scene_list+ decoupled PB/UMA allocator slots: compact per-pass VA stride to 64 MiB (8 slots) and isolatescene_listperPassSlot(bisect-safe on its own with overlap still off).DRM_ASAHI_FEATURE_FRAGMENT_BARRIERS+ deeper queues: enable TA(N)/FS(N-1) overlap via explicit hardware barriers, raiseFLIGHTS/SCHEDULER_CREDITSto 32 andAHEAD_RECORDSto 128, restrict each logical queue'sdrm_schedworkqueue to the creator cluster (matchingg17/job.rs), and add lock-freesyncobj_wait_hint().Wave 3 (commits 7–10, hardware-verified on
dcp55)Shared::stateacross retire tail delay + early fence signalling: stop EP0x20trailing retries immediately once any flight retires (armingshort_repollonly when active flights still retired nothing), releaseShared::stateand wake ready jobs before each 5 µs retry, and defer 1 ms housekeeping until afterjob::complete_all().DRM_ASAHI_FEATURE_COMPUTE_NO_QUERY_ACCESS: advertise and honorCOMPUTE_NO_QUERY_ACCESSso ordinary compute dispatches pipeline behind timed compute work while query copy/reset shaders wait for host timestamp publication.Firmware::flush_doorbells()(preserving FS-before-TA ordering and fail-stopping on post-visibility doorbell error).ComputeWorkand renderscene: write only the activeComputeWorkspans (0..0x150and0x720..0xf08: 2,360 B vs 8,192 B, -71%) on clean same-VM reuse and populate only the active 2 KiBsceneprefix (vs 16 KiB, -87.5%) per render pass.Hardware Verification (
dcp55, Mac mini M4 J773g, macOS 27.2 / 26B123, 4K60 HDMI)Tested on
7.3.0-ARCH-m4drv8withaurora.kocarrying all 10 commits (Wave 1–3) + Mesamesa-m4-mega-59b5202e(Borealis + G16G single-sample colour compression). Across the full boot (314,336 GPU jobs;247,674across the 24-rowgpubench-vksweep withtpc_clear_bytes=0,grouped=14,376,overlaps=202,661,reused=113,529 / fresh=3): 0 recoveries, 0 hang kills, 0 dmesg GPU errors,health: failed 0 degraded 0.vk-mustpass-short): 87/87 Pass (0 Fail, 0 Crash, 0 Timeout, 0 GPU errors).compute.light+ concurrent 2-procrender.light+compute.heavy(compute_cross_vm=6,264,render_cross_vm=77) PASS, 0 recoveries, 0 errors.rd=12, offscreen KWin Wayland, native GL):rc=0, 163.80 FPS (still154.77 /spin131.38 /fly194.91 FPS,mean 6.105 ms, 8,191 frames, 0 GPU errors).24-Row
gpubench-vkvs Native macOS 27.2 (us/sub, lower is better;*= Linux beats/ties native macOS 27.2 — 7/24 rows)render.light (tri)render.light (tri)render.light (tri)render.light (tri)render.light (tri)render.heavy (fill, 1080p)render.heavy (fill, 1080p)render.heavy (fill, 1080p)render.heavy (fill, 1080p)render.heavy (fill, 1080p)render.heavy (geom, 1080p)render.heavy (geom, 1080p)compute.lightcompute.lightcompute.lightcompute.lightcompute.heavy (alu)compute.heavy (alu)compute.heavy (bw)compute.heavy (bw)copy.bw (fill)copy.bw (fill)copy.bw (buf2buf)copy.bw (buf2buf)Host Verification
python3 tools/testing/selftests/drivers/gpu/asahi-m4/run-host-tests.py: 24/24 PASS at Commit 5 alone (bisect verification) and atHEAD(all 10 commits).scripts/checkpatch.pl --strict --git origin/djdev/g16-272..HEAD: 0 errors, 0 warnings, 0 checks across all 10 commits.Keeping as draft pending the fanless M4 MacBook Air (J713) idle/video power + 10-minute sustained thermal row.
Assisted-by: LLM (Gemini:4 Neon)