Skip to content

drm/asahi: g16: Wave 2–5 GPU submit, concurrency, DVFS, ZLS/prefixes, and hang-recovery fixes - #2787

Open
DjDeveloperr wants to merge 17 commits into
djdev/g16-perf-272-prfrom
djdev/g16-perf-wave2-5
Open

DjDeveloperr wants to merge 17 commits into
djdev/g16-perf-272-prfrom
djdev/g16-perf-wave2-5

Conversation

@DjDeveloperr

@DjDeveloperr DjDeveloperr commented Nov 22, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Seventeen-commit R2 (5610a84b974e5) G16 (M4 / T8132, 26B123 firmware) GPU performance, concurrency, and hang-recovery series stacked cleanly on #2741 (djdev/g16-perf-272-pr at 65a87c442d99), introducing zero module knobs or debugfs switches and addressing all 10 review items from @RyanTheTide (#issuecomment-7078017919):

  1. bbd2af5c36515 Inline retirement & poll trim: retire completed flights inline on submit and trim poll-loop overhead (documents why 1 ms Session::check() deferral is safe while publish_checked() keeps per-submit Session::transport()).
  2. 042b94b1ea2fb 16 render pass slots & clean-reuse work trims: expand render pass slots to 16 inside 0x7020000000..0x7040000000 (32 MiB stride = 28 MiB pass layout + 4 MiB guard) and trim clean-reuse WC writes.
  3. 752c66d26f295 Batched VM coverage checks: batch VM range checks and trim per-SUBMIT CPU overhead.
  4. 612dabbe9f4fe Concurrent compute + render flights: allow concurrent same-VM compute and render flights with cross-engine Hold barriers (walking all commands of multi-command flights), independent per-class progress/hang deadlines (deadline_ns vs compute_deadline_ns), and squashed alt-class drain-fairness guard (!needs_engine_drain(&job)) + EAGAIN turn preservation so every commit bisects cleanly.
  5. 1cad4da5fb6e7 CLPC idle-hold hysteresis & fast-ramp DVFS governor: 2 ms CLPC_IDLE_HOLD_NS (half of clpc.c's 4 ms sample cadence), 5 ms RAMP_WINDOW_NS (== SAMPLE_NS), >= 70% / >= 40% thresholds matching demand(), and clamped Governor::demand under thermal limits.
  6. 7b54ab53b397b Off-lock slot pre-scrubbing: pre-scrub displaced render pass slots off-lock.
  7. 6d376201cd880 Ready inline drm_sched bypass & off-lock timeline signaling: bypass drm_sched on ready inline submits and signal timeline fences off-lock.
  8. 058731de1f0af Independent queue limits: expose per-class queue limits with per-file QueueCredit isolation in usage::Client so one client cannot starve other processes.
  9. 70756169505cd Barrier-free compute horizon batching: batch barrier-free compute horizons into one doorbell while tracking per-member completion stamps for cross-engine waits and recovery snapshots.
  10. 70c082c6d03a7 Inactive WC poll skip & submit allocation trims: skip inactive WC poll reads (hold[0] poll storm -93.2%) and trim submit allocations.
  11. 379a034591e92 Stamp-only barriers for pipelined same-VM renders: use stamp-only barriers for pipelined same-VM renders while keeping cross-engine barriers Hold.
  12. 469144bdac255 Compressed ZLS depth/stencil support: enable compressed ZLS depth and stencil attachments.
  13. 33a7c5223bd63 DRM_ASAHI_RENDER_STAGE_PREFIXES_V2: support v2 render stage prefixes (feature bit 33).
  14. f5a4612e16f69 Batched multi-pass render validation & publication: validate and publish multi-pass render batches together.
  15. c4e5905be6fc0 Lazy render extension slots on overlap or burst: replace 25 ms burst gap with Cache::active_renders overlap tracking + 2 ms sub-ms burst / multi-pass expansion so steady 60/120 Hz depth-1 clients stay on 2 base slots.
  16. ca433600f9e7b Quarantine-triggered extension expansion (live-caught fix): expand extension compute/render slots when a base slot is quarantined so fresh single-command clients do not deadlock on ECANCELED.
  17. 5610a84b974e5 Lazy recovery PB/UMA allocator slots (PB_SLOTS = 6, live-caught fix): keep non-fault VMs at BASE_PB_SLOTS = 2 (zero extra RAM) and lazily allocate per-slot recovery PoolExtArena allocators only after a render hang quarantines an allocator slot, preventing premature render_failed=1 / degraded=1 / device wedged after a single render hang.

Verification

Host checks (5610a84b974e5)

  • scripts/checkpatch.pl --strict --ignore FILE_PATH_CHANGES: 0 errors, 0 warnings, 0 checks across all 17 commits.
  • run-host-tests.py: 25 / 25 PASS (including new tests for per-class hang deadlines, multi-command cross-engine barriers, 60 Hz DVFS non-pumping & clamped cooling ceiling, per-file QueueCredit, 60/120 Hz render non-expansion, base-slot quarantine expansion, and multi-hang PB/UMA recovery).
  • 0 new rustc warnings across the full kernel build.

Live hardware (Apple M4 (G16G B0), J773g, 0x8132, 26B123 firmware)

  • Live bidirectional infinite-loop concurrent hang recovery proof (dcp77-v15-recovery-proof, 7.3.0-ARCH-m4prod15):
    • Direction A (compute infinite-loop hang + concurrent render.heavy d=4): recoveries 0 -> 1 (526.9 ms), concurrent render rc=0 (33,622 submits), immediate fresh compute.heavy d=1 / compute.light d=1 / render.heavy d=1 all rc=0 with 0 ECANCELED.
    • Direction B (render infinite-loop hang mux: apple-display-crossbar: replace the T602x selector fields instead of OR-ing them #1 + concurrent compute.heavy d=4): recoveries 1 -> 2 (113.6 ms), concurrent compute rc=0 (16,179 submits), render_failed=0, degraded=0, no [drm] device wedged, immediate fresh render.heavy d=1 + render.postfx (CRC=EXACT 668d6f0a) + compute.heavy d=1 all rc=0.
    • Direction B2 (second consecutive render hang — lazy PB slot 3+ scaling): recoveries 2 -> 3 (128.9 ms), concurrent compute rc=0 (16,170 submits), render_failed=0, degraded=0, pb: bad 0, post-3-recovery vk-mustpass-short 87/87 PASS, 0 kernel BUG/WARN/Oops.
    • (Note: 5610a84b974e5 differs from the dcp77-tested commit af25205059308 solely by adding #[allow(dead_code)] on PbOwnerArena in resources.rs:293-300, zero codegen change.)
  • Performance, CTS & S2R (dcp71 / dcp72 / dcp73a / dcp77):
    • gpubench Wave-1 57/57 OK (0.8027× M4 Metal), concurrent.sram 10/10 PASS (1.00–1.04× Metal, EXACT CRC), hold[0] poll-storm -93.2%.
    • vk-mustpass-short 87/87 PASS, dEQP-VK.draw.*indirect* 3,710/0 PASS, gles-smoke 3,086/0 PASS, real Deep Sleep S2R (apple-sleep: back from sleep, boot object resumes 0 -> 1) + post-wake vk-mustpass-short 87/87 PASS.

Stacked on #2741 (djdev/g16-perf-272-pr); will retarget onto integration/aurora once #2741 lands.

Assisted-by: LLM (Gemini:4 Neon)

@RyanTheTide

Copy link
Copy Markdown
Contributor

Reviewed the 16 commits on top of #2741 (65a87c4..9ce6f75). No gates, knobs or taint wording found. Concerns, by commit:

  • 7a616dc (CLPC hold / fast ramp): RAMP_WINDOW_NS = 4 ms, CLPC_IDLE_HOLD_NS = 10 ms and the 50%/30% thresholds (g16/power/policy.rs:19,21,99-101) have no stated source.
    • The fast path is more aggressive than the main governor: >= 50% over 4 ms jumps to top, where demand() needs >= 70% over 20 ms. It runs on every submit (poll.rs:246-251). A 2-3 ms frame early in a window therefore sends the GPU to S9, and decay is one state per 20 ms. Light 60 Hz loads can keep pumping to the top state.
    • The CLPC hold keeps the memory/fabric floor up ~13 of every 16.7 ms for a 3 ms/frame compositor, and adds 10 ms poll re-arms at idle (poll.rs:122-141).
    • The cooling ceiling is still applied to the published state, which is good. But ramp_demand(self.demand.min(limit), util, self.top) (policy.rs:259) stores a demand above the thermal limit. When cooling relaxes one level, the next publish jumps to top instead of stepping. Please clamp the stored demand to limit.
    • Please derive these values from CLPC or the firmware tables, or show measurements on the fanless M4 Air (J713) at this head: idle power and wakeups/s, a 60 Hz desktop/video power A/B, and a 10-minute sustained GPU load (temperature, GPU clock, throttle residency), each A/B against 65a87c4.
  • 11f1a0c (concurrent flights): progress() (poll.rs:1288-1326) folds the oldest compute and oldest render stamps into one tuple, and any movement renews the single shared deadline (recovery.rs:117). A stuck render, for example one blocked on a c2ta Hold barrier, never times out while same-VM compute keeps advancing. Please keep a deadline per class, renewed only by that class's stamps.
  • 11f1a0c / 0947a95: Order::ready() (command.rs:153) now accepts a published cross-engine predecessor. That relies on cross_engine_wait() (engine.rs:1285) always emitting the barrier, but it only matches a flight whose first command is the waited sequence. Please confirm that a batched compute group, where the waited command is not f.first, can never publish without a barrier. A host test for that case would help.
  • 45c09aa: the commit message says the Hold render->render barriers held the next stage for predecessor teardown, and the switch to stamp-only (engine.rs:2296) is argued from G15's policy. There is no G16 row with a GPU fault or firmware recovery while compute and render flights overlap ("0 faults" means the path was not exercised). Please add a fault-injection run (for example an out-of-bounds shader in one queue while another queue renders) showing the right fences error and the rest recover.
  • 11f1a0c -> d05b17b: commit 4 introduces the cross-VM/InitBM starvation that commit 15 fixes (engine.rs:894, engine.rs:1113). Please squash the fix into commit 4 so the series bisects.
  • def0eb8: this adds a device-wide cap of 58 live logical queues across all DRM files (queue.rs:108, ENOSPC), where G16 had none before. One client can exhaust it and make QUEUE_CREATE fail for every other process, compositor included. Since logical queues own no firmware state, a per-file bound would avoid that.
  • 9fa951d: publish_checked() now runs only transport() (engine.rs:865). The ASC_RUN, pstate > top and UAT binding checks run every 1 ms (poll.rs:202-240). Please state in the commit why the added latency is safe, mainly for check_adopted_bindings.
  • 93b8888: 16 slots and the 32 MiB stride (hw.rs:641,678) look like a host layout choice that fills the 0x7020000000 window. Is there a firmware bound on retained/in-flight render passes or on the stride? Please say where 16 comes from.
  • 9ce6f75: with RENDER_BURST_GAP_NS = 25 ms (resources.rs:90), every 60 fps client gets the 14-slot extension arena (about 50 MiB per VM, kept for the VM's lifetime). Is that intended for every animating client, compositor included?
  • Nit: 15 new #[allow(dead_code)], several on functions that are used (policy.rs:223, slots.rs render_needs_full_work, stage_and_take_scrubs).

Testing: all hardware results are from the J773g Mac mini, which has a fan. Rows still needed at 9ce6f75: J713 power/thermal as above, J713 vk-mustpass-short and gles-smoke, and a GPU-fault recovery run under concurrent compute+render on either M4.

Assisted-by: LLM (Claude:Opus 5.5)

…ll overhead

Pipelined submissions previously never retired completed flights inside
submit_locked(), forcing a workqueue round-trip whenever in-flight slots
filled and producing hold[0] stalls and 300-500 us tail bubbles. At the
same time, every EP 0x20 interrupt triggered full housekeeping in
submit_locked(), every publish_checked() ran Session::check() (including
the cross-core UAT handoff lock and full TTB scan), every TA (0x70)
completion triggered a 50 us EVENT_RETIRE_POLLS busy-spin while FS was
still running, every retire_once() polled all nine log sub-rings and the
trace ring, and retired timestamps were re-read from WC memory for both
accounting and empty timestamp outputs.

Deferring the full Session::check() checks (ASC_RUN, pstate > top, and
check_adopted_bindings()) to the 1 ms housekeeping tick is safe because:
- publish_checked() still runs Session::transport() on every submit so
  RTKit crashes and latched transport errors stop publication immediately;
- host VM bind/unbind updates ttbs[slot] and owned_user_roots[slot]
  atomically under uat.inner.lock() before submit/publish, so
  check_adopted_bindings() detects only unexpected firmware/external
  TTBAT corruption rather than normal host binding races;
- the host never writes pstate > top (policy::limit() bounds every
  published selector to 1..=top), avoiding per-submit PPL handoff lock
  contention;
- full Session::check() still runs on every >=1 ms housekeeping tick, on
  the first submit after >=1 ms idle, and whenever recovery_pending() is
  true.

Fix the hot submit and retirement paths:
- Eagerly retire completed flights inside submit_locked() when an EP 0x20
  event arrived, housekeeping/recovery is due, or the oldest flight's
  terminal fw_stamp has reached its target, leaving evented cleared when
  work retires so queued poll work does not spin.
- Separate 1 ms housekeeping (Session::check, object health, power,
  auxiliary log/trace ring drain) from per-submit fast publication, using
  Session::transport() in publish_checked().
- Track fragment completion events (0x71) alongside stage events so
  TA-only (0x70) interrupts do not trigger receipt-tail busy-spins or
  short-repoll storms while FS is still executing, and step the initial
  receipt-tail backoff from 1 us -> 2 us -> 5 us.
- Split channel draining into fast event+stats draining on retirement and
  auxiliary log/trace draining on 1 ms housekeeping, skip ktime_get() on
  empty event rings, reuse retired_snapshot() GPU ticks in stat_retire(),
  and skip WC timestamp reads when timestamp_outputs is empty.

Signed-off-by: Diljit Singh <hey@djdev.me>
…ork writes

With RENDER_SLOTS = 8, render pipelining capped out at 7 in-flight passes
(stalling depth 8 and 16 workloads), while each render pass recomputed
Geometry::new() three times and rewrote the untouched zero middle gaps of
both ta_work (0x360..0x740, 992 bytes) and fs_work (0x460..0x780, 800
bytes) into write-combining memory on every clean same-VM reuse.

Both WORK_STRIDE and RENDER_SLOTS are host client-VA layout choices
rather than firmware bounds: firmware receives explicit 64-bit VAs for
each pass object rather than indexing by stride. Each pass slot spans
+0x0..+0x1c00000 (28 MiB: TILEMAP at +0x1400000 plus 8 MiB TILEMAP_SIZE),
which rounds to a 32 MiB power-of-two stride (4 MiB guard) and yields 16
slots inside the 512 MiB window 0x7020000000..0x7040000000 (matching
COMPUTE_UNITS = 16).

- Halve WORK_STRIDE / PASS_STRIDE from 64 MiB to 32 MiB and expand
  RENDER_SLOTS from 8 to 16 inside the existing
  0x7020000000..0x7040000000 window, raising pipelined render depth from
  7 to 15.
- Keep full 16 MiB TPC / 8 MiB tilemap capacity on the two base pass
  slots (0..BASE_RENDER_SLOTS) and size extension slots (2..16) to 2 MiB
  TPC / 1 MiB tilemap so bursting VMs support 15 concurrent passes up to
  4K multi-layer and 8K without inflating per-VM RAM.
- Track RenderOwners::last_full (mirroring ComputeOwners::last_full) and
  write only the active head and tail spans of ta_work and fs_work on
  clean same-VM reuse, saving 1,792 bytes of WC writes per render pass
  while full-scrub claims still write the complete images.
- Compute render Geometry once per publication and update only
  layer_metadata after claiming the pass slot.

Signed-off-by: Diljit Singh <hey@djdev.me>
Every SUBMIT ioctl currently takes 10-40 separate gpuvm.lock_inner()
holds across command root validation, attachment checks, and per-record
stream-gate walks, decodes the command stream twice, calls ktime_get()
on every submit even after extension arenas are allocated, and
unconditionally notifies the standby condvar even with zero waiters.

- Batch page-aligned VM range checks in g16/command.rs through a
  48-entry stack SpanBatch calling vm.covers_ranges_batch() once per
  render or compute command, preserving exact per-span
  EINVAL/ENOENT/EACCES error reporting on failure.
- Collapse command::prepare()'s two stream decode passes into one.
- Reuse page coverage across 256-byte window cache hits in g16/gate.rs
  instead of calling cover() on every record in the same page.
- Skip ktime_get() burst tracking in g16/resources.rs once extension
  arenas are already allocated, and skip standby_available.notify_all()
  when no waiter is blocked.
- Add a host unit test in test-stream-gate.py verifying window-cached
  coverage reuse.

Signed-off-by: Diljit Singh <hey@djdev.me>
…s-engine barriers

Previously, G16 serialized compute and render work strictly:
can_queue_behind() rejected any mixed-class overlap, Order::ready()
forced cross-engine dependencies to wait for full host retirement, and
poll.rs only retired flights[0], leaving the compute and TA/FS hardware
pipelines unable to overlap.

Enable concurrent same-VM compute and render flights with per-class
retirement, per-class hang deadlines, drain fairness, and hardware ring
barriers:

- Allow healthy same-VM non-InitBM compute and render commands to queue
  concurrently up to their respective per-class flight caps, while
  preserving cross-VM, InitBM, and query_access serialization.
- Gate alt-class fallback selection on !needs_engine_drain(&job) so
  class-cap overlap runs concurrently on primed same-VM queues while
  drain-requiring head jobs (unprimed same-VM InitBM render, cross-VM
  switch, failed VM, or doomed/recovery/InitBM flight in progress) are
  never starved; advance queue fairness turns only after
  publish_prefix() succeeds so transient EAGAIN does not burn a turn.
- Emit hardware Hold ring barriers for same-queue cross-engine
  dependencies (fs2c0 on the compute ring using the FS fw_stamp, and
  c2ta0 on the TA ring using the compute fw_stamp via a dedicated
  PassSlot::compute_barrier FwObject), walking all commands of a flight
  so multi-command compute batches resolve their exact member stamp and
  defer with EAGAIN if an unretired predecessor is missing.
- Retire the oldest compute flight and oldest render flight
  independently in poll.rs, track progress and administrative/warning
  deadlines per class (render deadline_ns vs compute_deadline_ns) so an
  advancing class never masks a hung flight on the other class, and
  convert Job::full_retired to a 64-bit command bitmask so mixed-class
  jobs can retire out of class order while preserving completion proofs.
- Read only the published engine's WC qstate WRITE register(s) during
  publication instead of touching both compute and render rings.
- Add host selftests covering concurrent compute+render publication,
  drain fairness and EAGAIN turn preservation, fs2c0/c2ta0 barriers
  across multi-command flights, out-of-order per-class retirement, and
  independent per-class progress/timeout enforcement.

Signed-off-by: Diljit Singh <hey@djdev.me>
…rnor

On depth-1 submission loops, the engine briefly drains between consecutive
jobs, causing Power::report_work(false) to toggle
apple_clpc_gpu_set_active() on every job boundary (triggering
mod_delayed_work(..., 0) flushes in drivers/soc/apple/clpc.c even though
CLPC itself already holds rails for APPLE_CLPC_HOLD_MS = 32 ms on a 4 ms
sampling cadence) and forcing the utilization governor to wait a full
20 ms window before stepping up from FLOOR.

Add short 2 ms CLPC idle-hold hysteresis and an asymmetric 5 ms fast-ramp
upward path to the G16 power governor:

- Hold the CLPC active hint for 2 ms (CLPC_IDLE_HOLD_NS, half of clpc.c's
  APPLE_CLPC_SAMPLE_MS = 4 ms) across brief sub-ms depth-1 inter-job gaps
  via policy::ClpcHold, releasing it from service_power() once expired and
  re-arming run_poll() while held idle. This suppresses redundant
  mod_delayed_work flushes without holding across 60 Hz (16.67 ms) or
  120 Hz (8.33 ms) frames.
- Release CLPC immediately via Power::release_clpc_now() on fail-stop
  (fail_flights) and system sleep commit (sleep_down).
- Add a 5 ms upward-only ramp window (RAMP_WINDOW_NS = SAMPLE_NS, aligned
  with the existing PMP sampling cadence) and ramp_demand() matching
  demand()'s upward thresholds (>= 70% -> top, >= 40% -> +1) so sustained
  bursts reach top within 5 ms while light 60 Hz desktop frames stay near
  FLOOR and 20 ms slow decay is preserved. Clamp stored Governor::demand to
  the active thermal/user limit on the fast-ramp path so cooling
  relaxation steps cleanly instead of jumping.
- Add unit tests in g16/power/policy.rs and wiring checks in
  test-thermal-policy.py covering fast ramp, 60 Hz non-pumping, decay
  preservation, clamped cooling ceilings, and CLPC idle-hold expiry/reset.

Signed-off-by: Diljit Singh <hey@djdev.me>
Every render pass claim previously ran PassSlot::scrub() synchronously
while holding the device mutex (Shared::state), stalling concurrent
submissions and retirement behind WC tilemap/heap/preemption zeroing.

Move clean same-VM pass slot scrubbing outside Shared::state with a
race-free synchronous fallback:

- Record clean render retirements in Slots::note_render_retired() and
  stage remaining idle displaced, cleanly retired render pass slots
  (never Published or undisplaced Retired slots) right before dropping
  Shared::state via stage_and_take_scrubs(), only when the engine is
  neither failed nor frozen. Staging after publication avoids
  self-blocking immediate slot reuse within the same lock hold and skips
  staging on pure compute workloads.
- Run staged scrubs in run_poll(), submit(), and try_submit_inline()
  after dropping Shared::state and after signalling retired completion
  fences via job::complete_all().
- Coordinate slot selection (pass_available, pick_pass, pick_pass_in)
  and off-lock scrub completion via SeqCst scrubbing/waiter atomics so
  in-flight scrubbing slots are skipped, pre-cleaned slots are
  preferred, and blocked waiters trigger an immediate poll_now().
- Skip synchronous WC fills in claim_pass_in()/claim_pass_for() when
  a same-VM slot was pre-cleaned, while falling back synchronously on
  first use, cross-VM switch, recovery invalidation, error/cancel, or
  any failed/dropped off-lock scrub.
- Wait for active off-lock scrubs to drain in sleep_quiet() before
  entering system sleep.
- Add host selftests in test-resources.py covering displaced-only
  staging, post-publish staging without blocking immediate reuse,
  off-lock scrub + zero-fill claim, waiter wakeup, dropped batch
  cleanup, and synchronous fallback on error/recovery/cross-VM.

Signed-off-by: Diljit Singh <hey@djdev.me>
…imelines off-lock

Ready submissions with already-signalled inputs previously still allocated
and armed a drm_sched job and woke the scheduler kworker even when inline
publication succeeded, and `try_submit_inline()` used `try_lock()` on
`Shared::state`, bouncing bursts during brief poll-lock windows onto
`drm_sched` wakeups and causing 350-520 us gap95 tail stalls on multi-depth
workloads. In addition, `Timeline::complete()` held `Timeline::pending`
across `dma_fence_signal()` callbacks.

Bypass `drm_sched` allocation and kworker wakeup in `Scheduler::push()`
when `prev_started` is true and `Job::try_start_inline()` succeeds,
rolling back `accepted` and falling through to `drm_sched` otherwise.
Track active firmware recovery in `Shared::recovering` (guarded via RAII
in `Firmware::recover()`) so `Shared::try_submit_inline()` can bail out
only during sleep or recovery and otherwise acquire `Shared::state`
directly. Finally, split `Timeline::complete()` so ready head fences are
drained in fixed stack batches under `signaller -> pending` and signalled
without holding `Timeline::pending`.

Extend `test-job-failure.py` and `test-fence-signalling.py` to cover
`Scheduler::push()` inline bypass/fallback rollback, recovery bailout,
and off-pending FIFO batch signalling.

Signed-off-by: Diljit Singh <hey@djdev.me>
Mirror the G15 logical-queue contract on G16:

- Add params::queue_limits() (max_queues = MAX_QUEUES = 58,
  max_in_flight_per_queue = FLIGHTS = 32) and return it from
  Gpu::queue_limits(), which enables DRM_ASAHI_FEATURE_INDEPENDENT_QUEUES
  and DRM_ASAHI_PARAM_GROUP_QUEUE_LIMITS in the shared frontend.
- Track live logical queues per open DRM file in usage::Client via an
  RAII QueueCredit guard (ENOSPC at MAX_QUEUES), stored after Scheduler
  in Queue so capacity returns only after scheduler drain.  Because G16
  logical queues own no firmware execution contexts, keeping QueueCredit
  per-file while file.rs enforces per-process queue_share::Share bounds
  ensures a client that opens MAX_QUEUES queues cannot starve a later
  process (such as a compositor) from creating its own queue.
- Extend test-publication.py and usage.rs unit tests to cover queue
  limits, INDEPENDENT_QUEUES, per-file QueueCredit isolation and
  release, and multi-queue compute+render concurrency on one VM.

Signed-off-by: Diljit Singh <hey@djdev.me>
Allow compute-only barrier-free command runs within a job and across
consecutive ready compute-only jobs on the same queue and VM to share one
firmware publication and one doorbell, bounded by free compute units,
available compute ring slots, and the compute flight budget.

Each batched compute member claims its own hardware compute unit, writes
its own submitted counter before the shared barrier, and records its own
completion stamp and work descriptor so retirement verifies every claimed
unit receipt, publishes timestamps for every grouped job, and lets
cross-engine waits and firmware recovery snapshots resolve interior stamps.

Signed-off-by: Diljit Singh <hey@djdev.me>
Avoid redundant uncached write-combining reads and barriers during poll
and retirement:
- Track per-class check state across retire_once() so retiring a burst of
  one class does not re-poll an already pending flight of the other class
  or double-poll a mixed flight.
- Collapse oldest_stamp_reached() and progress() into single flight passes
  that touch only active classes, and skip the empty-ring rmb() in event
  and statistics draining.
- Check FS firmware stamp before TA, and check notifier/descriptor receipts
  before reading scene or ITS timestamps so unsettled polls fail fast.

Trim hot-path SUBMIT ioctl heap allocations without unsafe code:
- Add inline 0..2 storage (Commands, Orders, Resolved, InputFences) that
  spills to KVec only above two entries, eliminating per-submit kmalloc
  traffic on single- and two-command submissions.
- Move attachment lists into the final render or compute command of a
  stream via core::mem::take instead of duplicating them with copy_list().

Signed-off-by: Diljit Singh <hey@djdev.me>
When pipelining a healthy same-VM non-InitBM render behind an in-flight
render (class_behind), prefix_render() encoded both prev_barrier
(TA->TA under early tiling or TA->FS otherwise) and fs_dep (FS->FS
under early tiling) with BarrierType::Hold (type 1), holding each
consecutive TA/FS stage for the predecessor software-job teardown even
though own_barrier on the same FS ring already uses BarrierType::Stamp
(type 0).

Use BarrierType::from_stamp_only(!init_buffer) for both same-class
render->render barriers (prev_barrier and fs_dep), matching own_barrier
and G15's proven RENDER_INTERNAL_HOLD = 0 policy while keeping
cross-engine barriers (fs2c and c2ta) strictly BarrierType::Hold on
G16's class-2 queues. G16 uses firmware-initiated phase recovery and
never writes G15 kill controls, and any failed/suspect/cross-VM/InitBM
transition still drains the engine first.

Signed-off-by: Diljit Singh <hey@djdev.me>
Wire the HAL200 compressed ZLS metadata registers (`fs::SEL[44..52]`) and
the four `JobParameters3` partial-render mirrors (`0x958`, `0x970`,
`0x988`, `0x9a0`) in `encode_fs_for()`, all of which lie inside the
clean-reuse active work spans (`0..fs::HEAD_ACTIVE_SIZE` and
`fs::TAIL_ACTIVE_START..fs::WORK_SIZE`).

Widen `zls_ctrl` admission in `g16/command.rs` to admit the four load and
store compression bits (`0x154`, mask `0x00000000060cc1fe`), enforce
symmetric load/store compression bits per plane, remove the `EOPNOTSUPP`
refusal on `comp_base`/`comp_stride`, and validate compressed metadata
alignment, stride encoding, and read-write VM span coverage through the
layer count via `SpanBatch`.

Add host selftests covering 1x, 2x, and layered compressed depth/stencil
admission, invalid control/alignment/span rejection, 25G83/26B123 FS
encoding parity, and clean-reuse scrubbing of compressed ZLS fields.

Signed-off-by: Diljit Singh <hey@djdev.me>
Accept the 480-byte drm_asahi_cmd_render_m5 render payload when
DRM_ASAHI_RENDER_STAGE_PREFIXES is set, decode the 32-byte
drm_asahi_render_stage_prefixes tail with compile-time layout assertions,
and record per-stage TA compute and render wait-through prefixes on Order
via narrow_tiling() (refusing any TA wait later than the FS wait, nonzero
reserved words, or combination with DRM_ASAHI_RENDER_FRAGMENT_BARRIERS).

Add per-pass ta_fs_barrier and fs_compute_barrier objects to PassSlot,
resolve independent TA and FS compute/render barrier stamps in
prefix_render() via plan_render_barriers(), and advertise
DRM_ASAHI_FEATURE_RENDER_STAGE_PREFIXES_V2 in G16 params so Mesa can
emit independent TA and FS waits on G16G.

Signed-off-by: Diljit Singh <hey@djdev.me>
Multi-pass render frames (shadow maps, deferred G-buffer + lighting,
post-processing, and Zink multi-pass submits) currently pay per-pass VM
span coverage locks, per-pass VDM root reads, and per-pass TA/FS ring
publications with individual SubmitRecords and UMA submitted counter
updates.

Batch validation and publication across consecutive render passes within
a single SUBMIT without delaying fence completion for separate jobs:

- Lift SpanBatch to Prepared::validate() so render passes within a
  SUBMIT deduplicate shared VM spans and flush coverage in one lock
  acquisition, and replace gate::vdm() with gate::vdm_batch() to verify
  all render VDM root words under a single Vm::read_lock() with a
  256-byte intra-page read window.
- Add ordered multi-pass slot tracking (Claims::pass_count), replace
  PoolSet::publish() with PoolSet::publish_batch() (submitted += 2 *
  renders), and add Job::can_fold_render_cmd() so consecutive ready
  non-InitBM render passes in one SUBMIT fold into a single Flight, one
  TA/FS producer advance, one SubmitRecord kick, and one UMA submitted
  update while preserving exact per-pass barriers (wait_stamp_for_with),
  recovery matching (r_ta_stamp/r_fs_stamp), per-pass scene ITS
  verification, and ordered pass-slot retirement.
- Add host selftests covering batched VDM root admission, submit-wide
  span deduplication, ordered wrapped multi-pass retirement and
  publish_batch(), same-batch render wait stamps, can_fold_render_cmd,
  and interior render-stamp recovery preparation.

Signed-off-by: Diljit Singh <hey@djdev.me>
Cache::prepare() previously gated RenderExtArena allocation on compute's
2 ms BURST_GAP_NS and 3-command BURST_COMMANDS thresholds.  Heavy
multi-millisecond pipelined renders arrive more than 2 ms apart and
would reset render_burst to 0 on every submit, staying capped at
BASE_RENDER_SLOTS (2 slots), whereas simply widening the idle gap to
cover 60/120 Hz frame periods would force every steady depth-1
compositor or window client to allocate 14 extension pass slots it never
uses.

Track live prepared render jobs per VM in Cache::active_renders
(incremented on returning Prepared and decremented in Prepared::drop),
keep the 2 ms BURST_GAP_NS for back-to-back single-pass bursts, and
expand RenderExtArena when a render overlaps an active in-flight render
(active_renders != 0), submits at least BASE_RENDER_SLOTS passes in one
SUBMIT, or sustains a sub-2 ms burst.  Add a host unit test covering
steady 60/120 fps depth-1 non-expansion, multi-ms overlap, sub-2 ms
bursts, and multi-pass submits.

Signed-off-by: Diljit Singh <hey@djdev.me>
… slot is quarantined

Fresh user VMs start with only BASE_COMPUTE_UNITS (2) compute units and
BASE_RENDER_SLOTS (2) render pass slots, allocating extension slots
lazily on bursts, overlap, or multi-pass submits.

After a hang recovery quarantines one base compute unit (0 or 1) while
the other base unit is the newest-retired undisplaced owner,
compute_exhausted_within(2) becomes true even though 14+ extension
compute units remain healthy. Every subsequent fresh single-command
compute client then submits with only 2 units, hits
compute_exhausted_within(2) in engine publication, and fails
immediately with ECANCELED without ever running on an extension unit to
displace the surviving base unit. The same deadlock occurs for fresh
single-pass render clients when a base pass slot is quarantined.

Track base-slot quarantine state in Lifetime::any_quarantined_within(),
publish lockless hints on Shared whenever Slots::quarantine() runs, and
pass them via PrepareHints into Cache::prepare() so fresh VMs expand
compute_ext / render_ext immediately once a base slot is quarantined.

Add host unit tests covering both compute and render base-slot
quarantine recovery and displacement.

Signed-off-by: Diljit Singh <hey@djdev.me>
…ender hangs

Physical render PB/UMA allocators (hardware UMA slot 33) were previously
capped at two fixed slots (parameter_id 0 and 1). Once both slots had
been used across VMs (for example compositor plus benchmark), a single
render hang on the active slot called pools.invalidate() and
pools.quarantine_flight(), leaving the other slot undisplaced and
immediately tripping Pools::exhausted() -> render_failed=1, degraded=1,
and device wedged even with 14+ healthy render pass slots remaining.

Expand the client-VA render allocator window to PB_SLOTS = 6
(parameter_id 0..6 in a 64 MiB stride across 0x7008000000..0x7020000000,
disjoint from base compute units, render pass slots, dynamic PB page
lists, and extension compute units). Keep non-fault VMs at
BASE_PB_SLOTS = 2 with zero extra RAM overhead; lazily allocate per-slot
PoolExtArena recovery allocators sharing the VM's existing PB heap only
after a render hang quarantines an allocator slot, signaled locklessly
via Shared::needed_pb_slots.

Generalize Pools selection, displacement, and exhaustion checks to
PB_SLOTS while preserving the two-slot working set in normal cross-VM
switching and per-job limit checks (pick_slot_within /
can_pipeline_within / exhausted_for_within) so pre-hang jobs cancel
cleanly instead of spinning on EAGAIN.

Add host unit tests covering the exact cross-VM single-render-hang
recovery and survivor displacement path, multi-hang survival up to four
quarantines, and clean exhaustion at the slot limit.

Signed-off-by: Diljit Singh <hey@djdev.me>
@DjDeveloperr
DjDeveloperr force-pushed the djdev/g16-perf-wave2-5 branch from 9ce6f75 to 5610a84 Compare November 22, 2026 15:46
@DjDeveloperr DjDeveloperr changed the title drm/asahi: g16: Wave 2–5 GPU submit, concurrency, DVFS, ZLS/prefixes, and fairness wins drm/asahi: g16: Wave 2–5 GPU submit, concurrency, DVFS, ZLS/prefixes, and hang-recovery fixes Nov 22, 2026
@DjDeveloperr

Copy link
Copy Markdown
Contributor Author

@RyanTheTide Thanks for the thorough review — pushed R2 (5610a84b974e5, 17 commits on djdev/g16-perf-wave2-5) addressing all 10 items (verified 25/25 host tests PASS, 0/0/0 checkpatch.pl --strict, and 0 new rustc warnings):

  1. 7a616dc7 → 1cad4da5fb6e7 (CLPC hold / fast ramp & thermal clamping):
    • Tightened CLPC_IDLE_HOLD_NS from 10 ms to 2 ms (half of drivers/soc/apple/clpc.c's APPLE_CLPC_SAMPLE_MS = 4 ms, vs CLPC's own APPLE_CLPC_HOLD_MS = 32 ms), suppressing redundant sub-ms depth-1 mod_delayed_work(..., 0) flushes without holding across 60 Hz (16.67 ms) or 120 Hz (8.33 ms) frames.
    • Aligned RAMP_WINDOW_NS to SAMPLE_NS (5 ms) and matched ramp_demand() thresholds to demand() (>= 70% → top, >= 40% → +1) so light 60 Hz desktop frames stay near FLOOR instead of pumping to S9.
    • Clamped stored Governor::demand to the active thermal/user limit on the fast-ramp path so cooling relaxation steps cleanly instead of jumping.
    • Added unit tests in g16/power/policy.rs + wiring checks in test-thermal-policy.py covering fast ramp, 60 Hz non-pumping, decay preservation, clamped cooling ceilings, and CLPC idle-hold expiry/reset. (Note: we do not have a fanless M4 Air J713 on this desk, so J713 idle/60 Hz/10-min sustained power+thermal A/B and J713 smoke rows remain open for a J713 machine.)
  2. 11f1a0c9 → 612dabbe9f4fe (per-class progress & hang deadlines):
    • Split progress tracking and administrative/warning deadlines per class (deadline_ns for render vs compute_deadline_ns for compute), renewed only by that class's stamps, so an advancing class can never mask a hung flight on the other class. Added host selftests covering independent per-class progress/timeout enforcement.
  3. 11f1a0c9 / 0947a956 → 612dabbe9f4fe & 70756169505cd (cross-engine barrier on interior batched compute commands):
    • cross_engine_wait() now walks all commands of a flight (not just f.first) so multi-command compute batches resolve their exact interior member stamp and emit the hardware Hold barrier (c2ta0/fs2c0), deferring with EAGAIN if an unretired predecessor is missing. Added host selftests covering fs2c0/c2ta0 barriers across multi-command flights.
  4. 45c09aa5 → 379a034591e92 + ca433600f9e7b + 5610a84b974e5 (live bidirectional concurrent infinite-loop hang recovery proof):
    • Running real infinite-loop GPU hangs on live M4 (dcp75b) caught two pre-existing g16 recovery bugs, both now fixed in this series with host tests and live M4 proof:
      • ca433600f9e7b: expand extension compute/render slots when a base slot is quarantined so fresh single-command clients do not deadlock on ECANCELED.
      • 5610a84b974e5: add lazy PB_SLOTS = 6 recovery PB/UMA allocator slots (keeping non-fault VMs at BASE_PB_SLOTS = 2 with zero extra RAM) so a single cross-VM render hang no longer trips Pools::exhausted() → render_failed=1 / degraded=1 / device wedged.
    • Live M4 G16G B0 proof (dcp77-v15-recovery-proof, 7.3.0-ARCH-m4prod15):
      • Direction A (live compute infinite-loop hang + concurrent render.heavy d=4): recoveries 0 -> 1 (526.9 ms), concurrent render rc=0 (33,622 submits), immediate fresh compute.heavy d=1 / compute.light d=1 / render.heavy d=1 all rc=0 with 0 ECANCELED.
      • Direction B (live render infinite-loop hang mux: apple-display-crossbar: replace the T602x selector fields instead of OR-ing them #1 + concurrent compute.heavy d=4): recoveries 1 -> 2 (113.6 ms), concurrent compute rc=0 (16,179 submits), render_failed=0, degraded=0, no [drm] device wedged, immediate fresh render.heavy d=1 + render.postfx (CRC=EXACT 668d6f0a) + compute.heavy d=1 all rc=0.
      • Direction B2 (second consecutive live render hang — lazy PB slot 3+ scaling): recoveries 2 -> 3 (128.9 ms), concurrent compute rc=0 (16,170 submits), render_failed=0, degraded=0, pb: bad 0, post-3-recovery vk-mustpass-short 87/87 PASS, 0 kernel BUG/WARN/Oops.
    • (Transparency note: 5610a84b974e5 differs from the dcp77-tested commit af25205059308 solely by adding #[allow(dead_code)] on PbOwnerArena at resources.rs:293-300, zero codegen change.)
  5. 11f1a0c9 -> d05b17bf → squashed into 612dabbe9f4fe:
    • Squashed the alt-class drain-fairness guard (!needs_engine_drain(&job)) and EAGAIN turn preservation directly into commit 4 (612dabbe9f4fe) so the series bisects cleanly with no intermediate cross-VM/InitBM starvation.
  6. def0eb85 → 058731de1f0af (per-file QueueCredit):
    • Moved QueueCredit from device-wide to per-open-DRM-file (usage::Client), stored after Scheduler in Queue so capacity returns only after scheduler drain. One client exhausting MAX_QUEUES (58) can no longer cause ENOSPC in other processes (compositor, etc.); added per-file isolation tests in usage.rs and test-publication.py.
  7. 9fa951d3 → bbd2af5c36515 (publish_checked() 1 ms Session::check() deferral):
    • Documented in the commit message why deferring ASC_RUN, pstate > top, and check_adopted_bindings() to the 1 ms tick is safe: publish_checked() still runs Session::transport() on every submit; host VM bind/unbind updates ttbs[slot] and owned_user_roots[slot] atomically under uat.inner.lock() before submit/publish; policy::limit() bounds every published pstate to 1..=top; and full Session::check() still runs on every >=1 ms tick, first submit after >=1 ms idle, and whenever recovery_pending() is true.
  8. 93b88884 → 042b94b1ea2fb (origin of 16 slots & 32 MiB stride):
    • Documented in the commit message: both WORK_STRIDE and RENDER_SLOTS are host client-VA layout choices (firmware takes explicit 64-bit VAs per pass object). Each pass slot spans +0x0..+0x1c00000 (28 MiB: TILEMAP at +0x1400000 + 8 MiB TILEMAP_SIZE), rounding to a 32 MiB power-of-two stride (4 MiB guard) and fitting 16 slots into 0x7020000000..0x7040000000 (512 MiB, matching COMPUTE_UNITS = 16).
  9. 9ce6f755 → c4e5905be6fc0 (no 25 ms burst gap forcing 50 MiB arenas on 60 fps clients):
    • Dropped RENDER_BURST_GAP_NS = 25 ms. Now tracks live prepared render jobs per VM in Cache::active_renders, keeps 2 ms BURST_GAP_NS, and expands RenderExtArena only on live overlap (active_renders != 0), multi-pass submits (>= BASE_RENDER_SLOTS), or sub-2 ms bursts. Steady 60/120 Hz depth-1 clients stay on 2 base slots with zero extension allocation (covered by a new host unit test).
  10. Nit (#[allow(dead_code)] cleanup):
    • Removed the redundant #[allow(dead_code)] annotations on used functions (policy.rs:223, render_needs_full_work, stage_and_take_scrubs), leaving only genuinely test/layout-only items (0 new rustc warnings).

Assisted-by: LLM (Gemini:4 Neon)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants