Skip to content

drm/asahi: g17: tie kick-ring views to a render pair's time in a VM - #2520

Open
funkzy wants to merge 1 commit into
aurora-silicon:integration/aurorafrom
funkzy:pr/g17-kick-ring-views
Open

funkzy wants to merge 1 commit into
aurora-silicon:integration/aurorafrom
funkzy:pr/g17-kick-ring-views

Conversation

@funkzy

@funkzy funkzy commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Problem

A render pair's two kick rings are mapped read-only into the VM of the client that runs on the pair. That view was handed to the VM and kept until the VM was closed, so after the pair moved to a queue of another client the previous VM still mapped the rings. A pair rebound in place to a context of another VM was also not cleared first.

Change

The read-only kick-ring views now belong to the pair's time in a VM, not to the VM.

  • The pair holds them while it runs that VM's contexts and moves between them. While it is parked it keeps them only for as long as the VM is open, so a later context of that VM reuses them.
  • Once the client has closed the VM, a parked pair gives them up: when it parks, when it comes back from a failed rebuild and, for pairs already parked, when the VM is closed. A parked pair never keeps a closed VM's page tables alive. The views are dropped off the device lock.
  • When the pair goes to a context of another VM, the rings are cleared, and the new owner's first entry is written only after the previous VM's views are gone. From the parked state, the previous VM's views are removed and the rings cleared before the new VM's views exist. On an in-place rebind, the rings are cleared first and the new VM's views are created while the previous VM's still exist; the previous VM's views are removed when the new binding is installed. Both are read-only and the cleared rings hold no entries, so neither VM can read the other's.
  • A rebind that fails keeps the previous VM's views. A pair drops its views before its rings, so a reused ring never finds an older view of its pages.

This applies to the render pairs of every G17 part, with no chip, board or firmware condition. It does not depend on #2501.

Since RyanTheTide's review: a parked pair now drops a closed VM's views (when it parks, after a failed rebuild, and through a new close hook for pairs already parked), and this description and the commit message say when the previous views are removed on each path.

Testing

On the J775D (Mac Studio, M5 Ultra), 27.0 firmware, integration/aurora cc14570 with #1973 and our other three open drm/asahi PRs, default command line, one boot (the kernel also carries a local fix for an appledrm oops at boot on current integration, not part of this PR). Two clients alternated on the render pairs for 165 s with the desktop stopped, while a third client held its pair: 4998 rounds each, every render completing (fence signalled; these clients do not read back), 0 client failures, recoveries or faults. The render CTS subset gives 571 Pass and 57 NotSupported, as on the same base without the series. KUnit (366 of 368 pass, two skips as on the base) and the upstream host tests (427 of 427 on base and head, no result differs) pass.

Regression run (ALARM has no SELinux, so there is no AVC count): 0 new failed units (iwd.service and systemd-networkd-persistent-storage.service fail on the read-only test root with and without the series) and 0 new kernel error lines against the boot without the series.

Still to run

  • The other G17 parts (a Neo or a J616s), which take the same change: two clients alternating on render pairs for a few minutes with correct renders, the render CTS subset, KUnit pass counts per suite, 0 failed units and 0 new kernel errors. Rows from RyanTheTide's lab or their owners welcome.
  • The in-place rebind path needs a park to fail and did not occur in these runs.
Full test results

On the Mac Studio (M5 Ultra), read-only test root:

  • This head, 27.0 firmware: pairmove clients A (256x256) and B (512x512) started and exited in turn, each submitting one render pass and waiting on its fence, for 165 s; a holder client rendered and kept its pair for the whole run. A 4998 ok and 0 failed, B 4998 ok and 0 failed, holder rendered and held. Kernel journal for the whole boot: 0 client failures, 0 lost devices, 0 recoveries, 0 firmware restarts, 0 faults, 0 timeouts, 0 oopses; the only refusals are the intended ENOMEM cases of the drm/asahi: g17: M5: render large passes instead of losing the device after 26 s, refuse those that cannot fit, and account their memory to the client #2501 tests run in the same boot.
  • Previous head, 27.2 firmware: a diagnostic build (not part of this PR) checked, at each move of a pair to a context of another VM, that the previous VM's page tables no longer mapped either kick ring. Every move it saw was clear. The desktop, Vulkan clears and the triangle benchmark ran normally, with no client failures.
  • Render CTS subset (628 cases): 571 Pass, 57 NotSupported, with and without the series.
  • KUnit: 79 suites, 366 of 368 cases pass, 2 skipped, against 75 suites and 353 of 355 on the base with the same two skips.
  • Host tests: 427 of 427 on base and head, no result differs.
  • Kernel log and units: kernel lines compared after removing timestamps and numbers; no new error line beyond the intended ENOMEM refusals. The failed units are iwd.service and systemd-networkd-persistent-storage.service in both boots.

Thanks to ewb for the report and the suggestion, and to RyanTheTide for the review.

  • Clean-room: no Apple code, strings, firmware, RE output or tuning values
  • No chip/board/firmware-version gate where a hardware property can decide
  • Works on the default kernel command line; no hacks, delays or lab-only paths
  • Safety limits come from the machine, not from constants in code
  • Hardware safety rules followed for every test boot
  • Third-party commits unchanged; reporters and testers credited
  • Commits: human Signed-off-by only; no co-author or Assisted-by trailers
  • Tested on the affected hardware; no regression

Assisted-by: LLM (Claude:Opus 5.5)

@funkzy
funkzy force-pushed the pr/g17-kick-ring-views branch from 7ddb50c to d140d08 Compare November 20, 2026 06:40
@funkzy funkzy changed the title drm/asahi: g17: Fix a process reading the next owner's render kicks after its pair moves drm/asahi: g17: tie kick-ring views to a render pair's time in a VM Nov 20, 2026
@funkzy
funkzy force-pushed the pr/g17-kick-ring-views branch from d140d08 to e85fd3f Compare November 20, 2026 21:27
@RyanTheTide

Copy link
Copy Markdown
Contributor

No blockers.

  1. drivers/gpu/drm/asahi/g17/queue/render/parked.rs:99 and :139: a parked pair now keeps its read-only views in the last client's VM until it is next bound to a context of another VM. Each KernelMapping holds an ARef to its GpuVm (mmu.rs:1497). So after that client closes its VM, the VM's page tables stay alive for as long as the pair sits parked. Before this change the views went with the VM's own driver mappings. The amount is bounded by the number of pairs, but it is a lifetime change. Consider dropping client_kicks at park, or in take/clear_for_vm, once the VM's status reports it closed.
  2. Description accuracy: on the in-place rebind (queue/render.rs:3168-3259), the new VM's views are created by prepare_client_alias() before the previous VM's views are dropped at :3259. The rings are zeroed first (:3171) and the views are read-only, so neither side can read the other's entries. Still, the sentence "removed ... before the new VM's views exist" is only true for the parked path. Please reword it so it matches the code.

What I checked: field order drops client_kicks before kicks in both Pair and ParkedPair. Queue::render now reuses retired storage only when it gets a fresh alias (Ok(Some(_)), kick.rs:615), so retired storage never comes back with a view still in place. A view the VM already has is accepted only when this pair holds it for the same VM; anything else is EFAULT (queue/render.rs:3181, parked.rs:201). The failed-rebind path keeps the previous views because views is only swapped in after install_memory succeeds.

HW row: one non-M5 G17 Mac (Neo or J616s) running this head: two clients alternating on the render pairs for a few minutes (e.g. vkcube plus a GL client), 0 client faults and 0 GPU recoveries. Also add the regression counts from the M5 Ultra run (0 AVC denials, 0 failed units, 0 new kernel errors).

Assisted-by: LLM (Claude:Opus 5.5)

A render pair's two kick rings are mapped read-only into the VM of the
client that runs on the pair. That view was handed to the VM and kept
until the VM was closed, so after the pair moved to a queue of another
client the previous VM still mapped the rings. A pair rebound in place
to a context of another VM was also not cleared first.

The views of the kick rings now belong to the pair's time in a VM
instead of to the VM. The pair holds them while it runs that VM's
contexts and moves between them, and keeps them while it is parked for
as long as the VM is open, so a later context of that VM reuses them.
Once the client has closed the VM, a parked pair gives them up: when it
parks, when it comes back from a failed rebuild, and, for pairs already
parked, when the VM is closed. A parked pair therefore never keeps a
closed VM's page tables alive. The views are dropped off the device
lock.

When the pair goes to a context of another VM, the rings are cleared
and the new owner's first entry is written only after the previous VM's
views are gone. From the parked state, the previous VM's views are
removed and the rings cleared before the new VM's views exist. On an
in-place rebind, the rings are cleared first, the new VM's views are
then created while the previous VM's still exist, and the previous
VM's views are removed when the new binding is installed; both are
read-only and the cleared rings hold no entries, so neither VM can read
the other's. Either way the pair is idle and the firmware consumed
every entry; a rebind that fails keeps the previous VM's views. A pair
drops its views before its rings, so a ring that is reused never finds
an older view of its pages.

Reported-by: Erik Bourget <erik@ebourget.net>
Suggested-by: Erik Bourget <erik@ebourget.net>
Suggested-by: Ryan Murray <ryan@aurorasilicon.org>
Signed-off-by: Jeff Queisser <funkzy@users.noreply.github.com>
@funkzy
funkzy force-pushed the pr/g17-kick-ring-views branch from e85fd3f to ad9ab51 Compare November 21, 2026 19:14
@funkzy

funkzy commented Nov 21, 2026

Copy link
Copy Markdown
Contributor Author

Thanks, Ryan. Both points are addressed in the commit, now at ad9ab51:

  1. Closed VM behind a parked pair. A parked pair now gives up its kick-ring views once the client has closed their VM, so it no longer keeps that VM's page tables alive. This happens when the pair parks, when it comes back from a failed rebuild, and, for pairs already parked, from a new Gpu::vm_closed hook that takes each such pair's views under the device lock and drops them after it. While the VM is open, a parked pair keeps its views, so a later context of the same VM reuses them.
  2. Description. The commit message and the PR description now say what happens on each path. From the parked state, the previous VM's views are removed and the rings cleared before the new VM's views exist. On an in-place rebind, the rings are cleared first, the new VM's views are created while the previous VM's still exist, and the previous VM's views are removed when the new binding is installed. Both are read-only and the cleared rings hold no entries. Either way the new owner's first entry is written only after the previous views are gone.

I've added your Suggested-by. git range-diff against the current head: the one commit changes in queue/render/parked.rs, queue/render.rs, runtime/render.rs, runtime/render/prepare.rs and g17/mod.rs for the closed-VM drop, and in its message.

J775D on this head (stacked with #1973 and our other three open drm/asahi PRs on integration/aurora cc14570, 27.0 firmware, default command line; the kernel also carries a local fix for an appledrm boot oops, see #1973):

  • Two clients alternating on the render pairs for 165 s, desktop stopped, a third client holding its pair: 4998 rounds each, every render completing (fence signalled; these clients do not read back), and 0 client failures, recoveries or faults in the boot.
  • Render CTS subset: 571 Pass, 57 NotSupported, the same as without the series.
  • KUnit: 366 of 368 pass (two skips, as on the base). Host tests: 427 of 427 on base and head.
  • Regression: 0 new failed units (iwd.service and systemd-networkd-persistent-storage.service fail on the read-only test root with and without the series) and 0 new kernel error lines. ALARM has no SELinux, so there is no AVC count.

Rows we can't run: could your lab, or a Neo or J616s owner, run this head? The change covers every G17 part's render pairs. The useful row is two clients alternating on render pairs for a few minutes with correct renders, the render CTS subset, KUnit pass counts per suite, 0 failed units and 0 new kernel errors.

@RyanTheTide

Copy link
Copy Markdown
Contributor

Re-review of ad9ab51: No blockers.

  1. Parked pair keeping a closed VM's page tables alive: resolved. Pair::park (drivers/gpu/drm/asahi/g17/queue/render/parked.rs, park) now hands the views to the RetiredBinding when the VM's status reports the client closed, a pair returning from a failed rebuild gives them up under the lock and drops them after it (g17/runtime/render/prepare.rs:616-642), and pairs already parked lose them in the new Gpu::vm_closed (g17/mod.rs:2506-2514), which takes one pair's views at a time under the device lock and drops them outside it. Vm::drop calls release_mirror(), which marks the client closed, before vm_closed() (file.rs:164-179), so the status check there always sees the close. RetiredBinding declares client_kicks first, so those views drop before its memory and context.
  2. Description of the in-place rebind: resolved; the PR body and commit message now describe the parked and in-place paths separately and match queue/render.rs.

HW row: the new run is on the M5 Ultra only, on an integration stack with other open PRs and a local appledrm fix. Still needed: one non-M5 G17 Mac (Neo or J616s) on this head, two clients alternating on the render pairs for a few minutes with correct renders, a client closing while its pair is parked, the render CTS subset unchanged, 0 client faults, 0 GPU recoveries, 0 failed units, 0 new kernel errors.

Assisted-by: LLM (Claude:Opus 5.5)

@DjDeveloperr DjDeveloperr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pre-PR checklist review (§1–§6) on live head ad9ab51fac730065c43bb6b149d7aba0c2f3757d (host-tests: pass).

Checklist Summary

  • §1 No model gating: Pass — applies uniformly to all G17 render pairs with zero chip/SoC/board/firmware gates.
  • §2 Production-ready, no hacks: Pass — default kernel command line, no new parameters, no sleeps/retries/lab-only paths.
  • §3 Safety: Pass — no hardcoded hardware safety constants.
  • §4 Clean-room & attribution: Pass — commit ad9ab51fac73 carries human author/committer and human Reported-by/Suggested-by/Signed-off-by trailers only; PR body includes Assisted-by:.
  • §6 Naming: Pass.
  • §5 Evidence: Holding for non-M5-Ultra G17 hardware proof.

Code Audit (0 blockers)

  • Drop ordering verified in Pair (kick_aliases and client_kicks precede kicks), ParkedPair (client_kicks precedes kicks), and RetiredBinding (client_kicks precedes memory and context).
  • Queue::render() (g17/kick.rs) now requires Ok(Some(alias)) to reuse retired ring storage, preventing reuse if a client unexpectedly retains an existing mapping.
  • Closed-VM view release is complete and race-free: file::Vm::drop() calls status.release_mirror() (marking client_closed) before self.gpu.vm_closed(&self.vm). Already-parked pairs release views one at a time in Gpu::vm_closed() outside state.lock(); pairs parking afterwards hand views to RetiredBinding in Pair::park(); pairs returning from a failed rebuild take closed views under state.lock() in install_parked_pair() and drop them after releasing the lock.
  • Cross-VM rebind (Pair::prepare_bind / Pair::bind_owner): rings are cleared before preparing the new VM's read-only views, unexpected pre-existing aliases fail with EFAULT, failed rebinds retain the prior VM's views, and bind_owner drops the previous VM's client_kicks before any new kick entry can be written (safe under state.lock() since old_context remains pinned in self.displaced and install_memory/drop(moved) already establish the state -> gpuvm lock order). Commit message and PR body accurately distinguish the parked vs in-place rebind ordering.

Holding For (§5)

0 code/policy/attribution/naming/CI blockers. Holding approval solely for the missing cross-chip G17 hardware row on this head (the J775D run was also stacked with #1973 and other open PRs):

  • One non-M5-Ultra G17 Mac (Neo G17P or J616s) on this head: alternating clients on render pairs with verified render output, a client closing while its pair is parked, render CTS subset matching baseline, KUnit pass counts, 0 failed units, and 0 new kernel errors.

Assisted-by: LLM (Gemini 4 Neon)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants