Skip to content

drm/asahi: bound the queue structures held for firmware poweroff - #2619

Draft
RyanTheTide wants to merge 1 commit into
integration/aurorafrom
gpu/g13-cap-retired-queues
Draft

RyanTheTide wants to merge 1 commit into
integration/aurorafrom
gpu/g13-cap-retired-queues

Conversation

@RyanTheTide

Copy link
Copy Markdown
Contributor

Problem

#2579 keeps the queue structures of invalidated contexts until a statistics batch ends with a PowerOff. Under sustained load the firmware reports each PowerOff in the same batch as the following PowerOn, and idle sends none, so that boundary almost never arrives. Every destroyed context then pins about 60 KiB. On the M1 Pro a context-churning load retained 20,000 contexts and added 1.4 GB of Shmem in 180 seconds; a 300 second row added 2.17 GB that never came back. The M2 (G14G) shows the same pattern.

Releasing at any PowerOff is not a fix: a lab build doing that crashed the G13S firmware again after about 13,000 churned contexts.

Change

Keep the PowerOff boundary and cap the list. When more than 1024 invalidated contexts are waiting, the ones invalidated longest ago are released.

Testing

Module-only aurora.ko on the unchanged fleet76 kernel (integration e00fb9a + #2579), one boot per row. Each row: render-churn mix x32 (about 1,000 short-lived rendering processes per run), then 600 s with two glmark2 clients keeping the GPU busy plus four churning processes, with memory sampled every 10 s.

Mac cap mix x32 600 s load firmware crash Shmem after load / after idle
M1 Pro none (#2579 only) 16/16 (x16) 32,956 processes (300 s) 0 +2.17 GB, not returned
M1 Pro 256 crash in run 7 not run 1 -
M1 Pro 512 32/32 68,003 processes 0 0.16 / 0.11 GB
M1 Pro 1024 32/32 67,875 processes 0 0.20 / 0.14 GB
M1 Pro 4096 32/32 67,671 processes 0 0.40 / 0.34 GB
M1 Pro 1024, this commit 32/32 67,854 processes 0 0.20 / 0.14 GB
M2 1024 32/32 50,843 processes 0 0.19 / 0.16 GB

For comparison, the stock fleet76 module crashed the M1 Pro firmware on 2 of 2 boots within about 1,250 churned contexts. Latency of a one-case Vulkan process (33-34 ms median) and glmark2 terrain (402-404 FPS) are the same as with #2579 alone. The commit also builds on the current integration/aurora tip.

Still to run

  • A machine that churns contexts faster than the M1 Pro (M1 Max/Ultra): the cap is a count, so its time window shrinks as the churn rate grows (here about 110-125 contexts per second, so 1024 is about 8 seconds).
Details for reviewers
  • The cap releases from the front of powered_contexts with drain(..excess) right after the push, under the same locks as the push. Order is invalidation order, so the front is the oldest.
  • Rows used a lab build with a module parameter for the cap and debug-gated log lines counting releases. On the M1 Pro the cap did nearly all of the releasing (62,000-96,000 contexts per boot against 7-16 PowerOff-batch releases); on the M2, 76,000 against 13.
  • The smallest cap that held was 512; 256 crashed after about 6,400 churned contexts (stock crashes after about 1,000). 1024 gives a 2x margin over 512 for about 30 MB more at the bound.
  • Peak Shmem during the 600 s rows (about 0.6-0.85 GB) includes the glmark2 clients' own buffers and is flat over the window.
  • Final row with the exact commit: module sha256 a170a7e5..., fleet76 tree + drm/asahi: free retired queue structures after firmware poweroff #2579 + this hunk only, 613/613 undefined symbols identical to the stock fleet76 module.

Assisted-by: LLM (Claude:Opus 5.5)

The queue structures of invalidated contexts are kept until a statistics
batch ends with a PowerOff. Under sustained load the firmware reports each
PowerOff together with the following PowerOn, so that boundary can be
minutes away, and idle sends none. A load that keeps creating and
destroying rendering contexts then pins about 60 KiB of kernel memory per
destroyed context: on the M1 Pro, 20,000 retained contexts and +1.4 GB of
Shmem in 180 seconds, and the same pattern on the M2 (G14G).

Releasing at every PowerOff is not safe: it brings the G13S firmware crash
back. Instead keep the boundary and cap the list. When more than 1024
contexts wait, release those invalidated longest ago. With a cap of 256
the firmware crash still occurred, after about 6,400 churned contexts;
512, 1024 and 4096 each ran about 100,000 churned contexts without one.

Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
@RyanTheTide
RyanTheTide marked this pull request as draft November 20, 2026 23:42
@RyanTheTide

Copy link
Copy Markdown
Contributor Author

Back to draft under our production-ready rule. A fixed count (1024) is a magic depth, not a release signal. It bounds memory, but it doesn't say when the firmware is really done with a queue, and the right count would differ per load and per GPU. The data here stands: #2579 alone retains +2.17 GB, and releasing at any PowerOff crashes the firmware. What's missing is the real completion point. A lane is tracing how macOS tears down a context's queues on the same hardware, to find the event the firmware gives when it stops reading them; that replaces the count. This stays a draft until then.

Assisted-by: LLM (Claude:Opus 5.5)

@RyanTheTide

Copy link
Copy Markdown
Contributor Author

Still draft. Results from looking for a real release signal on the M1 Pro (same fleet76 tree, module-only test builds, one boot per row, render-churn mix plus no-idle-gap load):

  • Channel advance doesn't work. Freeing a destroyed context's queue structures once every (pipe, priority) channel it used has completed a job submitted after the DestroyContext ack crashed the firmware in mix run 5, with releases 3-5 ms after the ack. So the data master moving on to newer work on that channel is not the point where the firmware lets go.
  • No host-visible firmware table holds the address. A test build scanned all five kernel firmware heaps every 250 ms for the addresses of destroyed queues' QueueInfo, ring state, ring, notifier list, notifier and context data. They only show up in per-context and per-command objects and in freed memory, never in the firmware-global buffers (init data, runtime buffers).
  • The firmware trace doesn't mention them either. With the firmware ktrace channel enabled, none of 274 tracked destroyed contexts was named in any ktrace event between its ack and its release (windows from 5 ms to 9 s). The same run (mix 16/16 + 300 s no-gap, 0 crashes) also had 147 releases at a batch ending in PowerOff, which is more evidence that the existing PowerOff point is safe.

So the reference the firmware still follows after the ack lives in firmware-internal state, and it doesn't surface through any channel the driver can see today. The release rule has to come from how the reference OS tears these queues down. That work is in progress; this PR stays a draft until it lands.

Rows and evidence
arm change result
A1 release when every used channel completed post-ack work firmware crash, mix run 5 (PC ...16d9c)
S1 #2579 release + heap scan (scan stalls idled the GPU: 208 PowerOffs in 91 s) firmware crash, mix run 5, 3 s after a 981-context release; confounded by the stalls, not counted against #2579
K1 #2579 release + firmware ktrace mix 16/16 + no-gap 300 s, 0 crashes; 147 PowerOff releases; 0 ktrace references to destroyed queues before release

Module base: fleet76 e00fb9a + #2579, loaded on a one-shot of the unchanged kernel. Same harness as the earlier cap rows.

@RyanTheTide

Copy link
Copy Markdown
Contributor Author

Still draft. Two more release rules tried on the M1 Pro (fleet78, module-only test builds, one boot each, same churn harness). Both crashed the firmware:

  • Forcing the PowerOff boundary doesn't help. When the firmware reported a PowerOff but was already powered on again, a test build held new GPU work until a stats batch ended powered off, so drm/asahi: free retired queue structures after firmware poweroff #2579's release ran with the GPU off. It crashed in mix run 13 (about 13,400 churned processes), the same as releasing one power cycle later did before (about 13,000). So having the GPU off at release time is not what makes drm/asahi: free retired queue structures after firmware poweroff #2579 safe.
  • Firmware slot reuse isn't the signal either. The firmware writes a small slot number (0-7) into each context's data. Releasing a destroyed context once both of its slot numbers had been given to newer contexts freed queues 1-155 contexts after the ack. That crashed after about 1,900 processes, close to stock.

Across all the rows so far, the crash rate drops with the age of the queue structures at release, and no host-visible event we've tested marks the end:

release point age at release processes before first crash
at the DestroyContext ack (stock) 0 about 1,000-1,250
firmware slot reused 1-155 contexts about 1,900
every used channel completed newer work 3-5 ms about 3,800
oldest beyond 256 retained about 2 s about 6,400
one power cycle later / forced power-off about 0.1-1 s about 13,000
oldest beyond 512 / 1024 / 4096 4 s or more none in about 100,000 each
#2579 (batch ends powered off) seconds to minutes none

So the firmware seems to let go of destroyed queues on its own internal schedule. The count in this PR bounds the memory but doesn't name that point. The firmware analysis of how the reference OS tears these queues down is still waiting on access, and this stays a draft until it comes back.

Rows and evidence
arm module result
Q1 quiesce at PowerOff fleet78 + #2579 + lab hold of new ops until a batch ends PowerOff (max 200 ms) firmware crash, mix run 13, PC ...16d9c, ESR 0x96000004; 0 hold timeouts
S2 slot reuse fleet78 + lab log of the slot bytes in the context data (bytes 0 and 1, default 0xff) + release once both were reused firmware crash, mix run 2, PC ...16d9c, FAR 0xfa08ffffffa8; slots 0-7, both bytes always equal, lowest free first

Both modules were built from the exact fleet78 tree with undefined symbols identical to the stock module and loaded on a one-shot of the unchanged kernel. Earlier rows (channel advance, heap scan, firmware trace, caps) are in the comments above.

Assisted-by: LLM (Claude:Opus 5.5)

@RyanTheTide

Copy link
Copy Markdown
Contributor Author

Closing: a fixed count bounds the memory but is not the point where the firmware is done with the queues; a replacement needs that release signal.

@RyanTheTide RyanTheTide reopened this Nov 22, 2026
@RyanTheTide

Copy link
Copy Markdown
Contributor Author

Reopened and kept as a draft; the comment above lists what it still needs before review.

@RyanTheTide

Copy link
Copy Markdown
Contributor Author

Two results from the M1 Pro (G13S), both on the same kernel tree as before.

The detach candidate doesn't discriminate. A test build logged each retained queue's scheduler field and ring pointers when its context was invalidated. All 60 queues from 20 destroyed contexts already showed the field back at "unassigned" (-1) and the ring fully drained (done == write pointer) at that moment. So "release once the firmware writes the field back" would fire at once, the same as releasing at invalidation, which is what crashed before. It's not the completion point we're looking for.

The Vulkan device-churn fault is not a queue-release problem. The same build released nothing (#2579's rule, and the stats batches never ended powered off), and the firmware still crashed 6 seconds into the churn load. The stock module on the production kernel gives the same fault, and it is still there on our newest fleet kernel. A minimal trigger, stock Mesa, each row on a fresh boot:

load result
one-case process loop of dEQP-VK.renderpasses.dynamic_rendering.primary_cmd_buff.suballocation.multisample_resolve.layers_3.a2b10g10r10_uint_pack32.samples_4_baseLayer1 730 processes, 0 crashes
3 long-running deqp-vk processes over renderpasses cases (layered and multisample) 85 runs, 0
8 of those 467 runs, 0
3 of those + an api.smoke.triangle process loop 1,174 processes, 0
1 of those + the multisample/layers loop 919 processes, 0
3 of those + the multisample/layers loop crash after 100 processes (239 s); on the newer kernel after 15 (142 s); with the test build after 20 (6 s)

Every crash has the same firmware PC and ESR 0x96000004. The fault address is 0xffffffa6 or 0x........ffffffa0: a 64-bit load whose low word is the upper half of a firmware kernel address. That looks like a pointer read 4 bytes off inside some structure, reached only when short-lived contexts doing layered multisample rendering overlap other busy contexts. So this PR's memory bound and that crash need separate fixes; the crash shouldn't be counted against or for a release rule.

Details
  • Test build: module-only aurora.ko on the unchanged kernel tree, observe-only (release rule unchanged), logging per retained queue: scheduler field, ring done/read/write pointers at retain, at every later invalidation, and at the PowerOff release.
  • Crash dump registers (first crash): x21 = 0xffffffa600189e00, x02 = 0x00000000ffffffa6 = FAR.
  • Churn loop: about 3 processes per second; long-running processes run a 1,500-case shard each and are restarted when they finish.
  • The fault needs the churned context itself to do layered multisample rendering: a triangle loop next to the same three processes ran 1,059 processes clean.
  • Not yet run: the same reproducer on G14G (M2) to tell whether it is G13-only.

Assisted-by: LLM (Claude:Opus 5.5)

@RyanTheTide

Copy link
Copy Markdown
Contributor Author

Follow-up to the minimal-trigger table above: the same load on a G14G does not fault.

MacBook Air M2 (J413, T8112), current fleet default kernel with stock asahi DRM and stock Mesa, dEQP-VK 1.4.7.1. Load: 3 renderpasses shard loaders, plus the one-case loop of multisample_resolve.layers_3.a2b10g10r10_uint_pack32.samples_4_baseLayer1, the shape that faulted the M1 Pro twice (after 100 trials / 239 s and 15 trials / 142 s).

  • M2, 480 s: 1357 trials and 174 loader runs, 0 firmware crashes, 0 GPU faults, every trial exit 0.
  • To confirm the load is real, the trigger case passes on the M2 and the first 400 shard cases come out 231 Pass / 169 NotSupported / 0 Fail.

So the PC ...16d9c fault looks specific to G13 firmware, or at least far rarer on G14G. That fits a firmware structure that G13 reads differently, not anything in the shared queue-release code.

Assisted-by: LLM (Claude:Opus 5.5)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant