Repository navigation
drm/asahi: bound the queue structures held for firmware poweroff - #2619
RyanTheTide wants to merge 1 commit into
Conversation
The queue structures of invalidated contexts are kept until a statistics batch ends with a PowerOff. Under sustained load the firmware reports each PowerOff together with the following PowerOn, so that boundary can be minutes away, and idle sends none. A load that keeps creating and destroying rendering contexts then pins about 60 KiB of kernel memory per destroyed context: on the M1 Pro, 20,000 retained contexts and +1.4 GB of Shmem in 180 seconds, and the same pattern on the M2 (G14G). Releasing at every PowerOff is not safe: it brings the G13S firmware crash back. Instead keep the boundary and cap the list. When more than 1024 contexts wait, release those invalidated longest ago. With a cap of 256 the firmware crash still occurred, after about 6,400 churned contexts; 512, 1024 and 4096 each ran about 100,000 churned contexts without one. Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
|
Back to draft under our production-ready rule. A fixed count (1024) is a magic depth, not a release signal. It bounds memory, but it doesn't say when the firmware is really done with a queue, and the right count would differ per load and per GPU. The data here stands: #2579 alone retains +2.17 GB, and releasing at any PowerOff crashes the firmware. What's missing is the real completion point. A lane is tracing how macOS tears down a context's queues on the same hardware, to find the event the firmware gives when it stops reading them; that replaces the count. This stays a draft until then. Assisted-by: LLM (Claude:Opus 5.5) |
|
Still draft. Results from looking for a real release signal on the M1 Pro (same fleet76 tree, module-only test builds, one boot per row, render-churn mix plus no-idle-gap load):
So the reference the firmware still follows after the ack lives in firmware-internal state, and it doesn't surface through any channel the driver can see today. The release rule has to come from how the reference OS tears these queues down. That work is in progress; this PR stays a draft until it lands. Rows and evidence
Module base: fleet76 e00fb9a + #2579, loaded on a one-shot of the unchanged kernel. Same harness as the earlier cap rows. |
|
Still draft. Two more release rules tried on the M1 Pro (fleet78, module-only test builds, one boot each, same churn harness). Both crashed the firmware:
Across all the rows so far, the crash rate drops with the age of the queue structures at release, and no host-visible event we've tested marks the end:
So the firmware seems to let go of destroyed queues on its own internal schedule. The count in this PR bounds the memory but doesn't name that point. The firmware analysis of how the reference OS tears these queues down is still waiting on access, and this stays a draft until it comes back. Rows and evidence
Both modules were built from the exact fleet78 tree with undefined symbols identical to the stock module and loaded on a one-shot of the unchanged kernel. Earlier rows (channel advance, heap scan, firmware trace, caps) are in the comments above. Assisted-by: LLM (Claude:Opus 5.5) |
|
Closing: a fixed count bounds the memory but is not the point where the firmware is done with the queues; a replacement needs that release signal. |
|
Reopened and kept as a draft; the comment above lists what it still needs before review. |
|
Two results from the M1 Pro (G13S), both on the same kernel tree as before. The detach candidate doesn't discriminate. A test build logged each retained queue's scheduler field and ring pointers when its context was invalidated. All 60 queues from 20 destroyed contexts already showed the field back at "unassigned" (-1) and the ring fully drained (done == write pointer) at that moment. So "release once the firmware writes the field back" would fire at once, the same as releasing at invalidation, which is what crashed before. It's not the completion point we're looking for. The Vulkan device-churn fault is not a queue-release problem. The same build released nothing (#2579's rule, and the stats batches never ended powered off), and the firmware still crashed 6 seconds into the churn load. The stock module on the production kernel gives the same fault, and it is still there on our newest fleet kernel. A minimal trigger, stock Mesa, each row on a fresh boot:
Every crash has the same firmware PC and ESR 0x96000004. The fault address is 0xffffffa6 or 0x........ffffffa0: a 64-bit load whose low word is the upper half of a firmware kernel address. That looks like a pointer read 4 bytes off inside some structure, reached only when short-lived contexts doing layered multisample rendering overlap other busy contexts. So this PR's memory bound and that crash need separate fixes; the crash shouldn't be counted against or for a release rule. Details
Assisted-by: LLM (Claude:Opus 5.5) |
|
Follow-up to the minimal-trigger table above: the same load on a G14G does not fault. MacBook Air M2 (J413, T8112), current fleet default kernel with stock asahi DRM and stock Mesa, dEQP-VK 1.4.7.1. Load: 3 renderpasses shard loaders, plus the one-case loop of
So the PC ...16d9c fault looks specific to G13 firmware, or at least far rarer on G14G. That fits a firmware structure that G13 reads differently, not anything in the shared queue-release code. Assisted-by: LLM (Claude:Opus 5.5) |
Problem
#2579 keeps the queue structures of invalidated contexts until a statistics batch ends with a PowerOff. Under sustained load the firmware reports each PowerOff in the same batch as the following PowerOn, and idle sends none, so that boundary almost never arrives. Every destroyed context then pins about 60 KiB. On the M1 Pro a context-churning load retained 20,000 contexts and added 1.4 GB of Shmem in 180 seconds; a 300 second row added 2.17 GB that never came back. The M2 (G14G) shows the same pattern.
Releasing at any PowerOff is not a fix: a lab build doing that crashed the G13S firmware again after about 13,000 churned contexts.
Change
Keep the PowerOff boundary and cap the list. When more than 1024 invalidated contexts are waiting, the ones invalidated longest ago are released.
Testing
Module-only aurora.ko on the unchanged fleet76 kernel (integration e00fb9a + #2579), one boot per row. Each row: render-churn mix x32 (about 1,000 short-lived rendering processes per run), then 600 s with two glmark2 clients keeping the GPU busy plus four churning processes, with memory sampled every 10 s.
For comparison, the stock fleet76 module crashed the M1 Pro firmware on 2 of 2 boots within about 1,250 churned contexts. Latency of a one-case Vulkan process (33-34 ms median) and glmark2 terrain (402-404 FPS) are the same as with #2579 alone. The commit also builds on the current integration/aurora tip.
Still to run
Details for reviewers
powered_contextswithdrain(..excess)right after the push, under the same locks as the push. Order is invalidation order, so the front is the oldest.Assisted-by: LLM (Claude:Opus 5.5)