Repository navigation
drm/asahi: free retired queue structures after firmware poweroff - #2579
Conversation
DestroyContext completion removes a context from firmware lookup, but it does not mean the running data masters have stopped reading the queue structures of that context. On G13S (M1 Pro) a load that creates and destroys rendering contexts quickly crashes the GPU firmware in about one of five runs: the firmware follows queue structures that were freed, and reused, right after the invalidation was acknowledged. Free the context data right after the acknowledged invalidation, as before, but keep the queue structures of each invalidated context until the firmware reports the next PowerOff on the statistics channel, and free them then. The statistics ring is drained under its lock before a context is queued, so a PowerOff that was already pending before the invalidation cannot release it, and only the last power transition of a batch counts, so an old PowerOff followed by PowerOn releases nothing. Only the queue structures are held. Holding the context data as well keeps its TVB buffer alive, and under heavy context churn the firmware powers off too rarely for that: the M1 Pro ran out of GPU memory within seconds. If the invalidation times out, or the bookkeeping allocation fails, leak the objects instead of freeing memory the firmware may still use. Link: #2506 Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>
DjDeveloperr
left a comment
There was a problem hiding this comment.
Approving the code. I'm not merging yet, because nothing has built this exact head. Integration CI doesn't compile drm/asahi: host-tests on be4ddeb was the repo-wide tool suite (451/451), and your hardware rows used a fleet74 + #2468 module build. Please post the buildbox result for be4ddeb in a comment: the SHA, rustc version, and the drivers/gpu/drm/asahi/ module build with its warning count. A build of be4ddeb merged into current integration/aurora would be even better, since memacct.rs and sysfs.c in this driver have changed since your base. I'll merge as soon as that's posted.
What I checked:
- Lifetime of the queue structures:
- After the acknowledged DestroyContext, only
datais dropped.retainedgoes ontopowered_contexts(gpu.rs:1731-1757). - The stats drain at
gpu.rs:1740-1741and the push both happen underrx_channels. The release path (channel.rs:744, called frompoll()under the same lock) therefore can't run between them, and a PowerOff that was already in the ring releases only older entries. - Released objects go to the heap allocator's garbage (
alloc.rs:604-627) and are reused only after the next flush. So nothing is reused between the PowerOff and the cache flush either.
- After the acknowledged DestroyContext, only
- Contexts reused before PowerOff: contexts aren't reused. A new context and its queues get fresh allocations, and a retired context's queue memory stays out of the allocator until its PowerOff and a later flush.
- Locks:
gpu.alloc()is only called from submission and queue creation (queue/mod.rs:1248,render.rs:281,compute.rs:52,gpu.rs:1658), never underrx_channels. Dropping heap objects infw_power_off()takes only the mm allocator's inner lock, which is a leaf. I see no new ordering problem. - If PowerOff never comes:
- Firmware-driven sleep: legacy GPUs already refuse it (
driver.rs:278). Over s2idle the firmware keeps running and powers the GPU off when idle, so a PowerOff still arrives. - Unbind: the new
stats.set_manager()(gpu.rs:851) adds a second strong-reference cycle next to the existingevent.set_manager()(:850), and RTKit holds the manager too. The legacy manager is never dropped on unbind today, so the retired list leaks along with everything else. That's no worse than before. - Timed-out invalidation (
:1727): leaking is the right call.
- Firmware-driven sleep: legacy GPUs already refuse it (
- Memory: the first version's 10.5 GB of pinned memory came from holding the context data with the TVB buffers. With only the queues held, a retired context costs about one
gpu_buf(0x2c18), plus its ring and state, per queue. Your 32 flat runs on the M1 Pro, with 219 PowerOffs in about 217 s, fit that.
Not blocking:
- Two comments still describe the old rule:
retain()atworkqueue.rs:175("until the firmware has invalidated this context") and theWorkQueueInnerdrop comment at:726-728. Please update them in a follow-up, or in this branch if you push again for the build. - Under a load with no idle gap, the list grows until the next PowerOff. A debug-level count of the list length at each release would make that easy to see in the soak.
Post-merge rows (we can't produce them here):
- G13C/G13X and G14X churn
- the no-idle-gap load (a game plus context churn), logging the list length
- a same-day control on the identical tree
- the G14G PowerOff rate, so the M2 memory picture matches the M1 Pro's
The revert is a single commit.
|
Buildbox build of the exact head
Build command: Results:
Logs (build script, full build, W=1, clippy, KUnit): https://gist.github.com/RyanTheTide/40db2224eb9e68cabbe50687044c10d2 |
|
G17S base row (M5 Pro, J714s): #2579 not applied, default kernel 7.3.0-m5-intg28d, default system Vulkan driver. The kit is your churn-mix kit (release
The health check passed 3/3 after every pass. GPU liveness after the arm was smoke 2/2 on the same boot, d04c650a. So G17S does not crash under this context churn on the default kernel: 0 of 10 passes, against the G13S 9/47 baseline. Our 8-client |
|
M5 (G17S) relevance, from code: #2579 can't change anything on G17S, so it doesn't need an M5 row to merge. I checked this while chasing the M5 8-client
So none of #2579's new code runs on M5, and an M5 run of it would measure only the unchanged G17 path. The M5 failure has a different cause, in G17's task-stage render path ( This says nothing for or against #2579 on G13/G14, where its code runs. |
|
@DjDeveloperr here are the post-merge rows from your review: the same-day control on the identical tree, and the no-idle-gap load with the list length logged. Machine: M1 Pro (G13S). Control. Both arms ran on the same fleet76 kernel (integration
So the fix holds on the identical tree, and stock still crashes within the first thousand or so destroyed contexts. Memory: the list is almost never released on G13S under load. In the 300 s no-idle-gap run (two offscreen glmark2 clients plus four churners), Shmem and Unevictable rose from 0.82 GB to 2.99 GB, with no GEM clients left. That is about 60 KB per destroyed context. To see why, I loaded a lab-only build that adds three debug-flag-gated log lines (release count, every 1000th queued context, and a line for a batch that saw a PowerOff but ended PowerOn):
Relaxing the boundary is not safe. I also tried a lab variant that releases at any PowerOff in a batch, in two stages (contexts queued before one PowerOff batch are freed at the next one). It crashed the firmware in mix run 14, after about 13,000 churn processes, 140 ms after a burst of releases (842 contexts, then 25 only 5 ms later). So a PowerOff reported mid-load does not mean the firmware is done with those queues. The current "batch must end powered off" rule is what keeps #2579 safe, and on G13S it rarely fires while the GPU is in use. Net: #2579 fixes the crash, and its memory cost is unbounded under sustained context churn: about 60 KB per destroyed context, kept until the batch that finally ends in PowerOff, then pinned in the heap. Ordinary desktop churn is small. A long session that creates and destroys many Vulkan devices (test suites, launchers) grows without bound. I don't have a safe release signal yet. Options I see: a firmware-side completion for the destroyed context's queues, or capping the list and stalling new context creation until a real power-off batch. Builds, method and data
Assisted-by: LLM (Claude:Opus 5.5) |
Problem
On the M1 Pro (G13S) the GPU firmware crashes under a load that creates and destroys rendering contexts quickly: a few long-running Vulkan renderers plus many short-lived rendering processes. #2468 removed the VA-0 faults from this load, but the firmware crash stayed: 9 of 47 runs on fleet74 with its first commit, and 16 of 30 on the integration kernel with all of it. After a crash the GPU is wedged until reboot. Tracked in #2506.
The driver frees a context's queue structures as soon as DestroyContext is acknowledged, and the firmware can still read them after that. Keeping the queue structures of the last 256 destroyed contexts stopped the crash, but a fixed depth is a delay, not a release rule.
Change
There is no chip or generation check. The rule applies to every GPU the driver supports.
Only the queue structures are held. A first version also held the context data, and with it the TVB buffer. Under this load the firmware does not power off often enough for that, and the M1 Pro ran out of GPU memory within seconds (details below).
Testing
Module builds of fleet74 + #2468 + this change, loaded on a one-shot boot of the unchanged fleet74 kernel (
modprobe.blacklist=aurora, then insmod). Same config as shipped, vermagic matches, same 595 undefined symbols as the stock module.Without this change the same mix crashed the firmware in 9 of 47 runs (fleet74 + the first #2468 commit) and 16 of 30 (integration kernel + all of #2468). At 9 in 47, 32 clean runs in a row by chance has a probability of about 0.1%.
Memory on the M1 Pro stayed flat: MemAvailable 12.0 to 12.2 GB for all 32 mix runs. The firmware reported 219 PowerOffs in about 217 s of that boot, so retired queue structures were freed about once a second.
The branch builds against integration/aurora with the same 16 warnings as before and no new clippy lints in the changed code.
Still to run
First version: holding the whole context until PowerOff ran out of memory
Same tree and load, but the retired context kept its context data (with the TVB buffer) until the next PowerOff:
VK_ERROR_OUT_OF_DEVICE_MEMORYatvkEndCommandBuffer.Holding only the queue structures (this PR) leaves memory flat (table above). With the first version Shmem was 10.5 GB after 4 runs. With this PR it was 1.06 GB after 32 runs, J=8 and the subset.
Load and counters
dEQP-VK.api.copy_and_blit.*.blit_image.*.3d.r8g8b8a8_srgb.*.dEQP-VK.geometry.layered.*,pipeline.monolithic.render_to_image.*,api.copy_and_blit.core.blit_image.simple_tests.*,api.copy_and_blit.core.resolve_image.*,renderpass.suballocation.multisample.*,multiview.renderpass.*.GPU timeout,unit: TE,Fault info,RTKit: co-processor has crashed. Whole-boot log on both Macs: no oops, WARN, hung task,leaking it for safetyorNo GPU managerlines.alloc()now takesrx_channelsand thenpowered_contextsfor each invalidated context. The receive path takesrx_channels, thenstats.poll(), which can callfw_power_off(), which takespowered_contexts. That is the same order in both places, and none of thegpu.alloc()callers holdrx_channels.Assisted-by: LLM (Claude:Opus 5.5)