Skip to content

drm/asahi: free retired queue structures after firmware poweroff - #2579

Merged
DjDeveloperr merged 1 commit into
integration/aurorafrom
gpu/g13-release-queues-after-poweroff
Nov 20, 2026
Merged

DjDeveloperr merged 1 commit into
integration/aurorafrom
gpu/g13-release-queues-after-poweroff

Conversation

@RyanTheTide

@RyanTheTide RyanTheTide commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Problem

On the M1 Pro (G13S) the GPU firmware crashes under a load that creates and destroys rendering contexts quickly: a few long-running Vulkan renderers plus many short-lived rendering processes. #2468 removed the VA-0 faults from this load, but the firmware crash stayed: 9 of 47 runs on fleet74 with its first commit, and 16 of 30 on the integration kernel with all of it. After a crash the GPU is wedged until reboot. Tracked in #2506.

The driver frees a context's queue structures as soon as DestroyContext is acknowledged, and the firmware can still read them after that. Keeping the queue structures of the last 256 destroyed contexts stopped the crash, but a fixed depth is a delay, not a release rule.

Change

  • After the acknowledged invalidation, free the context data right away (as now), but keep the context's queue structures on a list until the firmware reports the next PowerOff on the statistics channel. Free the list then.
  • Drain the statistics ring, under its lock, before a context goes on the list, so a PowerOff that was already pending cannot free it. A batch of messages only counts as a release point if its last power transition is a PowerOff.
  • If the invalidation times out or the list allocation fails, leak the objects instead of freeing memory the firmware may still use.
  • KUnit tests for the power-transition batching.

There is no chip or generation check. The rule applies to every GPU the driver supports.

Only the queue structures are held. A first version also held the context data, and with it the TVB buffer. Under this load the firmware does not power off often enough for that, and the M1 Pro ran out of GPU memory within seconds (details below).

Testing

Module builds of fleet74 + #2468 + this change, loaded on a one-shot boot of the unchanged fleet74 kernel (modprobe.blacklist=aurora, then insmod). Same config as shipped, vermagic matches, same 595 undefined symbols as the stock module.

Mac Render-churn mix J=8 3D-blit loop deqp-vk subset (2076) Faults / timeouts / fw crash
M1 Pro (G13S) 32/32 runs, 31,975 churn processes, 368,640/368,640 renders 20/20 runs, 460,800 case runs 1610 Pass / 466 NotSupported, same cases as stock 0 / 0 / 0
M2 (G14G) 4/4 runs, 4,215 churn processes 5/5 runs, 115,200 case runs 1610 Pass / 466 NotSupported, same cases as stock 0 / 0 / 0

Without this change the same mix crashed the firmware in 9 of 47 runs (fleet74 + the first #2468 commit) and 16 of 30 (integration kernel + all of #2468). At 9 in 47, 32 clean runs in a row by chance has a probability of about 0.1%.

Memory on the M1 Pro stayed flat: MemAvailable 12.0 to 12.2 GB for all 32 mix runs. The firmware reported 219 PowerOffs in about 217 s of that boot, so retired queue structures were freed about once a second.

The branch builds against integration/aurora with the same 16 warnings as before and no new clippy lints in the changed code.

Still to run

  • A same-day control run without the change on the identical tree (the crash rates above are from the earlier runs linked in drm/asahi: G13S GPU firmware crash after DestroyContext under client churn #2506).
  • M1 Max / Ultra (G13C/G13X) and M2 Pro/Max (G14X) rows.
  • Retention under a load that keeps the GPU busy without a break (for example a game running while other apps create and destroy contexts). Queue structures are only freed at a PowerOff, so in that case they wait for the next idle moment.
  • The KUnit suite was not run on hardware: the shipped config builds without KUnit. Its logic tests pass with host rustc.
First version: holding the whole context until PowerOff ran out of memory

Same tree and load, but the retired context kept its context data (with the TVB buffer) until the next PowerOff:

  • mix runs 1 to 4: no faults or crashes, but MemAvailable fell from 13.6 GB to 2.4 GB in about 35 s. Run 2 completed 11233/11520 renders and run 4 6925/11520. Then the 3-case health check failed.
  • After the load stopped, nothing was freed: Shmem and Unevictable stayed at 10.5 GB (pinned kernel GPU objects; no user BOs were left in debugfs). A new 3-case run failed with VK_ERROR_OUT_OF_DEVICE_MEMORY at vkEndCommandBuffer.
  • The last power transition the firmware reported was a PowerOn, and no statistics message arrived after it, so nothing reached a release point.

Holding only the queue structures (this PR) leaves memory flat (table above). With the first version Shmem was 10.5 GB after 4 runs. With this PR it was 1.06 GB after 32 runs, J=8 and the subset.

Load and counters
  • Render-churn mix: 4 long-lived deqp-vk processes, each running 2880 passing 3D-blit cases, plus 4 loops of short deqp-vk processes running 3 rendering 3D-blit cases each, until the long ones finish (about 9 s and about 1,000 short processes per run). A 3-case health check runs after every run.
  • J=8: 8 workers x 30 processes x 96 cases of dEQP-VK.api.copy_and_blit.*.blit_image.*.3d.r8g8b8a8_srgb.*.
  • Subset: dEQP-VK.geometry.layered.*, pipeline.monolithic.render_to_image.*, api.copy_and_blit.core.blit_image.simple_tests.*, api.copy_and_blit.core.resolve_image.*, renderpass.suballocation.multisample.*, multiview.renderpass.*.
  • Counted from the kernel log per run: GPU timeout, unit: TE, Fault info, RTKit: co-processor has crashed. Whole-boot log on both Macs: no oops, WARN, hung task, leaking it for safety or No GPU manager lines.
  • Locks: alloc() now takes rx_channels and then powered_contexts for each invalidated context. The receive path takes rx_channels, then stats.poll(), which can call fw_power_off(), which takes powered_contexts. That is the same order in both places, and none of the gpu.alloc() callers hold rx_channels.

Assisted-by: LLM (Claude:Opus 5.5)

DestroyContext completion removes a context from firmware lookup, but it
does not mean the running data masters have stopped reading the queue
structures of that context. On G13S (M1 Pro) a load that creates and
destroys rendering contexts quickly crashes the GPU firmware in about one
of five runs: the firmware follows queue structures that were freed, and
reused, right after the invalidation was acknowledged.

Free the context data right after the acknowledged invalidation, as
before, but keep the queue structures of each invalidated context until
the firmware reports the next PowerOff on the statistics channel, and free
them then. The statistics ring is drained under its lock before a context
is queued, so a PowerOff that was already pending before the invalidation
cannot release it, and only the last power transition of a batch counts,
so an old PowerOff followed by PowerOn releases nothing.

Only the queue structures are held. Holding the context data as well
keeps its TVB buffer alive, and under heavy context churn the firmware
powers off too rarely for that: the M1 Pro ran out of GPU memory within
seconds.

If the invalidation times out, or the bookkeeping allocation fails, leak
the objects instead of freeing memory the firmware may still use.

Link: #2506
Signed-off-by: Ryan Murray <ryan@aurorasilicon.org>

@DjDeveloperr DjDeveloperr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving the code. I'm not merging yet, because nothing has built this exact head. Integration CI doesn't compile drm/asahi: host-tests on be4ddeb was the repo-wide tool suite (451/451), and your hardware rows used a fleet74 + #2468 module build. Please post the buildbox result for be4ddeb in a comment: the SHA, rustc version, and the drivers/gpu/drm/asahi/ module build with its warning count. A build of be4ddeb merged into current integration/aurora would be even better, since memacct.rs and sysfs.c in this driver have changed since your base. I'll merge as soon as that's posted.

What I checked:

  • Lifetime of the queue structures:
    • After the acknowledged DestroyContext, only data is dropped. retained goes onto powered_contexts (gpu.rs:1731-1757).
    • The stats drain at gpu.rs:1740-1741 and the push both happen under rx_channels. The release path (channel.rs:744, called from poll() under the same lock) therefore can't run between them, and a PowerOff that was already in the ring releases only older entries.
    • Released objects go to the heap allocator's garbage (alloc.rs:604-627) and are reused only after the next flush. So nothing is reused between the PowerOff and the cache flush either.
  • Contexts reused before PowerOff: contexts aren't reused. A new context and its queues get fresh allocations, and a retired context's queue memory stays out of the allocator until its PowerOff and a later flush.
  • Locks: gpu.alloc() is only called from submission and queue creation (queue/mod.rs:1248, render.rs:281, compute.rs:52, gpu.rs:1658), never under rx_channels. Dropping heap objects in fw_power_off() takes only the mm allocator's inner lock, which is a leaf. I see no new ordering problem.
  • If PowerOff never comes:
    • Firmware-driven sleep: legacy GPUs already refuse it (driver.rs:278). Over s2idle the firmware keeps running and powers the GPU off when idle, so a PowerOff still arrives.
    • Unbind: the new stats.set_manager() (gpu.rs:851) adds a second strong-reference cycle next to the existing event.set_manager() (:850), and RTKit holds the manager too. The legacy manager is never dropped on unbind today, so the retired list leaks along with everything else. That's no worse than before.
    • Timed-out invalidation (:1727): leaking is the right call.
  • Memory: the first version's 10.5 GB of pinned memory came from holding the context data with the TVB buffers. With only the queues held, a retired context costs about one gpu_buf (0x2c18), plus its ring and state, per queue. Your 32 flat runs on the M1 Pro, with 219 PowerOffs in about 217 s, fit that.

Not blocking:

  1. Two comments still describe the old rule: retain() at workqueue.rs:175 ("until the firmware has invalidated this context") and the WorkQueueInner drop comment at :726-728. Please update them in a follow-up, or in this branch if you push again for the build.
  2. Under a load with no idle gap, the list grows until the next PowerOff. A debug-level count of the list length at each release would make that easy to see in the soak.

Post-merge rows (we can't produce them here):

  • G13C/G13X and G14X churn
  • the no-idle-gap load (a game plus context churn), logging the list length
  • a same-day control on the identical tree
  • the G14G PowerOff rate, so the M2 memory picture matches the M1 Pro's

The revert is a single commit.

@RyanTheTide

Copy link
Copy Markdown
Contributor Author

Buildbox build of the exact head be4ddeb4ffa95dd8898fdd866adca606264e21d1, merged into the current integration/aurora.

  • Merge base (integration/aurora tip): df62a8c4d38dc1888cff32653329ea88abf99c7c. The PR's own fork point is 391fac63 (68 commits behind).
  • Local merge: git merge --no-ff be4ddeb4 gives 422a0c18 with no conflicts. Nothing was pushed.
  • Config: Fedora 7.0.13-400.asahi.fc44 base + arch/arm64/configs/asahi.config (merge_config + olddefconfig), CONFIG_DRM_ASAHI=m, CONFIG_RUST=y
  • Toolchain: clang 22.1.6, rustc 1.97.1, bindgen 0.73.2

Build command:

make ARCH=arm64 LLVM=1 O=build LOCALVERSION=-aurora-pr2579 -j24 Image modules dtbs
make ... W=1 drivers/gpu/drm/asahi/
make ... CLIPPY=1 drivers/gpu/drm/asahi/
tools/testing/kunit/kunit.py run --arch=arm64 --kunitconfig=drivers/gpu/drm/asahi --make_options LLVM=1

Results:

Step Result
Full build (Image, modules, dtbs) rc=0. 32 warning: lines across the whole build, 0 of them in channel.rs, gpu.rs or workqueue.rs (all existing dead-code warnings in g17/, soc/apple, etc.)
aurora.ko links: 120 MB, vermagic 7.3.0-aurora-pr2579 SMP preempt mod_unload aarch64, depends apple-pmp,macsmc-hwmon,appledrm
W=1 drivers/gpu/drm/asahi rc=0, 16 warnings, 0 errors, 0 in the files this PR touches
CLIPPY=1 drivers/gpu/drm/asahi rc=0. 1 hit in a touched file (gpu.rs:307, undocumented_unsafe_blocks on unsafe impl HasWork), which comes from older code (6a6b371) and not from this PR. 0 hits on lines this PR adds
drm/asahi KUnit (arm64 QEMU) 354 tests: 352 passed, 0 failed, 2 skipped (apple_g17_rx_util_min_test needs CONFIG_UCLAMP_TASK; j713_accepts_all_loaded_images). The three new tests power_off_is_release_boundary, later_power_on_cancels_release and final_power_off_wins pass

Logs (build script, full build, W=1, clippy, KUnit): https://gist.github.com/RyanTheTide/40db2224eb9e68cabbe50687044c10d2

@DjDeveloperr
DjDeveloperr merged commit 789b208 into integration/aurora Nov 20, 2026
1 check passed
@DjDeveloperr

Copy link
Copy Markdown
Contributor

G17S base row (M5 Pro, J714s): #2579 not applied, default kernel 7.3.0-m5-intg28d, default system Vulkan driver.

The kit is your churn-mix kit (release untagged-cbbbcf0b6c6ec40b3e19, tar sha256 0bf8c6ce…). The case lists are used verbatim: big.txt, devinit.txt and three.txt. mix.sh/mixN.sh are adapted only for M5 paths: user alarm, our CTS 1.4.7.1 tree, and the default ICD instead of the asahi override. There are 10 passes of 4 renderers (one deqp-vk over big.txt each) plus 4 churners (devinit.txt create/destroy loops), with the three.txt health check after each pass.

pass secs render Pass churn procs watchdog give-ups GPU recoveries client failures runtime stopped fw crash
1–10 4–5 11520/11520 every pass 723–768 per pass 0 0 0 0 0

The health check passed 3/3 after every pass. GPU liveness after the arm was smoke 2/2 on the same boot, d04c650a. So G17S does not crash under this context churn on the default kernel: 0 of 10 passes, against the G13S 9/47 baseline. Our 8-client mesh_shader.ext.misc.many_task_work_groups_x repro, which ends with "GPU runtime stopped, device lost" (row on #2500), is therefore a different trigger than churn alone: long task work from several clients hitting the 2 s watchdog. #2579 is not built here.

@DjDeveloperr

Copy link
Copy Markdown
Contributor

M5 (G17S) relevance, from code: #2579 can't change anything on G17S, so it doesn't need an M5 row to merge.

I checked this while chasing the M5 8-client dEQP-VK.mesh_shader.ext.misc.many_task_work_groups_x failure (lost devices, then "GPU runtime stopped" for every client) at the tip this PR merges into (df62a8c).

  • Code this PR changes: drivers/gpu/drm/asahi/{channel.rs,gpu.rs,workqueue.rs}. That is GpuManager::ver, StatsChannel::ver and the legacy workqueue::RetiredContext/RetainedObject path, all used by G13 to G16.
  • G17S probe path: driver.rs probes g17::Gpu for G17S (probe_g17). g17::Gpu::manager() returns None (g17/mod.rs:2484).
  • G17's own code: G17 has its own channels and queues under g17/. Nothing under g17/ refers to GpuManager, StatsChannel, RetiredContext or RetainedObject, and its work queues are kernel::workqueue, not crate::workqueue.

So none of #2579's new code runs on M5, and an M5 run of it would measure only the unchanged G17 path. The M5 failure has a different cause, in G17's task-stage render path (g17/queue/render.rs) and its device-wide watchdog stop (#2500's area). I'll link that fix PR here once it's open.

This says nothing for or against #2579 on G13/G14, where its code runs.

@RyanTheTide

Copy link
Copy Markdown
Contributor Author

@DjDeveloperr here are the post-merge rows from your review: the same-day control on the identical tree, and the no-idle-gap load with the list length logged. Machine: M1 Pro (G13S).

Control. Both arms ran on the same fleet76 kernel (integration e00fb9a8, which predates this merge), each on a fresh boot with the stock module blacklisted and the arm's module loaded by hand. Stock is the installed fleet76 aurora.ko. #2579 is e00fb9a8 + be4ddeb4, built with the shipped config; its 613 undefined symbols are identical to stock.

arm load result
stock, boot 1 churn mix x16 firmware crash in run 1, 7 s in (about 1,000 churn processes)
stock, boot 2 no-idle-gap load firmware crash 16 s in (about 1,250 churn processes)
#2579 mix x16, then no-idle-gap 300 s 16/16 runs, 184,320/184,320 renders, about 49,000 churn processes; 0 crashes, timeouts or faults

So the fix holds on the identical tree, and stock still crashes within the first thousand or so destroyed contexts.

Memory: the list is almost never released on G13S under load. In the 300 s no-idle-gap run (two offscreen glmark2 clients plus four churners), Shmem and Unevictable rose from 0.82 GB to 2.99 GB, with no GEM clients left. That is about 60 KB per destroyed context. To see why, I loaded a lab-only build that adds three debug-flag-gated log lines (release count, every 1000th queued context, and a line for a batch that saw a PowerOff but ended PowerOn):

  • In 180 s of load, 34 stats batches contained a PowerOff and all 34 ended with a PowerOn, so nothing was released. 20,000+ contexts were retained, and Shmem went from 0.06 to 1.45 GB.
  • In every asahi_stats snapshot, the PowerOff and PowerOn timestamps are about 125-170 ns apart, so the firmware reports a PowerOff together with the next PowerOn. At idle it sends no PowerOff at all: nothing was released during 120 s of idle.
  • The first batch that did end in PowerOff was from a single small job after the idle period: "Releasing 20296 invalidated GPU contexts". Shmem fell only 58 MB, because the kernel heap keeps the backing it grew into. The peak retention therefore stays pinned until the module unloads.

Relaxing the boundary is not safe. I also tried a lab variant that releases at any PowerOff in a batch, in two stages (contexts queued before one PowerOff batch are freed at the next one). It crashed the firmware in mix run 14, after about 13,000 churn processes, 140 ms after a burst of releases (842 contexts, then 25 only 5 ms later). So a PowerOff reported mid-load does not mean the firmware is done with those queues. The current "batch must end powered off" rule is what keeps #2579 safe, and on G13S it rarely fires while the GPU is in use.

Net: #2579 fixes the crash, and its memory cost is unbounded under sustained context churn: about 60 KB per destroyed context, kept until the batch that finally ends in PowerOff, then pinned in the heap. Ordinary desktop churn is small. A long session that creates and destroys many Vulkan devices (test suites, launchers) grows without bound. I don't have a safe release signal yet. Options I see: a firmware-side completion for the destroyed context's queues, or capping the list and stalling new context creation until a real power-off batch.

Builds, method and data
  • Kernel: 7.3.0-aurora-fleet76 (integration e00fb9a8), unchanged, booted with modprobe.blacklist=aurora and the arm's module inserted by hand. Each arm is a fresh boot. A firmware crash wedges the GPU, so a stock arm ends at its first crash.
  • Modules (module-only M= builds, shipped fleet76 config, vermagic 7.3.0-aurora-fleet76, 15 warnings):
  • Churn mix: 4 deqp-vk renderers (2,880 3D blits each) plus 4 churners looping short deqp-vk processes (device and queues created, 3 blits, destroyed), x16, with a 3-case health check after each pass.
  • No-idle-gap: 2x glmark2-es2-gbm --off-screen -s 1920x1080 -b terrain --run-forever plus 4 churners. asahi_stats PowerOff/PowerOn counts and /proc/meminfo every 2 s. A 1-case deqp-vk process timed every 10 s as a latency probe: median 34-35 ms with drm/asahi: free retired queue structures after firmware poweroff #2579, 0 failures. Then a 120 s idle tail and one 1-case process.
  • PowerOffs during the drm/asahi: free retired queue structures after firmware poweroff #2579 300 s load: 64 (155 -> 219), each in a batch that ended with a PowerOn.
  • Lab-logging run: 34 "batch had a PowerOff but ends PowerOn: no release" lines, 0 releases during the load and the idle tail, "20000 invalidated GPU contexts retained", then "Releasing 20296" at the first batch that ended in PowerOff.
  • Two-stage run: runs 1-13 clean (0 timeouts, TE or Fault info), crash at run 14. The last release lines before the crash were 842 / 25 / 1 / 4 contexts, all within 160 ms.

Assisted-by: LLM (Claude:Opus 5.5)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants