Repository navigation
drm/asahi: g17: keep the GPU running for other clients when one client's commands hang or restart the firmware - #2500
Conversation
|
Update 11-20 10:3x: superseded by the clean-boot rerun in the next comment. The 8-client failure reproduces on a fresh boot and ends with "GPU runtime stopped, device lost". Correction (11-20 ~10:00 EST): the 8-client row I posted earlier is not valid evidence, so I've removed it. That run came right after three firmware recoveries from another case, What remains valid, on G17S (M5 Pro, J714s), Mesa 463da7d candidate DSO, #2500 not applied:
The intg26 kernel line was "G17: watchdog gave up on a compute command …: allowance 2000 ms, at the head of its queue for 2000 ms; not long work; no sharing renewals on this GPU", followed by ETIMEDOUT and a lost device. A clean 8-client rerun (shared first, on a fresh boot, without |
|
Clean-boot rerun on G17S. This supersedes my correction above: the 8-client failure is real, and it takes the whole GPU down. Setup: M5 Pro (J714s), kernel 7.3.0-m5-intg28d without #2500, Mesa 463da7d candidate DSO. Fresh boot: uptime 181 s, and 0 GPU-recovery or watchdog lines in the boot before the run.
Kernel, in order (our Linux log, trimmed):
Second candidate (11-20 10:4x): linux#2579 (RyanTheTide, head be4ddeb, "free retired queue structures after firmware poweroff"). It fixes a G13S firmware crash under context churn: context data is still freed at the DestroyContext ack, but queue structures are now kept until the next firmware PowerOff. It has no chip gate, so it reaches G17S, and on the M1 Pro crashes went from 9/47 to 0. The 8-client run above creates and destroys contexts while one client hangs, so both PRs are in scope. Neither is built here; this row is on the default kernel. A ~90 ms dispatch run by 8 clients ends with the whole GPU runtime stopped. The first client is failed "as its own" with no sharing renewal. Two VMs block on each other, and the others time out behind them. The runtime then stops for everyone. That looks like exactly the case #2500 targets. We have not built #2500. If a G17S row on its head would help, say so and we'll queue one. The earlier q-003 run, the one I called invalid above, followed the same path and ended with the same ENODEV. The |
|
Thanks for the clean-boot rerun, that sequence is very useful. Here is how it lines up with #2500, plus what we see on the J775D (M5 Ultra). Why the runtime stopped. Your 8 give-ups (1 at 2.06 s, 7 at 18.2 s, nothing retired in between) are exactly the storm limit: 8 watchdog expiries within 30 s with no progress. That is the path #2500 changes. What #2500 covers in your sequence. Each of your 8 give-ups is the first one for its command, and #2500 no longer counts those toward the limit. So with #2500 the runtime should keep running, and new processes should not get ENODEV. Your timing (2 s, then 2 s + 8 deferral polls = 18.2 s) matches the "10 VMs, each with a hung compute command" row in the PR body, where the runtime kept running. What it does not cover. #2500 does not change any deferral or give-up decision, so the 8 clients would still lose their devices. The two VMs deferring to each other. This is not a cycle the watchdog can get stuck in. Both engines take their publication number from one device-wide counter, and a command only defers behind another VM's command with a lower number. The oldest unfinished command therefore never defers and is decided on its own. vm 19 and vm 23 each had a compute command and a render command in flight, and each VM had one command older than one of the other's. vm 19's render was the oldest, so it was the first give-up at 2 s. #2579. As far as I can tell it does not reach G17S. It changes J775D rows (integ-c41 = integration e0ee46a + aura-pack4, 27.2 stub; 8 processes, one VkDevice each, all submitting one command at the same moment; each command alone takes the time shown):
No runtime stop on any kernel. Work retires between the give-ups here, which starts a new storm window. So your 90 ms case does not reproduce on the J775D, but the same first step does once the shared work adds up to more than about 2 s. On M5 a command that has not finished at its first watchdog expiry (2 s after it reaches the head of its queue) fails then, unless older work of another VM holds it back. Clients running side by side retire nothing until they end, so the oldest one is failed while all of them are still making progress. The sharing fix is a local commit, not pushed yet. On M5 it renews a command, up to the existing 30 s bound, while another VM has unfinished work running beside it. A lone spin still fails at 2 s. The cost: hangs beside each other now share the 30 s bound. 10 compute spins submitted together lose their devices from 30 s on, instead of from 2 s. The difference from your run. Here, after the first give-up, the other clients finish 0.1-0.2 s later. On G17S your other 7 did not finish in the next 16 s, so something on G17S kept them from running after the first give-up, which was a render there and a compute here. The sharing fix would not help with that: it would only move the first give-up from 2 s to about 30 s. One more thing we saw, not yet looked into: 4 x 600 ms render clients next to 4 x 600 ms compute clients gave 4 firmware restarts, and the 4 compute clients lost their devices. A G17S row would help, yes, in this order:
How long does one |
6e0a2c4 to
d5d966b
Compare
|
Assisted-by: LLM (Claude:Opus 5.5) |
…en watchdog_timeout() takes an argument
watchdog_publication_guard_tests() checks that timed_out() returns
early for a packet still in publication, then releases the packet's
backlog, and only then calls the shared watchdog_timeout() that counts
toward the timeout storm limit. It finds the last of these by the exact
text of the call, "self.shared.watchdog_timeout()", so it fails with
"substring not found" once the call takes an argument, although the
order it checks still holds.
Match the call by "self.shared.watchdog_timeout(" instead. The test
still requires the publication guard, release_backlog(&packet) and
watchdog_timeout() in that order in timed_out(), and passes on the
current code as on a call with arguments.
Signed-off-by: Jeff Queisser <funkzy@users.noreply.github.com>
|
#2500 on the M5 Pro (J714s, G17S): it keeps the runtime alive through the 8-client stall. Evidence only; I haven't changed this PR. Kernel: 7.3.0-m5-sship2 (d1d26ab) + this PR's two commits (3bc86c7, 6e0a2c4) + #2629 (796130c). The A/B also uses a test-only debugfs switch that turns #2629's ordering off, so the original stall happens again on top of #2500. Native warm build on the M5, one boot.
Without #2500, the same stall stopped the runtime with "GPU runtime stopped, device lost" for every client until reboot. That happened twice on clean boots of intg28d. In those runs the device-wide stop came from one ~32 ms burst of first give-ups by 8 different clients. That is the case this PR stops counting. With #2500 the stalled clients lose only their own devices, the GPU keeps serving others, and the mesh path came back once the stalled clients exited. #2629 removes this particular stall; #2500 is what keeps a stall like it from taking the GPU down for everyone. |
|
Independent review of #2500 at d5d966b (a fresh reviewer agent from the M5 GPU-recovery lane, read-only; summary below). funkzy, it's your PR, so I haven't changed anything; one question at the end. CI. Design point the reviewer treats as blocking unless you accept it: the device-wide stop no longer has a bound for a GPU that makes no progress for anyone, if each event lands on a new command or VM.
Non-blocking
Hardware (already posted above): on the M5 Pro this PR kept the runtime alive through the 8-client mesh stall, and new clients worked: #2500 (comment) Question: would you be OK merging #2500 once the CI string is fixed, with the J714s rows above plus the Neo row? If so, do you want the backstop first, or as a follow-up? I won't push to your branch. |
…t's commands hang Every watchdog expiry of a published command counts toward the device-wide restart-storm limit, and the eighth within 30 s with no retirement in between stops the GPU runtime until reboot. Settling a command's first expiry already fails its own client's work, or spares it behind another VM's command. Hung commands of one client, one in each of several of its VMs, therefore reach the limit and stop the GPU for every client, although settling each of them had already handled it. Count only an expiry that settling the command did not end. The job's timeout data now records that the watchdog expired it while it was published (PublishedExpiry). The first such expiry no longer counts toward the storm limit, and a later one counts as before. The limit still covers what it was added for: a settlement that never completes, a deferred render timeout whose transaction never returns, and watchdog expiries that end no work. Every expiry counted now was counted before. This code runs on every G17 GPU, G17P included. KUnit covers the published-expiry record: only an expiry after a published one is a repeat. Reported-by: Erik Bourget <erik@ebourget.net> Suggested-by: Erik Bourget <erik@ebourget.net> Signed-off-by: Jeff Queisser <funkzy@users.noreply.github.com>
…t's work restarts the firmware Every firmware restart counts toward the device-wide restart-storm limit before it is classified, and the eighth within 30 s with no retirement in between stops the GPU runtime until reboot. Each restart already fails the work of the VM it finds guilty. On M5 a render that spins restarts the firmware about 50 ms in, and a failed VM's renders that were already queued still run, each restarting the firmware again with the same VM to blame. The leftover forgiveness added for that does not apply on M5, where the driver cannot tie a restart to a command by its stamp. Restarts that one client's work causes can therefore stop the GPU for every client. Count a restart once it is classified, and not when it is charged to the VMs it finds guilty. A VM is charged for the first restart that finds it guilty, which costs its client the device, and then for at most one more restart per command it held at that point: it publishes nothing new, so its queued work bounds the restarts it can still cause. The commands are counted as the restart classifies them. A restart with no guilty VM (no culprit, unknown owners, blame only on work an earlier restart spared) counts as before, as does one that keeps blaming a VM for more than its work, so a firmware that keeps restarting with nobody to blame still stops the GPU after eight. Such a restart is now serviced before the limit stops the GPU, one handshake more than before. Every restart counted now was counted before. Reported-by: Erik Bourget <erik@ebourget.net> Suggested-by: Erik Bourget <erik@ebourget.net> Signed-off-by: Jeff Queisser <funkzy@users.noreply.github.com>
|
Thanks, dj, for the review, and for the M5 Pro (J714s) run: #2500 (comment) is the cross-chip row for this PR. Should we split this into two problems?There are two problems here:
One shared counter can't do both, because an idle GPU and a wedged one look the same to it. The restart row below shows it. I built the backstop you suggested; it counts every published expiry and every serviced restart while nothing retires anywhere, counting an event only when new work was published since the one before, and stops the runtime at the existing limit of eight. One unprivileged process restarting the firmware on fresh VMs, which is the security report's own case, then stops the runtime whenever nobody else is retiring work, as on an idle or locked desktop. So it brings back the device-wide stop this PR removes. The backstop is on a local branch, with the other fixes below; nothing is pushed until you've answered. What we recommend:
Questions for you:
Part (1) is already written locally, with KUnit tests: one file making fresh VMs loses its own device while another file keeps working, and first failures from 64 files at once spend nothing. Either way I'll rerun the rows below on the result, including the restart row on an idle desktop. What a restart costs other clients. A completed restart does drop in-flight work beyond the client that caused it. It fails the in-flight work of whichever client it blames, and on G17P also the other commands in flight at the time. On M5, other clients' in-flight work completes or is resubmitted by our recovery path. Here is that on the J775D: with 8 clients, compute and render mixed, there were four restarts about 100 ms apart. Each one failed one compute client whose 600 ms dispatch runs clean on its own. The render clients' commands were in flight across all four restarts and completed. So a restart loop still costs other clients even without a device stop: each restart pauses everyone's work for the handshake and can fail a bystander's command. Splitting per open file bounds that per open file. Reopening the device for a fresh budget remains a gap, and other drivers' per-file bans have the same gap. i915 keeps its ban score on the open file ( Other points, all in the two commits:
So the two commits now differ from the 3bc86c7 and 6e0a2c4 you tested by more than Ryan's point: the backstop, the rename, the new tests and the doc. Your J714s row is for the previous head. G17P: yes, this reaches G17P, and the commit messages now say so. A Neo row on the new head would be very welcome. Tests on the local head:
J775D (M5 Ultra):
The 8-client rows and the restart row ran through the 27.2 stub, the other two through the 27.0 stub. |
G17P row (Neo, J700 / T8140): no regressionThe two arms differ only by this PR's commits.
Shared setup
Results
Verdict and limitsVerdict: compared with A, E has no new Fail, DeviceLost or GPU recovery on G17P. Limits:
Assisted-by: LLM (Claude:Opus 5.5) |
|
@funkzy, all hardware rows for #2500 are in. It now waits only on your decision on three points:
I won't push to your branch. |
d5d966b to
769f5ea
Compare
|
Thanks, dj, for the G17P row and for collecting the rows.
|
|
Re-review of 769f5ea: No blockers.
Since d5d966b I compared the two versions of the series against the same base (2c35f50). Apart from the test string, the changes do not alter behaviour. Assisted-by: LLM (Claude:Opus 5.5) |
DjDeveloperr
left a comment
There was a problem hiding this comment.
Pre-PR checklist review (§1–§6) on live head 769f5ea7576f0e5328bc9337312838b36dd8e963 (host-tests: pass).
Checklist Summary
- §1 No model gating: Pass — applies across all G17 parts (G17P Neo and M5 family) with zero chip/SoC/board/firmware gates.
- §2 Production-ready, no hacks: Pass — default kernel command line, no new module parameters, no sleeps/retries/lab-only paths.
- §3 Safety: Pass — no hardcoded hardware safety constants; storm bounds remain latched on unattributed/repeated failures.
- §4 Clean-room & attribution: Pass — all 3 commits (
7a055b12f479,0429d625213e,769f5ea7576f) carry human author/committer and humanReported-by/Suggested-by/Signed-off-bytrailers only; PR body includesAssisted-by:. - §5 Evidence: Pass — hardware rows cover all affected G17 tiers: M5 Ultra (J775D, A/B render-spin & compute-hang isolation + render CTS 571/57 unchanged + KUnit), M5 Pro (J714s / G17S, runtime stays alive through 8-client mesh stall and serves new clients + forced compute timeout recovery), and A18 Pro (Neo J700 / G17P, exact A/B on 26,938
dEQP-VK.mesh_shader.ext.*cases + 72/72 focus probe, 0 recoveries/DeviceLost/kernel GPU errors). Verified the live head769f5ea7is behaviorally identical to the tested commits (only theg17-resource-tests.pystring fix,PublishedExpirywrapper with identical semantics, newasahi_g17_job_expiryKUnit test, and doc/comment alignment). - §6 Naming: Pass.
Code Audit (0 blockers)
tools/asahi/g17-resource-tests.py: Updating the substring match to'self.shared.watchdog_timeout('in commit 1 keepshost-testsgreen at every commit while still enforcing theis_published()guard ->release_backlog(&packet)->watchdog_timeout(again)order inruntime/backend.rs.g17/job.rs&g17/runtime.rs:PublishedExpiry::expire()returns the previous flag and latchesself.0 |= published, so unpublished waits never spend the storm budget, the first published expiry is handled per-client, a concurrent publish/watchdog race grants at most one extra free expiry, and any subsequent expiry passesagain = trueintotimeout_limit(). Covered by KUnitasahi_g17_job_expiry.g17/status.rs,g17/recovery.rs,g17/runtime/events.rs:classify()counts active commands per guilty VM viaqueues.classify_work()(verified once per active command across compute and render queues) and charges every guilty VM without short-circuiting (status.charge_restart(held.get()) && charged).VmStatus::charge_restart()safely boundsin_flighttoUNCHARGED - 1under the device mutex, andservice_runtime_recovery()invokesrestart_limit()whenever!result.charged(unattributed, unresolved, collateral-only, or over-budget VMs) with leftover forgiveness bounded byFORGIVE_LIMIT.Documentation/gpu/asahi-g17-recovery-safety.rstaccurately reflects the new accounting and nine-expiration stalled-settlement bound.- Follow-ups noted in thread (per-open-file budget and device-wide wedged-GPU backstop) are agreed upon out of band and tracked separately.
Assisted-by: LLM (Gemini 4 Neon)
0c6c7e7
into
aurora-silicon:integration/aurora
The G17 runtime stops the GPU until reboot once eight watchdog give-ups or firmware restarts land within 30 s with nothing retired in between (
recovery/safety.rs). Each of these already fails the work of the client it belongs to, but each also counts toward that device-wide limit. Several hangs or restarts from one client therefore stop the GPU for all clients. This series counts them against the client instead, and keeps the device-wide stop for failures that no client accounts for.Changes
The first commit updates
tools/asahi/g17-resource-tests.pyto matchwatchdog_timeout()taking an argument, so every commit passes host-tests.Every expiry and restart counted after this change was counted before it.
Testing
On the Mac Studio (M5 Ultra), with and without the series. One client ran the cases below while a second client submitted a small compute dispatch at +0, +30 and +60 s.
asahi_g17_job_expiryfor the expiry record).Rows from other Macs, by dj:
dEQP-VK.mesh_shader.ext.*(26,938 cases) identical with and without the series, with 0 recoveries and 0 DeviceLost. No hang was triggered there, so that row shows the change is inert on healthy runs.Notes for review
Assisted-by: LLM (Claude:Opus 5.5)