Skip to content

drm/asahi: g17: keep the GPU running for other clients when one client's commands hang or restart the firmware - #2500

Merged
RyanTheTide merged 3 commits into
aurora-silicon:integration/aurorafrom
funkzy:pr/g17-storm-own-budget
Nov 22, 2026
Merged

RyanTheTide merged 3 commits into
aurora-silicon:integration/aurorafrom
funkzy:pr/g17-storm-own-budget

Conversation

@funkzy

@funkzy funkzy commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

The G17 runtime stops the GPU until reboot once eight watchdog give-ups or firmware restarts land within 30 s with nothing retired in between (recovery/safety.rs). Each of these already fails the work of the client it belongs to, but each also counts toward that device-wide limit. Several hangs or restarts from one client therefore stop the GPU for all clients. This series counts them against the client instead, and keeps the device-wide stop for failures that no client accounts for.

Changes

The first commit updates tools/asahi/g17-resource-tests.py to match watchdog_timeout() taking an argument, so every commit passes host-tests.

  1. Watchdog give-ups. A command's first give-up no longer counts toward the storm limit: settling it already fails its client's work, or spares it behind another VM's command. An expiry after that, one the settlement did not end, counts as before.
  2. Firmware restarts. A restart is counted once it is classified, and not when it is charged to the VMs it finds guilty.
    • A VM is charged for the first restart that blames it, which costs its client the device.
    • After that it is charged for at most one more per command it held then: a failed VM publishes nothing new, but its already-queued work still runs. On M5 that queued work can restart the firmware again, and the existing leftover forgiveness does not apply there, because the driver cannot tie a restart to a command by its stamp.
    • Restarts with no guilty VM, or that keep blaming a VM beyond the work it held, count as before. A firmware that keeps restarting with no one to blame still stops the GPU after eight.

Every expiry and restart counted after this change was counted before it.

Testing

On the Mac Studio (M5 Ultra), with and without the series. One client ran the cases below while a second client submitted a small compute dispatch at +0, +30 and +60 s.

Case Without the series With it
12 render spins, each from a new VM the eighth restart stopped the runtime; the second client lost its device each spin's client lost its device; the second client completed all three dispatches
10 VMs, each with a hung compute command (desktop running, and desktop stopped) — all 10 lost their device within 18.2 s; the runtime kept running; the second client completed
One VM with 12 render spins queued — the same
  • A single hang of each engine still recovers, and the next client runs, before and after these cases.
  • Render CTS subset (628 cases): 571 Pass and 57 NotSupported, unchanged.
  • KUnit: the existing storm tests pass, and each change adds one (asahi_g17_job_expiry for the expiry record).
  • Host tests: no result differs between base and head.

Rows from other Macs, by dj:

  • M5 Pro (J714s): the runtime kept running through the 8-client stall, and new clients worked.
  • A18 Pro (Neo, G17P): dEQP-VK.mesh_shader.ext.* (26,938 cases) identical with and without the series, with 0 recoveries and 0 DeviceLost. No hang was triggered there, so that row shows the change is inert on healthy runs.

Notes for review

  • Restarts from many VMs. A client that keeps faulting from new VMs now only costs itself, but each of those restarts still pauses the other clients briefly. A follow-up charges restarts and expiries to the open file, however many VMs it creates.
  • A GPU that runs nothing for anyone. It no longer reaches the device-wide stop through first expiries alone. Each client loses its device in turn, until a repeat expiry or an unattributed restart reaches the limit. A second follow-up stops the GPU when recovery itself fails or times out with nothing retiring anywhere, treating a completed restart handshake as progress.

Assisted-by: LLM (Claude:Opus 5.5)

@DjDeveloperr

DjDeveloperr commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Update 11-20 10:3x: superseded by the clean-boot rerun in the next comment. The 8-client failure reproduces on a fresh boot and ends with "GPU runtime stopped, device lost".

Correction (11-20 ~10:00 EST): the 8-client row I posted earlier is not valid evidence, so I've removed it. That run came right after three firmware recoveries from another case, mesh_shader.ext.misc.work_group_ordering, in the same boot. The third recovery logged "a GPU recovery dropped groups that cannot run again in order; failing 2 groups". After that, the GPU looks wedged: the 8 clients made no progress, and the next batch a minute later failed at device init on every process (DRM_IOCTL_ASAHI_GET_PARAMS failed: No such device). Its watchdog lines describe a stuck GPU, not the sharing path.

What remains valid, on G17S (M5 Pro, J714s), Mesa 463da7d candidate DSO, #2500 not applied:

case kernel alone (1 client) shared
dEQP-VK.mesh_shader.ext.misc.many_task_work_groups_x 7.3.0-m5-intg28d Pass 3/3, 87–93 ms — (the run was invalid, see above)
same 7.3.0-m5-intg26 — DeviceLost once, in a batch of 8 CTS workers running different cases, with no earlier GPU recovery in that boot

The intg26 kernel line was "G17: watchdog gave up on a compute command …: allowance 2000 ms, at the head of its queue for 2000 ms; not long work; no sharing renewals on this GPU", followed by ETIMEDOUT and a lost device. A clean 8-client rerun (shared first, on a fresh boot, without work_group_ordering) is queued for our next M5 hold, and I'll post the result here.

@DjDeveloperr

DjDeveloperr commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Clean-boot rerun on G17S. This supersedes my correction above: the 8-client failure is real, and it takes the whole GPU down.

Setup: M5 Pro (J714s), kernel 7.3.0-m5-intg28d without #2500, Mesa 463da7d candidate DSO. Fresh boot: uptime 181 s, and 0 GPU-recovery or watchdog lines in the boot before the run.

dEQP-VK.mesh_shader.ext.misc.many_task_work_groups_x result
1 client (earlier boot) Pass 3/3, 87–93 ms
8 clients at once (clean boot) DeviceLost 8/8: 1 at 2.06 s, 7 at 18.2 s
any new client afterwards DRM_IOCTL_ASAHI_GET_PARAMS failed: No such device until reboot

Kernel, in order (our Linux log, trimmed):

  1. Two clients defer to each other. Compute watchdog owner=5 vm=19 … renewals=0 excused=0 decision=Defer behind earlier work of another VM: blocker_vm=23 … and Render watchdog owner=6 vm=23 … decision=Defer behind earlier work of another VM: blocker_vm=19 …. The other 6 also defer behind vm 23.
  2. "G17: watchdog gave up on a render command of owner 5: allowance 2000 ms, at the head of its queue for 2000 ms; not long work; no sharing renewals on this GPU …", then ETIMEDOUT and a lost device for that client.
  3. About 16 s later: renewals=8 … decision=Spare behind earlier work of another VM … blocker_in_allowance=0 and decision=Own … blocker_failed=1. Then "gave up … at the head of its queue for 18127–18132 ms; not long work; no sharing renewals on this GPU … deferred 8 polls behind older or pending work", and ETIMEDOUT for each remaining client.
  4. aurora 2300000000.gpu: GPU runtime stopped, device lost. The last clients then get EIO (g17/completion.rs:739/741). From then on every new process gets ENODEV at device init.

Second candidate (11-20 10:4x): linux#2579 (RyanTheTide, head be4ddeb, "free retired queue structures after firmware poweroff"). It fixes a G13S firmware crash under context churn: context data is still freed at the DestroyContext ack, but queue structures are now kept until the next firmware PowerOff. It has no chip gate, so it reaches G17S, and on the M1 Pro crashes went from 9/47 to 0. The 8-client run above creates and destroys contexts while one client hangs, so both PRs are in scope. Neither is built here; this row is on the default kernel.

A ~90 ms dispatch run by 8 clients ends with the whole GPU runtime stopped. The first client is failed "as its own" with no sharing renewal. Two VMs block on each other, and the others time out behind them. The runtime then stops for everyone. That looks like exactly the case #2500 targets. We have not built #2500. If a G17S row on its head would help, say so and we'll queue one.

The earlier q-003 run, the one I called invalid above, followed the same path and ended with the same ENODEV. The work_group_ordering recoveries before it were probably not what killed the GPU.

@funkzy

funkzy commented Nov 20, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the clean-boot rerun, that sequence is very useful. Here is how it lines up with #2500, plus what we see on the J775D (M5 Ultra).

Why the runtime stopped. Your 8 give-ups (1 at 2.06 s, 7 at 18.2 s, nothing retired in between) are exactly the storm limit: 8 watchdog expiries within 30 s with no progress. That is the path #2500 changes.

What #2500 covers in your sequence. Each of your 8 give-ups is the first one for its command, and #2500 no longer counts those toward the limit. So with #2500 the runtime should keep running, and new processes should not get ENODEV. Your timing (2 s, then 2 s + 8 deferral polls = 18.2 s) matches the "10 VMs, each with a hung compute command" row in the PR body, where the runtime kept running.

What it does not cover. #2500 does not change any deferral or give-up decision, so the 8 clients would still lose their devices.

The two VMs deferring to each other. This is not a cycle the watchdog can get stuck in. Both engines take their publication number from one device-wide counter, and a command only defers behind another VM's command with a lower number. The oldest unfinished command therefore never defers and is decided on its own. vm 19 and vm 23 each had a compute command and a render command in flight, and each VM had one command older than one of the other's. vm 19's render was the oldest, so it was the first give-up at 2 s.

#2579. As far as I can tell it does not reach G17S. It changes channel.rs, gpu.rs and workqueue.rs of the G13-G16 path, and the G17 driver does not use that manager.

J775D rows (integ-c41 = integration e0ee46a + aura-pack4, 27.2 stub; 8 processes, one VkDevice each, all submitting one command at the same moment; each command alone takes the time shown):

case without #2500 #2500 #2500 + sharing fix (below)
8 x 90 ms compute 8/8 pass, 0.67 s 8/8, 0.69 s 8/8, 0.67 s
8 x 90 ms, compute -> render -> compute in one submission 8/8, 0.70 s 8/8 8/8
8 x 90 ms, 4 compute + 4 render clients 8/8, 0.60 s 8/8 8/8
2, 4 x 30/90/300 ms compute, 8 x 30 ms all pass
8 x 300 ms compute 1 lost at 2.0 s, 7 pass by 2.2 s 1 lost at 2.0 s 8/8, 2.19 s
8 x 600 ms compute 2 lost (2.0 s, 4.0 s) 2 lost 8/8, 4.37 s
8 x 1500 ms compute 2 lost (2.0 s, 10.1 s) 8/8, 10.9 s
8 x 300 ms render 8/8, 1.71 s 8/8, 1.72 s
one lone compute spin lost at 2.0 s lost at 2.0 s

No runtime stop on any kernel. Work retires between the give-ups here, which starts a new storm window.

So your 90 ms case does not reproduce on the J775D, but the same first step does once the shared work adds up to more than about 2 s. On M5 a command that has not finished at its first watchdog expiry (2 s after it reaches the head of its queue) fails then, unless older work of another VM holds it back. Clients running side by side retire nothing until they end, so the oldest one is failed while all of them are still making progress.

The sharing fix is a local commit, not pushed yet. On M5 it renews a command, up to the existing 30 s bound, while another VM has unfinished work running beside it. A lone spin still fails at 2 s. The cost: hangs beside each other now share the 30 s bound. 10 compute spins submitted together lose their devices from 30 s on, instead of from 2 s.

The difference from your run. Here, after the first give-up, the other clients finish 0.1-0.2 s later. On G17S your other 7 did not finish in the next 16 s, so something on G17S kept them from running after the first give-up, which was a render there and a compute here. The sharing fix would not help with that: it would only move the first give-up from 2 s to about 30 s.

One more thing we saw, not yet looked into: 4 x 600 ms render clients next to 4 x 600 ms compute clients gave 4 firmware restarts, and the 4 compute clients lost their devices.

A G17S row would help, yes, in this order:

  1. drm/asahi: g17: keep the GPU running for other clients when one client's commands hang or restart the firmware #2500's head with your 8-client case. We expect 8/8 lost, no "GPU runtime stopped", and a new client afterwards working. That is the part drm/asahi: g17: keep the GPU running for other clients when one client's commands hang or restart the firmware #2500 claims.
  2. If you have room, the same with the sharing fix on top (we can push it as a branch). If all 8 then pass, your first give-up was a sharing timeout. If they still stop at about 30 s, the GPU stopped running them, and the kernel lines around the first give-up would be the thing to look at.

How long does one many_task_work_groups_x submission take on the GPU, as opposed to the whole case? That would tell us how far your 8 clients are from the 2 s line.

@RyanTheTide

Copy link
Copy Markdown
Contributor

host-tests fails in tools/asahi/g17-resource-tests.py line 318 (watchdog_publication_guard_tests): body.index('self.release_backlog(&packet);') raises ValueError: substring not found. The test pins the order guard -> release_backlog(&packet) -> watchdog_timeout(), and this head no longer has that call spelled that way. Either keep the call or update the test to the new ordering it should guard.

Assisted-by: LLM (Claude:Opus 5.5)

…en watchdog_timeout() takes an argument

watchdog_publication_guard_tests() checks that timed_out() returns
early for a packet still in publication, then releases the packet's
backlog, and only then calls the shared watchdog_timeout() that counts
toward the timeout storm limit. It finds the last of these by the exact
text of the call, "self.shared.watchdog_timeout()", so it fails with
"substring not found" once the call takes an argument, although the
order it checks still holds.

Match the call by "self.shared.watchdog_timeout(" instead. The test
still requires the publication guard, release_backlog(&packet) and
watchdog_timeout() in that order in timed_out(), and passes on the
current code as on a call with arguments.

Signed-off-by: Jeff Queisser <funkzy@users.noreply.github.com>
@DjDeveloperr

Copy link
Copy Markdown
Contributor

#2500 on the M5 Pro (J714s, G17S): it keeps the runtime alive through the 8-client stall. Evidence only; I haven't changed this PR.

Kernel: 7.3.0-m5-sship2 (d1d26ab) + this PR's two commits (3bc86c7, 6e0a2c4) + #2629 (796130c). The A/B also uses a test-only debugfs switch that turns #2629's ordering off, so the original stall happens again on top of #2500. Native warm build on the M5, one boot.

run result GPU runtime new clients
8× many_task_work_groups_x, #2629's ordering off DeviceLost 8/8 at 18.2 s (the stall from my earlier comments) not stopped default driver smoke 2/2
forced compute timeout (shader spin) DEVICE_LOST at 2.0 s not stopped 2/2 Pass
forced compute timeout, right after the stall DEVICE_LOST at 18.1 s (deferred behind the stalled work) not stopped 2/2 Pass
work_group_ordering, then a new mesh client Pass, Pass not stopped —

Without #2500, the same stall stopped the runtime with "GPU runtime stopped, device lost" for every client until reboot. That happened twice on clean boots of intg28d.

In those runs the device-wide stop came from one ~32 ms burst of first give-ups by 8 different clients. That is the case this PR stops counting.

With #2500 the stalled clients lose only their own devices, the GPU keeps serving others, and the mesh path came back once the stalled clients exited. #2629 removes this particular stall; #2500 is what keeps a stall like it from taking the GPU down for everyone.

@DjDeveloperr

Copy link
Copy Markdown
Contributor

Independent review of #2500 at d5d966b (a fresh reviewer agent from the M5 GPU-recovery lane, read-only; summary below). funkzy, it's your PR, so I haven't changed anything; one question at the end.

CI. host-tests fails only in g17-resource-tests: tools/asahi/g17-resource-tests.py:318 still looks for the text self.shared.watchdog_timeout() in timed_out, which this PR changed to watchdog_timeout(again). Matching 'self.shared.watchdog_timeout(again)' there should turn it green. The other 450 host tests passed. Because the failure aborted the rest of that script, its timeout_render/timeout_compute checks didn't run.

Design point the reviewer treats as blocking unless you accept it: the device-wide stop no longer has a bound for a GPU that makes no progress for anyone, if each event lands on a new command or VM.

  • A command's first published expiry is free, and settling it ends the job. So a wedged GPU with a client that keeps resubmitting fresh commands (a respawning compositor, say) never reaches the limit. Before this PR that stopped after about 8 expiries in 16 s.
  • Restarts are charged per VM, so every VM_CREATE gets 1 + held free restarts. If attribution keeps blaming whatever queue is running, restarts on fresh VMs never count, and each handshake can poll for up to 2 × 10 s under the device mutex.
  • That contradicts recovery/safety.rs's "Fail closed … even across client loss" and the doc's "limits host recovery load".
  • The reviewer's suggestion: keep a second, device-wide backstop that notes every published expiry and every serviced restart, charged or not, with note_unless_progress and a higher limit, so a GPU retiring nothing for anyone still stops. Whether that limit should come from a measured signal rather than a fixed number, given the no-magic-constants rule, is your call.

Non-blocking

  • again semantics: correct, including the publish/watchdog race (at most one extra free expiry) and replay.
  • charge_restart: sound. No overflow; every guilty VM is charged; it relies on the device mutex, which holds.
  • Missing tests: 1ba6f37 adds no KUnit test for the job.rs flag, though the description says each change adds one.
  • G17P scope: the change also reaches G17P, where leftover forgiveness now applies only after a VM's budget is spent. A Neo row is queued in the Neo lane (arm E).
  • Naming: Timeout::gave_up shares its name with the Backend::gave_up method.
  • Stale text: a recovery.rs test comment still says "A first failure counts toward the storm". Doc lines 56–57 and 75–76, and "eight two-second expirations" (now nine for a stalled settlement), are stale.

Hardware (already posted above): on the M5 Pro this PR kept the runtime alive through the 8-client mesh stall, and new clients worked: #2500 (comment)

Question: would you be OK merging #2500 once the CI string is fixed, with the J714s rows above plus the Neo row? If so, do you want the backstop first, or as a follow-up? I won't push to your branch.

…t's commands hang

Every watchdog expiry of a published command counts toward the
device-wide restart-storm limit, and the eighth within 30 s with no
retirement in between stops the GPU runtime until reboot. Settling a
command's first expiry already fails its own client's work, or spares
it behind another VM's command. Hung commands of one client, one in
each of several of its VMs, therefore reach the limit and stop the GPU
for every client, although settling each of them had already handled
it.

Count only an expiry that settling the command did not end. The job's
timeout data now records that the watchdog expired it while it was
published (PublishedExpiry). The first such expiry no longer counts
toward the storm limit, and a later one counts as before. The limit
still covers what it was added for: a settlement that never completes,
a deferred render timeout whose transaction never returns, and watchdog
expiries that end no work. Every expiry counted now was counted before.

This code runs on every G17 GPU, G17P included. KUnit covers the
published-expiry record: only an expiry after a published one is a
repeat.

Reported-by: Erik Bourget <erik@ebourget.net>
Suggested-by: Erik Bourget <erik@ebourget.net>
Signed-off-by: Jeff Queisser <funkzy@users.noreply.github.com>
…t's work restarts the firmware

Every firmware restart counts toward the device-wide restart-storm
limit before it is classified, and the eighth within 30 s with no
retirement in between stops the GPU runtime until reboot. Each restart
already fails the work of the VM it finds guilty. On M5 a render that
spins restarts the firmware about 50 ms in, and a failed VM's renders
that were already queued still run, each restarting the firmware again
with the same VM to blame. The leftover forgiveness added for that does
not apply on M5, where the driver cannot tie a restart to a command by
its stamp. Restarts that one client's work causes can therefore stop
the GPU for every client.

Count a restart once it is classified, and not when it is charged to
the VMs it finds guilty. A VM is charged for the first restart that
finds it guilty, which costs its client the device, and then for at
most one more restart per command it held at that point: it publishes
nothing new, so its queued work bounds the restarts it can still cause.
The commands are counted as the restart classifies them.

A restart with no guilty VM (no culprit, unknown owners, blame only on
work an earlier restart spared) counts as before, as does one that
keeps blaming a VM for more than its work, so a firmware that keeps
restarting with nobody to blame still stops the GPU after eight. Such a
restart is now serviced before the limit stops the GPU, one handshake
more than before. Every restart counted now was counted before.

Reported-by: Erik Bourget <erik@ebourget.net>
Suggested-by: Erik Bourget <erik@ebourget.net>
Signed-off-by: Jeff Queisser <funkzy@users.noreply.github.com>
@funkzy

funkzy commented Nov 21, 2026

Copy link
Copy Markdown
Contributor Author

Thanks, dj, for the review, and for the M5 Pro (J714s) run: #2500 (comment) is the cross-chip row for this PR.

Should we split this into two problems?

There are two problems here:

  1. One client causing failures shouldn't stop the GPU for everyone. That is the security report, and what this PR fixes.
  2. A GPU that really makes no progress should still stop. That is your review point.

One shared counter can't do both, because an idle GPU and a wedged one look the same to it. The restart row below shows it. I built the backstop you suggested; it counts every published expiry and every serviced restart while nothing retires anywhere, counting an event only when new work was published since the one before, and stops the runtime at the existing limit of eight. One unprivileged process restarting the firmware on fresh VMs, which is the security report's own case, then stops the runtime whenever nobody else is retiring work, as on an idle or locked desktop. So it brings back the device-wide stop this PR removes. The backstop is on a local branch, with the other fixes below; nothing is pushed until you've answered.

What we recommend:

  1. Per open file. Restarts and expiries are charged to a budget on the client's open file, not on each VM. New VMs don't refill it. When it is spent, that file loses its device and nobody else is affected.
  2. Judged by whether recovery succeeds. A completed restart handshake counts as progress, since a firmware that comes back from a restart is not wedged. Only expiries and restarts that fail or time out, with nothing retiring anywhere, count toward the device stop.

Questions for you:

  1. Do you want them split this way: (1) per open file, and (2) judged by whether recovery succeeds?
  2. Should (2) go in this PR, or in your recovery lane, with drm/asahi: g17: keep the GPU running for other clients when one client's commands hang or restart the firmware #2500 carrying only (1)?

Part (1) is already written locally, with KUnit tests: one file making fresh VMs loses its own device while another file keeps working, and first failures from 64 files at once spend nothing. Either way I'll rerun the rows below on the result, including the restart row on an idle desktop.

What a restart costs other clients. A completed restart does drop in-flight work beyond the client that caused it. It fails the in-flight work of whichever client it blames, and on G17P also the other commands in flight at the time. On M5, other clients' in-flight work completes or is resubmitted by our recovery path. Here is that on the J775D: with 8 clients, compute and render mixed, there were four restarts about 100 ms apart. Each one failed one compute client whose 600 ms dispatch runs clean on its own. The render clients' commands were in flight across all four restarts and completed. So a restart loop still costs other clients even without a device stop: each restart pauses everyone's work for the handshake and can fail a bystander's command. Splitting per open file bounds that per open file. Reopening the device for a fresh budget remains a gap, and other drivers' per-file bans have the same gap. i915 keeps its ban score on the open file (drm_i915_file_private::ban_score, raised in client_mark_guilty()), refuses context creation from a banned file with -EIO, and a new open starts at zero.

Other points, all in the two commits:

  • Timeout::gave_up is now published_expiry, a small PublishedExpiry type.
  • job.rs gets its KUnit test (asahi_g17_job_expiry).
  • The stale recovery.rs comment is fixed, and so is the doc: lines 56-57 and 75-76, and the window text now says nine two-second expirations for a stalled settlement, its first and then the eight that count.
  • The CI string: a commit of ours ahead of the two matches the call by self.shared.watchdog_timeout(. The test still requires the publication guard, release_backlog(&packet) and watchdog_timeout() in that order in timed_out(). It goes first, so every commit in the PR passes it.

So the two commits now differ from the 3bc86c7 and 6e0a2c4 you tested by more than Ryan's point: the backstop, the rename, the new tests and the doc. Your J714s row is for the previous head.

G17P: yes, this reaches G17P, and the commit messages now say so. A Neo row on the new head would be very welcome.

Tests on the local head:

  • run-host-tests.py, base against head: 427 of 427 pass on both, and no result differs.
  • KUnit: 344 of 346 pass (339 of 341 on the base); the two skips are the same on both.

J775D (M5 Ultra):

case result
8 clients, one ~300 ms compute command each at once 1 lost its own device, runtime alive
the same at ~600 ms 2 lost their own device, runtime alive
8 clients, compute and render mixed 4 lost their own device, runtime alive, a new client works
ten VMs each hanging a compute command at once, plus a client submitting every 30 s the ten lost their device in 18.2 s; runtime alive; the other client kept working
one process, a render that restarts the firmware on each of twelve fresh VMs in turn, desktop idle the backstop stopped the runtime at the ninth restart, 0.49 s after the first
one process, a compute spin on each of 24 fresh VMs in turn, desktop stopped each spin waited about 18 s behind the previous one, so the 76 s run reached six expiries and no stop; at that pace the stop comes at the ninth, after about 165 s (not measured)

The 8-client rows and the restart row ran through the 27.2 stub, the other two through the 27.0 stub.

@DjDeveloperr

Copy link
Copy Markdown
Contributor

G17P row (Neo, J700 / T8140): no regression

The two arms differ only by this PR's commits.

A (baseline) E (this PR)
Kernel linux e0ee46a1 (the kernel-aurora 7.3.0-43 source) e0ee46a1 + 1ba6f371 + d5d966b9, both picked cleanly onto e0ee46a1
Release 7.3.0.n2629a 7.3.0.n2500e
aurora.ko sha256 8164fea50baf5887 4476552d6af11d65

Shared setup

  • Config: the kernel-aurora-core-7.3.0-43 RPM config plus DEBUG_INFO_NONE, run through olddefconfig. Both arms use the same .config. Built with clang 23.1.5 and rustc 1.100.0.
  • Mesa: Borealis from the signed mesa-aurora 26.3.0-35 RPMs, in a private prefix. libvulkan_asahi.so sha256 is 767f3e176e4c03df, driverInfo git-15fe5010c1. Same for both arms.
  • Run: one-shot boot of each kit. deqp-vk (sha256 d3ef839af652df22) ran dEQP-VK.mesh_shader.ext.*, 26,938 cases, at 8 parallel workers. Then three rounds of a focus probe: 8 deqp-vk processes started together, each running misc.many_task_work_groups_{x,y,z}.

Results

Arm Pass Fail Crash Timeout ResourceError NotSupported Batch wall time
A 26,567 0 0 0 1 370 46 s
E 26,567 0 0 0 1 370 46 s
  • ResourceError: dEQP-VK.mesh_shader.ext.misc.work_group_ordering in both arms. It is the only non-Pass case in either arm. At 2 workers on this machine it passes (0.33 s), and at 8 workers it is a ResourceError in A as well, so it does not come from this change.
  • Focus probe: 72/72 Pass in both arms, 0.8–0.9 s per round.
  • Kernel log (journalctl -k -b of each kit boot): 0 GPU recoveries, 0 device lost, 0 client failures, 0 GPU faults and 0 GPU runtime stopped lines, in both arms.
  • GPU liveness: checked before and after each batch with vulkaninfo (system and candidate driver) plus one fresh-process compute dispatch. It passed in every check.

Verdict and limits

Verdict: compared with A, E has no new Fail, DeviceLost or GPU recovery on G17P.

Limits:

  • No command hung and no firmware restart happened in either arm, so the recovery path this PR changes was never triggered. The row shows only that the change is inert on healthy G17P runs.
  • There is one batch per arm.
  • No forced-hang check was run, because there is no hang injector for this machine.

Assisted-by: LLM (Claude:Opus 5.5)

@DjDeveloperr

Copy link
Copy Markdown
Contributor

@funkzy, all hardware rows for #2500 are in. It now waits only on your decision on three points:

  1. CI: tools/asahi/g17-resource-tests.py:318 still matches self.shared.watchdog_timeout(). Change it to self.shared.watchdog_timeout(again) and host-tests should go green; the other 450 tests already pass.
  2. Consent: can it merge with the J714s rows (runtime kept alive through the 8-client stall, new clients worked) and the G17P row above (26,938 mesh cases identical, 0 recoveries or DeviceLost)?
  3. Backstop: do you want the review's backstop for a GPU wedged for every client (a device-wide stop when nothing retires for anyone) in this PR, or as a follow-up?

I won't push to your branch.

@funkzy
funkzy force-pushed the pr/g17-storm-own-budget branch from d5d966b to 769f5ea Compare November 21, 2026 08:36
@funkzy

funkzy commented Nov 21, 2026

Copy link
Copy Markdown
Contributor Author

Thanks, dj, for the G17P row and for collecting the rows.

  1. CI: pushed. The test fix is now the first commit, so every commit in the PR passes host-tests. The two commits after it also carry the review points from the earlier reply: Timeout::gave_up becomes PublishedExpiry, job.rs gets its KUnit test (asahi_g17_job_expiry), and the recovery.rs comment and the doc are fixed. None of that changes behaviour: the expiry logic reads the published state before the call and passes the earlier value as again, exactly as before. So the J714s and G17P rows apply to the new head.

  2. Consent: yes, it can merge with those rows.

  3. Backstop: as a follow-up, please. We'll send two:

    • the per-open-file budget, so a file's restarts and expiries are charged to that file however many VMs it creates (written, with KUnit tests);
    • the device stop judged by whether recovery succeeds, which is the review's backstop: a completed restart handshake counts as progress, and only failed or timed-out recovery with nothing retiring anywhere counts toward stopping the GPU.

    If you'd rather the second one sit in your recovery lane, that works for us too.

@RyanTheTide

Copy link
Copy Markdown
Contributor

Re-review of 769f5ea: No blockers.

  1. Resolved: the host-tests failure in watchdog_publication_guard_tests. tools/asahi/g17-resource-tests.py:318 now matches self.shared.watchdog_timeout(, and the order it guards still holds at the head: the publication guard at drivers/gpu/drm/asahi/g17/runtime/backend.rs:324, self.release_backlog(&packet); at :327, self.shared.watchdog_timeout(again) at :328. The timeout_render/timeout_compute checks that the earlier failure skipped also pass against render/settle.rs and compute_control.rs at this head. The fix is the first commit, so each commit passes on its own.

Since d5d966b I compared the two versions of the series against the same base (2c35f50). Apart from the test string, the changes do not alter behaviour. PublishedExpiry::expire() (drivers/gpu/drm/asahi/g17/job.rs) returns the old flag and then ORs in published, which is the same order as the earlier inline gave_up code. So a publication that races the call still gets one free expiry of its own, and the J714s and Neo G17P rows taken on 1ba6f37/d5d966b9 still apply. The new asahi_g17_job_expiry KUnit test covers expiries before publication, the first published expiry, later ones, and the case where a later unpublished read leaves the flag set. The recovery.rs test comment and the doc text (nine expirations for a stalled settlement) now match the code. Commits carry only the author's Signed-off-by.

Assisted-by: LLM (Claude:Opus 5.5)

@DjDeveloperr DjDeveloperr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pre-PR checklist review (§1–§6) on live head 769f5ea7576f0e5328bc9337312838b36dd8e963 (host-tests: pass).

Checklist Summary

  • §1 No model gating: Pass — applies across all G17 parts (G17P Neo and M5 family) with zero chip/SoC/board/firmware gates.
  • §2 Production-ready, no hacks: Pass — default kernel command line, no new module parameters, no sleeps/retries/lab-only paths.
  • §3 Safety: Pass — no hardcoded hardware safety constants; storm bounds remain latched on unattributed/repeated failures.
  • §4 Clean-room & attribution: Pass — all 3 commits (7a055b12f479, 0429d625213e, 769f5ea7576f) carry human author/committer and human Reported-by/Suggested-by/Signed-off-by trailers only; PR body includes Assisted-by:.
  • §5 Evidence: Pass — hardware rows cover all affected G17 tiers: M5 Ultra (J775D, A/B render-spin & compute-hang isolation + render CTS 571/57 unchanged + KUnit), M5 Pro (J714s / G17S, runtime stays alive through 8-client mesh stall and serves new clients + forced compute timeout recovery), and A18 Pro (Neo J700 / G17P, exact A/B on 26,938 dEQP-VK.mesh_shader.ext.* cases + 72/72 focus probe, 0 recoveries/DeviceLost/kernel GPU errors). Verified the live head 769f5ea7 is behaviorally identical to the tested commits (only the g17-resource-tests.py string fix, PublishedExpiry wrapper with identical semantics, new asahi_g17_job_expiry KUnit test, and doc/comment alignment).
  • §6 Naming: Pass.

Code Audit (0 blockers)

  • tools/asahi/g17-resource-tests.py: Updating the substring match to 'self.shared.watchdog_timeout(' in commit 1 keeps host-tests green at every commit while still enforcing the is_published() guard -> release_backlog(&packet) -> watchdog_timeout(again) order in runtime/backend.rs.
  • g17/job.rs & g17/runtime.rs: PublishedExpiry::expire() returns the previous flag and latches self.0 |= published, so unpublished waits never spend the storm budget, the first published expiry is handled per-client, a concurrent publish/watchdog race grants at most one extra free expiry, and any subsequent expiry passes again = true into timeout_limit(). Covered by KUnit asahi_g17_job_expiry.
  • g17/status.rs, g17/recovery.rs, g17/runtime/events.rs: classify() counts active commands per guilty VM via queues.classify_work() (verified once per active command across compute and render queues) and charges every guilty VM without short-circuiting (status.charge_restart(held.get()) && charged). VmStatus::charge_restart() safely bounds in_flight to UNCHARGED - 1 under the device mutex, and service_runtime_recovery() invokes restart_limit() whenever !result.charged (unattributed, unresolved, collateral-only, or over-budget VMs) with leftover forgiveness bounded by FORGIVE_LIMIT.
  • Documentation/gpu/asahi-g17-recovery-safety.rst accurately reflects the new accounting and nine-expiration stalled-settlement bound.
  • Follow-ups noted in thread (per-open-file budget and device-wide wedged-GPU backstop) are agreed upon out of band and tracked separately.

Assisted-by: LLM (Gemini 4 Neon)

@RyanTheTide
RyanTheTide merged commit 0c6c7e7 into aurora-silicon:integration/aurora Nov 22, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants