Skip to content

drm/asahi: g17: order task-stage tiling behind the last task-stage fragment - #2629

Merged
DjDeveloperr merged 1 commit into
integration/aurorafrom
djdev/g17-task-stage-tiling-order
Nov 21, 2026
Merged

DjDeveloperr merged 1 commit into
integration/aurorafrom
djdev/g17-task-stage-tiling-order

Conversation

@DjDeveloperr

@DjDeveloperr DjDeveloperr commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Problem

On the M5 Pro (J714s, G17S), 8 concurrent clients of dEQP-VK.mesh_shader.ext.misc.many_task_work_groups_x all lose their device, and then the GPU runtime stops for every client until reboot ("GPU runtime stopped, device lost", then ENODEV at device init). It reproduced twice on a clean boot of 7.3.0-m5-intg28d. One client alone passes 3/3 in 87–93 ms.

What the kernel log shows:

  • None of the 8 renders finished.
  • For each stuck render:
    • its tiling kick started (the first four 8–11 ms apart);
    • both rings were consumed (done == read == write);
    • the fragment-end word was never written;
    • no completion stamp ever arrived.
  • Other work kept retiring ("last command retired ~1 s ago"), so the GPU was not wedged as a whole.
  • The watchdog failed the first client after 2 s and the other seven after about 18 s, which is the full deferral budget.
  • The device-wide stop came from a ~32 ms burst of those first give-ups from different clients: 8 or more inside the storm window. That stop is drm/asahi: g17: keep the GPU running for other clients when one client's commands hang or restart the firmware #2500's area.

A code read found no host-side dependency between the clients. The "two VMs defer to each other" lines come from timeout::blocker() skipping a job's own VM's older head, not from a real cross-client wait.

Change

  • The order. A render with an API task stage now waits, on its tiling kick and in class 2, for the fragment kick of the last task-stage render published on another pair.
    • Two clients' task-stage tiling never runs at once.
    • A pair's own earlier kicks get no such wait, because its tiling ring already runs them in order. Single-client overlap is unchanged.
    • The order follows from the work, not the board. Renders without a task stage are not ordered.
    • The J813's existing rule (every render waits for the last fragment kick, 7595d75, macOS 27.0's order there) is unchanged.
  • Release. The order is released when a pair holding a task-stage group fails or is quarantined, or when the firmware drops its groups (the replay path). Later task-stage renders then never wait for a fragment kick that may never complete. The order only sequences tiling and carries no data, so dropping it is always safe. After such a failure, the next task-stage render may overlap one still running elsewhere, once. Task-stage renders already published behind a failed kick stay behind it in the firmware until it completes or their own watchdogs end them.
  • Timeout record. It now prints the four stage words (tiling start/end, fragment start/end), so a stalled render shows which half stopped. It reports up to 8 timeouts per boot instead of 4 (a named constant). This is only a rate limit on that diagnostic log line, like the existing bound it replaces. It doesn't bound the order, the watchdog or any other behaviour; the order is released only on the real signals above.
  • KUnit. asahi_g17_render_cross_pair_order covers the order per board and the dependency choice, including the own-pair exclusion.
  • Host test. tools/asahi/g17-quarantine-drain-tests.py compiles the real quarantine() against a mock Pair. The mock gains a no-op release_task_order_if_held(), because that model doesn't cover the order (KUnit does). All 90 g17 host tests pass locally.
  • G17P. Unaffected: G17P rejects render resources (command.rs), so it never has task-stage renders and its kicks are byte-identical.

macOS reference

  • Ordinary renders. From the J714s macOS 27.0 captures (3,598 TA kicks across seven dumps), no TA kick waits on another pair's 3D kick. The only TA→3D wait is on the kick's own pair (render n−1's 3D), and only in app processes.
  • Task-stage renders. None of those captures contains a task-stage render.
  • Pending capture. A macOS capture of several processes drawing object+mesh pipelines at once is queued in the M5 trace lane. If it shows macOS ordering task-stage work differently, this PR will copy that rule before merge.

Testing

  • M5 Pro (J714s), head 796130c. Row.
    • 5× 8 concurrent clients: 40/40 Pass, 0 DeviceLost, and the runtime was never stopped.
    • With the order switched off (test kernel only), 8/8 DeviceLost at 18.2 s: the bug is back.
    • The stall's stage words show the task-stage tiling hanging.
  • M5 Pro, current head ab5c8a9: pending. It adds the review fixes: release when a pair holding a task-stage group fails, is quarantined or has its groups dropped, and no own-pair wait. The run repeats the 5× 8-client run and adds one client whose task shader never ends, beside 4 healthy task-stage clients, with and without the order.
  • G17P. No row is needed: the change can't reach G17P (see above).

Scope

🤖 Generated with Claude Code

@RyanTheTide

Copy link
Copy Markdown
Contributor

An early read while the rows are pending:

  • Isolation: before this, a task-stage render on a non-J813 Mac depended on nothing outside its own client. Now its tiling waits on whichever client published the last task-stage fragment kick, and the comment on release_task_order() says "work already ordered behind it fails behind it as before". So one client whose task-stage render hangs, buggy or deliberately, takes the next task-stage render of every other client down with it, up to the watchdog's 2 s each. That is the J813's trade-off too, but there it came from macOS's order. Here it is new for G17P and every M5. If the M5 capture shows macOS doesn't order these, a per-client order with a device-wide cap on concurrent task-stage tiling might fix the storm without coupling clients' failures.
  • Release paths: release_task_order() runs only from the failure path (render.rs, at the active check). What if the render that recorded LAST_TASK_FRAGMENT never reaches that path, because its client closes its VM or queue while the render is still published but not yet run? The next task-stage render from another client would then wait for a kick that won't complete, or that matches a reused queue's kick, until its own watchdog fires. Clearing the word when the recording pair is torn down would cover that.
  • Inherited from the J813 code, but now on every Mac: the filter .filter(|d| d.word() != 0 && d.qid() != qids[0]) compares the recorded fragment kick's queue with this render's tiling queue (qids[0]). The recorded word is always a fragment kick (KickDependency::new(qids[1], ...)), so the filter never drops a render's own pair. If the intent is to skip the own pair (already ordered by its own barriers), this should be qids[1]. If waiting on the own pair is intended, the filter can go.
  • CrossPairOrder::of(j813) keeps a board flag selecting the order. Since the task-stage rule now follows the work, could the J813 rule be expressed the same way (a firmware or ABI property of that backend) rather than the board?

The KUnit cases cover the selection and releases(). A case for a release on a different pair leaving the order in place would also pin the compare-exchange path.

Assisted-by: LLM (Claude:Opus 5.5)

@DjDeveloperr

Copy link
Copy Markdown
Contributor Author

M5 Pro (J714s, G17S) row for 796130c: fixed. Task-stage ordering on: 40 of 40 clients pass. Ordering off: 8 of 8 lose their device.

Setup

Results

run clients result per case runtime stopped new clients afterwards
PR order, runs 1–5 8 at once Pass 8/8 every run (40/40), DeviceLost 0 82–151 ms no default driver smoke 2/2; a new mesh client Pass (~85–93 ms)
ordering off (test switch 0) 8 at once DeviceLost 8/8 at 18.2 s (the original failure) — no (#2500) default smoke 2/2; a new mesh client DeviceLost at 18.2 s while the stalled work was still there
forced compute timeout (spin), PR order 1 DEVICE_LOST at 2.0 s — no 2/2 new clients Pass
work_group_ordering, after the stall 1 Pass 1/1 (1.2 s) — no a new mesh client Pass

Before this PR: on the default kernel without it, the same 8-client run lost all 8 devices on two clean boots. The runtime then stopped for every client until reboot.

Where the stalled renders stop

The ordering-off run used this PR's new timeout record:

  • stages [tiling start, tiling end, 0, 0]: the fragment half never started.
  • Tiling end minus tiling start is 2.0 s for the first client and 18.1 s for the rest, which are exactly the watchdog give-up times. So the task-stage tiling hangs while several clients' task-stage tiling runs at once, and its end word is written only when the render is failed.
  • stamps [0x600, 0x600] with both rings consumed, the same picture as the original incident.

Not covered by this row

  • A forced GPU page fault. The injector's write to VA 0x100 completed without a fault under the default soft-fault mode, so that row says nothing either way.

The M5 boot partition was restored byte for byte afterwards, and the GPU was live at the end of the slot.

Still pending before merge: the G17P (Neo) row (mesh_shader.ext.* at 8 workers), the independent review, and the macOS multi-process mesh capture (the order changes to copy macOS if it differs).

…agment

Eight clients running one task-stage (mesh shading with an API task stage) render each
on J714s never finished: every render's tiling kick started (the first four 8 to 11 ms
apart), both rings were consumed (done == read == write), the fragment end word was never
written and no completion stamp ever came, while other work kept retiring. One client
alone finishes the same render in about 90 ms. The watchdog failed the first client after
2 s and the other seven after 18 s, and those give-ups stopped the runtime for every
client until reboot.

On the J813 the tiling kick of every render already waits for the last fragment kick
published on any pair, as macOS 27.0 orders them there (7595d75). Order renders with
an API task stage the same way everywhere else, among themselves: a task-stage render's
tiling kick waits, in class 2, for the fragment kick of the last task-stage render
published on another pair. Two clients' task-stage tiling then never runs at once. A
pair's own earlier kicks need no such wait: its tiling ring already runs them in order.
The order follows from the work, not the board; renders without a task stage are not
ordered, as before, and the J813 order is unchanged.

The order is released when a pair holding a task-stage group fails or is quarantined, or
when the firmware drops its groups, so later task-stage renders do not wait for a
fragment kick that may never complete. It only sequences tiling and carries no data, so
dropping it is always safe; after such a failure the next task-stage render may overlap
one still running elsewhere, once.

The render timeout record now also prints the four M5 stage words (tiling start and end,
fragment start and end), so a stalled render says which half stopped, and it reports up to
eight timeouts per boot instead of four.

Signed-off-by: Diljit Singh <hey@djdev.me>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@DjDeveloperr
DjDeveloperr force-pushed the djdev/g17-task-stage-tiling-order branch from fa5833e to ab5c8a9 Compare November 21, 2026 03:19
@DjDeveloperr

Copy link
Copy Markdown
Contributor Author

M5 Pro (J714s) row for the current head ab5c8a9: clean. 48/48 8-client runs pass, and the order now also contains a hung task shader instead of letting it spread.

Kit: built at slot start on the running default 7.3.0-m5-sship4 (fd2587c) + #2500 (d5d966b) + this PR's exact head ab5c8a9 + the test-only order switch (not in this PR). Native warm build on the M5, packed with the default's own DTB, one boot.

run result runtime stopped new clients afterwards
8× many_task_work_groups_x, 6 runs, this PR's order Pass 8/8 every run (48/48), 0 DeviceLost, 82–96 ms per case no default smoke 2/2, a new mesh client Pass
one client whose task shader never ends + 4 healthy task-stage clients, this PR's order healthy 4/4 Pass (82–104 ms); the hung client DEVICE_LOST; the firmware recovered it ("recovery cause gen 0 … firmware blames queue 13") no default smoke 2/2, a new mesh client Pass
the same, order off healthy 0/4: DeviceLost at 2.1 s and 18.2 s; no firmware recovery no (#2500) default smoke 2/2; a new mesh client DeviceLost at 18.2 s
forced compute timeout (spin) DEVICE_LOST at 2.0 s no 2/2 Pass

The review asked whether a hung task-stage render would chain every later client behind it under this order. On hardware it's the other way round:

  • With the order: the healthy clients are unaffected, because the firmware recovers the hung render.
  • Without the order: the hung task stage takes the concurrent task-stage clients down with it.

Not covered by this row: a non-task render hanging ahead of a task-stage render on the same pair. The release path for that is in quarantine()/fail_after_observation(), but my injector only hangs a task stage.

After the slot: the default sship4 was verified, the ESP was byte-identical to the slot start, and the GPU was live at the end.

@DjDeveloperr

Copy link
Copy Markdown
Contributor Author

Independent review: passed at ab5c8a9 (fresh read-only reviewer agent; three rounds)

  • Round 1 at 796130c, one blocking issue. The order was released only when the failing render held the newest task-stage fragment, so one hung task render could chain later clients behind it. Fixed: release whenever a pair holding a task-stage group fails, is quarantined, or has its groups dropped, plus on announce failure.
  • Round 2. The own-pair exclusion is safe: a pair's tiling ring orders its own kicks, so no two pairs' task-stage tiling kicks overlap. One remaining path (quarantine through a non-task group, or fail_render_vm) is now covered by release_task_order_if_held() at the top of quarantine().
  • Round 3 at 1722932: no blocking issue. I applied its optional items (release on announce failure, doc order). The CI harness g17-quarantine-drain-tests mock got the new no-op method; host-tests are green at ab5c8a9.
  • Confirmed unchanged: the J813 order is byte-for-byte the same, and G17P is unaffected because render resources are rejected there. Locking is by the device mutex.
  • Non-blocking, left as is: when the tiling barrier array is already full, the order is skipped silently (5 slots, existing limit).

Hardware rows: #2629 (comment) (796130c) and #2629 (comment) (ab5c8a9).

Still outstanding, as a follow-up and not a merge gate: a macOS capture of several processes running task-stage renders, to confirm or replace this order with macOS's own (the trace lane is re-running it).

@DjDeveloperr
DjDeveloperr marked this pull request as ready for review November 21, 2026 04:08
@DjDeveloperr
DjDeveloperr merged commit 107047f into integration/aurora Nov 21, 2026
1 check passed
@DjDeveloperr

DjDeveloperr commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor Author

macOS reference: task-stage kick order across processes (J714s, macOS 27.0)

#2629 orders task-stage tiling across pairs. That order was empirical: the M5 A/B backed it, but no macOS capture had a task-stage render. We now have two captures from the M5 Pro. Each ran macOS under the HV tracer with three processes drawing object+mesh pipelines into offscreen targets at the same time, plus a fourth process drawing a mesh-only pipeline as a control.

capture load per frame (one process alone) task-stage TA kicks decoded deps naming another process's kick #2629's wait target named
light 2048 object × 8 mesh threadgroups, ~1-2 ms 158 (54 / 53 / 51 per process) 0 0 of 156
heavy 65536 object × 8 mesh threadgroups, 15-20 ms 6 (2 per process) 0 0 of 5

The order macOS follows

The same order held in both captures and on every kick:

  • Task-stage TA kick: it waits on exactly one thing, the previous kick on its own TA ring (class 1).
  • Its 3D kick: it waits on its own TA kick (class 1) and the previous 3D kick on its own 3D ring (class 3).
  • Across processes: no kick names another process's kick, either TA to 3D or TA to TA, and there are no shared event bits. The J813 order (wait for the last 3D published on any pair) is not used here either: 0 of 158 light, 0 of 6 heavy.
  • Mesh-only renders: without an object stage, the render is a plain TA kick with no task-stage buffer. Its dependencies are the same per-ring ones.
  • Task-stage buffer: each process binds its own 32 MiB, the same size our driver binds per pair. It doesn't grow under heavy load.

macOS doesn't serialise task-stage tiling across processes on the host side

  • Light load: the firmware's own start/end stamps of different processes' task-stage TA kicks overlap in 264 pairs, by up to about 0.7 ms.
    • A task-stage TA that was ready while another process's task-stage TA was running started beside it 233 times.
    • It waited 28 times, with nothing in either kick to explain the wait. That is firmware arbitration, not a dependency.
  • Heavy load: the stamps only cover each kick's last segment, so they can't show whether kicks overlap. The dependencies are the same as under light load.

What this means for #2629

macOS has no cross-process order for task-stage work, so #2629's wait isn't macOS's. Copying macOS literally would mean removing it. #2629's own A/B, though, lost 8/8 clients without the wait. So the wait is hiding some other way our task-stage kicks differ from macOS's, and these captures don't show that difference in the kick dependencies.

They do show one candidate difference in how the TA command is bound. It is the same for task-stage and plain renders, at both loads:

  • macOS: on some TA kicks it binds the whole TA register program in the kick entry (the checkpoint covers every register) and marks the TA command to match. This always happens on the first two kicks of every ring, plus sporadically later.
  • When: every one of the 25 marked kicks started right after another process's TA on the hardware. No rule the host could apply at publication time explains the set.
  • Who writes it: this looks like the firmware marking the command when it starts it, not a choice the macOS driver makes. If so, our firmware may already do the same for our kicks.
  • Our driver: it always binds the shorter form.

(Edit: an earlier version of this comment called this a load-dependent flag on task-stage TA kicks. That was a sampling artifact: the heavy capture only held the first two kicks of each ring.)

Follow-up: a PR that drops the wait to match macOS has to come together with whatever the wait is hiding.

  1. Read back our own TA commands after an interleaved 8-client run with the wait off (test kit only), and see whether the firmware marks them the same way.
  2. If it does, this is not the difference, and the cause lies outside what these captures show.
  3. If it doesn't, copy the marking and re-run the 8-client A/B on the M5 without the wait.

Until then #2629's order stays as merged.

Assisted-by: LLM (Claude:Opus 5.5)

@DjDeveloperr

Copy link
Copy Markdown
Contributor Author

Follow-up TA command read-back on J714s (M5 Pro, T6050, G17S):

Tested on 7.3.0-m5-sship7 (cc145705, which includes this PR's merge 107047f6) plus test-only read-back of the TA command word our driver initializes to zero and a test-only switch to disable the cross-pair wait.

  • With the merged wait on: two 8-client many_task_work_groups_x runs passed 8/8 (183 ms and 167 ms, 0 DeviceLost, 0 runtime stops).
  • Early-boot completed renders: 0/600 plain renders (across two render contexts, including 30 cross-context TA transitions) were marked. The 600-entry per-boot log cap saturated during startup before the two 8-client runs.
  • With the cross-pair wait disabled: 0/8 passed (8 DeviceLost, 18.2 s, GPU runtime stopped). On 5 of 5 timed-out render commands, that TA command word came back marked (nonzero) even though our driver wrote zero. All 5 stalled renders had both tiling start and end timestamps written and neither fragment timestamp written before the watchdog timed out and stopped the runtime.

Conclusion: the GPU firmware writes that TA command marking itself on our Linux commands too. That closes it as a candidate driver difference to copy; this PR's cross-pair wait stays in place while the underlying cause of the stalled fragment start under concurrent cross-client task-stage tiling is investigated elsewhere.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants