Skip to content

drm/asahi: g17: one zeroed clustering-metadata copy per in-flight render - #2170

Draft
AxxzyWasTaken wants to merge 5 commits into
aurora-silicon:integration/aurorafrom
AxxzyWasTaken:pr/t6041-cluster-meta-per-render
Draft

AxxzyWasTaken wants to merge 5 commits into
aurora-silicon:integration/aurorafrom
AxxzyWasTaken:pr/t6041-cluster-meta-per-render

Conversation

@AxxzyWasTaken

@AxxzyWasTaken AxxzyWasTaken commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Depends on #2021. This branch sits on #2021's branch (1e62e2c25c38), so GitHub shows #2021's four commits first; only the last commit belongs to this PR. I'll rebase it once #2021 lands.

A render target keeps one TPC object for all of its in-flight renders. The 25G83 clustering metadata (GTP 4-7) sits at its end and is never cleared; only the per-slot records are zeroed. Clustered tiling can then read metadata left by another render, or written by one still in flight. G15 gives each clustered pass a zeroed buffer.

This gives each render slot its own copy of the clustering metadata, zeroed before the render uses it. The TPC and the tile maps stay in one object shared by the target's renders.

My first version of this in my tree copied the whole TPC object per slot. With 32 renders in flight that took 47 MiB for a 1728x1117 target, and grown targets keep their old copies until those renders retire. SCP: Secret Laboratory filled the client's 320 MiB target window within minutes: target growth failed with ENOSPC, then the render submit gave up with EBUSY and the game saw a lost device. This version copies only the metadata: 0x70000 bytes (448 KiB) per render at that size, 14 MiB for 32 renders, from a page-aligned base so every sub-buffer keeps its alignment.

The shared TPC object (holding the TPC and GTP 1-3) is sized to GTP 4 rounded up to a page, not down, so the tile maps never spill past the object when GTP 4 is not page-aligned (for example 32x32 utiles at 3456x2234). Unit tests check that every sub-buffer lies in its object with its alignment across sizes, utile shapes, layer counts and cluster counts. (The earlier separate overflow-fix commit has been folded into this single commit for bisectability.)

The TPC has to stay shared. Two kernels that gave each render its own TPC copy as well, one of them with the overflow fixed, hung the GPU in SCP: Secret Laboratory (reason 0).

The two-cluster M4 Pro (T6040) uses the same 25G83 layout, so it gets the per-render copy too. I couldn't test it there. M5 is unchanged.

Testing

All on a J616C (32-core M4 Max), 25G83, under Hyprland, as the daily module in my tree on top of other work that isn't in this PR:

  • Whole-object version: my daily from 16 Nov. It was part of the build that ran Roblox (Vulkan) for 11+ minutes with 0 GPU recoveries (see drm/asahi: g17: tile clustered on the 32-core M4 Max (T6041 25G83) #2021). SCP:SL then failed within 1-5 minutes while spectating, as described above.
  • Earlier version before rounding shared size up: my daily from 17 Nov. SCP:SL spectating ran with no GPU errors, but Minecraft with a shader pack in a window near 3456x2234 hung the GPU (reason 3, no MMU fault) within about 35 s, on two Mesa builds.
  • Per-render TPC and metadata, overflow fixed: Minecraft ran, SCP:SL hung after about 73 s (reason 0, tiler heap not starved). Not pursued.
  • Current logic: my daily since 18 Nov. Minecraft with the shader pack for 2 minutes and SCP:SL spectating for about 4 minutes, 0 GPU recoveries.
  • Not tested on its own. I haven't run clustered tiling without the zeroed copy since drm/asahi: g17: tile clustered on the 32-core M4 Max (T6041 25G83) #2021's pool fix, so I can't say that a game needs it. It's a correctness change toward what G15 does.
  • This branch (6d2f59d650ff): aurora.o compiles clean with W=1 and CONFIG_KUNIT=y, and tools/asahi/run-host-tests.py --only g17 passes 90/90. Not booted as-is yet.

It conflicts in one line of queue/render.rs (the gtp_va call) with the PR that tiles small passes unclustered. Whichever lands second needs a one-line rebase.

Assisted-by: LLM (Google:Gemini 4 Neon)

@RyanTheTide

Copy link
Copy Markdown
Contributor

The production boot stub pins at macOS 27.2 (26B123), so please retarget this to 27.2. The clustering metadata layout here comes from the 26.6.2 (25G83) GPU firmware; please take it from the T6041 GPU image the 27.2 stub loads.

@AxxzyWasTaken

Copy link
Copy Markdown
Contributor Author

Thanks. As far as I can see, 26.6.2 is still what M4 Max installs get today, so I'd like to keep this on 25G83 for now and port it once there's a 27.2 base for the T6041:

When a T6041 board moves to 26B123 in the installer, I'll port this and the four-cluster series (#2021, #2171, #2172) onto the 27.2 base and re-test on my J616c. If you'd rather hold these until then, that's fine too.

Disclosure: prepared with an AI tool (the commits carry Assisted-by: LLM); facts checked against the repos named above.

@AxxzyWasTaken

Copy link
Copy Markdown
Contributor Author

I found an overflow in this PR's metadata-copy commit and pushed a fix as a new commit on top ("drm/asahi: g17: hold the last tile map in the shared TPC object"). The earlier commits are unchanged.

  • The overflow: the shared TPC object (TPC and GTP 1-3) was sized to GTP 4 rounded down to a page, where the per-render metadata copies start. The tile maps run up to GTP 4 itself. With 32x32 utiles at 3456x2234 the TPC is 0x46000 bytes, GTP 4 isn't page aligned, and the last 8 KiB of the tile maps lay past the object, in its unmapped guard page. No MMU fault was reported; I don't know why. The fix sizes the object to GTP 4 rounded up (shared_end()) and checks that bound in gtp_va(). M5 is unchanged.
  • The test: that commit's <= 0x2_0000 bound on the metadata copy was wrong. The copy is 0x70000 bytes at 1728x1117, because metadata 4 (GTP 6) has room for 64 partial renders. The tests now check that every sub-buffer lies in its object with its alignment across sizes, utile shapes, layer counts and cluster counts, and a new test covers 3456x2234 with 32x32 utiles. I've corrected the "about 64 KiB" in the description too.
  • The TPC has to stay shared. Two kernels that gave each render its own TPC copy as well, one of them with this overflow fixed, hung the GPU in SCP:SL (reason 0). This PR keeps one TPC per target.

Testing on my J616C (32-core M4 Max, 25G83):

  • My daily kernel with this PR's metadata-copy commit hung the GPU (reason 3, no MMU fault) within about 35 s of Minecraft with a shader pack in a window near 3456x2234, on two Mesa builds.
  • The same kernel with only this fix added, my daily now: Minecraft with the shader pack for 2 minutes and SCP:SL spectating for about 4 minutes, 0 GPU recoveries.
  • This branch: compiles (M=drivers/gpu/drm/asahi), tools/asahi/run-host-tests.py --only g17 90/90, and the target.rs layout tests pass in a host harness (6/6; the old bound fails there). Not booted as-is.

#2171 and #2172 aren't affected: they sit on #2021 alone, where the TPC object holds the whole parameter block and is sized to its end.

Disclosure: prepared with an AI tool (the commits carry Assisted-by: LLM); tested as described.

T6041 ran its 25G83 tiling passes in TA mode 1 (0x1c890 = 1), which
tiles non-clustered: one of the four clusters runs the tiling and
vertex work of a pass. Vertex shaders reached a quarter of the ALU rate
of the GPU (2.95 vs 12.3 TFLOPS), and geometry-bound games were capped
by it.

Tile clustered instead, with what a four-cluster pass needs:

- Clustering metadata sized for more than two clusters, as the M2
  Pro/Max path sizes it (meta1 per block of tiles, meta2-4 fixed).
- The four-cluster segment words of cluster_control(): 0x1c890 =
  0x280540, 0x1c918 = 4, and 32 + 16 x clusters = 0x60 free parameter
  buffer pages in 0x1c9a8. The two-cluster capture (0x200440 / 4 /
  0x3c) stays for two clusters.
- A per-build tiler heap normal maximum: the four-cluster 25G83 build
  may use every slot of the block table. A clustered RD32 Minecraft
  frame needs about 1600-2200 blocks; at the 0x45e normal maximum the
  pass never starts and the firmware recovers with reason 0.

25F84 keeps TA mode 1, and every other build keeps its values.

Measured on the J616C (32-core M4 Max, 25G83): vertex-shader ALU 2.95
-> 10-11.5 TFLOPS; Sodium-style terrain 1.93 -> 0.68 ms per frame with
identical images; Minecraft RD32 105 -> ~300 fps, with no GPU
recoveries over several sessions, heavy and concurrent stress included.
G17 host tests 31/31.

Signed-off-by: AxxzyWasTaken <axxzytte@gmail.com>
With four-cluster tiling the 32-core M4 Max runs vertex shaders on every
cluster at once. Roblox (Vulkan) then hangs the GPU about once a minute:
the firmware kills a small depth/stencil render as no-progress (reason
0), with no MMU fault and a nearly empty tiler heap. The same hang showed
up on this Mac with the earlier driver until its render pool had 96
blocks.

Give firmware variants a smallest render pool, 44 blocks as before, and
96 for T6041 25G83. Owners grow to it on their first render, as they grow
for ray-query scratch.

Signed-off-by: AxxzyWasTaken <axxzytte@gmail.com>
…them

macOS 26.6.2 on the 32-core M4 Max (25G83), captured under a hypervisor
while running a game, spaces a clustered pass's tiler metadata 2, 3 and 4
0x1180, 0x1400 and 0x1600 apart and writes 0x5c free parameter-buffer
pages. Metadata 2 was 0xc00, the M2 Pro/Max size, so the tiler could
write past it into metadata 3. Use 0x1180 and 0x5c; metadata 3 and 4 keep
their larger sizes.

Signed-off-by: AxxzyWasTaken <axxzytte@gmail.com>
Four-cluster passes size clustering metadata 1 as the M2 Pro/Max path
does, 0x44 bytes per block of tiles, with 0x80, the two-cluster M4 Pro
size, as the floor. Metadata 1 is the last sub-buffer, and the macOS
26.6.2 capture on the M4 Max does not show its size. Give it a 16 KiB
floor instead.

The per-block size stays the M2 Pro/Max formula; the floor is our own
choice, not a macOS value.

Signed-off-by: AxxzyWasTaken <axxzytte@gmail.com>
@RyanTheTide

Copy link
Copy Markdown
Contributor

Looks good; approve once this exact head has booted on the M4 Max and on a two-cluster M4 Pro.

  1. drivers/gpu/drm/asahi/g17/buffer/target.rs: d7c3861 on its own sizes the shared TPC object to GTP 4 rounded down. With 32x32 utiles at 3456x2234 the last tile map then runs past the object; that is the reason-3 hang 17a8d6f fixes. If both land as they are, a bisect can stop on a commit that hangs the GPU. Fold 17a8d6f into d7c3861 when you rebase onto drm/asahi: g17: tile clustered on the 32-core M4 Max (T6041 25G83) #2021, and drop the "about 64 KiB" claim from that commit message (it is 0x70000 at 1728x1117).
  2. The T6040 (two-cluster M4 Pro) takes this path too. prepare() (target.rs:389) zeroes a per-render metadata copy whenever !m5 && gtp[4].is_some(), and that holds for the two-cluster 25G83 layout as well. Nothing has run there. The lab J616s is a T6040, so it can carry the row.

Checked and correct: the shared/per-render split in gtp_va() (target.rs:364). Offsets below meta_base() resolve in the shared object, sized to shared_end() rounded up. Offsets above resolve in the slot's copy, which is checked against meta_stride. The copy is zeroed only after the slot's EBUSY check. ensure_capacity() (target.rs:444) retires the old metadata object while renders still own it, so in-flight renders keep their addresses. This sits on #2021 (open), so its merge order follows that PR.

HW row: this head on #2021, booted as-is (not a daily tree with other work). On the J616C (M4 Max, 25G83): Minecraft with the shader pack near 3456x2234 for 10 min and SCP:SL for 10 min, 0 GPU recoveries. On a T6040 (J616s or J614s): the same pair, or the G17 Vulkan CTS render subset, with 0 recoveries. Also 0 AVC, 0 failed units and no new kernel errors on both.

Assisted-by: LLM (Claude:Opus 5.5)

A render target keeps one TPC object, with the 25G83 clustering metadata
(GTP 4-7) after it, for all of its in-flight renders, and never clears
the metadata; only the per-slot records are zeroed. Clustered tiling can
then read metadata left by another render, or written by one still in
flight.

Give each render slot its own copy of the clustering metadata, as the
records have, and zero it before the render uses it, as G15 gives each
clustered pass a zeroed buffer. Only the metadata is copied: the TPC and
the tile maps stay shared among the target's renders. A copy of the
whole object per slot took 47 MiB for a 1728x1117 target with 32 renders
in flight, and SCP: Secret Laboratory filled the 320 MiB target window
within minutes (target growth failed with ENOSPC until the render submit
gave up with EBUSY). The metadata copy is 0x70000 bytes at 1728x1117
(metadata 4, GTP 6, has room for 64 partial renders) and starts at a
page-aligned base, so every sub-buffer keeps its alignment. M5 is
unchanged.

The shared TPC object, which holds the TPC and GTP 1-3, is sized to GTP
4 rounded up to a page, not down. The tile maps (GTP 1) run up to GTP 4
itself, so when GTP 4 is not page aligned their end would lie past the
object. With 32x32 utiles at 3456x2234 the TPC is 0x46000 bytes and the
last 8 KiB of the tile maps would fall into the unmapped guard page
after it; that hung the GPU (reason 3, no MMU fault reported) within
about 35 seconds of Minecraft with a shader pack in a window near that
size. gtp_va() checks the bound. The TPC itself has to stay shared by
the target's renders: two kernels that also gave each render its own
copy of the TPC hung the GPU in SCP: Secret Laboratory (reason 0).

The tests check that every sub-buffer lies in its object with its
alignment across sizes, utile shapes, layer counts and cluster counts,
including 3456x2234 with 32x32 utiles.

Signed-off-by: AxxzyWasTaken <axxzytte@gmail.com>
@AxxzyWasTaken
AxxzyWasTaken force-pushed the pr/t6041-cluster-meta-per-render branch from 17a8d6f to 6d2f59d Compare November 21, 2026 21:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants