Repository navigation
drm/asahi: g17: M5: render large passes instead of losing the device after 26 s, refuse those that cannot fit, and account their memory to the client - #2501
Conversation
9d1e69b to
48c5f89
Compare
8a3871f to
daa0d08
Compare
|
No blockers. I checked these and they read correct:
Two nits:
HW row: an M5, M5 Pro or M5 Max (for example J815 or K114c) on this head, since commits 1 and 2 apply there. Run 16384x16384x22 and 512x512x2048 single-sample passes (correct readback), one over-4 GiB pass (SUBMIT returns ENOMEM, no lost device) and the same render CTS subset (no Pass to Fail). Check the client's fdinfo memory grows by the pair size during the large pass and returns afterwards. Pass: 0 failed units and no new kernel errors besides the intended ENOMEM line. Assisted-by: LLM (Claude:Opus 5.5) |
…compact window before allocating them On M5 a render pair's target records, TPC object and initial parameter buffer are sized from the client's render pass. The record and TPC limits apply only to the other G17 parts, so on M5 these sizes were bounded only by integer overflow. A pair allocated and zeroed each object in full before mapping it into the client's compact window, so a layout too large for the window was allocated first and then failed at the mapping, and the render was retried while the runtime waited for capacity. The other parts are bounded by the window their targets live in. On M5 that window is the client's compact window, which the firmware addresses with 32-bit offsets: 4 GiB. A pair maps there its target records (one stride of the page-rounded record for each of the 32 command slots), its TPC object, its initial parameter blocks at 0x88000 bytes each, and about 3.8 MiB of objects its layout does not size: the context state of each command slot, the parameter buffer's scene scratch and discard, its block manager's page list, its metrics entry and its pool page list. Each object is followed by its guard page where it has one and placed at 32 KiB alignment. The discard object's size is named once, for the bound and for the pair that allocates it. A layout whose pair does not fit in the window can never be mapped, so deriving the layout, which SUBMIT already does for every M5 render, now refuses it with ENOMEM before anything is queued or allocated. The bound depends on the pass's tile size and on whether it is marked as drawing few primitives, which sets how much of the TPC object is needed. On the M5 Ultra, for a single-sample pass with 32x32 tiles, the size the Borealis Vulkan driver uses for such a pass, this admits a 16384x16384 target of up to 22 layers when the pass draws more than four primitives, and up to 53 layers when it draws four or fewer; 2048 layers of a target up to 1024x1024, or up to 2048x2048 with few primitives. With 16x16 tiles it admits up to 5 layers at 16384x16384 and 2048 layers up to 512x512. KUnit pins these sizes and their first refused neighbours. The window is shared by the client's pairs, so a layout that fits on its own can still find too little room. Before allocating a target object or parameter-buffer blocks, a pair now checks that the window has a hole for the object and room for all the blocks, and fails with ENOSPC, as the mapping would, without allocating them. Other G17 parts are unchanged. Reported-by: Erik Bourget <erik@ebourget.net> Suggested-by: Erik Bourget <erik@ebourget.net> Suggested-by: Ryan Murray <ryan@aurorasilicon.org> Signed-off-by: Jeff Queisser <funkzy@users.noreply.github.com>
…izes it On M5 a render pair's target records, TPC object and parameter buffer are sized from its client's passes and geometry: up to the 4 GiB of the client's compact window for each pair, and a client can hold a pair on each of its queues. They were allocated as kernel memory, which counts toward the device total but is not refused and is attributed to no client, so the client's memory limit and fdinfo did not include it. Charge that backing as the M5 pool backing is charged: admitted against the client's and the device's memory limits before it is allocated, as a user buffer of the same size would be, and counted as the client's held memory instead of kernel memory until it is freed. This covers the target records and TPC object when a pair is built for a client and when a larger target replaces them, and the parameter buffer's initial blocks and every growth. A refusal fails the allocation with ENOMEM: building or growing the pair for a render fails that render, and a growth the firmware asks for is declined, as when the buffer is at its largest. The client's fdinfo now includes this memory. Other G17 parts are unchanged. Reported-by: Erik Bourget <erik@ebourget.net> Suggested-by: Erik Bourget <erik@ebourget.net> Signed-off-by: Jeff Queisser <funkzy@users.noreply.github.com>
…one render pair, so it renders instead of failing after 26 s On the M5 Ultra a render queue may hold two render pairs. Each pair maps its own target records, TPC object and parameter blocks, sized from the pass, in the client's 4 GiB compact window. The first render of a pass warms the second pair at the same layout, and admission spreads a queue's renders over both. A pass whose pair takes more than half the window, such as a single-sample 512x512 pass of 2048 layers or a 16384x16384 pass of 22 layers (2.2 to 2.4 GiB a pair), never rendered: building the second pair failed with ENOSPC, the render waited for room that its own first pair held, and after about 26 seconds failed with EBUSY, which the client saw as a lost device. Skipping the second pair when it does not fit is not enough: with three such renders in flight the third went to the unbuilt pair and failed the same way. Decide per render how many pairs its queue uses: both for a layout the window holds twice, else one (g17::pairs::for_window). SUBMIT, which already derives the layout, keeps the window bytes of one pair on the command, and the decision reads them there. A render that runs on one pair goes, through admission, only to the pair that took its queue's previous such render, so the large layout's objects exist in one pair, and it claims and warms no sibling. The scheduler's admission check applies the same rule, so it waits for that pair rather than for any. Smaller renders of the same queue still use both pairs. The optional warm-up of a sibling for which the window or the client's memory limit has no room is skipped as a busy one is, and the render runs on its pair. The rule only narrows a queue's renders to a slot it already holds: a held, quarantined or killed slot is handled as before, and no slot goes to another queue. On the M5 Ultra, Borealis (Vulkan) renders of 512x512x2048 (one, and two, three or four in flight), 16384x16384x22, 2048x2048x2047 and 1152x1152x2048 are now correct; heavy 1080p fill and geometry rows at depth 8 still run on both pairs, at the same time per submit as before (38.2 and 36.9 us) and 7-10% faster than on one pair. KUnit covers the decision and the admission routing: renders that run on one pair stay on the same pair up to its depth and never move to the other, and a new queue whose renders need one pair claims no sibling. Other G17 parts, which hold one pair per queue, are unchanged. Fixes: b1c9a21 ("drm/asahi: g17: restore two render pairs per queue on G17D within the shared QID capacity, 9-15% faster heavy 1080p rows at depth 8") Suggested-by: Ryan Murray <ryan@aurorasilicon.org> Signed-off-by: Jeff Queisser <funkzy@users.noreply.github.com>
daa0d08 to
5d6427e
Compare
|
Thanks, Ryan. Both nits are fixed in the commits, now at 5d6427e, with no change in behaviour:
I've added your J775D on this head (stacked with #1973 and our other three open drm/asahi PRs on integration/aurora cc14570, 27.0 firmware, default command line; the kernel also carries a local fix for an appledrm boot oops, see #1973):
Rows we can't run: could your lab, or an owner of an M5, M5 Pro or M5 Max, run this head? These take the first two commits (the third only matters where a queue holds two pairs). The useful row is the same cases: 16384x16384x22 and 512x512x2048 with correct readback, a pass over 4 GiB returning ENOMEM without a lost device, fdinfo growing and returning, the render CTS subset, 0 failed units and 0 new kernel errors. |
|
Re-review of 5d6427e: No blockers.
The second commit is unchanged since the last review. HW row: the new numbers are from the M5 Ultra, on an integration stack with other open PRs and a local appledrm fix. Still needed: an M5, M5 Pro or M5 Max (one queue holding one pair) on this head: 16384x16384x22 and 512x512x2048 with correct readback, a pass over 4 GiB returning ENOMEM with no lost device, fdinfo growing during the pass and returning after, the render CTS subset unchanged, 0 failed units, 0 new kernel errors apart from the intended ENOMEM line. Assisted-by: LLM (Claude:Opus 5.5) |
DjDeveloperr
left a comment
There was a problem hiding this comment.
Pre-PR checklist review (§1–§6) on live head 5d6427e7cba76199c56b2b76cce8af2ac15abf4a (host-tests: pass).
Checklist Summary
- §1 No model gating: Pass — uses the existing M5 firmware path and architectural 32-bit compact-window size; commit 3 keys on queue pair count (
queue > 1), not chip IDs. - §2 Production-ready, no hacks: Pass — default command line, no new params/hacks.
- §3 Safety: Pass — no hardcoded hardware safety constants.
- §4 Clean-room & attribution: Pass — all 3 commits (
312bbde50816,aeed8c38c412,5d6427e7cba7) carry human author/committer and humanReported-by/Suggested-by/Signed-off-bytrailers only; PR body includesAssisted-by:. - §6 Naming: Pass — uses Borealis and Aurora naming.
- §5 Evidence: Still needs a 1-pair-per-queue M5/M5 Pro/M5 Max row for commits 1–2 in addition to the code fixes below.
Blocking Code Findings
-
Stale
State::wide[slot]can route a >2 GiB render onto the second pair and reproduce the 26 sENOSPC/lost-device failure (drivers/gpu/drm/asahi/g17/queue/admission.rs):self.wide[slot]is only written inState::pick()whenpairs == 1(self.wide[usize::from(slot)] = owner), and is never cleared whenself.owners[slot]changes or is cleared (take_idle,claim_sibling,reassign_unused_spare,cancel_empty_slot), nor are other slots cleared when a newwideslot is assigned forowner.- Concrete failure path on a 2-pair queue (owner
O):Opicks slot1for a 1-pair render (pairs == 1) ->wide[1] = O.- Slot
1leavesOviacancel_empty_slot(1, O)(failed/cancelled build before publication) orreassign_unused_spare(1, Some(O), ...)->owners[1]becomes0or another owner, whilewide[1]remainsO. O's next >2 GiB render takes slot2->owners[2] = O,wide[2] = O, and slot2builds and maps the >2 GiB target in the 4 GiB compact window.Oruns a small render (pairs == 2) andclaim_siblingclaims slot1back forO. Nowowners[1] == O,owners[2] == O, and bothwide[1] == Oandwide[2] == O.O's next >2 GiB render runs(0..cfg::RENDER_SLOTS).find(|slot| self.owners[*slot] == owner && self.wide[*slot] == owner), which returns the lowest-index slot (slot1) instead of slot2. Slot1then tries to map a second >2 GiB pair beside slot2's live >2 GiB pair in the 4 GiB compact window ->ENOSPC-> 26 s timeout / lost device.
- Fix: Clear
self.wide[slot] = 0wheneverself.owners[slot]is reassigned or cleared (take_idle,claim_sibling,reassign_unused_spare,cancel_empty_slot), and clear any priorwide[other] == ownerwhen recordingself.wide[slot] = owner. Add a KUnit case covering slot loss + sibling reacquire with a live wide slot at a higher index.
-
pair_window_bytes()omits task-stagemesh_storage(32 MiB default) mapped inM5_COMPACT(drivers/gpu/drm/asahi/g17/queue/render/m5_render_submission.rs):- For any render pass using a task shader (
programs.uses_task()),Memory::mesh_storage_register()(drivers/gpu/drm/asahi/g17/queue/render/memory.rs:672-685) mapsm5_mesh_storage_chunks * 64 KiB(default0x200chunks = 32 MiB, plus 16 KiB guard rounded to 64 KiB alignment) intoBlockGeometry::M5_COMPACTviaCompact::new_aligned(). pair_window_bytes()does not account for this reservation whenresources.as_deref().is_some_and(|p| p.uses_task())andm5_mesh_storage_chunks > 0. As a result, a task-shader pass whose pair fits in(4 GiB - 32 MiB, 4 GiB]passes SUBMIT instead of failing withENOMEM, and a pass in(2 GiB - 32 MiB, 2 GiB]getsfor_window() == 2even though two pairs plus mesh storage exceed the 4 GiB compact window.- Fix: Include the aligned
mesh_storagecompact-window reservation (whenuses_task()andm5_mesh_storage_chunks != 0, using a shared helper/constant withMemory::mesh_storage_register) inpair_window_bytes()and add a KUnit check.
- For any render pass using a task shader (
Verified Non-Blocking Items
KernelObject::client_backing()(g17/object.rs) properly orders fields (mapping,obj,ptr,size,charge) so GPU unmap and GEM teardown precede releasing the client charge, and RAIICommitrolls back admission on allocation failure.Object::new(buffer/target.rs) andTvb::allocate(buffer.rs) both pre-check compact-window room (check_m5_room) before allocating and cover both initial creation and growth (Target::ensure_capacity/Tvb::grow).DISCARD_BYTESdeduplication and SUBMIT-timeValidated::Render::pair_windowcaching are clean.
Also Needed Before Merge (§5)
- After fixing the two code issues above, re-prove on J775D and obtain a 1-pair-per-queue M5 / M5 Pro / M5 Max row on the updated head covering commits 1–2 (
16384x16384x22,512x512x2048readback, over-4 GiBENOMEMwith no lost device, fdinfo growth/release, render CTS subset, 0 failed units, 0 new kernel errors beyond the intendedENOMEM).
Assisted-by: LLM (Gemini 4 Neon)
Problem
On M5 a render pair's target records, TPC object and initial parameter buffer are sized from the client's render pass, with no cap: the objects were allocated in full before they were mapped into the client's 4 GiB compact window, and they were accounted as kernel memory rather than to the client. On the Mac Studio (M5 Ultra, J775D) a pass whose pair took more than half the window, such as a single-sample 512x512 pass of 2048 layers, never rendered: the queue's second render pair did not fit beside the first, and after about 26 s the client saw a lost device.
Change
Three commits in
drivers/gpu/drm/asahi/g17:Which parts it affects: the M5 render path (the G17 parts whose firmware uses the compact window), selected by the existing M5 firmware path in the driver; this series adds no chip, board or firmware condition of its own. The window size is the firmware interface's (32-bit offsets), not a tuned value. Commit 3 only matters where a queue holds two pairs, today the M5 Ultra; every other G17 part holds one pair per queue and is unchanged. No new parameter or default.
Testing
On the J775D (Mac Studio, M5 Ultra), 27.0 firmware, integration/aurora cc14570 with #1973 and our other three open drm/asahi PRs, default command line, one boot (the kernel also carries a local fix for an appledrm oops at boot on current integration, not part of this PR). Borealis, single-sample R8, all four corners read back:
The client's fdinfo grows during each pass (to 8201 MiB at 16384x16384x22 and 2984 MiB at 512x512x2048, image included) and the device's committed memory returns to its starting value within seconds of the client's exit. The render CTS subset gives 571 Pass and 57 NotSupported, as on the same base without the series. KUnit (366 of 368 pass, two skips as on the base) and the upstream host tests (427 of 427 on base and head, no result differs) pass.
Regression run (ALARM has no SELinux, so there is no AVC count): 0 new failed units (iwd.service and systemd-networkd-persistent-storage.service fail on the read-only test root with and without the series) and 0 new kernel error lines against the boot without the series, apart from the intended
submit failed ... ENOMEMof the over-4 GiB cases.Since RyanTheTide's review: the discard object's size is named once, for the bound and for the pair that allocates it (first commit), and the pair window is computed once at SUBMIT, kept with the command and used when the render's pairs are chosen (third commit). Behaviour is unchanged;
git range-diffshows only those lines and the messages. The earlier rows on 27.2 firmware (below) were taken on the previous head.Still to run
Full description of the three commits
Other G17 parts are unchanged.
Full test results
On the Mac Studio (M5 Ultra), 27.2 firmware, on the integration branch at the previous head, with Borealis (single-sample R8, one draw per pass, all four corners read back):
Known limits
Thanks to ewb, who reported the uncapped M5 target sizes and suggested refusing and charging them, and to RyanTheTide for the review.
Assisted-by: LLM (Claude:Opus 5.5)