Repository navigation
drm/asahi: g17: scale the tiler heap's normal maximum with memory - #2724
jgoyvaerts wants to merge 4 commits into
Conversation
|
Good writeup. The sizing comes from a memory share measured on the machine, not a per-model table, and the 15k-40k triangle numbers make the case. An adversarial read of 0a2a5c2:
Assisted-by: LLM (Claude:Opus 5.5) |
DjDeveloperr
left a comment
There was a problem hiding this comment.
Adversarial review of head 0a2a5c2b8f6543f30ab09bf035d9f621f0581b16 against the pre-PR checklist. Holding approval and requesting changes on the memory-accounting/VA-exhaustion implications and missing hardware evidence on this head.
Code & resource-limit findings
-
Per-heap RAM multiplication is unattributed to clients and bypasses GPU memory admission limits.
Following up on Ryan's point #1 directly fromgem.rsandmemacct.rs:Tvb::prepare_growth()allocates each extension viaKernelObject::backing()->gem::new_kernel_object_wc()->memacct::Charge::kernel(&memacct::LEDGER, size)(gem.rs:324). In this driver:- Kernel objects are not charged to any client (
Client) or processmm(MM_SHMEMPAGES), so the OOM killer does not score the offending client for them. - Kernel objects bypass both
LEDGER.limitandLEDGER.client_limit(memacct.rs:215: "Kernel objects share the device limit but are never refused"), while increasingkernel_residentand shrinking the budget left for other clients' user BOs. set_mapping_held(&gem, true)pins the shmem pages asAS_INACCESSIBLE | AS_UNEVICTABLE(unswappable, unmigratable), and off-lock growth runs in scheduler/workqueue context rather than reliably under the submitting task's memcg.
BecauseLayout::neo(memory_bytes()?)applies per physical render pair (RENDER_SLOTS = 64, and one logical queue can hold 2 pairs), raising the cap from 140 MiB to 256 MiB (16 GiB Mac), 384 MiB (24 GiB), or 419 MiB (≥26.2 GiB) multiplies per active heavy pair — e.g., 4–8 heavy pairs on a 16 GiB Mac can pin 1–2 GiB of unswappable, unattributed, limit-bypassing kernel memory.
- Kernel objects are not charged to any client (
-
Whole-extension shrink interacts poorly with the larger 3× growth steps.
Inbuffer.rs:1012-1050, 1266-1285:- While a client stays active on light frames (
now - last_busy < 10s),shrink_target()earnsSHRINK_STEP_BLOCKS = 4(512 KiB/s) and only shrinks oncecredit >= last(the entire last extension), resettingcredit = 0on any period above 1/4 capacity. A single 3× growth jump from e.g. 900 -> 2,700 blocks creates a 1,800-block (225 MiB) extension that takes 7.5 minutes of uninterrupted low load to release while active. - Because
Manager::shrink()can only drop whole trailingKernelObjectextensions (kept - size < target), ifshrink_floor()sits anywhere above the pre-jump size (e.g. 950 blocks after a 900 -> 2,700 jump), that 225 MiB extension cannot be shrunk even after 10 s idle untilbusy_peakdrops below 450 blocks.
Raisingnormal_blocksup to 3× makes both the peak footprint and this coarse-granularity retention substantially heavier. Please either bound/attribute TVB growth (or cap individual extension chunk size so shrink can reclaim incrementally) or demonstrate multi-client pressure behavior on a 16 GiB Mac.
- While a client stays active on light frames (
-
Multi-pair compact-VA spillover into
TARGETSwithin a single VM (hw/t8140.rs:346-355).
compact_ranges()providesTVB_MAX_BLOCKSslots inTVB_GROWTH(pair 0) andTVB_MAX_BLOCKSinsecond(pair 1), and then falls back toTARGETS(320 MiB, shared with render targets). Previously at 1,118 blocks/pair, up to 6 render pairs in one VM fit in the two TVB arenas without touchingTARGETS. At 3,353 blocks/pair (≥26.2 GiB Macs), just 2 heavy pairs in one VM fill both TVB arenas completely, so a 3rd growing render pair in the same VM spills TVB blocks intoTARGETSand then fails withENOSPC. -
Minor cleanups:
buffer::memory_bytes()(buffer.rs:319) duplicatesmemacct::total_ram_bytes()(memacct.rs:104).GraphArgs.normal_blocksis populated for both NEO and M5 (buffer.rs:819), validated/used byfw::Graph::init(), and silently ignored byLayout::initialize()(which readsself.normal_blocksinstead).
Evidence & checklist blockers
- No hardware boot or regression run on current head
0a2a5c2b(Checklist §5): the J616s numbers in the PR body are from an older superseded revision (#2186) onm4.166(which also carried#1902and#1986), and no regression run (0 SELinux denials, 0 failed units, 0 new kernel errors) is reported for this head. - No conformance run for a performance change (Checklist §5): checklist requires a same-machine A/B with conformance unchanged on every GPU generation the change applies to.
- Untested scaled range (9–26 GiB) & A18 Pro: as noted by Ryan and in the PR body, no 16 GiB / 24 GiB Mac has exercised the non-clamped formula (
BM+0x80= 2,048 / 3,072), and the J700 (A18 Pro) readback/no-regression row is still pending. - CI:
host-testspassed on0a2a5c2b. - Checklist §1–§4 & §6: Pass on code (RAM-scaled policy with no chip/board/firmware gates, default cmdline, human-only
Signed-off-byin commit, clean-room live measurements). Minor attribution nit: PR body ends withAssisted-by: LLM (Claude:Opus 5.5)instead ofAnthropic:Claude Opus 5.5.
Assisted-by: LLM (Google:Gemini 4 Neon)
A growth of the tiler heap is one backing object, and a shrink can only give back whole trailing growth objects. A heap that grew from 900 to 2,700 blocks in one step holds a single 1,800-block (225 MiB) extension: the active shrink returns it only after 450 periods of shrink credit, and the idle shrink cannot return it at all while its floor sits above 900 blocks. Build each growth from extensions of at most 64 blocks (8 MiB). The active shrink then gives back 8 MiB every 16 periods of credit, and the idle shrink stops within 8 MiB of its floor. A full block table is 53 extensions. Signed-off-by: Jonathan Goyvaerts <jonathan.goyvaerts@gmail.com>
… they go Tiler-heap blocks are placed in the first of the compact client arenas with room: the render-alias window, the two tiler-heap growth arenas (3,321 and 3,353 blocks), then the target-record window. Growth that fills both growth arenas, for instance three pairs of one VM at 3,353 blocks, then takes target-record space, and the next target record or growth of that VM fails with ENOSPC, and the render with it. Place growth only in the two growth arenas. A growth that runs out of them or of memory keeps the whole extensions it made, and the tiling pass partial-renders within that heap, as it does when a firmware grow request is refused. An idle growth still fails if the heap would stay below the blocks the render's layout requires. After a growth stopped short, the host grows the heap beyond those blocks again only a shrink period later, as the busy growth already does after a refusal: every attempt allocates and maps afresh. Signed-off-by: Jonathan Goyvaerts <jonathan.goyvaerts@gmail.com>
Tiler-heap growth is allocated as kernel objects: they count towards the device total, but are charged to no client, are never refused by the device or the per-client limit, and are pinned. The client sizes this growth through its renders, and every render pair of every client can hold its own, so many heavy pairs can pin a lot of memory that no limit bounds and no client is accountable for. Admit each growth extension of a client VM's pair against that client's and the device's limits, and count it as the client's held memory, as the M5 pools are. A refused extension ends the growth there: the tiling pass partial-renders within the heap it has, and an idle growth fails with ENOMEM only if the heap would stay below what the render's layout requires. Kernel pairs keep kernel objects. Signed-off-by: Jonathan Goyvaerts <jonathan.goyvaerts@gmail.com>
The tiler heap of a 25G83 render grows on the host only: before a render the host grows it to three times the last high-water, capped at the heap's normal maximum, which also goes to the firmware in BM+0x80. A scene larger than that cap partial-renders on every frame. The cap is the fixed 0x45e (1,118 blocks, 140 MiB) on every NEO-layout heap, whatever the memory. Make the cap a 64th of the memory si_meminfo() reports (memacct's total_ram_bytes(), which also sizes the memory limits), clamped to NORMAL_MAX_BLOCKS..=TVB_MAX_BLOCKS (the block table). BM+0x7c, the heap's capacity, is unchanged. The cap holds for each render pair; the growth up to it is charged to the pair's client and bounded by its and the device's memory limits, and by the VM's growth arenas. Both graph layouts take the normal maximum from GraphArgs. The share comes from three measurements: - On the 48 GiB J616s, macOS 26.6.2 runs the GPU with BM+0x80 = 6,579 blocks (read live), 1/60 of the memory. The fixed 1,118 is 1/59 of 8 GiB. A 64th stays below both, so above the floor the cap is never a larger share of memory than the firmware is known to run with. - At 8.7 GiB and below the clamp keeps 1,118: BM+0x80 stays 0x45e * 4 pages, and the 8 GiB A18 Pro (and its mesh floor) is unchanged. - On the J616s (46.6 GiB usable) a 64th is 5,959 blocks, clamped to the table's 3,353. An off-screen GLES test draws N screen-spanning triangles into a stored 3456x2234 RGBA8 + D24S8 target and waits for each frame. With 1,118, a triangle costs 3.0 us up to N = 14,500 and 9.7 us from N = 15,000. With 3,353, 3.0 us holds up to N = 40,000: N = 20,000 takes 55 ms a frame instead of 193 ms, N = 30,000 83 ms instead of 281 ms. At N = 45,000 the scene passes 3,353 blocks and partial-renders as before. A 16 GiB Mac gets 2,048 blocks, a 24 GiB Mac 3,072; from about 26.2 GiB the table's 3,353 is the limit. Signed-off-by: Jonathan Goyvaerts <jonathan.goyvaerts@gmail.com>
0a2a5c2 to
001d783
Compare
|
Thanks both. New head
Cleanups: Tests on this head: each commit compile-checked with arm64 defconfig + asahi.config, @RyanTheTide, we have no 16 GiB M4 or J700. Could you run the 16 GiB tiler test with Assisted-by: LLM (Claude:Opus 5.5) |
|
Thanks, this answers the earlier review. At 001d783:
No new code issues found. Assisted-by: LLM (Claude:Opus 5.5) |
Supersedes #2186. The sizing is rewritten as this driver's own policy.
On 25G83 the tiler heap grows on the host only. Before a render, the host grows it to 3x the last high-water, capped at the heap's normal maximum. The driver also writes that cap to the firmware in
BM+0x80. A scene bigger than the cap partial-renders on every frame. Today the cap is a fixed0x45e(1,118 blocks, 140 MiB) on every NEO-layout heap, whatever the memory.Series
cfg::tvb_growth_ranges()). A growth that runs out of arena space or memory keeps the whole extensions it made, and the pass partial-renders within that heap, as it does after a refused firmware grow request. An idle growth still fails if the heap would stay below what the render's layout requires. After a growth stops short, the host grows beyond the required blocks again only a shrink period later (the existing busy retry, nowretry).memacct::LEDGER.admit()against the client's and the device's limits (defaults 50% and 67% of RAM) and counted as the client's held memory (fdinfodrm-resident-system,gpu_memory), as the M5 pools are. A refusal ends the growth there, as in 2. Pages are not added to a process'sMM_SHMEMPAGES, as for the M5 pools, because growth also runs from the firmware grow worker. memcg charging of the shmem pages is unchanged. The primary backing, sized by the render target, stays a kernel object.Policy
The normal maximum is a 64th of
si_meminfo()memory, clamped tofw::NORMAL_MAX_BLOCKS..=cfg::TVB_MAX_BLOCKS(1,118..=3,353 blocks). There is no rounding. The cap is per render pair; 1-3 bound what all pairs of a client and of the device hold together.BM+0x7c(the heap capacity, 3,354 slots) is unchanged. The memory ismemacct::total_ram_bytes(), which also sizes the memory limits.si_meminfo)BM+0x80stays0x45e * 4pages)The A18 Pro (8 GiB) keeps 1,118, and its mesh floor (
mesh_floor()) stays pinned tofw::NORMAL_MAX_BLOCKS. An M4 Max gets 3,353, the value #2021 sets for it by hand.Why a 64th
All three reasons come from our own measurements:
BM+0x7c= 19,737 andBM+0x80= 6,579 blocks (read live). That is 1/60 of the memory. The fixed 1,118 is 1/59 of 8 GiB. A 64th stays below both. Above the floor,BM+0x80is never a larger share of memory than the firmware is known to run with.Cleanups: the duplicate memory helper is gone, and
Layout::initialize()(M5 graph) now readsGraphArgs::normal_blocks, asfw::Graph::init()does.Measured on the J616s (M4 Pro, 48 GiB, GPU firmware 25G83)
The J616s runs the macOS 26.6.2 GPU firmware (
GPU firmware: macOS 26.6.2 (25G83, iBoot mBoot-18000.161.10)in the kernel log). All measurements here are from 25G83.The test is off-screen GLES: N screen-spanning triangles into a stored 3456x2234 RGBA8 + D24S8 target, waiting for each frame (
glFinish). There were 0 GPU recoveries. "Before" is a 1,118-block cap and "after" is a 3,353-block cap. Both were measured on our m4.166 test kernel with the earlier revision (#2186), which wrote the same 3,353 on this Mac. m4.166 also carries #1902 and #1986, which only affect pairs that never go idle.On this head (
001d7834eb78, on integration/aurora606d37ec77a5), no hardware run yet:defconfig+asahi.config,LLVM=1,W=1,drivers/gpu/drm/asahi. Clean; no warnings in the changed files.CONFIG_UCLAMP_TASK), 1 failed:apple-isp-hw.isp_test_sensor_sysfs, an ISP test outside this series.asahi_g17_tvb_limitspasses.tools/asahi/run-host-tests.py --only g17: 90/90.Pending: J616s boot with this head (next test kernel)
All GPU runs go through our GPU queue. 0 GPU recoveries, 0 AVC, 0 failed units and no new kernel warnings in every row.
BM+0x7c/BM+0x80of a live NEO heap: 13,416 / 13,412 pages (3,354 slots, 3,353 blocks).gpu_memoryand each client's fdinfodrm-resident-system: tiler growth now shows up per client. In the one-VM case, the third queue's growth stops at the arena edge, and its frames keep rendering (slower) without ENOSPC.gpu_mem_limit_pctlowered so the client limit bites.refusedrises, the heap stops short of 3,353, frames keep rendering, no warnings. The limit is restored afterwards.HEAP_SHRINK), and to its floor after 10 s idle.HW rows still wanted (machines we don't have)
BM+0x80read back as 8,192 pages (2,048 blocks), and rows 3 and 4 if possible. @RyanTheTide, could you run it?BM+0x80readback (4,472 pages, 1,118 blocks), plus 0 GPU recoveries / AVC / failed units, as you suggested. Series 1-3 also apply there, so a GLES run with a heavy scene would cover the charged growth.Not done
Assisted-by: LLM (Claude:Opus 5.5)