Repository navigation
drm/asahi: keep timestamp buffers out of the shared firmware range, and cap bound objects per file - #2531
drm/asahi: keep timestamp buffers out of the shared firmware range, and cap bound objects per file#2531funkzy wants to merge 2 commits into
Conversation
df120a7 to
d92d363
Compare
|
No blockers. What I checked at d92d363. On G17 the only consumer of a bound timestamp buffer is drivers/gpu/drm/asahi/g17/completion.rs:50-66. It goes through Nit: file.rs:1593-1602 checks the 4096 cap only after HW row: the run lists the M5 Ultra only. The Assisted-by: LLM (Claude:Opus 5.5) |
On M5, and on the other G17 GPUs once the 64 MiB timestamp window is full, GEM_BIND_OBJECT mapped a timestamp buffer into the kernel VM's dynamic range, the range every firmware object (queues, kick rings, render state) is allocated from. The driver never gives the firmware that address: outside the window the host writes every stamp through the buffer's CPU mapping. The mapping only held shared address space, as much as the bound range, for as long as the buffer stayed bound. Give such a buffer only its CPU mapping (timestamp::Buffer::Host), as G16 and G18 keep theirs without a firmware alias. It refuses the same ranges and VM-private objects the alias did. Buffers mapped in the window, and every other GPU family, are unchanged. Share the bound-range check with Buffer::cpu, and the VM-private test with gem::validate_vm_binding, and test the range check in a new asahi_timestamp_buffer KUnit suite. Reported-by: Erik Bourget <erik@ebourget.net> Suggested-by: Erik Bourget <erik@ebourget.net> Signed-off-by: Jeff Queisser <funkzy@users.noreply.github.com>
…bounded GEM_BIND_OBJECT stores a record in the file's object table for every bind, and the number of records per file was not limited. Each record is a small kernel allocation that is not charged to the client's memory limit, and a timestamp buffer also keeps a CPU mapping of its backing for as long as it is bound. Refuse a bind past 4096 objects in one file with EBUSY, as the table's allocator reports a full range. The bind takes its table slot before it maps the buffer, so a refused bind gets no window alias or CPU mapping at all. Mesa binds one object per GL context, one per timestamped query pool and one debug buffer, so 4096 leaves room for hundreds of pools. Allocate the record, the buffer it names and the table's nodes with GFP_KERNEL_ACCOUNT, so they count towards the memory cgroup of the task that binds. This applies to every GPU family and to every GEM_BIND_OBJECT usage: the table is shared code. Reported-by: Erik Bourget <erik@ebourget.net> Suggested-by: Erik Bourget <erik@ebourget.net> Suggested-by: Ryan Murray <ryan@aurorasilicon.org> Signed-off-by: Jeff Queisser <funkzy@users.noreply.github.com>
d92d363 to
60e5bda
Compare
|
Thanks, Ryan. The nit is fixed in the second commit, now at 60e5bda: J775D on this head (stacked with #1973 and our other three open drm/asahi PRs on integration/aurora cc14570, 27.0 firmware, default command line; the kernel also carries a local fix for an appledrm boot oops, see #1973):
Rows we can't run: we only have the M5 Ultra, where every bind takes the host path. Could your lab, or a J616s or Neo owner, run this head with the timestamp window filled (bind timestamp buffers until binds land outside it)? The row we need is |
|
Re-review of 60e5bda: No blockers.
The first commit is unchanged since the last review; the second changes only in HW row: the new numbers are from the M5 Ultra, which takes only the host path, on an integration stack with other open PRs and a local appledrm fix. Still needed: J616s or Neo on this head with the 64 MiB window filled, Assisted-by: LLM (Claude:Opus 5.5) |
DjDeveloperr
left a comment
There was a problem hiding this comment.
Adversarial pre-PR checklist review (§1–§6) on 60e5bdaae4cec3e495996a83d412b5bf8e3ef61b: 0 code, policy, attribution, naming, or CI blockers; holding for §5 hardware proof rows on non-M5 G17 and G13.
Verified against surrounding source:
- §1 No model gating:
Buffer::HostvsBuffer::Firmware(window: true) is selected by whether the buffer fits in the timestamp window (firmware.uat.is_m5()was pre-existing), and the 4096-object cap lives in shareddrivers/gpu/drm/asahi/file.rsacross every GPU family. - §2 / §3 Code & safety verification:
bound_size()(drivers/gpu/drm/asahi/timestamp.rs:16-22) enforces non-empty,UAT_PGMSK-aligned, in-bounds ranges with overflow checks and KUnit coverage (asahi_timestamp_buffer).Buffer::host()(timestamp.rs:72-83) refusesis_vm_private()withEINVAL, maps viaTimestampVmap::new(&bo.gem)->owned_vmap::<0>()(which refuses external/user_memoryBOs atrust/kernel/drm/gem/shmem.rs:261and pins shmem pages viadrm_gem_shmem_vmap_locked), and wraps the buffer incleanup::Deferredsodma_resv_lock-taking drop runs off fence-producer paths. Verifieddrivers/gpu/drm/asahi/mmu/residency.rs:481, 519, 615only walk VM-private BOs (for_each_private_bo), so omittingKernelMappingPinonHostBuffercannot race with idle reclaim or purge.Buffer::timestamp_offset()(timestamp.rs:153-166) checksoffset + 8 <= buffer.size(ERANGE/EOVERFLOW) and returnsbuffer.start + offset;window_va()returnsNoneandfirmware_mapping()/timestamp_mapping()returnErr(EINVAL). Verifieddrivers/gpu/drm/asahi/g17/completion.rs:45-65, 160-188routeswindow_va == Noneexclusively towrite_timestamp().do_gem_bind_object()(drivers/gpu/drm/asahi/file.rs:1595-1609) reserves the XArray slot (1..=4096,GFP_KERNEL_ACCOUNT) beforemap_timestamp_buffer(), so binds past 4096 fail withEBUSYbefore creating any window alias or vmap; failed mapping releases the slot onReservationdrop;resv.fill(obj)?unlocks the XArray before dropping any rejectedStoreError. Verifiedrust/kernel/xarray.rs:321, 558:remove()on an unfilled reservation coercesXA_ZERO_ENTRYtoNULL(ENOENT), and any concurrent race fails cleanly withEBUSY. (cleanup::Deferred::newatcleanup.rs:93still usesGFP_KERNELfor the work item wrapper, which is pre-existing, small, and now bounded by the 4096/file cap — non-blocking.)
- §4 Clean-room & attribution: Both commits (
a9d3d1658565,60e5bdaae4ce) carry human author/committer + humanSigned-off-by(plusReported-by/Suggested-by) and zero AI trailers; PR body includesAssisted-by:. - §6 Naming: Clean.
- CI:
host-testspassed (sole workflow matchingdrivers/gpu/drm/asahi/**).
Holding for §5 hardware proof rows on 60e5bdaae4ce:
Your J775D (M5 Ultra) run covers the host-only path, cap enforcement, KUnit, and render CTS, though on a stacked branch with other open PRs + a local appledrm fix, and M5 Ultra does not exercise the 64 MiB timestamp-window spillover path. Still needed:
- Non-M5 G17 (J616s or Neo) on this head with the 64 MiB timestamp window filled so binds spill into
Buffer::Host:dEQP-VK.query_pool.*timestamp*and a GLGL_TIMESTAMPquery test unchanged before/after, out-of-window binds succeeding with monotonic timestamps, KUnit pass counts, 0 failed units, and 0 new kernel errors. - One G13 Mac (M1/M2) running the same timestamp tests unchanged (covering the shared 4096-cap path), with clean regression counts.
Assisted-by: LLM (Gemini 4 Neon)
|
Thanks, dj. Here is the J775D row on this PR's own base: integration/aurora b21dcc6 plus this PR, and only what the board needs for a GPU, which is #1973 (the J775D GPU node). The baseline is the same kernel without this PR. 27.0 firmware, default command line, read-only root, one boot each.
The two CTS failures are the stage-stamp ordering cases ( On 27.2 firmware, on this branch's merge-base 61295c2, the cap, timestamp-buffer, CTS and GL rows came out the same, with and without this PR. Those kernels also need #1650 and two of our GPU-driver commits that aren't public yet, so the body lists them only as extra evidence, as it does the earlier stacked row. The spill path isn't exercised here. The M5 Ultra has no timestamp window, so all of its timestamp binds take the host-written path, which is what the rows above test. The body now has a "Why this shape" section, and its Testing section leads with this row. On RyanTheTide, could you run the two open rows: this head on your J616s or J700 with the timestamp window held full, and the same timestamp tests on your M1 Pro? We only have the M5 Ultra, which has no window. To hold the window: it is one 64 MiB range per device, so one process that keeps more than 64 MiB of timestamp binds open (for example five 16 MiB buffers bound with Assisted-by: LLM (Claude:Opus 5.5) |
Problem
On M5, and on the other G17 GPUs once the 64 MiB timestamp window is full,
GEM_BIND_OBJECTmapped a timestamp buffer into the kernel VM's dynamic range, the range every firmware object (queues, kick rings, render state) is allocated from. The driver never gave the firmware that address: outside the window the host writes every stamp through the buffer's CPU mapping. So the mapping only held shared address space for as long as the buffer stayed bound, and the number of objects a file could bind was not limited.Why this shape
map_timestamp_buffer(). G13/G14 (gpu.rs), G15 and M3 give it a firmware mapping, because their firmware writes the stamps. G16 and G18 keep only a CPU mapping (Buffer::cpu), because the host writes them. The separate M5 backend (m5/, T8142) refuses timestamp buffers (ENOTSUPP), so only the slot reservation below is new there. G17 maps into the 64 MiB window when the buffer fits. Otherwise, and always on the G17 code's M5 firmware graph (uat.is_m5()), it mapped the buffer into the kernel VM's dynamic range instead, although the host writes those stamps and the firmware is never given that address. The number of bound objects had no limit anywhere: the per-file object table infile.rs, which every family shares, accepted any number of slots.Change
GFP_KERNEL_ACCOUNT. Mesa binds one small buffer per GL context, one per timestamped query pool and one debug buffer, so 4096 leaves room for hundreds of query pools. The object table is shared code, so this applies to every GPU family.The host-written path is selected by whether the buffer fits the window, not by chip or firmware. Since RyanTheTide's review, the cap is checked before the mapping (second commit).
Testing
On this PR's own base (kit
pr2531-1, baseline kitpr2531-1b): J775D (Mac Studio, M5 Ultra), 27.0 firmware, default command line, read-only root, one boot each. The kernel is integration/aurora b21dcc6 (its head when the kits were made) plus this PR and only what the board needs for a GPU: #1973, the J775D GPU node (integration has none for this board yet). The baseline is the same kernel without this PR. Boot loader: Aura main 3efa4a81 with aura#407 and aura#408, both since merged.pr2531-1bandpr2531-1dEQP-VK.query_pool.*timestamp*GL_TIMESTAMP/GL_TIME_ELAPSED(3 runs of 10 rounds)asahi_timestamp_buffer1 of 1Without this PR the 223 binds fill the kernel VM's dynamic range down to a remainder smaller than one 256 MiB bind; a queue's firmware objects are small and still fit there, so the second client's queue creation succeeds, while the next timestamp bind does not.
Regression run on those kits (ALARM has no SELinux, so there is no AVC count): 0 new failed units (iwd.service and systemd-networkd-persistent-storage.service fail on the read-only root in both boots) and 0 new GPU or kernel errors. The only kernel lines that differ between the two boots are USB enumeration retries, two in the boot without this PR and one in the boot with it, all before any test ran; the other 7 error lines are the same in both.
Not exercised here: the spill from a full 64 MiB window to the host-written path. The M5 Ultra has no timestamp window, so every timestamp bind on it takes the host-written path (the rows above), and filling the window needs a non-M5 G17.
Extra evidence, not this PR's own row: the same tests on 27.2 firmware and on the earlier stacked kernel, in the details below.
Still to run
dEQP-VK.query_pool.*timestamp*and a GLGL_TIMESTAMPquery test with the same results before and after, binds outside the window succeeding and their stamps read back monotonic, KUnit pass counts per suite, 0 failed units and 0 new kernel errors.Full description of the two commits
timestamp::Buffer::Host: it keeps a CPU mapping of the bound range and no firmware alias. The host writes its stamps through that mapping (the only consumer is the G17 completion path, which takes object offsets). It refuses the same ranges and VM-private objects the alias did. The bound-range check is shared withBuffer::cpuand the VM-private test withgem::validate_vm_binding; a new KUnit suite,asahi_timestamp_buffer, tests the range check.GEM_BIND_OBJECTreserves a slot in the file's object table (up to 4096) first, then maps the buffer, then fills the slot; a bind past the cap fails with EBUSY before any mapping exists, and a failed mapping releases the slot. The record, the buffer it names and the table's nodes are allocated withGFP_KERNEL_ACCOUNT.Full test results
Own base, 27.0 firmware (kit
pr2531-1: integration/aurora b21dcc6 + #1973 + this PR; baseline kitpr2531-1b: the same without this PR), Mac Studio (M5 Ultra), read-only root, one boot each, 2026-11-22:timestamporcalibrat, 1149 in all:pipeline.monolithic.timestamp(278),multiview.*queries*_and_timestamps*(144),wsi.*.present_timing(54), one API case, and the graphics timestamp groups of the other six pipeline variants (672). Both kernels: 1093 Pass, 2 Fail, 54 NotSupported, with the same two failures. The two failures arepipeline.monolithic.timestamp.misc_tests.timestamp_stage_top_of_pipe_stageandtimestamp_stage_host_stage(the stage stamp reads 0.8-3.3 us before the preceding guard stamp), the same on both kernels, so they are not from this change. The 54 NotSupported are the present-timing cases, which need a window system the test boot lacks. This mustpass has noquery_pooltimestamp group: of its 19276query_poolcases none names a timestamp, anddeqp-vk --deqp-case='dEQP-VK.query_pool.*timestamp*'selects 0 cases.GL_TIME_ELAPSEDandGL_TIMESTAMP; 10 rounds of aGL_TIME_ELAPSEDquery around 1 to 512 full-target triangles withGL_TIMESTAMPqueries before and after. In every round the stamps increase and the elapsed time lies inside the two stamps' difference; 3 runs, 30 of 30 rounds pass on both kernels (512 triangles: 1.27-1.45 ms).apple_g17_rx_util_min_test, which needsCONFIG_UCLAMP_TASK, andj713_accepts_all_loaded_images); without it 76 suites, 375 of 377, the same two skips. The one new suite isasahi_timestamp_buffer, 1 of 1; every other suite's counts are the same.Extra evidence, not this PR's own row: 27.2 firmware (kits
o2531pando2531b, on integration/aurora 61295c2, this branch's merge-base; boot loader Aura main 3efa4a81 with aura#407 and aura#408, both since merged). Besides #1973's binding and GPU node, these kits carry sithglan's #1650 (rolotrealanis98's #1173, the 27.2 firmware initialisation data) and two GPU-driver commits of ours that are not public yet and that the J775D needs on 27.2: 11b7aff60364 "drm/asahi: t6050: Run the Mac Studio's (M5 Ultra) GPU under the 27.2 boot stub: admit its 26B123 G17D firmware" and 2f4c1c6028c4 "drm/asahi: t6050: Give the M5 Ultra's 27.2 firmware the power-controller status config of its own 26B123 layout". With and without this PR: the same cap (4096 then EBUSY; 4112 without), timestamp-buffer (256 of 256 at +0 MiB; 223 then ENOSPC without), timestamp CTS (1093 Pass, 2 Fail, 54 NotSupported, identical case by case), GL (30 of 30), render CTS (571/57) and failed-unit results. The differences from the 27.0 rows: the two CTS failures' stamps were 0.5-1.9 us early (0.8-3.3 us on 27.0); KUnit had fewer suites on this older base, 73 suites and 354 of 356 with the PR against 72 and 353 of 355 without (the PR adds the same one suite); and the boot with this PR logged one USB enumeration retry at 6 s, before any test.Extra evidence, stacked kernel (kit
rr21r, the earlier row: integration/aurora cc14570 + the appledrm fix since merged as #2733 + #1973 + #2530 + this PR + #2501 + #2520, 27.0 firmware; baseline cc14570 + the appledrm fix + #1973): cap 4096 of 4112 then EBUSY; 256 of 256 timestamp binds at +0 MiB, a second client's QUEUE_CREATE succeeding before and after close; render CTS subset 571 Pass, 57 NotSupported with and without the series; KUnit 79 suites, 366 of 368, against 75 suites and 353 of 355; host tests 436 of 436 on base and head; no new error line beyond the intended ENOMEM refusals of #2501's render tests in the same boot.Thanks to ewb, who reported the unbounded binds and suggested the fix, and to RyanTheTide for the review.
Assisted-by: LLM (Claude:Opus 5.5)