Repository navigation
Conversation
7c99f6e to
22b9bb2
Compare
|
No blockers. Checked the ledger split (
HW row: one G13 or G14 Mac (M1/M2) on this head: Assisted-by: LLM (Claude:Opus 5.5) |
A client VM's page tables are allocated as the client maps and kept until the VM is destroyed, but they were not counted by the GPU memory limits, not shown in the client's fdinfo and not charged to any memory cgroup. Because sparse and single-page binds map a small object over a large range, the table memory a client causes was not bounded by the memory it holds. Charge the tables to the client that created the VM, as a held user buffer of the same size would be. The root and the tables that exist when the VM gets its client are counted at once. Each later table is admitted against the client's and the device's memory limits once it is allocated, after every other step that can fail, so a table that is not installed leaves no charge behind. A refusal frees the table and fails the bind or kernel mapping that needed it with ENOMEM. From then on the tables are also allocated with GFP_KERNEL_ACCOUNT, so they count towards the memory cgroup of the task that maps. The charge is released when the page table is freed with the VM. Kernel VMs are not charged. Legitimate use stays far below the limits: the Borealis Vulkan driver's sparse address space on M5 is 16 GiB per device, 8 MiB of tables (16 MiB with its read-only shadow). This applies to the client VMs of every GPU family. Reported-by: Erik Bourget <erik@ebourget.net> Suggested-by: Erik Bourget <erik@ebourget.net> Suggested-by: Ryan Murray <ryan@aurorasilicon.org> Signed-off-by: Jeff Queisser <funkzy@users.noreply.github.com>
…es them Each mapping in a VM is recorded in a GPUVA node until it is unmapped, and those nodes were not charged anywhere. Allocate the nodes that binds and unbinds use with GFP_KERNEL_ACCOUNT, so they count towards the memory cgroup of the task that asks for the operation. This applies to every GPU family. Reported-by: Erik Bourget <erik@ebourget.net> Suggested-by: Erik Bourget <erik@ebourget.net> Signed-off-by: Jeff Queisser <funkzy@users.noreply.github.com>
Besides its page tables, each VM holds kernel memory that is not charged to its client: its GPUVM, its kernel allocators and per-VM objects. The number of VMs one file could create was not limited. Refuse VM_CREATE past 64 VMs in one file with EBUSY, as the ID allocator reports a full range. Mesa creates one VM per file. The cap is per file, as the existing memory limits are: a client that opens the device again gets another 64 VMs in the new file, each file accounted on its own. This applies to every GPU family. Reported-by: Erik Bourget <erik@ebourget.net> Suggested-by: Erik Bourget <erik@ebourget.net> Suggested-by: Ryan Murray <ryan@aurorasilicon.org> Signed-off-by: Jeff Queisser <funkzy@users.noreply.github.com>
22b9bb2 to
6386898
Compare
|
Thanks, Ryan. Both points are fixed in the commits, now at 6386898.
J775D on this head (stacked with #1973 and our other three open drm/asahi PRs on integration/aurora cc14570, 27.0 firmware, default command line; the kernel also carries a local fix for an appledrm boot oops, see #1973):
Rows we can't run: could your lab, or an M1 or M2 owner, run this head on a G13 or G14 Mac? The useful row is |
|
Re-review of 6386898: No blockers.
Nothing else changed since the last review (second commit unchanged, third only in its message). Trailers are a human Signed-off-by plus Suggested-by only. HW row: still only the M5 Ultra, on an integration stack with other open PRs and a local appledrm fix. Needed: one G13 or G14 Mac (M1/M2) on this head alone, Assisted-by: LLM (Claude:Opus 5.5) |
DjDeveloperr
left a comment
There was a problem hiding this comment.
Pre-PR checklist review (§1–§6) on live head 63868987c8536c7612d1c4cdf198c39ad37a8b50 (host-tests: pass).
Checklist Summary
- §1 No model gating: Pass — shared
file.rs,memacct.rs,mmu.rs,mmu/bind.rs, andpgtable.rspaths apply uniformly across all GPU families with zero chip/SoC/board/firmware gates. - §2 Production-ready, no hacks: Pass — works on default kernel command line, no new module parameters, no sleeps/retries/lab-only paths.
- §3 Safety: Pass — no hardcoded hardware safety constants;
MAX_VMS_PER_FILE = 64is a per-file DoS/resource bound matching the existing per-file memory-limit model. - §4 Clean-room & attribution: Pass — all 3 commits (
7d82a3723bc8,62174055d25f,63868987c853) carry human author/committer and humanSigned-off-by/Reported-by/Suggested-bytrailers only; PR body includesAssisted-by:. - §6 Naming: Pass — uses Borealis and Aurora naming.
- §5 Evidence: Holding for cross-family hardware proof.
Code Audit (0 blockers)
memacct.rs:Ledger::reserve()split fromadmit()preserves rollback-on-device-limit-failure semantics;Usage::{new, try_add, add, hold}andDropkeepclient.{committed,resident}andledger.{user_committed,user_resident}balanced.pgtable.rs:UatPageTable::with_pages()allocates withGFP_KERNEL_ACCOUNTonce charged, runsclean_pocanddriver_tables.reserve(1, GFP_KERNEL)beforecharge.try_add(UAT_PGSZ as u64), and only then records viapush_within_capacityand incrementstables. Any failure frees the page with no phantom charge. Verifiedfree_table()is only reachable fromUatPageTable::drop(). KUnitasahi_pgtable_chargecovers normal lifecycle, client-limit refusal, and borrowed-root refusal.mmu.rs&mmu/bind.rs: Verified all threeGpuVaAlloc::newallocation sites useGFP_KERNEL_ACCOUNT.file.rs: Verifiedvm_xa.lock().reserve_limit(1..=MAX_VMS_PER_FILE, GFP_KERNEL)returns-EBUSY(__xa_allocinlib/xarray.c) when full, andfailed_vmcleans up the newly createdvm(releasing its initial root-table charge) on refusal.
Holding For (§5)
0 code/policy/attribution/naming/CI blockers. Holding approval solely for the missing cross-family hardware row on this head (this change applies to all GPU families, and the J775D run was stacked with #1973 and other open PRs):
- One non-M5 Mac (e.g. G13/G14 M1 or M2) running this head:
dEQP-VK.sparse_resources.*and render CTS subset matching baseline, fdinfo sparse page-table charge and release onVM_DESTROY, 0 SELinux denials (if enabled), 0 failed units, and 0 new kernel errors.
Assisted-by: LLM (Gemini 4 Neon)
Problem
A client VM's page tables, and the GPUVA nodes that record its mappings, were not counted by the GPU memory limits, not shown in the client's fdinfo and not charged to any memory cgroup, and the number of VMs one file could create was not limited. Sparse and single-page binds map a small object over a large range, so the table memory a client causes was not bounded by the memory it holds.
Change
Three commits in
drivers/gpu/drm/asahi: page tables are charged to the client that created the VM (admitted against its limits once allocated, after every other step that can fail, refusals fail the bind with ENOMEM, allocated withGFP_KERNEL_ACCOUNT); GPUVA nodes are allocated withGFP_KERNEL_ACCOUNT; and VM_CREATE past 64 VMs in one file fails with EBUSY. Mesa creates one VM per file, and Borealis's sparse space on M5 needs 8 MiB of tables (16 MiB with its read-only shadow).It applies to the client VMs of every GPU family; there is no chip, board or firmware condition. The 64-VM cap is a new per-file limit, like the existing memory limits, a policy bound rather than a hardware value (see the questions below).
Since RyanTheTide's review: a table is now charged only after the last step that can fail, so a failed table leaves no charge behind. The first commit has a new KUnit case for the refusal on a root that records its own tables. The third commit's message says the cap is per file.
Testing
On the J775D (Mac Studio, M5 Ultra), 27.0 firmware, integration/aurora cc14570 with #1973 and our other three open drm/asahi PRs, default command line, one boot (the kernel also carries a local fix for an appledrm oops at boot on current integration, not part of this PR). A bind of one page over 1 TiB is charged to the client in fdinfo, the GPU memory counters and its memory cgroup (512.2 MiB each, the size of the tables) until the VM is destroyed, then all three return to their starting values; a sparse-tile bind reads the same. The 65th VM in one file is refused with EBUSY. The render CTS subset gives 571 Pass and 57 NotSupported, as on the same base without the series. KUnit (366 of 368 pass, two skips as on the base) and the upstream host tests (436 of 436 on base and head, no result differs) pass.
Regression run (ALARM has no SELinux, so there is no AVC count): 0 new failed units (iwd.service and systemd-networkd-persistent-storage.service fail on the read-only test root with and without the series) and 0 new kernel error lines against the boot without the series.
Still to run
dEQP-VK.sparse_resources.*and the render CTS subset with the same results as the base, fdinfo showing the sparse table charge and its release, 0 new kernel errors and 0 failed units. Rows from RyanTheTide's lab or an M1/M2 owner welcome.Full description of the three commits
GFP_KERNEL_ACCOUNT. The charge is released when the VM is destroyed. Kernel VMs are not charged.GFP_KERNEL_ACCOUNT.Legitimate use stays far below the limits: Borealis's sparse address space on M5 is 16 GiB per device (8 MiB of tables, 16 MiB with its read-only shadow).
This applies to every GPU family. On non-M5 parts Borealis advertises a 512 GiB sparse space, which with its shadow would be charged up to 512 MiB of tables if filled completely.
Full test results
On the Mac Studio (M5 Ultra), 27.0 firmware, read-only test root, with and without the series (the boot without it is integration/aurora cc14570 with #1973):
SPARSE_TILE: +512.2, +512.2 and +513.0 MiB, then back to within 1.1 MiB.asahi_pgtable_charge3 of 3, including the newa_refused_table_is_neither_charged_nor_recorded.Questions
Thanks to ewb, who reported the uncharged page-table memory and suggested the charge, and to RyanTheTide for the review.
Assisted-by: LLM (Claude:Opus 5.5)