Skip to content

mm: populate a subpage window's uffd-wp markers with the faulting page - #2225

Merged
RyanTheTide merged 2 commits into
aurora-silicon:integration/aurorafrom
jgoyvaerts:m4pro/mm-subpage-uffd-wp
Nov 22, 2026
Merged

RyanTheTide merged 2 commits into
aurora-silicon:integration/aurorafrom
jgoyvaerts:m4pro/mm-subpage-uffd-wp

Conversation

@jgoyvaerts

@jgoyvaerts jgoyvaerts commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

In an mm whose user page size is smaller than the native page (CONFIG_MM_SUBPAGE: 4K processes on a 16K kernel, such as everything running under FEX), a write fault on unpopulated anonymous memory fills the whole native window from one fresh native page. Each 4K page lands at its own offset, so the window is physically in VA order.

A uffd-wp marker (UFFD_FEATURE_WP_UNPOPULATED) is unpopulated memory too, but anon_subpage_missing_mask() counted only none PTEs as missing.

  • Effect: in a VMA write-protected before its first touch, every neighbour holds a marker, so each first write took a native page of its own for one 4K page. That is four times the memory, and no window was ever one native page in order.
  • Who it hurts: a device that maps native pages can then use none of it. One example is a GPU with 16K pages taking user memory through GEM_USERPTR. A write-watch built on PAGEMAP_SCAN sets up exactly this kind of memory; we hit it with one under FEX.

This change counts a uffd-wp marker in a uffd-wp VMA as missing, and gives the neighbour it stood for the uffd-wp bit, as the faulting page already gets from its own marker. The populated neighbours stay write-protected and unwritten for PAGEMAP_SCAN. Markers in a VMA that is no longer registered are left as they were. Only a marker that is exactly PTE_MARKER_UFFD_WP counts: handle_pte_marker() ranks poison and guard above uffd-wp, so a slot that carries either keeps faulting as before.

The populated neighbours are present: they count in RSS and report PAGE_IS_PRESENT, and a UFFDIO_COPY into one in a WP-only VMA returns -EEXIST where upstream would have filled the marker. Both are already true for none neighbours on this path.

The second commit adds a case to tools/testing/selftests/arm64/user4k/uffd-wp-init.c. The existing markers() case reads before it writes, so its first touch maps the zero page and never reaches this path. The new case covers a native-aligned window, write-protected with WP_ASYNC | WP_UNPOPULATED before any touch, then written once in its second 4K page:

  • it checks that all four 4K pages are present in one native page, in order;
  • it checks that the three neighbours carry the uffd-wp bit and the written page does not;
  • it then writes page 0 and checks that only page 0's bit clears (pages 2 and 3 keep theirs) and that the physical layout is unchanged.

Before/after (qemu, 16K kernel, user4k=on)

The same static uffd-wp-init binary as /init, on the same kernel config (arm64_user4k.config + USERFAULTFD, NET, UNIX, MEMCG, FAILSLAB), KVM, -cpu max -smp 4.

kernel 16K worker 4K worker result
base 0abc2f3 (no fix) 13/13 + status ok first 6 cases ok, then not ok - uffd line=120 UFFD CONTRACT FAIL
head 7655e71 13/13 + status ok 14/14 + status ok UFFD CONTRACT PASS, 29/29

On base, line 120 is the present-bit check in physical(), reached from start = physical(a, ps) on page 0 of marker_window: page 0 is still a marker. Base output, 4K worker:

ok - uffd ABI 4K unpopulated WP async=1 per-leaf pagemap state and close
ok - uffd ABI 4K poison: SIGBUS, MADV_FREE recovery, neighbours
not ok - uffd line=120 errno=17
not ok - uffd ABI 4K status=0x100
UFFD CONTRACT FAIL

Head output, 4K worker:

ok - uffd ABI 4K poison: SIGBUS, MADV_FREE recovery, neighbours
ok - uffd ABI 4K unpopulated WP first write: one native page, neighbours WP
ok - uffd target 4K remote pager 16K shared=0 SCM_RIGHTS copy/zero/batch uses target granule
...
ok - uffd ABI 4K status=0
UFFD CONTRACT PASS

Regression check, same qemu setup, base vs head

All of these give identical results on base and head:

  • KUnit arm64-user4k* (user4k=on, no arm64.user4k_test=1): 226 tests, 116 passed, 109 skipped, 1 failed. The failure is user4k_move_migration_race_test, the pre-existing one already reported (drm/asahi: GEM_USERPTR in 4K (user4k) processes #2114). The IOMMUFD cases that stall need the opt-in, so they were not run (pre-existing, see the comment above).
  • lazyfree-fixture-init: 8/8 in each granule.
  • In a 4K process: anon-hole-refill --native-16k 8/8, anon-hole-refill-race 3/3 (with swap), clipped-firstwrite --native-16k PASS, first-write --native-16k 4 pass and 2 skip (regular-file arms skip on tmpfs), and cow-pool 7 pass and 2 skip (memlock limit). anon-hole-refill-reclaim skips (needs --external-reclaim).
  • uffd-move-init: 12 ok on both, then the same not ok - MOVE line=180 errno=16 (EBUSY) in both granules on base too. This is a qemu environment or pre-existing issue, not this change.

mm/memory.o builds with W=1 and no warnings. checkpatch reports 0 errors and 0 warnings on both commits.

Tested on the J616s (M4 Pro, 16K kernel), earlier revision

  • Probe in a 4K process: 8 MiB of anonymous memory, write-protected through userfaultfd with WP_ASYNC | WP_UNPOPULATED before its first touch, then written once per 4K page. Before, 0 of 512 16K blocks were one native page in order; after, 512 of 512. Memory registered after its first touch, or never registered, gave 512 of 512 either way. PAGEMAP_SCAN's written/unwritten report stayed exact, native and 4K.
  • In use: this has run in our daily kernels since 11-16. That revision tested the UFFD_WP bit instead of the exact marker, which is the same for every marker the kernel creates today.

Not done

  • Read-before-write memory: memory that is read before it is written is unchanged. The read maps the zero page, and the copy on write lands in any free COW-pool slot.
  • The exact-marker hardening has no test: nothing creates a guard or poison marker combined with uffd-wp today.

Assisted-by: LLM (Claude:Opus 5.5)

@jgoyvaerts

Copy link
Copy Markdown
Contributor Author

Base control for the full arm64-user4k stall reported on #2114 (the same mm change on base 677be39, head 4c15fea), carried over here as noted when #2114 was closed. @DjDeveloperr asked for it there.

Setup: arm64_user4k.config + USERFAULTFD=y, booted with user4k=on arm64.user4k_test=1, KVM qemu with 1 vCPU, each run capped at 1500 s. Without arm64.user4k_test=1 the IOMMUFD tests skip, and both SHAs finish identically in about 1 s.

Results:

  • Bare base 677be39: stalls in user4k_iommufd_lifetime_test in 4 of 4 runs.
  • drm/asahi: GEM_USERPTR in 4K (user4k) processes #2114's head: stalls in the same test with the same signature. Tests 1–60 give identical results to base, with 0 failures.
  • The mm commit alone (695bc5e): behaves like base.
  • Which granule case hangs (4K or 64K) varies between runs of the same binary.

Signature: the test kthread loops forever on a level-2 write translation fault at TEST_VA in the first copy_to_user() after vm_mmap() (ESR 0x96000046, FAR 0x400000). After KUnit's per-test timeout, it then blocks in kthread_stop(), so user4k_iommufd_rollback_test never runs.

Why this change isn't involved:

So the stall is pre-existing and independent of #2225/#2226.

user4k_move_migration_race_test (arm64-user4k-nonlinear-swap) also fails on both SHAs, with moves 1 == initial 1. That failure is pre-existing too.

@DjDeveloperr DjDeveloperr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed as core mm. The change is right and contained:

  • It only reaches do_anonymous_subpage(). With no VM_UFFD_WP the mask and the PTEs written are identical to base: every non-target slot in current_mask is then pte_none, so the new marker test is false. Native-16K processes and non-uffd users see no change.
  • The PTL-held recompute (mm/memory.c:6944) and the per-neighbour marker test in the set loop (6972-6973) both read under the PTL, so they agree. A racing population only shrinks current_mask, and the dense-path abort is skipped for armed VMAs, so nothing is lost.
  • arm64 pte_mkuffd() write-protects, so a populated neighbour behaves like the marker it replaced: a WP fault in sync mode, a bit clear in async mode, and the marker comes back on a zap.

The stall control you posted settles #2114's question: base stalls 4/4, and the mm hunk here is byte-identical to 695bc5e. By construction this cannot reach a test that registers no userfaultfd.

Before merge, please post the negative run for the new case. Same uffd-wp-init binary, 4K worker mode, on a 16K kernel without 834e0e6 (0abc2f3 or a fleet kernel without the fix). I expect marker_window to fail at physical(a, ps) on page 0 with the earlier cases passing, then 29/29 on this head.

Two small things, neither blocking:

  1. Match the marker exactly (softleaf_to_marker(...) == PTE_MARKER_UFFD_WP) rather than "has the UFFD_WP bit". handle_pte_marker() ranks GUARD and POISONED above UFFD_WP, and this path would turn such a slot into a zero page. Nothing creates the combination today, so this is hardening.
  2. One sentence in the commit message: neighbours are now present (RSS, PAGE_IS_PRESENT), and a UFFDIO_COPY into one in a WP-only VMA returns -EEXIST where upstream would accept it. Both are already true for none neighbours on this path.

Optional for the test: after the first write, write a[0] and check that its bit 57 clears, pages 2 and 3 keep theirs, and physical() is unchanged.

@RyanTheTide

Copy link
Copy Markdown
Contributor

Thanks, the change reads right as core mm: it only reaches do_anonymous_subpage(), and without VM_UFFD_WP the PTEs written are the same as before.

One request before merge: could you add the uffd-wp case as a test that fails on a kernel without this fix (a subpage window registered for write-protect, first touch on one subpage, then check that the other subpages still report the marker)? A selftest under tools/testing/selftests/mm/ or a small reproducer in the PR body is fine, as long as it shows the before/after.

In an mm whose user page size is smaller than the native page, a write
fault on unpopulated anonymous memory fills the whole native window
from one fresh native page, each 4K page at its own offset. That keeps
the window physically in VA order, and costs one native page for four
user pages.

A uffd-wp marker (UFFD_FEATURE_WP_UNPOPULATED) is unpopulated memory
too, but anon_subpage_missing_mask() counted only none PTEs as missing.
In a VMA write-protected before its first touch, every neighbour holds
a marker, so each first write took a native page of its own for one
4K page: four times the memory, and no window was ever one native page
in order. A device that maps native pages (a GPU with 16K pages taking
user memory through GEM_USERPTR) can then use none of it.

Count a uffd-wp marker in a uffd-wp VMA as missing, and give the
neighbour it stood for the uffd-wp bit, as the faulting page already
gets it from its own marker: the populated neighbours stay write
protected and unwritten for PAGEMAP_SCAN. Markers in a VMA that is no
longer registered are left as they were. Only a marker that is exactly
PTE_MARKER_UFFD_WP counts: handle_pte_marker() ranks poison and guard
above uffd-wp, so a slot carrying either keeps faulting as before.

The populated neighbours are present: they count in RSS and report
PAGE_IS_PRESENT, and a UFFDIO_COPY into one in a WP-only VMA returns
-EEXIST where it would have filled the marker. Both were already true
for none neighbours on this path.

Measured with a probe in a 4K process on an M4 Pro (16K kernel): 8 MiB
of anonymous memory, write-protected through userfaultfd with
WP_ASYNC | WP_UNPOPULATED before its first touch, then written once per
4K page. Before, 0 of 512 16K blocks were one native page in order;
memory registered after its first touch, or never registered, gave
512 of 512.

Signed-off-by: Jonathan Goyvaerts <jonathan.goyvaerts@gmail.com>
Add a uffd-wp-init case that write-protects a native-aligned window with
WP_ASYNC | WP_UNPOPULATED before any touch, then writes one byte to its
second user page. pagemap must then show every user page of the window
present in one native page in VA order, with the uffd-wp bit kept on the
neighbours and cleared on the written page. A second write, to the first
user page, must then clear only that page's bit and leave the window's
physical layout unchanged. markers() never reached this path: its
window is not native-aligned, and its first touch is a read that maps
the zero page, so its write is a copy-on-write fault.

Signed-off-by: Jonathan Goyvaerts <jonathan.goyvaerts@gmail.com>
@jgoyvaerts
jgoyvaerts force-pushed the m4pro/mm-subpage-uffd-wp branch from c139961 to 7655e71 Compare November 20, 2026 05:18
@jgoyvaerts

Copy link
Copy Markdown
Contributor Author

Updated to 7655e71 (force-pushed, same two commits, base 0abc2f3).

Negative run: the same static uffd-wp-init binary as /init, the same kernel config on both kernels (arm64_user4k.config + USERFAULTFD/NET/UNIX), qemu KVM with user4k=on:

  • base 0abc2f3: the 16K worker passes 13/13. In the 4K worker the first 6 cases pass, then not ok - uffd line=120 errno=17: that is the present-bit check in physical(), from start = physical(a, ps) on page 0 of marker_window. Result: UFFD CONTRACT FAIL.
  • head: 29/29, UFFD CONTRACT PASS.

The exact output is in the PR body. @RyanTheTide, this covers your request: the case is in tools/testing/selftests/arm64/user4k/uffd-wp-init.c rather than selftests/mm, because it needs the 4K-on-16K worker that harness provides, and the PR body has its before/after output.

1. Exact marker match: new pte_is_only_uffd_wp_marker() (softleaf_is_marker() && softleaf_to_marker() == PTE_MARKER_UFFD_WP, mm/memory.c:6842). It is used both in the mask (6864), which covers the PTL-held recompute, and in the per-neighbour test in the set loop (6981). A slot that carries a guard or poison marker together with uffd-wp is no longer counted as missing, so it keeps faulting as before.

2. Commit message: added a paragraph. The populated neighbours are present (RSS, PAGE_IS_PRESENT), and UFFDIO_COPY into one in a WP-only VMA returns -EEXIST where it would have filled the marker. Both were already true for none neighbours on this path.

Optional test extension: done (uffd-wp-init.c:470). After the first write, a[0] is written. The test then checks that page 0's bit 57 clears, pages 2 and 3 keep theirs, and physical() is unchanged for all four pages. It is included in the head's 29/29.

No regressions: base and head give identical results in the same qemu setup:

  • KUnit arm64-user4k* (no arm64.user4k_test=1, so not the iommufd stall): 226 tests, 116 pass, 109 skip, and the 1 known user4k_move_migration_race_test failure.
  • lazyfree-fixture-init: 8/8 per granule.
  • In a 4K process: anon-hole-refill, anon-hole-refill-race, clipped-firstwrite, first-write and cow-pool all pass.
  • uffd-move-init: the same MOVE line=180 EBUSY on both kernels.

checkpatch reports 0 errors and 0 warnings on both commits, and mm/memory.o builds clean with W=1.

@RyanTheTide

Copy link
Copy Markdown
Contributor

Thanks, the before/after is exactly what we asked for: the 4K worker fails UFFD CONTRACT on the base at the present-bit check and passes 29/29 on this head, with the same binary and config on both. The exact-marker helper and the unchanged KUnit and fixture results answer the remaining questions on our side.

@jgoyvaerts
jgoyvaerts force-pushed the m4pro/mm-subpage-uffd-wp branch from 7655e71 to 6643e60 Compare November 21, 2026 06:14
@jgoyvaerts

Copy link
Copy Markdown
Contributor Author

Commit messages only, for the pre-PR checklist: the commits no longer carry Assisted-by (item 4), and the disclosure is now the PR body's last line. The tree is identical to 7655e71689 (git diff 7655e71689 6643e60c5b is empty). New head 6643e60c5b.

Assisted-by: LLM (Claude:Opus 5.5)

@DjDeveloperr DjDeveloperr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review of head 6643e60c5bfd85ff319f17dfa06d3d7371b94d1c against the pre-PR checklist (following our earlier comment on c139961).

Resolution of earlier items

  • Exact marker match: pte_is_only_uffd_wp_marker() (mm/memory.c:6842) checks softleaf_is_marker(entry) && softleaf_to_marker(entry) == PTE_MARKER_UFFD_WP in both anon_subpage_missing_mask() (pre-PTL snapshot and PTL-held recompute) and the per-neighbour set loop (mm/memory.c:6980-6982). Guard/poison markers are preserved, and softleaf_from_pte() returns none for present/none PTEs.
  • Commit message: 6ac4b1169a2f now documents RSS / PAGE_IS_PRESENT and UFFDIO_COPY -EEXIST semantics on populated neighbours.
  • Selftest & negative control: marker_window() (tools/testing/selftests/arm64/user4k/uffd-wp-init.c:443-479) now also checks the second write to a[0] clears only page 0's bit 57 ((i > ps)) while keeping the native physical layout intact. The posted KVM arm64 16K-kernel user4k=on run reproduces the exact expected failure on base 0abc2f385e54 (line=120 errno=17, UFFD CONTRACT FAIL) and passes 29/29 (UFFD CONTRACT PASS) on this tree (7655e71689 -> 6643e60c5b, commit messages only), with identical KUnit arm64-user4k* and user4k selftest regression results, backed by the J616s 512/512 + PAGEMAP_SCAN bare-metal probe.

Checklist & CI

  • 0 blockers. CI (host-tests) is green.
  • §1–§4 & §6: Core mm/ change with no model gating or hacks; both commits carry human author/committer and human Signed-off-by: only, with PR-body disclosure present.

Assisted-by: LLM (Gemini 4 Neon)

@RyanTheTide

Copy link
Copy Markdown
Contributor

Merge-train adversarial review (MERGETRAIN-LINUX2) at 6643e60: NO BLOCKERS. With Dj's approval at this head, that makes the joint verdict MERGE. Before the merge we will boot it on our Macs, as one test kernel of integration 24dc422 plus #2222 and #2225: M1 Pro, M2, J616s, Neo and the J615. Results will follow here.

Full review

#2225 mm: populate a subpage window's uffd-wp markers with the faulting page @6643e60c

VERDICT: NO BLOCKERS

Commits: 6ac4b11 (mm), 6643e60 (selftest). Line numbers are in mm/memory.c at 6643e60.

What I checked and found to hold:

  • Scope: only do_anonymous_subpage() is affected, which is reached from do_anonymous_page() write faults when mm_page_size < PAGE_SIZE (:7077-7078). With no VM_UFFD_WP, uffd_wp (:6880) is false and the mask at :6864 is the same as on base. In the set loop at :6980-6981, every non-target bit of current_mask is then a none PTE, so pte_is_only_uffd_wp_marker() is false and the PTEs written are identical to base. Native-16K processes are untouched.
  • MISSING mode: userfaultfd_missing(vma) (:6942) still returns to userspace before anything is populated, so a MISSING+WP VMA never has its marker neighbours filled behind its back.
  • Marker test: pte_is_only_uffd_wp_marker() (:6842) is safe on present and none PTEs, because softleaf_from_pte() returns none for both (include/linux/leafops.h:55). The exact == PTE_MARKER_UFFD_WP match keeps guard and poison combinations on their handle_pte_marker() path (memory.c:5693).
  • Locking and races: the PTL-held recompute (:6950-6952) and the per-slot marker read (:6981) both run under the PTL, so they agree. A neighbour that changes from marker to present between the snapshot and the PTL only shrinks current_mask. Its unused slot is put at release. A neighbour that changes to a marker adds a bit and aborts through current_mask & ~mask. userfaultfd_wp(vma) is stable under the fault's mmap or VMA lock (unregister needs the mmap write lock).
  • WP semantics: arm64 pte_mkuffd() write-protects (arch/arm64/include/asm/pgtable.h:353-356). A populated neighbour therefore raises a WP fault in sync mode and clears its bit in async mode, as the replaced marker would have. PAGEMAP_SCAN PAGE_IS_WRITTEN stays false for neighbours. Accounting is right: markers were not in RSS, and add_mm_counter(..., mapped) covers the newly present PTEs. Because the VMA is armed, the pool is never used (:6907), so the window is a fresh native folio at the per-offset slots, which is what the test asserts.
  • Selftest: marker_window() (uffd-wp-init.c:443) runs only in the 4K worker (ps < native, :823-824). The bit-57 expectations at :467 and :473 match the fault sequence: the target gets its bit from its own marker, and the retried write clears it in async mode. The PR body records the fail on base and the pass on head.

Blockers: none.

Nits:

  1. For anon VMAs, a zap does not reinstall a uffd-wp marker: pte_install_uffd_wp_if_needed() returns early for anonymous VMAs. So Dj's line "the marker comes back on a zap" does not apply here. MADV_DONTNEED on a populated neighbour drops its WP state, which is the same as upstream for any populated anon page. This is not a regression and only a correction to the review text.
  2. Behaviour visible to userspace that the commit message lists only in part: mincore() and /proc/pid/smaps Rss now count the neighbours of a WP-before-touch window after one write. The commit message mentions RSS and PAGE_IS_PRESENT. CRIU-style dumpers that use PAGEMAP_SCAN will dump them as present but unwritten zero pages. This is acceptable and matches what none neighbours already do.
  3. The selftest does not check that pages 2 and 3 still read zero after population. A one-line CHECK(!a[2 * ps] && !a[3 * ps]) would also cover the slot zeroing on this path.

Missing rows:

  • The current head (exact-marker revision) has only qemu runs. The J616s probe and daily-kernel use were on the earlier UFFD_WP-bit revision. The author is right that this is the same for every marker created today, but no current-head row exists on any Mac.
  • No end-to-end row with the motivating consumer (a FEX write-watch plus GEM_USERPTR, drm/asahi: GEM_USERPTR in processes with a smaller page size #2226) on the current head. The code is SoC-independent, so one 16K-kernel Mac row on the head would be enough.

Commit hygiene: both commits are by Jonathan Goyvaerts with a human Signed-off-by, and there are no agent trailers in the commits (they were removed in the message-only push, and the tree is identical to 7655e71 according to the thread). Provenance: there are no Apple-internal names.

@RyanTheTide

Copy link
Copy Markdown
Contributor

Hardware rows (MERGETRAIN-LINUX2), test kernel 7.3.0-aurora-mt3 = integration/aurora 24dc422 + #2222 c2e95d3 + #2225 6643e60, on the default fleet cmdline. Each Mac got 3 one-shot warm boots and went back to its default:

Mac boots failed/avc/oops/fwcrash s2idle dEQP-VK smoke AIC effective affinity
M1 Pro (T6000) 3/3 0/0/0/0 4/4 4986 P (= fleet79) 142 IRQs on CPUs 0-9 (fleet78: all 142 empty)
M2 (T8112) 3/3 0/0/0/0 4/4 4986 P (= fleet79) 62/77 at boot, 77/77 after the hotplug refresh (fleet78: all 77 empty)
Neo (T8140) 3/3 0/0/0/0 4/4 8951 P (= fleet79) unchanged: 74 on CPU0, 12 empty
M3 Air J615 (T8122) 3/3 0/0/0/0 3/3 (no GPU on this kernel) 37/50 on CPUs 0-7 (default boot: all 53 empty)
  • CPU1 hotplug: Neo offline/online OK, with NVMe I/O while CPU1 was offline. M1 Pro, M2 and J615 refuse the offline (rc=1, EOPNOTSUPP), and production fleet78 refuses it the same way, so the refresh path only ran on the Neo.
  • No new kernel errors on any Mac. The J615 set is the same as on the tip kernel (AOP and ISP pieces from open PRs).
  • mm: populate a subpage window's uffd-wp markers with the faulting page #2225: its path needs 4K userspace (user4k=on), which the default cmdline doesn't enable, so on these Macs this is a no-regression row only.
  • The J616s (M4 Pro) row follows when the Mac is free.

Evidence: traces/kernel/20261122-mt3-fleetboot (lab store).

@RyanTheTide
RyanTheTide merged commit 71abea7 into aurora-silicon:integration/aurora Nov 22, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants