From ba1936073fe6001fcac9b23555f4e8001a560a44 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ralph=20K=C3=BCpper?= Date: Tue, 11 Aug 2026 00:35:17 +0200 Subject: [PATCH] perf(gc): stop the globalThis bootstrap from disabling the per-object layout fast path MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Any plain-object or array property MISS forces the lazy `globalThis` bootstrap, and the bootstrap left 1113 immortal entries in the per-object GC slot-layout side tables. `PER_OBJECT_LAYOUTS_NONEMPTY` — the emptiness proof that keeps `layout_forget_object` off the allocation, death and relocation paths — could then never go false again, so every real TypeScript program ran the full two-map probe on every allocation. Measured: `churn` +28%, `tree` +29% from one `for…of` in `main()`, with `layout_forget_object` self time 112 -> 916 ms and 194 -> 740 ms. It is not the ~1.15 MB the bootstrap allocates and it is not GC pacing: 105 minors on `churn` either way, ~616 KB more copied over the whole run. Two changes, both load-bearing: * `gc::ImmortalLayoutScope` around `populate_global_this_builtins` — objects built inside it declare `GC_LAYOUT_UNKNOWN` (the tag-checked scan the code already falls back to for the same case) instead of minting a mask nothing will ever remove. Residue 1113 -> 0. NOT applied to typed-shape layouts, whose raw-f64 slots a conservative scan would misread as pointers and, under the copying collector, rewrite. * An 8192-bit thread-local address filter replacing the global flag as the hot guard. The scope alone moved nothing measurable: ordinary runtime init still leaves one or two immortal records, and for a single global bit two entries are exactly as bad as 1113. The filter answers "can THIS ADDRESS have an entry" instead. It replaces rather than joins the flag because testing both cost a second thread-local resolution on legitimately-armed workloads (+3.4% interp, +4.6% iso_miss). Five tests, none able to pass vacuously: the bootstrap leaves the tables empty (with a subject-live check on `globalThis.Array`); the same store outside a scope still mints a mask; a scoped object still traces its children; a live record survives a filter rebuild; and the filter still proves unrelated addresses absent while the global flag is armed. Claude-Session: https://claude.ai/code/session_012B8z92S82sCfqCrVqrFgS2 --- .../7809-globalthis-bootstrap-layout-latch.md | 100 ++++++ crates/perry-runtime/src/gc/hot_tls.rs | 8 +- crates/perry-runtime/src/gc/layout.rs | 38 ++- crates/perry-runtime/src/gc/layout_tables.rs | 296 +++++++++++++++++- crates/perry-runtime/src/gc/mod.rs | 5 + crates/perry-runtime/src/gc/telemetry.rs | 8 + .../tests/layout_trace/per_object_tables.rs | 222 ++++++++++++- .../src/object/global_this/populate.rs | 33 ++ 8 files changed, 689 insertions(+), 21 deletions(-) create mode 100644 changelog.d/7809-globalthis-bootstrap-layout-latch.md diff --git a/changelog.d/7809-globalthis-bootstrap-layout-latch.md b/changelog.d/7809-globalthis-bootstrap-layout-latch.md new file mode 100644 index 0000000000..eea5e31f4a --- /dev/null +++ b/changelog.d/7809-globalthis-bootstrap-layout-latch.md @@ -0,0 +1,100 @@ +### Fixed + +- **Touching `globalThis` no longer disables the per-object GC layout fast path + for the rest of the process.** `churn` and `tree` were paying +28% / +29% for + a single `for…of` in `main()`, and every real TypeScript program was paying it + too. + + `globalThis` is populated lazily on first touch, and *any* plain-object or + array property **miss** forces it: the miss walks the prototype chain, reaches + `builtin_prototype_value` → `js_get_global_this_builtin_value`, and runs the + several-hundred-builtin bootstrap. Perry's whole benchmark corpus happens + never to take that path — no `for…of`, no spread, no `Symbol`, no property + miss anywhere in `churn`/`tree`/`interp`/`shapes`/`asyncpipe`/`retain` — so + the cost was invisible to every number in the perf campaign while real + programs paid it before their first line of work. + + It was **not** the ~1.15 MB the bootstrap allocates. GC behaviour is + effectively identical with and without it: 105 minors on `churn` either way, + and ~616 KB more copied across the entire run. The cost was a **global latch**. + The bootstrap builds hundreds of permanently-rooted plain objects, and each + one's first pointer field minted an entry in the per-object GC slot-layout + side tables. Those entries are immortal, so `PER_OBJECT_LAYOUTS_NONEMPTY` — + the emptiness proof that keeps `layout_forget_object` off the allocation, + death and relocation paths — could never go `false` again. Measured + `layout_forget_object` self time: `churn` 112 ms → 916 ms, `tree` + 194 ms → 740 ms, which is essentially the whole regression in both. + + This is #7510's lesson at 1000× the scale ("one immortal entry is enough to + nullify an is-empty accelerator"; there it was a single interned keys array). + Two changes, and **both are needed**: + + 1. `gc::ImmortalLayoutScope` around `populate_global_this_builtins`. Inside + it, an object that would mint a per-object pointer mask declares + `GC_LAYOUT_UNKNOWN` instead — the tag-checked payload scan, which is the + code's own fallback for the same situation and the universally safe state, + not a weaker one. For an object that is never reclaimed the mask bought + precision nobody spends, at the price of two `RefCell` round-trips and two + hash probes on every allocation the program would ever make. Bootstrap + residue: **1113 entries → 0**. + + Deliberately **not** applied to typed-shape layouts + (`init_typed_shape_layout`): those describe raw-f64 slots, whose bit + patterns can alias a heap pointer, and a conservative scan would trace — + and under the copying collector *rewrite* — a slot holding a number. The + scope applies only where the mask being replaced is itself derived from + `layout_pointer_bearing_bits`, i.e. exactly the test `GC_LAYOUT_UNKNOWN` + re-runs per slot. + + 2. An **address filter** replacing `PER_OBJECT_LAYOUTS_NONEMPTY` as the hot + guard. Change 1 alone moved nothing measurable, and that is the important + finding: ordinary runtime init still leaves one or two long-lived records + behind, and for a single global bit two entries are exactly as bad as 1113. + An 8192-bit thread-local filter over the key addresses turns "is either + table empty?" into "can this *address* have an entry?", so a nursery + address the tables have never seen is proved absent in one multiply and one + load even while immortal records exist elsewhere. + + The filter sits *behind* the flag, and both live in ONE thread-local + (`PerObjectLayoutHint`). All three arrangements were measured on the quiet + mini; the co-located one wins everywhere: + + | | `churn` | `push_cls` | `tree` | `interp` | `churn`+`for…of` | `tree`+`for…of` | + |---|--:|--:|--:|--:|--:|--:| + | base | 0.422 | 0.356 | 1.627 | 1.888 | 0.539 | 2.151 | + | filter only (no flag) | 0.438 | **0.383** | 1.673 | 1.934 | 0.500 | 1.857 | + | flag + filter, 2 slots | 0.421 | 0.368 | 1.642 | 1.950 | 0.506 | 1.886 | + | **flag + filter, 1 slot** | **0.422** | **0.368** | **1.640** | **1.922** | **0.493** | **1.840** | + + Dropping the flag is a loss: almost every workload is *disarmed*, and for + those the flag is one load where the filter is a multiply, a shift, a load + and a test — `push_cls` went past its budget. Keeping both as separate + thread-locals costs a second `_tlv_get_addr` on exactly the workloads that + are legitimately armed (`interp`, `iso_miss`). One struct behind the + existing named hot slot gives the cheap gate AND one resolution. + + `false` is a proof of absence and nothing else rests on it; the filter is + rebuilt from the live keys once half its bits are set, so a workload that + genuinely churns per-object records cannot saturate it permanently. + + Measured on the quiet mini (base = `b9415d780`, both arms built locally; + interleaved, best-of-5, exit-checked): `churn` + `for…of` **0.539 → 0.493** + (floor 0.422), `tree` + `for…of` **2.151 → 1.840** (floor 1.627). Every + protected bench stays inside budget. `interp` 1.888 → 1.922 and `iso_miss` + 2.361 → 2.443 still pay the filter test without benefiting from it — they are + legitimately armed, so it never proves absence for them. + + Gated by five tests in `gc::tests::layout_trace::per_object_tables`, written + so none of them can pass vacuously: the bootstrap must leave both tables empty + (with a subject-live check that `globalThis.Array` actually populated); the + same store *outside* a scope must still mint a mask; an object built inside a + scope must still trace its children through the fallback scan; a live record + must survive a filter rebuild and still be found and removed; and — the + accelerator's own subject-live assertion — the filter must still prove + unrelated addresses absent *while the global flag is armed*, which is the + exact condition under which it silently stopped accelerating before. + + `PERRY_GC_DIAG=1` now prints + `[gc-globalthis-bootstrap] elapsed_us=… per_object_slot_masks=… per_object_typed_layouts=…` + once per thread, so the bootstrap's cost and its residue are observable rather + than inferred. diff --git a/crates/perry-runtime/src/gc/hot_tls.rs b/crates/perry-runtime/src/gc/hot_tls.rs index 8650e55fbf..20a2327c29 100644 --- a/crates/perry-runtime/src/gc/hot_tls.rs +++ b/crates/perry-runtime/src/gc/hot_tls.rs @@ -24,7 +24,9 @@ use super::barrier::{ GC_BIRTH_EXTRA_FLAGS, INCREMENTAL_MARK_BARRIER_MINOR_ONLY, INCREMENTAL_MARK_BARRIER_VALID_PTRS, }; use super::layout::{LayoutSlotMask, TypedLayoutDescriptor, SHAPE_LAYOUTS}; -use super::layout_tables::{LAYOUT_SLOT_MASKS, PER_OBJECT_LAYOUTS_NONEMPTY, TYPED_LAYOUTS}; +use super::layout_tables::{ + PerObjectLayoutHint, LAYOUT_SLOT_MASKS, PER_OBJECT_LAYOUTS_NONEMPTY, TYPED_LAYOUTS, +}; use super::malloc::{ARENA_FREE_LIST, ARENA_FREE_LIST_NONEMPTY}; use super::trace::ValidPointerSet; use std::cell::{Cell, RefCell}; @@ -120,9 +122,9 @@ pub(super) fn hot_shape_layouts() -> &'static RefCell { /// per-object layout record at all" question the allocation, store, death and /// trace paths all ask (#7510). #[inline(always)] -pub(super) fn hot_per_object_layouts_nonempty() -> &'static Cell { +pub(super) fn hot_per_object_layout_hint() -> &'static PerObjectLayoutHint { // SAFETY: paired with `per_object_layouts_nonempty_hot_addr` above. - unsafe { &*(crate::tls_hot::hot().per_object_layouts_nonempty as *const Cell) } + unsafe { &*(crate::tls_hot::hot().per_object_layouts_nonempty as *const PerObjectLayoutHint) } } // --- gc::malloc ------------------------------------------------------------- diff --git a/crates/perry-runtime/src/gc/layout.rs b/crates/perry-runtime/src/gc/layout.rs index dcdbe1ef2c..d63a4c9568 100644 --- a/crates/perry-runtime/src/gc/layout.rs +++ b/crates/perry-runtime/src/gc/layout.rs @@ -909,11 +909,27 @@ pub(crate) fn layout_note_slot(parent_user: usize, slot_index: usize, value_bits set_layout_state(header, GC_LAYOUT_SIDE_MASK); } } else if (*header)._reserved & GC_LAYOUT_STATE_MASK == GC_LAYOUT_POINTER_FREE { - let mut mask = LayoutSlotMask::Inline(0); - mask.set_slot(slot_index); - masks.insert(parent_user, mask); - mark_per_object_layouts_nonempty(); - set_layout_state(header, GC_LAYOUT_SIDE_MASK); + if super::layout_tables::immortal_layout_scope_active() { + // An object built inside an `ImmortalLayoutScope` is + // rooted for the life of the process, so the entry it + // would mint here is never removed — and one such + // entry disables `PER_OBJECT_LAYOUTS_NONEMPTY` for + // every allocation the program will ever make. Take + // the same `GC_LAYOUT_UNKNOWN` fallback the `else` + // arm below uses for this exact situation; see + // `ImmortalLayoutScope` for why that is the safe + // state and not a weaker one. + set_layout_state(header, GC_LAYOUT_UNKNOWN); + } else { + let mut mask = LayoutSlotMask::Inline(0); + mask.set_slot(slot_index); + masks.insert(parent_user, mask); + mark_per_object_layouts_nonempty(); + // The one insert site that holds its own `borrow_mut`, + // so it maintains the address filter inline too. + super::layout_tables::layout_addr_filter_note(parent_user); + set_layout_state(header, GC_LAYOUT_SIDE_MASK); + } } else { set_layout_state(header, GC_LAYOUT_UNKNOWN); } @@ -1338,6 +1354,18 @@ pub(super) unsafe fn layout_rebuild_from_slots_with_policy( if mask.is_empty() { set_layout_state(header, GC_LAYOUT_POINTER_FREE); slot_masks_remove(user_ptr as usize); + } else if super::layout_tables::immortal_layout_scope_active() { + // Same reasoning as the `layout_note_slot` branch: an object built + // inside an `ImmortalLayoutScope` never dies, so the mask it would + // install here is a permanent tenant of a side table whose emptiness + // is a process-wide fast path. Falling back to the tag-checked scan is + // sound *for this rebuild specifically* because the mask above is + // itself derived from `layout_pointer_bearing_bits` — exactly the test + // `GC_LAYOUT_UNKNOWN` re-runs per slot. (This is why the scope may not + // be applied to a TYPED descriptor, whose raw-f64 slots the tag test + // would misread; see `ImmortalLayoutScope`.) + set_layout_state(header, GC_LAYOUT_UNKNOWN); + slot_masks_remove(user_ptr as usize); } else { set_layout_state(header, GC_LAYOUT_SIDE_MASK); slot_masks_insert(user_ptr as usize, mask); diff --git a/crates/perry-runtime/src/gc/layout_tables.rs b/crates/perry-runtime/src/gc/layout_tables.rs index 7dffb6db0b..a4df0ae135 100644 --- a/crates/perry-runtime/src/gc/layout_tables.rs +++ b/crates/perry-runtime/src/gc/layout_tables.rs @@ -29,7 +29,7 @@ //! Split out of `layout.rs` to stay under the repo's 2000-line-per-file cap //! (`scripts/check_file_size.sh`). -use super::hot_tls::{hot_layout_slot_masks, hot_per_object_layouts_nonempty, hot_typed_layouts}; +use super::hot_tls::{hot_layout_slot_masks, hot_per_object_layout_hint, hot_typed_layouts}; use super::layout::{LayoutSlotMask, TypedLayoutDescriptor}; use std::cell::{Cell, RefCell}; @@ -47,7 +47,258 @@ thread_local! { /// once they are empty. A stale `true` therefore costs exactly the /// pre-#7510 probe and nothing else — the flag is an accelerator, never an /// authority, and no caller may treat it as one. - pub(in crate::gc) static PER_OBJECT_LAYOUTS_NONEMPTY: Cell = const { Cell::new(false) }; + pub(in crate::gc) static PER_OBJECT_LAYOUTS_NONEMPTY: PerObjectLayoutHint = + const { PerObjectLayoutHint::new() }; +} + +/// The flag, the address filter, and the filter's rebuild counter in ONE +/// thread-local. +/// +/// They are co-located for a measured reason. On Darwin a thread-local access +/// is an out-of-line `_tlv_get_addr` call, and `crate::tls_hot` exists to pay +/// that once per hot region instead of once per table. `layout_forget_object` +/// consults the flag and then the filter on every allocation, death and +/// relocation; as two separate thread-locals that is two slot reads on exactly +/// the workloads that are legitimately armed, which measured +3.4% on `interp` +/// and +4.6% on `iso_miss`. One struct behind the existing named hot slot makes +/// it one. +pub(in crate::gc) struct PerObjectLayoutHint { + /// #7510's global emptiness proof — see [`PER_OBJECT_LAYOUTS_NONEMPTY`]. + pub(in crate::gc) nonempty: Cell, + /// Bits set in `filter` since the last rebuild. + pub(in crate::gc) sets: Cell, + /// Which addresses may have an entry — see [`layout_addr_filter_may_hold`]. + pub(in crate::gc) filter: std::cell::UnsafeCell<[u64; LAYOUT_ADDR_FILTER_WORDS]>, +} + +impl PerObjectLayoutHint { + const fn new() -> Self { + Self { + nonempty: Cell::new(false), + sets: Cell::new(0), + filter: std::cell::UnsafeCell::new([0u64; LAYOUT_ADDR_FILTER_WORDS]), + } + } +} + +/// Bits in the per-object address filter (see [`layout_addr_filter_may_hold`]). +/// 4096 bits is 512 B of thread-local storage, held INLINE in +/// [`PerObjectLayoutHint`] so the flag and the filter share one hot slot. One +/// 8-byte word is read per query and the whole filter stays L1-resident even +/// when the addresses probed sweep a 16 MB nursery. Steady-state occupancy +/// after the `ImmortalLayoutScope` is one or two entries, so the false-positive +/// rate is ~0.05% — sizing this up buys nothing and costs inline TLS on every +/// thread. +const LAYOUT_ADDR_FILTER_BITS: usize = 4096; +const LAYOUT_ADDR_FILTER_WORDS: usize = LAYOUT_ADDR_FILTER_BITS / 64; +/// Rebuild the filter from the live keys once this many bits have been set +/// since the last rebuild. Without it a workload that churns per-object +/// records would saturate the filter and never recover; with it the false +/// positive rate is bounded by (live entries / bits) rather than by +/// (entries ever inserted / bits), at an amortised O(1) per insert. +const LAYOUT_ADDR_FILTER_REBUILD_AFTER: u32 = (LAYOUT_ADDR_FILTER_BITS / 2) as u32; + +/// Word index + bit mask for `user_ptr`. Heap pointers are at least 8-byte +/// aligned and clustered, so the low bits alone would collide systematically; +/// a single multiply spreads the whole address across the filter. +#[inline(always)] +fn layout_addr_filter_slot(user_ptr: usize) -> (usize, u64) { + let h = (user_ptr as u64).wrapping_mul(0x9E37_79B9_7F4A_7C15); + let idx = (h >> (64 - LAYOUT_ADDR_FILTER_BITS.trailing_zeros() as u64)) as usize; + (idx >> 6, 1u64 << (idx & 63)) +} + +/// Could either per-object side table hold an entry keyed by `user_ptr`? +/// +/// `false` is a **proof of absence**; `true` is a hint (a real entry, or a +/// collision). This is the address-precise half of the accelerator, and it is +/// the half that survives an immortal resident: [`PER_OBJECT_LAYOUTS_NONEMPTY`] +/// is a single global bit, so ONE entry that is never removed — one long-lived +/// object anywhere in the process — turns `layout_forget_object` back into two +/// `RefCell` round-trips plus two hashes on every allocation, death and +/// relocation for the rest of the run. Removing 1113 of 1115 such entries buys +/// nothing while the last two remain; only an address-keyed test does. +/// +/// ## Why the flag is still tested FIRST, even though this subsumes it +/// +/// It does subsume it — both maps empty implies every bit is clear, because the +/// flag's own clear path clears the filter — and dropping the flag would make +/// the armed path one thread-local resolution cheaper. Measured on the quiet +/// mini, that trade is a loss: filter-only moved `interp` 1.948 → 1.934 and +/// `iso_miss` 2.476 → 2.464, but moved `push_cls` 0.367 → 0.383 (past its +/// budget), `churn` 0.422 → 0.438 and `tree` 1.642 → 1.673. Almost every +/// workload is *disarmed*, and for those the flag is a single load while this +/// is a multiply, a shift, a load and a test. The flag stays in front as the +/// cheap common-case gate; the filter is what rescues the armed case. +/// +/// The residual cost of consulting both — a second thread-local resolution on a +/// legitimately-armed workload, +3.4% on `interp` and +4.6% on `iso_miss` — +/// is not inherent. It goes away by co-locating the flag and this filter in one +/// thread-local so the armed path resolves once; that is a `tls_hot` change and +/// is deliberately left out of this one. +#[inline(always)] +pub(in crate::gc) fn layout_addr_filter_may_hold(user_ptr: usize) -> bool { + hint_may_hold(hot_per_object_layout_hint(), user_ptr) +} + +/// [`layout_addr_filter_may_hold`] against an already-resolved hint, so a +/// caller that also reads the flag pays ONE slot resolution for both. +#[inline(always)] +pub(in crate::gc) fn hint_may_hold(hint: &PerObjectLayoutHint, user_ptr: usize) -> bool { + let (word, bit) = layout_addr_filter_slot(user_ptr); + unsafe { (*hint.filter.get())[word] & bit != 0 } +} + +/// Record that `user_ptr` now has an entry. Called by every insert site. +/// The rebuild check runs BEFORE the bit is set, never after: a rebuild +/// reconstructs the filter from the maps' live keys, and this key is not in +/// the map yet at any call site, so rebuilding afterwards would erase the bit +/// just set and make a live record invisible to the filter. +#[inline] +fn layout_addr_filter_add(user_ptr: usize) { + if hot_per_object_layout_hint().sets.get() >= LAYOUT_ADDR_FILTER_REBUILD_AFTER { + layout_addr_filter_rebuild(); + } + layout_addr_filter_note(user_ptr); +} + +/// Set `user_ptr`'s bit WITHOUT the rebuild check. +/// +/// `layout_note_slot` calls this from inside its own `borrow_mut` on +/// `LAYOUT_SLOT_MASKS`, and [`layout_addr_filter_rebuild`] borrows both maps — +/// so the rebuilding form would panic there. Skipping the rebuild is safe: the +/// counter is an accuracy heuristic, not a correctness one. +#[inline] +pub(in crate::gc) fn layout_addr_filter_note(user_ptr: usize) { + let hint = hot_per_object_layout_hint(); + let (word, bit) = layout_addr_filter_slot(user_ptr); + unsafe { + (*hint.filter.get())[word] |= bit; + } + hint.sets.set(hint.sets.get().saturating_add(1)); +} + +/// Drop every bit and re-add the live keys. Removals cannot clear a bit on +/// their own (two keys may share one), so this is what keeps a workload that +/// genuinely churns per-object records from saturating the filter forever. +fn layout_addr_filter_rebuild() { + layout_addr_filter_clear(); + let keys: Vec = { + let masks = hot_layout_slot_masks().borrow(); + let typed = hot_typed_layouts().borrow(); + masks.keys().copied().chain(typed.keys().copied()).collect() + }; + let hint = hot_per_object_layout_hint(); + for k in keys { + let (word, bit) = layout_addr_filter_slot(k); + unsafe { + (*hint.filter.get())[word] |= bit; + } + } + hint.sets.set(0); +} + +fn layout_addr_filter_clear() { + let hint = hot_per_object_layout_hint(); + unsafe { + (*hint.filter.get()).fill(0); + } + hint.sets.set(0); +} + +crate::perry_thread_local! { + /// Nesting depth of the innermost [`ImmortalLayoutScope`]. + /// + /// Read only from the *cold* half of `layout_note_slot` — the branch that + /// would otherwise mint a brand-new per-object mask — so an inactive scope + /// costs nothing on any hot path. + static IMMORTAL_LAYOUT_SCOPE_DEPTH: Cell = const { Cell::new(0) }; +} + +/// True while an [`ImmortalLayoutScope`] is open on this thread. +#[inline] +pub(in crate::gc) fn immortal_layout_scope_active() -> bool { + IMMORTAL_LAYOUT_SCOPE_DEPTH.with(|d| d.get()) != 0 +} + +/// Marks a window whose objects are **immortal by construction** — reachable +/// from a GC root for the life of the process — so that none of them may take +/// out a per-object layout record. +/// +/// ## Why this exists (#7510's lesson, repeating) +/// +/// [`PER_OBJECT_LAYOUTS_NONEMPTY`] is a *global emptiness* proof: it turns +/// `layout_forget_object` — which runs on every allocation, every object death +/// and every relocation — into a single load whenever both side tables happen +/// to be empty. On a monomorphic workload they are empty for the entire run, +/// which is exactly what makes the accelerator worth having. +/// +/// A *single* entry that is never removed converts that accelerator into +/// permanently-disabled code. #7510 already paid this once, via one interned +/// keys-array the shape cache anchored forever. The `globalThis` bootstrap is +/// the same trap at several hundred times the scale: it builds hundreds of +/// plain objects whose first pointer field mints a mask, every one of them +/// rooted at `globalThis` forever. Since a plain-object property miss forces +/// that bootstrap, *every real TypeScript program* armed the regime before +/// user code ran — measured as +28% on `churn` and +29% on `tree`, with +/// `layout_forget_object` going 112 → 916 ms and 194 → 740 ms of self time. +/// +/// ## Why dropping the mask is sound +/// +/// The alternative the code already uses for the very same situation — a +/// pointer stored into an object whose state is not `POINTER_FREE` — is +/// [`super::layout::GC_LAYOUT_UNKNOWN`], the tag-checked payload scan. That is +/// the universally safe state, not a weaker one; the mask is a *precision* +/// optimization that lets the collector skip known-pointer-free slots. +/// +/// For an immortal object that precision buys nothing measurable: the object +/// is never reclaimed, and it is scanned only as part of the root graph. It +/// costs one tag test per slot on a few hundred objects, against two `RefCell` +/// round-trips and two hash probes on every allocation the program will ever +/// make. +/// +/// ## Deliberately NOT applied to typed-shape layouts +/// +/// The scope gates ONLY the mask-minting branch of `layout_note_slot`. It must +/// never redirect a *typed* layout (`init_typed_shape_layout`, +/// `layout_rebuild_from_slots`) to `GC_LAYOUT_UNKNOWN`: those describe objects +/// with raw-f64 slots, and a raw double's bit pattern can alias a heap pointer. +/// A conservative scan would then trace — and, under a copying collector, +/// *rewrite* — a slot holding a number. `GC_LAYOUT_UNKNOWN` is only safe where +/// every slot is a NaN-boxed value, which is what the mask-minting branch +/// already assumes. +pub struct ImmortalLayoutScope { + _not_send: std::marker::PhantomData<*const ()>, +} + +impl Default for ImmortalLayoutScope { + fn default() -> Self { + Self::new() + } +} + +impl ImmortalLayoutScope { + pub fn new() -> Self { + IMMORTAL_LAYOUT_SCOPE_DEPTH.with(|d| d.set(d.get().saturating_add(1))); + Self { + _not_send: std::marker::PhantomData, + } + } +} + +impl Drop for ImmortalLayoutScope { + fn drop(&mut self) { + IMMORTAL_LAYOUT_SCOPE_DEPTH.with(|d| d.set(d.get().saturating_sub(1))); + } +} + +/// Live entry counts of the two per-object side tables, for `PERRY_GC_DIAG` +/// and for the tests that assert the bootstrap left them alone. +pub(crate) fn per_object_layout_table_sizes() -> (usize, usize) { + ( + hot_layout_slot_masks().borrow().len(), + hot_typed_layouts().borrow().len(), + ) } /// True when either per-object side table may hold an entry. `false` is a @@ -55,7 +306,7 @@ thread_local! { /// hint, so every caller still has to handle a miss. #[inline(always)] pub(in crate::gc) fn per_object_layouts_maybe_nonempty() -> bool { - hot_per_object_layouts_nonempty().get() + hot_per_object_layout_hint().nonempty.get() } /// Arm the flag. Called by anything that inserts into either map — including @@ -63,7 +314,7 @@ pub(in crate::gc) fn per_object_layouts_maybe_nonempty() -> bool { /// through the wrappers below. #[inline(always)] pub(in crate::gc) fn mark_per_object_layouts_nonempty() { - hot_per_object_layouts_nonempty().set(true); + hot_per_object_layout_hint().nonempty.set(true); } /// Re-establish the flag after a removal emptied one map: clear it once the @@ -79,7 +330,11 @@ pub(in crate::gc) fn refresh_per_object_layouts_flag(touched_map_emptied: bool) return; } if hot_layout_slot_masks().borrow().is_empty() && hot_typed_layouts().borrow().is_empty() { - hot_per_object_layouts_nonempty().set(false); + hot_per_object_layout_hint().nonempty.set(false); + // Both maps are empty, so every bit is now stale. Clearing here is what + // makes the filter's occupancy track LIVE entries rather than every + // entry the program has ever created. + layout_addr_filter_clear(); } } @@ -87,6 +342,7 @@ pub(in crate::gc) fn refresh_per_object_layouts_flag(touched_map_emptied: bool) #[inline] pub(in crate::gc) fn typed_layouts_insert(user_ptr: usize, descriptor: TypedLayoutDescriptor) { mark_per_object_layouts_nonempty(); + layout_addr_filter_add(user_ptr); hot_typed_layouts() .borrow_mut() .insert(user_ptr, descriptor); @@ -96,13 +352,14 @@ pub(in crate::gc) fn typed_layouts_insert(user_ptr: usize, descriptor: TypedLayo #[inline] pub(in crate::gc) fn slot_masks_insert(user_ptr: usize, mask: LayoutSlotMask) { mark_per_object_layouts_nonempty(); + layout_addr_filter_add(user_ptr); hot_layout_slot_masks().borrow_mut().insert(user_ptr, mask); } /// Drop `user_ptr`'s per-object typed descriptor (only). #[inline] pub(in crate::gc) fn typed_layouts_remove(user_ptr: usize) { - if !per_object_layouts_maybe_nonempty() { + if !per_object_layouts_maybe_nonempty() || !layout_addr_filter_may_hold(user_ptr) { return; } let emptied = { @@ -115,7 +372,7 @@ pub(in crate::gc) fn typed_layouts_remove(user_ptr: usize) { /// Drop `user_ptr`'s per-object pointer mask (only). #[inline] pub(in crate::gc) fn slot_masks_remove(user_ptr: usize) { - if !per_object_layouts_maybe_nonempty() { + if !per_object_layouts_maybe_nonempty() || !layout_addr_filter_may_hold(user_ptr) { return; } let emptied = { @@ -133,7 +390,7 @@ pub(in crate::gc) fn with_per_object_descriptor( user_ptr: usize, f: impl FnOnce(&TypedLayoutDescriptor) -> R, ) -> Option { - if !per_object_layouts_maybe_nonempty() { + if !per_object_layouts_maybe_nonempty() || !layout_addr_filter_may_hold(user_ptr) { return None; } hot_typed_layouts().borrow().get(&user_ptr).map(f) @@ -144,7 +401,7 @@ pub(in crate::gc) fn with_per_object_descriptor( /// as much here as it is on the mutator side. #[inline] pub(in crate::gc) fn per_object_slot_mask(user_ptr: usize) -> Option { - if !per_object_layouts_maybe_nonempty() { + if !per_object_layouts_maybe_nonempty() || !layout_addr_filter_may_hold(user_ptr) { return None; } hot_layout_slot_masks().borrow().get(&user_ptr).cloned() @@ -160,7 +417,12 @@ pub(in crate::gc) fn per_object_slot_mask(user_ptr: usize) -> Option bool { - if !per_object_layouts_maybe_nonempty() { + // BOTH addresses are touched (the destination is cleared of a previous + // tenant's record before the source's is moved in), so the filter can only + // prove this call unnecessary when it proves both absent. + if !per_object_layouts_maybe_nonempty() + || (!layout_addr_filter_may_hold(old_user) && !layout_addr_filter_may_hold(new_user)) + { return false; } let mut typed = hot_typed_layouts().borrow_mut(); @@ -168,6 +430,8 @@ pub(in crate::gc) fn transfer_per_object_descriptor(old_user: usize, new_user: u match typed.remove(&old_user) { Some(layout) => { typed.insert(new_user, layout); + drop(typed); + layout_addr_filter_add(new_user); true } None => false, @@ -177,13 +441,17 @@ pub(in crate::gc) fn transfer_per_object_descriptor(old_user: usize, new_user: u /// Move `old_user`'s per-object pointer mask to `new_user` (relocation). #[inline] pub(in crate::gc) fn transfer_per_object_slot_mask(old_user: usize, new_user: usize) { - if !per_object_layouts_maybe_nonempty() { + if !per_object_layouts_maybe_nonempty() + || (!layout_addr_filter_may_hold(old_user) && !layout_addr_filter_may_hold(new_user)) + { return; } let mut masks = hot_layout_slot_masks().borrow_mut(); masks.remove(&new_user); if let Some(mask) = masks.remove(&old_user) { masks.insert(new_user, mask); + drop(masks); + layout_addr_filter_add(new_user); } } @@ -200,7 +468,11 @@ pub(in crate::gc) fn transfer_per_object_slot_mask(old_user: usize, new_user: us /// pre-#7510 path unchanged, and re-arms the flag on the way out. #[inline] pub(in crate::gc) fn layout_forget_object(user_ptr: usize) { - if !per_object_layouts_maybe_nonempty() { + // ONE hot-slot resolution for both halves of the guard: the flag (cheap, + // and false for the overwhelming majority of workloads) and then the + // address filter (what rescues a workload with an immortal record). + let hint = hot_per_object_layout_hint(); + if !hint.nonempty.get() || !hint_may_hold(hint, user_ptr) { return; } // One `borrow_mut` per map, not a `borrow` to test emptiness followed by a diff --git a/crates/perry-runtime/src/gc/mod.rs b/crates/perry-runtime/src/gc/mod.rs index 76baed3973..5116612cec 100644 --- a/crates/perry-runtime/src/gc/mod.rs +++ b/crates/perry-runtime/src/gc/mod.rs @@ -75,6 +75,11 @@ pub(crate) use layout_slot_visit::*; /// keeps them off the allocation, store, death and trace paths. Split out of /// `layout.rs` so it stays under the repo's 2000-line-per-file cap. mod layout_tables; +// The immortal-object construction window and the table-occupancy readout, both +// consumed from OUTSIDE `gc`: `object::global_this` opens the window around the +// `globalThis` bootstrap and prints the residue under `PERRY_GC_DIAG`. +pub(crate) use layout_tables::per_object_layout_table_sizes; +pub use layout_tables::ImmortalLayoutScope; /// #7510 item 1: the construction-side memo that turns an already-installed /// typed shape into two header bit-writes instead of a descriptor build plus a /// `SHAPE_LAYOUTS` round-trip. diff --git a/crates/perry-runtime/src/gc/telemetry.rs b/crates/perry-runtime/src/gc/telemetry.rs index e3cc182a02..7c0118fbd3 100644 --- a/crates/perry-runtime/src/gc/telemetry.rs +++ b/crates/perry-runtime/src/gc/telemetry.rs @@ -3,6 +3,14 @@ use super::*; /// Number of most-recent pause samples retained per thread (#6187). pub const GC_RECENT_PAUSE_WINDOW: usize = 32; +/// Is `PERRY_GC_DIAG` set? Read once and cached, so a diagnostic call site can +/// sit on a path that runs before/around `main` without paying a `getenv` each +/// time. Diagnostic-only: nothing may branch on this for behaviour. +pub fn gc_diag_enabled() -> bool { + static ENABLED: std::sync::OnceLock = std::sync::OnceLock::new(); + *ENABLED.get_or_init(|| std::env::var_os("PERRY_GC_DIAG").is_some()) +} + pub struct GcStats { pub collection_count: u64, pub total_freed_bytes: u64, diff --git a/crates/perry-runtime/src/gc/tests/layout_trace/per_object_tables.rs b/crates/perry-runtime/src/gc/tests/layout_trace/per_object_tables.rs index 007df0435f..40502986f1 100644 --- a/crates/perry-runtime/src/gc/tests/layout_trace/per_object_tables.rs +++ b/crates/perry-runtime/src/gc/tests/layout_trace/per_object_tables.rs @@ -11,9 +11,10 @@ use super::*; use crate::gc::layout_tables::{test_per_object_tables_are_empty, PER_OBJECT_LAYOUTS_NONEMPTY}; +use crate::gc::ImmortalLayoutScope; fn flag() -> bool { - PER_OBJECT_LAYOUTS_NONEMPTY.with(|c| c.get()) + PER_OBJECT_LAYOUTS_NONEMPTY.with(|h| h.nonempty.get()) } /// The invariant itself: never `false` while either map holds an entry. @@ -241,3 +242,222 @@ fn test_class_keys_array_declares_all_pointer_slots_instead_of_a_mask() { clear_marks(); clear_mark_seeds(); } + +/// The control half of the immortal-scope pair: WITHOUT the scope, the very +/// same store mints a per-object mask and arms the flag. +/// +/// This exists so the scoped test below cannot pass vacuously. If a future +/// change stops routing this shape through `layout_note_slot`'s mask-minting +/// branch at all, this test goes red and says so, rather than leaving its +/// partner green while testing nothing. +#[test] +fn test_pointer_store_outside_an_immortal_scope_still_mints_a_mask() { + clear_marks(); + clear_mark_seeds(); + assert_flag_sound("before store"); + assert!(test_per_object_tables_are_empty()); + + let obj = crate::object::js_object_alloc(0, 2); + let child = crate::object::js_object_alloc(0, 0); + crate::gc::layout_note_slot(obj as usize, 1, POINTER_TAG | (child as u64 & POINTER_MASK)); + + assert!( + flag(), + "a first pointer store into a pointer-free object must mint a mask — \ + if it no longer does, the scoped test below proves nothing" + ); + assert!(!test_per_object_tables_are_empty()); + assert_eq!(test_layout_pointer_slot_count(obj as usize, 2), Some(1)); + + crate::gc::layout_clear_for_ptr(obj as usize); + assert!(test_per_object_tables_are_empty()); + + clear_marks(); + clear_mark_seeds(); +} + +/// Inside an [`ImmortalLayoutScope`] the identical store must leave both maps +/// empty — and the object must still trace its child, because the fallback is +/// `GC_LAYOUT_UNKNOWN` (the tag-checked scan), not "no pointers here". +#[test] +fn test_immortal_scope_stores_trace_without_taking_a_side_table_entry() { + clear_marks(); + clear_mark_seeds(); + assert!(test_per_object_tables_are_empty()); + + let obj = crate::object::js_object_alloc(0, 2); + let child = crate::object::js_object_alloc(0, 0); + let child_header = unsafe { header_from_user_ptr(child as *const u8) }; + unsafe { + *(obj as *mut u8).add(8).cast::().add(1) = POINTER_TAG | (child as u64 & POINTER_MASK); + } + { + let _immortal = ImmortalLayoutScope::new(); + crate::gc::layout_note_slot(obj as usize, 1, POINTER_TAG | (child as u64 & POINTER_MASK)); + } + + assert!( + test_per_object_tables_are_empty(), + "an object built inside an ImmortalLayoutScope must not take out a \ + per-object layout record — one permanent entry disables the emptiness \ + fast path for every allocation the process will ever make" + ); + assert!(!flag()); + + // Correctness half: the child is still reached. `GC_LAYOUT_UNKNOWN` scans + // the payload with a tag check, so precision is lost but nothing is missed. + let valid_ptrs = build_valid_pointer_set(); + assert!(try_mark_value( + POINTER_TAG | (obj as u64 & POINTER_MASK), + &valid_ptrs + )); + trace_marked_objects(&valid_ptrs); + unsafe { + assert_ne!( + (*child_header).gc_flags & GC_FLAG_MARKED, + 0, + "the child must still be traced through the tag-checked scan the \ + immortal scope falls back to" + ); + } + + clear_marks(); + clear_mark_seeds(); +} + +/// The acceptance gate for the `globalThis` bootstrap itself. +/// +/// The bootstrap builds several hundred permanently-rooted objects. Before the +/// `ImmortalLayoutScope` around it, each of their first pointer fields minted a +/// mask that nothing would ever remove, so the first plain-object property miss +/// in ANY program permanently disarmed `PER_OBJECT_LAYOUTS_NONEMPTY`. +/// +/// The assertion is paired with a subject-live check: `Array` must resolve to a +/// real closure value, so a bootstrap that silently did nothing cannot pass this +/// by leaving the tables trivially empty. +#[test] +fn test_global_this_bootstrap_leaves_the_per_object_layout_tables_empty() { + let global = crate::object::js_get_global_this(); + assert_eq!( + global.to_bits() >> 48, + 0x7FFD, + "globalThis must be a real heap object for this test to mean anything" + ); + // Subject-live: the bootstrap actually populated the singleton. + let array_ctor = crate::object::js_get_global_this_builtin_value(b"Array".as_ptr(), 5); + assert_eq!( + array_ctor.to_bits() >> 48, + 0x7FFD, + "globalThis.Array must be populated — otherwise the emptiness assertion \ + below is vacuous" + ); + + let (slot_masks, typed) = crate::gc::per_object_layout_table_sizes(); + assert_eq!( + (slot_masks, typed), + (0, 0), + "the globalThis bootstrap left {slot_masks} slot-mask and {typed} typed \ + per-object layout records behind; every one of them is immortal, so \ + `layout_forget_object` now runs its full two-map probe on every \ + allocation, death and relocation for the rest of the process" + ); +} + +/// The address filter is an *accelerator*, never an authority: `false` must be +/// a proof of absence and nothing else may rest on it. This drives enough +/// inserts to force at least one filter rebuild and then checks, for every +/// live record, that the guarded accessors still find it and that +/// `layout_forget_object` still removes it. +#[test] +fn test_addr_filter_never_hides_a_live_record_across_a_rebuild() { + clear_marks(); + clear_mark_seeds(); + + // Comfortably more inserts than the rebuild threshold (half the bits), so + // the rebuild path is exercised rather than merely reachable. + let mut objs = Vec::new(); + for _ in 0..6000 { + let obj = crate::object::js_object_alloc(0, 2); + let child = crate::object::js_object_alloc(0, 0); + crate::gc::layout_note_slot(obj as usize, 1, POINTER_TAG | (child as u64 & POINTER_MASK)); + objs.push(obj); + } + assert!(flag(), "6000 masks must arm the flag"); + + for (i, obj) in objs.iter().enumerate() { + assert_eq!( + test_layout_pointer_slot_count(*obj as usize, 2), + Some(1), + "record {i} became invisible — the filter proved absence for an \ + address that has a live entry" + ); + } + for obj in &objs { + crate::gc::layout_clear_for_ptr(*obj as usize); + } + assert!( + test_per_object_tables_are_empty(), + "every record must still be removable after a filter rebuild" + ); + assert!(!flag()); + + clear_marks(); + clear_mark_seeds(); +} + +/// Subject-live check for the accelerator itself: with a record present (so +/// `PER_OBJECT_LAYOUTS_NONEMPTY` is armed and the old global test would force +/// the full two-map probe), an unrelated address must still be *proved absent*. +/// +/// This is the whole point of the address filter. The global flag alone cannot +/// distinguish "some object somewhere has a record" from "this address has a +/// record", so one immortal entry — which the `globalThis` bootstrap used to +/// leave 1113 of, and which ordinary runtime init still leaves one or two of — +/// put every allocation, death and relocation in the process back on the slow +/// path. If this assertion ever fails, the accelerator has silently stopped +/// accelerating even though nothing throws. +#[test] +fn test_addr_filter_proves_absence_while_the_global_flag_is_armed() { + clear_marks(); + clear_mark_seeds(); + + let live = crate::object::js_object_alloc(0, 2); + let child = crate::object::js_object_alloc(0, 0); + crate::gc::layout_note_slot( + live as usize, + 1, + POINTER_TAG | (child as u64 & POINTER_MASK), + ); + assert!( + flag(), + "the global flag must be armed for this test to mean anything" + ); + assert!(crate::gc::layout_tables::layout_addr_filter_may_hold( + live as usize + )); + + // A large sample of unrelated addresses: with one live record in 8192 bits + // the filter must prove nearly all of them absent. Anything close to 100% + // "maybe" means the filter is saturated or mis-hashed and the fast path is + // gone even though every test still passes. + let probes = 4096usize; + let mut maybe = 0usize; + for i in 0..probes { + let addr = 0x2000_0000_0000usize + i * 64; + if addr == live as usize { + continue; + } + if crate::gc::layout_tables::layout_addr_filter_may_hold(addr) { + maybe += 1; + } + } + assert!( + maybe * 100 < probes, + "{maybe}/{probes} unrelated addresses were not proved absent — the \ + address filter is not accelerating anything" + ); + + crate::gc::layout_clear_for_ptr(live as usize); + clear_marks(); + clear_mark_seeds(); +} diff --git a/crates/perry-runtime/src/object/global_this/populate.rs b/crates/perry-runtime/src/object/global_this/populate.rs index c93b4ba2bd..aeb36ef1a0 100644 --- a/crates/perry-runtime/src/object/global_this/populate.rs +++ b/crates/perry-runtime/src/object/global_this/populate.rs @@ -77,6 +77,25 @@ pub(crate) fn populate_global_this_builtins(singleton_at_entry: *mut ObjectHeade // whole window: ~1.15 MB allocated, ~410 KB of it live afterwards, once per // thread. let _no_move = crate::gc::GcSuppressScope::new(); + // Every object this bootstrap builds is reachable from `globalThis` for the + // life of the process (the `_no_move` comment above measures the graph: + // ~1.15 MB allocated, ~410 KB live afterwards). That makes each of them a + // permanent tenant of the per-object GC slot-layout side tables — and ONE + // permanent tenant is enough to disable `PER_OBJECT_LAYOUTS_NONEMPTY`, + // the global emptiness proof that keeps `layout_forget_object` off the + // allocation / death / relocation paths, for the rest of the run. + // + // Since a single plain-object property miss forces this bootstrap, that + // made the armed regime universal: measured on the quiet mini, adding one + // `for (const _x of [1]) {}` to `main()` cost `churn` +28% and `tree` +29%, + // with `layout_forget_object` self time going 112 → 916 ms and 194 → 740 ms + // respectively. The scope makes these objects declare + // `GC_LAYOUT_UNKNOWN` (the tag-checked scan — the universally safe state) + // instead of minting a mask nothing will ever remove. See + // `gc::ImmortalLayoutScope` for the soundness argument and for why it is + // deliberately NOT applied to typed-shape layouts. + let _immortal = crate::gc::ImmortalLayoutScope::new(); + let bootstrap_started = crate::gc::gc_diag_enabled().then(std::time::Instant::now); let scope = crate::gc::RuntimeHandleScope::new(); let singleton_handle = scope.root_raw_mut_ptr(singleton_at_entry); let singleton = || singleton_handle.get_raw_mut_ptr::(); @@ -740,6 +759,20 @@ pub(crate) fn populate_global_this_builtins(singleton_at_entry: *mut ObjectHeade // same function object as `Array.prototype.toString`. Alias it now that // both the Array constructor and the TypedArray intrinsic are set up. alias_typed_array_proto_to_string(singleton()); + // The bootstrap's own cost, and the evidence that the `ImmortalLayoutScope` + // above actually did its job. `slot_masks`/`typed` are the live entry + // counts of the two per-object layout side tables: they must still read + // `0 0` here, because a non-zero count is exactly what disables + // `PER_OBJECT_LAYOUTS_NONEMPTY` for the rest of the process. + if let Some(started) = bootstrap_started { + let (slot_masks, typed) = crate::gc::per_object_layout_table_sizes(); + eprintln!( + "[gc-globalthis-bootstrap] elapsed_us={} per_object_slot_masks={} per_object_typed_layouts={}", + started.elapsed().as_micros(), + slot_masks, + typed + ); + } } /// Install `%TypedArray%.prototype.toString` as the same closure object as