diff --git a/changelog.d/7015-gc-safepoint-cycle-completion.md b/changelog.d/7015-gc-safepoint-cycle-completion.md new file mode 100644 index 0000000000..1073f1ad14 --- /dev/null +++ b/changelog.d/7015-gc-safepoint-cycle-completion.md @@ -0,0 +1,56 @@ +### Fixed + +- **GC: a budgeted incremental cycle that nothing drives now completes (#6978).** + The budgeted stepper had a budget but no completion guarantee. It is driven + by exactly two things — allocation-point mutator assists (`gc_check_trigger`) + and host safepoints (`gc_runtime_safepoint`, from the microtask checkpoint + and the stdlib pump) — so a program that stops allocating stopped driving it + and the cycle parked for the life of the process. + + Measured on `test_gap_repsel_canonical_i32` under `PERRY_GC_HEAP_LIMIT=8` + (release, macOS arm64): `gc_check_trigger` runs **twice in the whole + process**. The trigger is `ArenaBytes` — this program never calls + `gc_malloc` at all (`malloc_count=0` against a 100 000 trigger), so the + malloc-trigger mechanism #6978 hypothesised is not involved. The second call + arms the cycle and pays one 256-unit assist, and no allocation-point + opportunity ever comes again; the host safepoints that follow each pay the + fixed 2 048-unit slice and get the cycle through `BuildValidPointerSet` and + one step into `RootScan` before the process exits. The assist *budget* was + never the binding constraint — the number of opportunities was, and a cycle + has seven resumable phases with one `step()` advancing at most one of them, + so completion needs a bounded number of calls rather than more units. + + A parked cycle is not inert: nothing is reclaimed, the arming trigger is + never re-baselined, every subsequent allocation is born **black** so the + parked cycle can never collect it, the incremental mark barrier stays armed + on every store, and `gc_safepoint_moving_minor` (the precise-root copying + minor at the outermost microtask boundary) early-returns on + `gc_budgeted_cycle_active()` — so the parked cycle also disabled the + collector's own moving path. Net effect on a compiled program in the shipped + configuration: zero completed collections. + + The host safepoint now carries the guarantee — the one point in the process + where the mutator has yielded and the JS stack has unwound. A cycle may span + `GC_CYCLE_HOST_SAFEPOINT_LIMIT` (2) safepoints on the ordinary bounded + budget; at the next one the safepoint drives it to completion, in a loop + bounded exactly like `gc_drain_active_budgeted_cycle`. Kill switch + `PERRY_GC_SAFEPOINT_FINISH=0`. + + The limit does not bind on healthy workloads, measured: on + `test_gap_repsel_gc_stress` the budgeted cycles complete on mutator assists + alone (20 / 25 / 35 assist steps, **zero** host safepoints), and an async + probe that yields every 256 iterations while allocating reports + `safepoint_steps=0` in every configuration. + + `PERRY_GC_TRACE=1` cycle counts under `PERRY_GC_HEAP_LIMIT=8` with no other + GC env var: `test_gap_repsel_canonical_i32` 0 → 1, + `test_gap_repsel_ptr_shape_locals` 0 → 1, `test_gap_repsel_gc_stress` + 21 → 22, all still byte-exact against node 26.5.0. + `scripts/gc_repsel_matrix.sh --arms all --pressure 8` (440 cells, same + binaries A/B'd through the kill switch): `PASS=229 UNVER=190 XFAIL=1 + FAIL=20` → `PASS=400 UNVER=19 XFAIL=1 FAIL=20`, the only cell transition + being `UNVER → PASS` × 171, with an identical FAIL set (#6981) and no + `PASS → non-PASS`. Costs one collection's worth of work that previously + never ran (+1.2 – 2.7 % wall clock where a heap budget makes a trigger due, + 0 % without one) and does **not** regress max pause: the added cycle's + largest step is 16.0 ms against the run's pre-existing 54.6 ms maximum. diff --git a/crates/perry-runtime/src/gc/policy.rs b/crates/perry-runtime/src/gc/policy.rs index 2f2ef7baae..bd5acd65a3 100644 --- a/crates/perry-runtime/src/gc/policy.rs +++ b/crates/perry-runtime/src/gc/policy.rs @@ -1757,6 +1757,12 @@ fn gc_finish_budgeted_cycle(mut cycle: BudgetedGcCycle) -> JsGcStepResult { } } GC_BUDGETED_CYCLE_ACTIVE.with(|active| active.set(false)); + // #6978: the host-safepoint allowance is charged PER CYCLE. Reset it on + // every completion, whatever completed the cycle — a mutator assist can + // finish one cycle and arm the next entirely between two safepoints, and + // the next cycle must not inherit its predecessor's spent allowance and be + // finished on its very first safepoint. + gc_reset_host_safepoint_starvation(); gc_step_result( JS_GC_STEP_STATUS_COMPLETED, GcCyclePhase::Complete.ffi_code(), @@ -1884,12 +1890,54 @@ fn gc_mutator_assist_step_work_units_inner_with_progress( gc_budgeted_step_work_units_inner_with_progress(work_units, start_progress_kind) } +/// Host safepoint: the mutator has yielded (microtask checkpoint / stdlib +/// pump) and the JS stack has unwound. +/// +/// Ordinarily this pays one `NormalIncremental`-budgeted slice. #6978: once a +/// cycle has been offered `GC_CYCLE_HOST_SAFEPOINT_LIMIT` such slices without +/// completing, nothing else in the process is obliged to finish it — the +/// allocation-point assists that would have are only emitted by allocation +/// the program may never do again — so finish it here. One `step()` advances +/// at most one phase even with an unbounded budget, so completion is a loop, +/// bounded exactly like `gc_drain_active_budgeted_cycle`: seven phases plus +/// slack, and a blocked stepper (suppression / unsafe zone / root lock) +/// reports SKIPPED without progress, so bail then rather than spin. pub(crate) fn gc_runtime_safepoint() -> JsGcStepResult { let budget = gc_progress_contract().budget_for(GcProgressKind::NormalIncremental); let Some(work_units) = budget.work_units else { return gc_budgeted_status_result(); }; - gc_budgeted_step_work_units_inner_with_progress(work_units, GcProgressKind::NormalIncremental) + if !gc_host_safepoint_starvation_due() { + return gc_budgeted_step_work_units_inner_with_progress( + work_units, + GcProgressKind::NormalIncremental, + ); + } + if std::env::var_os("PERRY_GC_DIAG").is_some() { + eprintln!( + "[gc-safepoint] finishing a cycle parked across {GC_CYCLE_HOST_SAFEPOINT_LIMIT} host safepoints" + ); + } + let mut result = gc_budgeted_step_work_units_inner_with_progress( + usize::MAX, + GcProgressKind::NormalIncremental, + ); + for _ in 0..64 { + if result.status != JS_GC_STEP_STATUS_ACTIVE { + break; + } + result = gc_budgeted_step_work_units_inner_with_progress( + usize::MAX, + GcProgressKind::NormalIncremental, + ); + } + // A completion resets the allowance in `gc_finish_budgeted_cycle`. Do NOT + // reset it here on any other exit: if the stepper was blocked (suppression + // / unsafe zone / root lock) or the phase loop ran out, the cycle is still + // parked, and clearing the count would send it back through another + // `GC_CYCLE_HOST_SAFEPOINT_LIMIT` bounded slices before retrying — on a + // program with few safepoints, possibly never. + result } fn write_gc_step_result(out: *mut JsGcStepResult, result: JsGcStepResult) -> u32 { diff --git a/crates/perry-runtime/src/gc/progress.rs b/crates/perry-runtime/src/gc/progress.rs index d4c45d34dd..489a13b82d 100644 --- a/crates/perry-runtime/src/gc/progress.rs +++ b/crates/perry-runtime/src/gc/progress.rs @@ -26,6 +26,95 @@ pub const GC_MUTATOR_ASSIST_SOFT_PAUSE_US: u64 = 500; /// workloads never see the scaled assists. pub const GC_ASSIST_DEBT_BYTES_PER_WORK_UNIT: u64 = 32; +/// COMPLETION GUARANTEE (#6978): the most host safepoints one budgeted cycle +/// may span on the ordinary bounded budget before the safepoint finishes it +/// outright. +/// +/// The budgeted stepper has a *budget* but no *completion guarantee*, and it +/// is driven by exactly two things: allocation-point mutator assists +/// (`gc_check_trigger`) and host safepoints (`gc_runtime_safepoint`, called +/// from the microtask checkpoint and the stdlib pump). A program that stops +/// allocating stops driving it. Measured on `test_gap_repsel_canonical_i32` +/// under `PERRY_GC_HEAP_LIMIT=8` (release, macOS arm64): `gc_check_trigger` +/// runs **twice in the whole process** — the second call arms an `ArenaBytes` +/// cycle and pays one 256-unit assist, and no allocation-point opportunity +/// ever comes again. The host safepoints that follow each paid the fixed +/// `GC_NORMAL_INCREMENTAL_WORK_UNITS` slice, which got the cycle through +/// `BuildValidPointerSet` and one step into `RootScan` before the process +/// exited. Note a fatter per-step budget cannot rescue this on its own: a +/// cycle has seven resumable phases and one `step()` advances at most one of +/// them, so completion needs a bounded *number of calls*, not more units. +/// +/// A PARKED CYCLE IS NOT INERT. Until it completes: +/// * nothing is reclaimed and the arming trigger is never re-baselined; +/// * every subsequent allocation is born BLACK (`gc_birth_extra_flags`), so +/// the parked cycle can never collect it; +/// * the incremental mark barrier stays armed on every store; and +/// * `gc_safepoint_moving_minor` — the precise-root copying minor at the +/// outermost microtask boundary — returns early on +/// `gc_budgeted_cycle_active()`, so the parked cycle also disables the +/// collector's own moving path for the rest of the process. +/// Net effect on a compiled program in the shipped configuration: ZERO +/// completed collections. +/// +/// A host safepoint is where the mutator has yielded and the JS stack has +/// unwound — the cheapest and safest point in the process to finish a +/// collection. So: a cycle may span this many host safepoints on the ordinary +/// bounded budget; at the next one the safepoint drives it to completion. +/// Measured cost on the corpus member that actually collects +/// (`test_gap_repsel_gc_stress`, `--pressure 8`): its cycles complete on +/// mutator assists alone (20 / 25 / 35 assist steps, **zero** host +/// safepoints), so a healthy allocating workload never reaches this limit and +/// pays nothing. The bound binds only when the mutator has stopped driving +/// the collector — precisely when parking forever is the alternative. +/// +/// Kill switch: `PERRY_GC_SAFEPOINT_FINISH=0` (also `off` / `false`). +pub const GC_CYCLE_HOST_SAFEPOINT_LIMIT: u32 = 2; + +std::thread_local! { + /// How many host safepoints the currently-active budgeted cycle has + /// already been offered. Reset whenever a safepoint finds no active cycle + /// — which covers completion through every path (mutator assist, host + /// safepoint, the drain before a synchronous collection). + static GC_CYCLE_HOST_SAFEPOINTS: std::cell::Cell = + const { std::cell::Cell::new(0) }; +} + +/// Kill switch for the #6978 completion guarantee. Default ON; `0` / `off` / +/// `false` restores the pre-fix behaviour, where every host safepoint takes +/// one bounded step and a cycle nobody drives parks for the life of the +/// process. +fn gc_safepoint_finish_enabled() -> bool { + static CACHED: std::sync::OnceLock = std::sync::OnceLock::new(); + *CACHED.get_or_init(|| { + !matches!( + std::env::var("PERRY_GC_SAFEPOINT_FINISH").as_deref(), + Ok("0") | Ok("off") | Ok("false") + ) + }) +} + +/// Count this host safepoint against the active budgeted cycle and report +/// whether the cycle has now outlived `GC_CYCLE_HOST_SAFEPOINT_LIMIT` of them +/// — i.e. whether this safepoint must finish it instead of taking another +/// bounded slice. +pub(super) fn gc_host_safepoint_starvation_due() -> bool { + if !super::gc_budgeted_cycle_active() || !gc_safepoint_finish_enabled() { + gc_reset_host_safepoint_starvation(); + return false; + } + let seen = GC_CYCLE_HOST_SAFEPOINTS.with(|seen| { + let next = seen.get().saturating_add(1); + seen.set(next); + next + }); + seen > GC_CYCLE_HOST_SAFEPOINT_LIMIT +} + +pub(super) fn gc_reset_host_safepoint_starvation() { + GC_CYCLE_HOST_SAFEPOINTS.with(|seen| seen.set(0)); +} + /// Runtime-visible classification for GC progress. /// /// Only `NormalIncremental` and `MutatorAssist` satisfy the low-pause diff --git a/crates/perry-runtime/src/gc/tests/host_safepoints.rs b/crates/perry-runtime/src/gc/tests/host_safepoints.rs index becb7beced..dc8953d621 100644 --- a/crates/perry-runtime/src/gc/tests/host_safepoints.rs +++ b/crates/perry-runtime/src/gc/tests/host_safepoints.rs @@ -141,6 +141,135 @@ fn repeated_runtime_safepoints_complete_cycle_rebaseline_debt_and_preserve_roots } } +/// #6978: a budgeted cycle that nothing drives must still complete. +/// +/// The stepper has a budget but no completion guarantee. Its only two drivers +/// are allocation-point mutator assists and host safepoints, so a program +/// that stops allocating parks its cycle — measured on a compiled program, +/// `gc_check_trigger` ran twice in the whole process, and the seven host +/// safepoints that followed each paid one fixed `NormalIncremental` slice and +/// left the cycle in `RootScan`. Parked is not inert: nothing is reclaimed, +/// the arming trigger is never re-baselined, every later allocation is born +/// black, and `gc_safepoint_moving_minor` early-returns on +/// `gc_budgeted_cycle_active()` for the rest of the process. +/// +/// So the host safepoint carries the guarantee: a cycle may span +/// `GC_CYCLE_HOST_SAFEPOINT_LIMIT` safepoints on the ordinary bounded budget, +/// and the next one finishes it. Both halves are asserted — the early +/// safepoints must NOT finish it (that would make incremental mode +/// synchronous), and the one past the limit must, WITHOUT any further +/// allocation to assist it. +#[test] +fn a_cycle_that_nothing_drives_is_finished_within_a_bounded_number_of_host_safepoints() { + let _guard = CopyingNurseryTestGuard::new(1); + let trigger_guard = GcTriggerThresholdTestGuard::suppress_automatic_triggers(); + reset_old_reclaim_pressure(); + make_arena_pressure(&trigger_guard, b"host_safepoint_starved_live"); + + let before = gc_collection_count(); + let started = gc_runtime_safepoint(); + assert_eq!(started.status, JS_GC_STEP_STATUS_ACTIVE); + for offered in 1..=GC_CYCLE_HOST_SAFEPOINT_LIMIT { + let result = gc_runtime_safepoint(); + assert_eq!( + result.status, JS_GC_STEP_STATUS_ACTIVE, + "host safepoint {offered} of {GC_CYCLE_HOST_SAFEPOINT_LIMIT} must stay an \ + ordinary bounded step -- finishing early would make incremental mode synchronous" + ); + assert_eq!(gc_collection_count(), before); + } + + // No allocation has happened since the cycle armed, so no mutator assist + // can ever come. This safepoint is the completion guarantee. + let completed = gc_runtime_safepoint(); + assert_eq!( + completed.status, JS_GC_STEP_STATUS_COMPLETED, + "a cycle offered more than {GC_CYCLE_HOST_SAFEPOINT_LIMIT} host safepoints without \ + completing must be finished by the safepoint, not parked" + ); + assert_eq!(completed.completed, 1); + assert_eq!(completed.active, 0); + assert!(gc_collection_count() > before); + assert_eq!( + js_gc_step_status(std::ptr::null_mut()), + JS_GC_STEP_STATUS_IDLE, + "no budgeted cycle may remain parked after the completion guarantee fires" + ); + assert!( + GC_NEXT_TRIGGER_BYTES.with(|trigger| trigger.get()) > crate::arena::arena_total_bytes(), + "the finished cycle must re-baseline the arming trigger like any other completion" + ); + + let live_after = (js_shadow_slot_get(0) & POINTER_MASK) as *const crate::StringHeader; + unsafe { + assert_string_bytes(live_after, b"host_safepoint_starved_live"); + } +} + +/// #6978 follow-up: the host-safepoint allowance is charged PER CYCLE. +/// +/// A mutator assist can finish one cycle and arm the next entirely between two +/// host safepoints. If the counter were only cleared when a safepoint happens +/// to observe an idle collector, the second cycle would inherit its +/// predecessor's spent allowance and be driven to completion on its very first +/// safepoint — silently synchronous. The FFI stepper stands in here for the +/// mutator assist: it drives cycles without going through +/// `gc_runtime_safepoint`, which is exactly the off-safepoint path. +#[test] +fn the_host_safepoint_allowance_is_charged_per_cycle_not_carried_across_cycles() { + let _guard = CopyingNurseryTestGuard::new(1); + let trigger_guard = GcTriggerThresholdTestGuard::suppress_automatic_triggers(); + reset_old_reclaim_pressure(); + + // Cycle A: spend the whole safepoint allowance without completing it. + make_arena_pressure(&trigger_guard, b"host_safepoint_cycle_a_live"); + assert_eq!(gc_runtime_safepoint().status, JS_GC_STEP_STATUS_ACTIVE); + for _ in 0..GC_CYCLE_HOST_SAFEPOINT_LIMIT { + assert_eq!(gc_runtime_safepoint().status, JS_GC_STEP_STATUS_ACTIVE); + } + + // ...then finish A OFF-safepoint, and arm + start B off-safepoint too, so + // no safepoint ever observes the collector idle in between. + let mut status = JsGcStepResult::default(); + let mut finished_off_safepoint = false; + for _ in 0..64 { + if js_gc_step_work_units(u64::MAX, &mut status) != JS_GC_STEP_STATUS_ACTIVE { + finished_off_safepoint = true; + break; + } + } + assert!( + finished_off_safepoint, + "cycle A should complete through the host-driven stepper" + ); + make_arena_pressure(&trigger_guard, b"host_safepoint_cycle_b_live"); + assert_eq!( + js_gc_step_work_units(1, &mut status), + JS_GC_STEP_STATUS_ACTIVE, + "cycle B should start off-safepoint, the way a mutator assist starts one" + ); + + // Cycle B must get its OWN allowance, not A's leftovers. + for offered in 1..=GC_CYCLE_HOST_SAFEPOINT_LIMIT { + assert_eq!( + gc_runtime_safepoint().status, + JS_GC_STEP_STATUS_ACTIVE, + "safepoint {offered} of the SECOND cycle must still be an ordinary bounded \ + step -- the allowance is per cycle, not per process" + ); + } + assert_eq!( + gc_runtime_safepoint().status, + JS_GC_STEP_STATUS_COMPLETED, + "and the guarantee must still fire for the second cycle" + ); + + let live_after = (js_shadow_slot_get(0) & POINTER_MASK) as *const crate::StringHeader; + unsafe { + assert_string_bytes(live_after, b"host_safepoint_cycle_b_live"); + } +} + #[test] fn microtask_runner_tail_pays_bounded_safepoint_under_pressure() { let _guard = CopyingNurseryTestGuard::new(1); diff --git a/docs/src/internals/memory-model.md b/docs/src/internals/memory-model.md index c2479ecd04..a7df300950 100644 --- a/docs/src/internals/memory-model.md +++ b/docs/src/internals/memory-model.md @@ -117,7 +117,8 @@ Idle nursery blocks observed empty for 2 GC cycles are `dealloc`'d back to the O | `PERRY_GC_FORCE_EVACUATE=1` | With generated write barriers active and policy evacuation allowed, stress-copy every marked non-pinned nursery object instead of only tenured survivors. | | `PERRY_GC_VERIFY_EVACUATION=1` | After an evacuation that actually forwards objects, panic if any mutable live slot still points at a forwarded nursery object after rewrite. | | `PERRY_WRITE_BARRIERS=0` / `off` / `false` | Disable codegen-emitted write barriers at compile time and runtime exact helper barriers at runtime for benchmark/debug bisection. Unset, `=1`, `=on`, and `=true` keep barriers enabled. | -| `PERRY_GC_DIAG=1` | Print per-cycle diagnostics, including one evacuation-policy line for cycles where evacuation was considered and for `barriers_inactive` skips. | +| `PERRY_GC_SAFEPOINT_FINISH=0` / `off` / `false` | Disable the host-safepoint completion guarantee (#6978). A budgeted incremental cycle is driven only by allocation-point mutator assists and host safepoints; a program that stops allocating stops driving it, and a parked cycle reclaims nothing, keeps the mark barrier armed, makes every later allocation born-black, and blocks the moving safepoint minor for the rest of the process. By default a cycle may span `GC_CYCLE_HOST_SAFEPOINT_LIMIT` host safepoints on the ordinary bounded budget and the next one finishes it. Bisection only. | +| `PERRY_GC_DIAG=1` | Print per-cycle diagnostics, including one evacuation-policy line for cycles where evacuation was considered and for `barriers_inactive` skips, and a `[gc-safepoint] finishing a cycle parked across N host safepoints` line whenever the completion guarantee above fires. | ## Why this design