fix(gc): bound and heap-scale the untraced-promotion budget (#7902)

changelog.d/7944-untraced-promotion-bound.md

@@ -0,0 +1,58 @@
+Bounded the untraced whole-block promotion path's worst-case retained garbage, and
+stopped it reporting bytes nobody has looked at as measured-live.
+
+Untraced promotion (#7888) skips the trace when the *previous* cycle measured a
+near-fully-live young generation, and then promotes everything as
+`PromotionLiveness::AssumeAllLive`. Four things made the resulting exposure much larger
+than the 32 MiB footprint the comments claimed, and invisible to the pacing that would
+otherwise have noticed:
+
+**1. The dead-byte charge was zero on exactly the workloads that reach the path.**
+`note_untraced_promotion()` extrapolated dead bytes from `LAST_YOUNG_SURVIVAL_PERMILLE`
+verbatim, so a stationary 1000‰ reading implied `1000 − 1000 = 0` and
+`PROMOTED_DEAD_BUDGET_BYTES` was never charged. The predictor is by construction the
+*previous* cycle's answer and says nothing about the cohort being promoted now — charging
+zero is not "no garbage", it is "no answer". The extrapolation is now clamped at
+`UNTRACED_PROMOTION_SURVIVAL_PERMILLE`, the worst ratio the decision itself admits. That
+is also the figure `UNTRACED_PROMOTION_SURVIVAL_PERMILLE`'s own doc already derived its
+1.28 MB bound from: the doc described a bound the code did not enforce.
+
+**2. The remaining bound was unbounded above.** `untraced_promotion_budget_bytes()` was
+`max(128 MiB, old-gen-at-last-measurement)`, so on a large live old heap an abrupt
+live→dead phase change could park an old-heap-sized cohort of assumed-live garbage before
+anything re-measured. It is now
+`min(max(floor, old-gen-at-last-measurement), ceiling)` with an explicit
+`UNTRACED_PROMOTION_CEILING_BYTES`. The budget *is* the worst-case retained-garbage bound
+— every byte it admits is assumed live — and it is now statable.
+
+**3. The 128 MiB floor ignored a configured heap budget.** Both floor and ceiling now run
+through `budget_scaled_with`, giving a quarter and a half of `PERRY_GC_HEAP_LIMIT`
+respectively. A device heap smaller than 128 MiB no longer carries a 128 MiB
+assumed-live allowance.
+
+**4. Assumed-live bytes were credited to the *clean* old-reclaim baseline.**
+`credit_promoted_bytes_to_old_baseline()` exists because promoted bytes are "live by
+construction" — a marked-liveness claim an untraced cycle does not make. Crediting them
+told old-reclaim pacing that an unexamined cohort was clean, deferring the very collection
+that could decide it. Untraced promotions no longer feed that baseline.
+
+**Recovery on contradiction.** When the forced measuring cycle lands and measures *below*
+the untraced threshold while an untraced run is outstanding, `note_young_survival()` now
+sets `GC_OLD_RECLAIM_PENDING`. Nothing else would: the traced minor measures only its own
+young generation, so it can neither identify nor reclaim a cohort the preceding untraced
+cycles already moved into old-gen, and a phase-changed program's heap has stopped growing
+so growth pressure may never fire.
+
+**Tests** (`gc::tests::promote_in_place`): a stationary 1000‰ predictor still charges the
+threshold's implied dead bytes and still closes the composite decision once the footprint
+cap is spent; the budget is asserted in both the unconstrained arm (floor, proportional
+middle, ceiling) and the constrained arm (a quarter of the budget, never more than half
+even against an old generation that fills it) through a pure
+`untraced_promotion_budget_with` so neither arm needs the process environment; and the
+contradiction path is asserted in all three states — a confirming measurement schedules
+nothing, a contradicting one with an outstanding untraced run schedules the reclaim, and
+a low measurement with no untraced run behind it schedules nothing.
+
+Still open from the issue: the end-to-end `1000‰ live phase → dead churn phase` ratchet
+asserting old-gen growth, full-GC timing, RSS and `heapUsed` against the documented bound.
+That is a benchmark-host artifact rather than a unit test.

crates/perry-runtime/src/gc/copying.rs

@@ -1802,7 +1802,16 @@ pub(super) fn gc_collect_minor_copying_fast_path_with_eligibility(
     // old-reclaim baseline BEFORE the pressure check below, or the check reads
     // the stale baseline and schedules a full that is guaranteed to free
     // nothing (see `credit_promoted_bytes_to_old_baseline`).
-    credit_promoted_bytes_to_old_baseline(collector.stats.promoted_bytes);
+    //
+    // #7902: "live by construction" is a MARKED-liveness claim, and an untraced
+    // promotion makes none — it uses `PromotionLiveness::AssumeAllLive`. Those
+    // bytes are the uncertain class the untraced budget bounds, so crediting
+    // them here would tell old-reclaim pacing that a cohort nobody has looked
+    // at is clean, and defer the very collection that could decide it. Charge
+    // only what a traced cycle actually marked.
+    if !untraced {
+        credit_promoted_bytes_to_old_baseline(collector.stats.promoted_bytes);
+    }
     // Everything outside from-space retains its pre-minor accounting. Remove
     // the entire Eden/active-survivor high-water, then add back exactly the
     // objects that survived by copy or promotion. This also preserves objects

crates/perry-runtime/src/gc/policy.rs

@@ -1613,6 +1613,22 @@ pub(super) fn maybe_schedule_old_reclaim_after_copied_minor() {
     }
 }
 
+/// #7902: a traced cycle contradicted the predictor that admitted `bytes` of
+/// untraced (assumed-live) promotion, so schedule the old-gen reclaim that can
+/// actually decide their liveness.
+///
+/// Nothing else will: the traced cycle measures only its own young generation,
+/// so it can neither identify nor reclaim a cohort the preceding untraced
+/// cycles already moved into old-gen. Left alone the bytes sit there until
+/// growth pressure fires — which it may not, because a phase-changed program's
+/// heap has stopped growing.
+pub(super) fn request_old_reclaim_for_untraced_promotions(bytes: usize) {
+    if bytes == 0 {
+        return;
+    }
+    GC_OLD_RECLAIM_PENDING.with(|pending| pending.set(true));
+}
+
 pub(super) fn finish_full_old_reclaim_baseline() {
     // Baseline includes external side-buffer bytes (#6010) so the growth
     // delta in `old_reclaim_pressure_due` stays unit-consistent.

crates/perry-runtime/src/gc/promote_in_place.rs

@@ -94,18 +94,46 @@ pub(super) const PROMOTE_SURVIVAL_THRESHOLD_PERMILLE: u64 = 950;
 /// assumed-live-but-dead bytes here against 0.128 MB at 999 — both far under
 /// the 32 MB cap, which means the binding bound is the untraced-bytes budget in
 /// either case, and that is unchanged.
+///
+/// #7902: that arithmetic is what the code does only because `permille` is now
+/// CLAMPED to this threshold (see [`implied_dead_bytes`]). Taken from the last
+/// measurement verbatim, a stationary 1000‰ reading charged zero and the
+/// paragraph above described a bound nothing enforced. The untraced-bytes
+/// budget remains the binding bound, and it is now itself capped — see
+/// [`untraced_promotion_budget_bytes`].
 pub(super) const UNTRACED_PROMOTION_SURVIVAL_PERMILLE: u64 = 990;
 
-/// Floor for the untraced-promotion budget — see
+/// Floor for the untraced-promotion budget on an UNCONSTRAINED heap — see
 /// [`untraced_promotion_budget_bytes`].
 ///
 /// Two young-cap ceilings (2 × 64 MB). The traced path's own misprediction
 /// bound is "at most ONE nursery of retained garbage before the policy turns
 /// itself off" (see the module docs); this is that bound doubled, which is
 /// what buys a fully-live workload a whole run of free cycles instead of
 /// re-measuring every other one.
+///
+/// #7902: this is a DEFAULT, not a minimum. A process with a configured heap
+/// budget gets a quarter of it instead — 128 MB of assumed-live retention is
+/// larger than some intended device heaps outright, and the floor was being
+/// applied unconditionally.
 pub(super) const UNTRACED_PROMOTION_FLOOR_BYTES: usize = 128 * 1024 * 1024;
 
+/// Hard ceiling on the untraced-promotion budget, i.e. on the worst-case
+/// retained-garbage exposure of the untraced path (#7902).
+///
+/// The budget's relative half (`old-gen at the last measurement`) exists so a
+/// program with a large genuinely-live old heap can keep running free cycles.
+/// But that half was uncapped, so on a multi-GB old generation an abrupt
+/// live→dead phase change could park an old-heap-sized cohort of assumed-live
+/// garbage before anything re-measured. The exposure is now bounded by
+/// `min(max(floor, old-gen-at-last-measurement), this)` — an explicit,
+/// statable worst case rather than "whatever the heap happens to be".
+///
+/// 4 × the unconstrained floor: it keeps the proportional behaviour across the
+/// range where it is cheap (old-gen up to 512 MB) and stops it exactly where a
+/// misprediction would cost more than one forced traced cycle is worth.
+pub(super) const UNTRACED_PROMOTION_CEILING_BYTES: usize = 512 * 1024 * 1024;
+
 /// Running cap on dead bytes promoted in place since the last full collection.
 ///
 /// The per-cycle bound above is self-correcting; this bounds the pathological
@@ -216,21 +244,55 @@ pub(super) fn should_promote_young_in_place() -> bool {
         .is_some_and(|permille| permille >= PROMOTE_SURVIVAL_THRESHOLD_PERMILLE)
 }
 
-/// How many untraced-promoted bytes may accumulate before a cycle has to
-/// measure again.
+/// The untraced-promotion budget, and therefore **the worst-case retained
+/// garbage of the untraced path**: every byte it admits is *assumed* live, so
+/// after an abrupt live→dead phase change every one of them can be garbage
+/// until the forced measuring cycle and a following full collection.
+///
+/// `min(max(floor, old-gen as it stood at the last MEASUREMENT), ceiling)`:
 ///
-/// `max(floor, old-gen as it stood at the last MEASUREMENT)`: an untraced run
-/// may not more than double the old generation it started from. The relative
-/// half lets a program with a large genuinely-live old heap keep running free
-/// cycles — its exposure is proportional to memory it already holds — and the
-/// absolute floor keeps the rule from re-measuring constantly while old-gen is
-/// still small.
+/// * the relative half lets a program with a large genuinely-live old heap keep
+///   running free cycles — its exposure is proportional to memory it already
+///   holds;
+/// * the floor keeps the rule from re-measuring constantly while old-gen is
+///   still small, and (#7902) scales to a configured heap budget rather than
+///   parking a flat 128 MB on a device heap smaller than that;
+/// * the ceiling (#7902) makes the worst case statable. Without it the bound
+///   grew with old-gen without limit, so a phase-changing server could park a
+///   whole old-heap's worth of dead-but-accounted-live memory.
 ///
 /// It has to be the size at the last measurement, not the size now: the
 /// untraced bytes ARE old-gen bytes, so comparing against the current figure
 /// compares a quantity with itself and the relative half can never fire.
-fn untraced_promotion_budget_bytes() -> usize {
-    UNTRACED_PROMOTION_FLOOR_BYTES.max(OLD_GEN_AT_LAST_MEASUREMENT.with(Cell::get))
+pub(super) fn untraced_promotion_budget_bytes() -> usize {
+    untraced_promotion_budget_with(
+        super::gc_heap_budget_bytes(),
+        OLD_GEN_AT_LAST_MEASUREMENT.with(Cell::get),
+    )
+}
+
+/// Pure form of [`untraced_promotion_budget_bytes`], so both the constrained
+/// and unconstrained arms are asserted without poking the process environment.
+pub(super) fn untraced_promotion_budget_with(
+    heap_budget: Option<usize>,
+    old_gen_at_last_measurement: usize,
+) -> usize {
+    // A quarter of a constrained budget; the historical 128 MB otherwise. One
+    // young cap (4 MB) is the floor's own floor — below that the policy would
+    // re-measure on essentially every cycle and #7888 would not exist.
+    let floor = super::budget_scaled_with(
+        heap_budget,
+        UNTRACED_PROMOTION_FLOOR_BYTES,
+        1,
+        4,
+        4 * 1024 * 1024,
+    );
+    // The ceiling scales the same way, so a constrained process never admits
+    // more retained garbage than its own budget allows.
+    let ceiling =
+        super::budget_scaled_with(heap_budget, UNTRACED_PROMOTION_CEILING_BYTES, 1, 2, floor)
+            .max(floor);
+    floor.max(old_gen_at_last_measurement).min(ceiling)
 }
 
 /// Should this promoting cycle also skip the trace?
@@ -275,18 +337,39 @@ pub(super) fn should_promote_young_untraced() -> bool {
 ///   footprint bound; they differ only in whether the dead figure is measured
 ///   or extrapolated, and the untraced budget is what bounds the extrapolation.
 pub(super) fn note_untraced_promotion(promoted_bytes: usize, promoted_objects: usize) {
-    let dead_permille =
-        1000u64.saturating_sub(LAST_YOUNG_SURVIVAL_PERMILLE.with(Cell::get).unwrap_or(0));
-    let implied_dead = (promoted_bytes as u64)
-        .saturating_mul(dead_permille)
-        .checked_div(1000)
-        .unwrap_or(0) as usize;
-    PROMOTED_DEAD_BYTES.with(|c| c.set(c.get().saturating_add(implied_dead)));
+    PROMOTED_DEAD_BYTES.with(|c| {
+        c.set(c.get().saturating_add(implied_dead_bytes(
+            promoted_bytes,
+            LAST_YOUNG_SURVIVAL_PERMILLE.with(Cell::get),
+        )))
+    });
     UNTRACED_PROMOTED_BYTES.with(|c| c.set(c.get().saturating_add(promoted_bytes)));
     UNTRACED_PROMOTION_CYCLES.with(|c| c.set(c.get().saturating_add(1)));
     UNTRACED_PROMOTED_OBJECTS.with(|c| c.set(c.get().saturating_add(promoted_objects as u64)));
 }
 
+/// Dead bytes an untraced promotion of `promoted_bytes` implies (#7902).
+///
+/// The extrapolation is capped at [`UNTRACED_PROMOTION_SURVIVAL_PERMILLE`], not
+/// taken from the last measurement verbatim. A stationary 1000‰ measurement
+/// says nothing about the cycle being promoted — it is by construction the
+/// PREVIOUS cycle's answer — so charging `1000 − 1000 = 0` disarmed
+/// [`PROMOTED_DEAD_BUDGET_BYTES`] entirely on exactly the workloads that enter
+/// this path. The most optimistic honest assumption is the worst ratio the
+/// decision itself admits, which is also the figure
+/// [`UNTRACED_PROMOTION_SURVIVAL_PERMILLE`]'s own doc computes its 1.28 MB
+/// bound from: before this the doc and the code disagreed.
+fn implied_dead_bytes(promoted_bytes: usize, last_survival_permille: Option<u64>) -> usize {
+    let assumed_survival = last_survival_permille
+        .unwrap_or(0)
+        .min(UNTRACED_PROMOTION_SURVIVAL_PERMILLE);
+    let dead_permille = 1000u64.saturating_sub(assumed_survival);
+    (promoted_bytes as u64)
+        .saturating_mul(dead_permille)
+        .checked_div(1000)
+        .unwrap_or(0) as usize
+}
+
 /// Cycles that promoted without tracing, and the objects they promoted.
 pub fn untraced_promotion_cycles() -> u64 {
     UNTRACED_PROMOTION_CYCLES.with(Cell::get)
@@ -309,7 +392,7 @@ pub(crate) fn untraced_promoted_bytes_since_measurement() -> usize {
 pub(super) fn note_young_survival(young_bytes: usize, live_bytes: usize) {
     // A real measurement landed, so the untraced run it ends is settled and the
     // next run's budget is taken against the heap this one leaves behind.
-    UNTRACED_PROMOTED_BYTES.with(|c| c.set(0));
+    let untraced_run_bytes = UNTRACED_PROMOTED_BYTES.replace(0);
     OLD_GEN_AT_LAST_MEASUREMENT.with(|c| c.set(crate::arena::old_gen_in_use_bytes()));
     if young_bytes == 0 {
         return;
@@ -320,6 +403,16 @@ pub(super) fn note_young_survival(young_bytes: usize, live_bytes: usize) {
         .unwrap_or(0)
         .min(1000);
     LAST_YOUNG_SURVIVAL_PERMILLE.with(|c| c.set(Some(permille)));
+    // #7902: this measurement is the FIRST evidence about the cohort the
+    // preceding untraced cycles promoted on faith. If it contradicts the
+    // predictor that admitted them, that cohort is probably garbage sitting in
+    // old-gen — and nothing else will look at it, because the traced cycle
+    // measures only its own young generation and old-reclaim pacing was told
+    // the promoted bytes were live. Ask for the old-gen reclaim now instead of
+    // waiting for growth pressure to notice a heap that is not growing.
+    if untraced_run_bytes > 0 && permille < UNTRACED_PROMOTION_SURVIVAL_PERMILLE {
+        super::request_old_reclaim_for_untraced_promotions(untraced_run_bytes);
+    }
 }
 
 /// Charge the dead bytes an in-place promotion just moved into old-gen against