test(next): add the warm-process soak arm #8163's residual needs

[proggeramlug]

Refs #8163. Adds the acceptance instrument that issue needs, and shows why the existing one cannot serve.

Why

fixture.sh's cold-start loop makes two verifier passes per process. Two facts about that, both measured here:

1. In NORMAL mode, two passes run ZERO copying minors. The normal cold-start arm therefore cannot exercise a moving-GC holder at all — only the forced arm moves anything. That is a stronger statement than "insensitive": there is nothing for a rooting bug to survive.
2. [Next.js/dylib] Forced-evacuation App Route arm: stale closure from a holder outside the GC heap #8163's residual costs ~1 broken request per 100 verifier passes in a warm process, so even a hundred passes is not a verdict: at p = 0.01 a clean 100-pass run happens ~37% of the time on a known-broken build — and two of five measured 100-pass runs did come back clean while the bug was present.

So "the fixture is green" has never been evidence about this class, and closing #8163 on it would be the failure mode CLAUDE.md names — a gate that cannot fail.

What

run_warm_soak: one warm process, N verifier passes, default GC (forcing evacuation would measure the arm that is already fixed).

• OFF by default (PERRY_NEXT_ROUTE_WARM_PASSES=0) because a meaningful N is slow. This is the close-condition instrument for [Next.js/dylib] Forced-evacuation App Route arm: stale closure from a holder outside the GC heap #8163, not a per-run check.

• Prints the confidence its N actually buys instead of letting any green run imply elimination — rule of three, ~3/p passes for 95%:

  N	verdict printed
  10	10% — NOT sufficient, a broken build passes this often
  100	63% — NOT sufficient
  300	95% — sufficient to claim elimination
  460	99% — sufficient

• Asserts its subject was live (copying minors > 0), the same rule the forced arm's evacuation-liveness assert applies.

• Reports an observed failure ahead of the liveness complaint. I had these the other way round at first, and a real broken request got masked behind "exercised nothing" — caught by running the arm, not by reading it.

Validation — every path exercised against real provider images

run	result
N=0	prints guidance, returns 0
pre-fix providers, N=100	caught it — pass 19, 1 host TypeError, 21 copying minors
merged-fix providers, N=100	caught the residual — pass 3, 22 minors
merged-fix providers, N=10	caught the residual — pass 6, 2 minors
N=2, clean	0 failures, 0 copying minors → liveness assert fires (this is finding #1 above)
liveness assert, sabotaged grep	fires with "exercised nothing"
confidence print	verified at N = 10/100/300/460

The middle two rows are an independent local reproduction of the residual that @proggeramlug measured on the bench mini — same signature, different host, on the merged fix.

Note for whoever closes #8163

The pass numbers that fail (3, 6, 12, 19 across runs) track collection boundaries, not request counts: N=10 produced 2 copying minors and one failure. The per-collection failure rate is much higher than the per-pass rate, so if the hunt wants a faster reproducer, drive collections rather than requests.

Summary by CodeRabbit

• Tests

  • Added an optional warm-process soak test for Next.js route releases.
  • Repeated verification passes detect intermittent failures, collection issues, and copying-minor activity.
  • Reports observed failures and statistical confidence against residual failure rates.
  • Remains disabled by default and supports configurable pass counts.

• Documentation

  • Added release documentation covering configuration, validation scenarios, and expected behavior.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 2047c905-2360-4800-a4a2-eba6c3ea091a

📥 Commits

Reviewing files that changed from the base of the PR and between 6d26e2c and 38f3d38.

📒 Files selected for processing (1)

• tests/release/packages/next-app-route/fixture.sh

🚧 Files skipped from review as they are similar to previous changes (1)

• tests/release/packages/next-app-route/fixture.sh

Included review availability: Your plan includes up to 8 reviews per rolling hour; 2 remain after this review.

---

📝 Walkthrough

Walkthrough

The Next App Route fixture adds an opt-in warm-process soak arm. It repeats verifier passes, checks request and GC results, reports statistical confidence, and runs after cold-start validation. The changelog documents its behavior and validation cases.

Changes

Next App Route warm-soak validation

Layer / File(s)	Summary
Warm-soak execution and integration tests/release/packages/next-app-route/fixture.sh, changelog.d/8215-next-route-warm-soak.md	The fixture adds configurable repeated verification in one normal-GC process. It tracks request failures, host TypeErrors, and copying-minor activity. It rejects observed failures or insufficient collection activity. The phase runs after cold-start validation. The changelog records configuration, assertions, statistical limits, and validation results.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to 38f3d

This change adds an opt-in warm-process soak test without altering default behavior, and no actionable merge-blocking risk remains after normal checks and review.

Possibly related PRs

• PerryTS/perry#7015: The warm-soak verifies repeated warm-process passes and observes copying-minor activity.
• PerryTS/perry#7687: The warm-soak exercises repeated allocation-point GC behavior.
• PerryTS/perry#8161: The change extends the Next.js App Route fixture with warm-process validation.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the addition of the warm-process soak arm for the Next.js test and references issue #8163.
Description check	✅ Passed	The description explains the purpose, implementation, linked issue, and validation results, although it omits some template headings and checklist items.
Linked Issues check	✅ Passed	The warm-process soak adds an opt-in acceptance path for #8163 with repeated verification, liveness checks, and residual-failure detection.
Out of Scope Changes check	✅ Passed	The changes are limited to the next-app-route fixture and its changelog, both directly related to the #8163 acceptance instrument.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch gc/8163-warm-soak-instrument

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[proggeramlug]

Validation status vs current main, stated explicitly

The arm in this PR was validated against real provider images built from b8d32ab19, plus the pinned app dylib — every path exercised, both asserts sabotage-tested (see the PR body).

It cannot currently be re-validated at HEAD, and not for any reason belonging to this change: #8228 means the in-process LLVM backend fails 5 of the fixture's 104 modules on current main (insertelement <2 x i64> from #8204's module-init header compose has no case in perry-codegen/src/dialect/mod.rs), and app-route.runtime.prod.js is among them — so fixture.sh step 5 cannot produce an app dylib on main at all right now. That gates every arm of this fixture equally, not just the new one.

This PR is orthogonal to that: it adds a shell function and touches no compiled code, so it neither causes nor worsens #8228, and landing it does not make the fixture any less runnable than it already is. Flagging it so a future reader does not mistake "validated" for "validated against HEAD" — the artifacts it was proven on predate #8204.