[Next.js/dylib] Warm App Route server retains ~1,100 async box cells per request: 113 KB/req live Rust heap, minor-GC pause grows 59 ms -> 0.5 s

[proggeramlug]

Scoped 2026-08-16. This issue now tracks mechanism (a) only — the async-activation box cells. The two mechanisms below have independent fixes and were blocking each other:

• (b) the append-only Proxy registry moved to GC: the Proxy registry is append-only — every new Proxy() roots its target graph for the life of the thread (#8213 mechanism (b)) #8230.
• (Proxy id band exhaustion is a wild pointer, not an error: the 65,536th new Proxy() in a thread returns a payload the next property read dereferences (SIGSEGV) #8218 / PR fix(proxy): refuse to mint a proxy id past the end of the id band (#8218) #8219) the band-exhaustion crash that (b) leads to — the 65,536th new Proxy() in a thread returned a payload the next property read dereferenced.

Closing condition here is (a): #8208 lands, and the fixture slope (RSS + [rss-diag-tables] box= over 100+ passes) is measured before/after on the #8034 route. The measurements below are kept whole because they were taken together.

Serving the pinned #8034 production App Route fixture (release layout: providers + app dylib from clean main 3c95020f8, C host), one warm process's RSS climbs 160 MB -> 510-620 MB and [gc-step] post_in_use climbs 5.4 -> 39.7 MB over 100 verifier passes (2,100 requests) with no plateau — first noted in the #8040 DoD audit and #8163's 2026-08-16 comment ("it wants its own home"). This issue is that home, with the diagnosis. All numbers measured 2026-08-16 on the bench mini (M1, 8 GB), 21 requests/pass (20 concurrent GETs + 1 POST via verify.mjs), PERRY_GC_DIAG=1 (+PERRY_GC_TRACE=1 where noted). Logs: ~/perry-bench.noindex/logs-rss/ on the mini; throwaway instrumented providers (counting #[global_allocator] wrapper + per-root-source retained-BFS, runtime crate only) in ~/perry-bench.noindex/build-rss-diag/, patch in the session scratchpad — not shipped.

1. The growth is per-REQUEST (~150 KB/req), not per-connection (~0.6 KB/conn)

client	connections	RSS pass 10	RSS pass 100	slope
stock verify.mjs loop (21 fresh conns/pass)	~21/pass	301 MB	609 MB	163 KB/request
http.Agent {keepAlive, maxSockets:1}, sequential	1 total	323 MB	603 MB	148 KB/request
20 persistent sockets, concurrent	20 total	—	600 MB	same
2,100 open/close, no request completes	2,100	—	+1.3 MB total	0.62 KB/conn
2,100 partial request head then close	2,100	—	+1.4 MB total	0.65 KB/conn

One keep-alive socket reproduces the stock slope exactly, and connection churn alone is noise-level. ext-http's per-connection tracking is clean (its GC scanner exposes only 10 live pointer roots at pass ~92).

2. The retained heap is real, rooted retention — 11 full collections reclaimed 0.00 MB

post_in_use climbs 5.4 -> ~50.4 MB over the first ~50 passes, then creeps ~4 KB/cycle. Across two 140-pass runs, all 11 fulls (arena_bytes + old_gen_bytes major pacing) reclaimed nothing from the arena:

full #46:  in_use 48.96 -> 48.96 MB  (pause 592 ms)
full #125: in_use 50.45 -> 50.45 MB  (pause 450 ms)
full #214: in_use 50.73 -> 50.75 MB  (pause 551 ms)
full #259: in_use 50.87 -> 50.87 MB  (pause 868 ms)

The fulls do clear the malloc-object registry (97,950 -> 400 objects at one full — almost all GC_TYPE_REGEXP headers; 12.6 MB allocated / 12.2 MB freed over a run), so RegExp churn collects fine and is not the retainer.

3. Off-heap: 100% of the RSS delta is mimalloc-owned VM, and most of it is LIVE Rust heap

vmmap --summary (one process, pass 0/10/50/100): Memory Tag 240 (mimalloc, os_tag=240) resident grows 95.0 -> 225.6 -> 421.8 -> 526.3 MB; libsystem MALLOC zones stay ~0.5 MB; stack/__DATA/__PERRY_GCMAP flat.

The counting-allocator build decomposes Tag 240 (values at GC cycle 1 ≈ pass 2 vs cycle 33 ≈ pass 119, ~2,450 requests apart, rss-idiag-host.log):

	cycle 1	cycle 33	delta
RSS	292 MB	670 MB	+378 MB
live Rust heap (exact, from the wrapper)	177.5 MB	454.2 MB	+277 MB = ~113 KB/request live
of which GC-arena blocks	25.2 MB	91.2 MB	+66 MB (plateaus with the heap)
non-arena live	152.3 MB	363.0 MB	+211 MB = ~86 KB/request
RSS − live (mimalloc committed-but-free)	115 MB	216 MB	+101 MB

The committed-but-free overlay is real but reclaimable: in a 400-pass run, when a concurrent build put the host under memory pressure, RSS fell 710 -> 405-413 MB in ~2 passes with no behavior change, then re-inflated when pressure eased — macOS MADV_FREE lag + fragmentation, the same mechanism PR #8208 measured on asyncpipe_big.

Live-histogram growth by size class (cycle 1 -> 33):

• 8-byte allocations: 107,666 -> 3,052,271 (0.7 -> 23.5 MB). Box cells: the runtime's box registries grew 68,284 -> 2,686,743 (+1,068 cells/request) plus i32/bool boxes (+60/request) — 94% of the 8 B growth.
• Power-of-2 giant tables step up alongside: at cycle 33 single allocations of 69.2, 37.7, 19.8 MB plus 3x ~11.6 MB (8-16 MB bucket). The 69.2 MB allocation matches a hashbrown usize table at 8M-bucket capacity — the box-registry/GC-census family; not individually named by this instrument.
• 64 B: +76k/+4.9 MB; 384 B: +4.7 MB; 16 B: +303k/+4.2 MB.
• The ≤1 MB bucket grew +69 allocations/+70 MB ≈ exactly the +66 arena blocks.

4. Which structures retain (counted, not guessed)

Side-table growth per request (cycle 1 -> 33, all monotone, none per-connection):

table	growth/request	end count (2,478 req)
BOX_REGISTRY (+i32/bool)	~1,128 cells	2.84 M
PROXIES (proxy.rs)	4.0 entries	10,037 live, 0 revoked
descriptor side tables (props+accessors)	14	49,159
closure dynamic props (fn.x = ...)	3.9	14,742
symbol side table	3.0	7,690
streams id quarantine	bounded	11,741 roots (cap 16,384)
timers / task queue / promise contexts / ALS guards	0	0-22

Per-root-source reachability over the live heap (retained-BFS, cycle 20 ≈ pass 92, live = 36.78 MB / 548k objects; independent walk then marginal in the same order):

root source	reach (independent)	marginal
crate::r#box::scan_box_roots_mut (2.10 M slots, 291,736 live ptrs)	20.41 MB (55%)	20.41 MB
symbol side table	6.74 MB	6.28 MB
crate::proxy::scan_proxy_roots_mut	3.24 MB	2.91 MB
module globals (actual app state)	2.37 MB	1.58 MB
descriptors + closures + streams + rest	~5 MB	~3.4 MB

So the "retained requests" decompose into two mechanisms:

(a) Async-activation box cells — the dominant term in all three symptoms. Every request leaks ~1,100 8-byte box cells whose registry entries are permanent (#7933 cleared values but kept cells+entries). They account for 55% of the retained JS heap (stale values in cells the clearing pass could not touch), ~25 MB of cells + tens of MB of registry table on the Rust heap, and 96% of the minor-GC root-scan slots — which is why minor pause grows 59 ms -> ~460-550 ms over 140 passes (scan slots 261k -> 6.5M; the collector rescans every box ever allocated, every minor). In a long run this compounds: once the trigger regime tightens, the host runs ~70 collections/pass at ~0.5 s each and throughput collapses ~15x.

(b) The Proxy registry is append-only and roots its targets forever. js_proxy_new pushes into the thread-local PROXIES: Vec<Option<Box<ProxyEntry>>> (id = v.len(), no slot reuse); nothing but explicit revocation ever detaches an entry (the only slot.take() is #[cfg(test)]), and scan_proxy_roots_mut marks every entry's target+handler as strong GC roots each cycle. A Perry proxy is an id-band handle, not a heap object, so the GC has no death signal for it — in Node an unreferenced Proxy is ordinary garbage. Next creates ~4 proxies/request on this route (headers/cookies/mutableCookies/searchParams adapters; chunks/430.js alone has 34 new Proxy( sites), each pinning its request's header/cookie graph: ~10k immortal proxies by 2,500 requests, 2.9 MB retained at ~1,900, growing without bound. The symbol/descriptor side-table growth is largely downstream of (a)+(b): #8174 gave owner-keyed tables death stories, but these owners never die.

Does PR #8208 cover this?

Mechanism (a), yes — fix/async-state-rss-accumulation releases + pools exactly these cells, de-registers them, and its [box-stats] gate asserts constant residue. If it works on this workload it should remove the box share of the Rust heap, the box 55% of the JS-heap retention, and the pause growth. Caveat that must be A/B'd on this fixture: its release only covers closure-unobservable locals and skips a body entirely when the escape scan is poisoned; Next's minified route code is closure-dense, so the asyncpipe numbers (releases == allocs) do not transfer automatically. Acceptance for this issue should be the fixture slope: RSS and [rss-diag-tables] box= over 100+ passes, before/after.

Mechanism (b) is NOT covered by #8208 and needs its own fix: give Proxy entries a lifetime — either back proxies with a real GC heap object (death hook drops the registry slot) or key the registry entries weakly so an unreferenced proxy id can be collected. Not a small change; the registry is load-bearing for lookup() misclassification defense.

Not covered by either: the ~100-215 MB mimalloc committed-but-free overlay (returned only under OS memory pressure). Once (a)+(b) stop the churn's live tail, purge_delay/page-retirement tuning can be evaluated separately.

Reproduction

• Stock: ~/mini-batch-rss.sh <tag> 100 PERRY_GC_DIAG=1 (mini; runs verify.mjs N times against one warm host, records per-pass RSS + snapshots).
• Client-mode variants (keep-alive / conn-only) + instrumented providers: ~/mini-batch-rss2.sh <tag> <stock|ka1|ka20|conn|partial> <N> [env..] with PROVIDERS=~/perry-bench.noindex/build-rss-diag and PERRY_RSS_DIAG=1 PERRY_RSS_DIAG_RETAINED_EVERY=20.
• The [Next.js/dylib] Forced-evacuation App Route arm: stale closure from a holder outside the GC heap #8163 stale-closure defect fired 2x during the instrumented run (unchanged by instrumentation).

Refs: #8040 (DoD audit that surfaced this), #8037 ("do not retain a request" acceptance bullet), #8163 (separate defect, same runs), #7933 / PR #8208 (box-cell accumulation and its in-flight fix).

[proggeramlug]

Measured input from the #8208 side: async box cells account for roughly a quarter of the per-request growth, and the post_in_use half cannot be them at all. Posting here rather than in a side channel so whoever picks this up has it.

Box cells are 38–42 bytes, measured two independent ways

• Marginal RSS across asyncpipe BATCHES 120 → 1200: 42.2 B/cell
• vmmap tag-240 (mimalloc) at BATCHES 1200: 37.8 B/cell

That is cell + registry + allocator overhead, not the payload alone.

At the ~1,100 cells/request reported here, that is 41–45 KB/request of the ~150 KB — about 27–30%.

#8208 removes that share, and it should be harvested in this workload: a warm event-driven server reaches an empty task queue between requests, which is the flush boundary the release path keys on. On asyncpipe the same fix takes malloc residue from linear-in-batches to constant 65,915 across a 10× range (slope zero), peak RSS 148.9 → 79.7 MB, vmmap tag-240 141.1 → 71.6 MB.

The post_in_use 5.4 → 39.7 MB is a different bug, and it is the larger half

Box cells cannot explain any of the arena growth, for two independent reasons:

1. They are not in the arena. Box cells are std::alloc, entirely malloc-side. In the asyncpipe reproduction the arena sat at ~250 KB while 118 MB accumulated in mimalloc and no full collection ever fired — the tracked-malloc registry read 0 for the whole run. Arena occupancy and box-cell retention are disjoint accounting.
2. A retained cell holds undefined. Post-perf(codegen): async-to-generator boxes are never cleared — completed activations retain 96% of asyncpipe's young generation #7933 a completed activation's cell contains no GC reference, and the control cells are i32/bool. So a retained box cell cannot keep a GC object alive even in principle.

A 34 MB rise in post_in_use is therefore a second retention path, in the GC heap, that #8208 does not touch.

Suggested next measurement

Split the two axes rather than treating ~150 KB/request as one number:

• malloc-side (vmmap tag 240, or a counting #[global_allocator]) — expect ~45 KB/req, and expect it to go away with fix(async): release and reuse a completed plain-async activation's box cells (#7933 follow-up) #8208.
• arena-side ([gc-step] post_in_use, PERRY_GC_DIAG=1) — expect the remaining ~105 KB/req, which is the real subject of this issue.

If those two numbers come out as ~45 / ~105, the box-cell share is accounted for and the hunt narrows to whatever is holding GC objects live across requests. If the arena side is much smaller than ~105 KB/req, then something else is malloc-side too and #8208 is not the whole of that half either.

The per-root-source retained-BFS instrument described in the issue body is the right tool for the arena half — it attributes which root keeps the objects, which neither RSS nor post_in_use can.

Caveats on these numbers

Measured on asyncpipe, not on this fixture, so the 38–42 B/cell figure transfers but the ~1,100 cells/request does not — that is this issue's own measurement and I have not reproduced it. The conclusion that box cells are outside the arena is structural rather than workload-dependent.

Refs #8208, #7933.

[proggeramlug]

Status check, 2026-08-16 — not fixed, and mechanism (b) has a sharper edge than this issue records.

Mechanism (a), box cells (55% of the retained JS heap): unchanged on main. PR #8208 is still open. The caveat in the issue body stands — its release skips closure-visible locals and Next's minified route code is closure-dense, so the asyncpipe "releases == allocs" result does not transfer automatically. Acceptance is still the fixture slope, before/after.

Mechanism (b), the append-only Proxy registry: unchanged on main (js_proxy_new still pushes with no removal path). While confirming that, the registry turned out to fail hard, not just leak:

The band is [PROXY_ID_BAND_START, HANDLE_BAND_MAX) = [0xF0000, 0x100000) — exactly 65,536 ids — and js_proxy_new never bound-checked PROXIES.len(). Id 65,536 encodes to HANDLE_BAND_MAX itself, which addr_class::is_proxy_id_band rejects and addr_class::is_above_handle_band accepts, so every classifier in the tree reads it as a dereferenceable heap address. Measured: 65,600 proxies created then read back segfaults (exit 139) at exactly the first out-of-band id, keep[65535].

Because the registry is cumulative rather than a live count, the ~4 proxies/request measured here put that ceiling at roughly 16k requests — the warm route was already at 10,037 entries (15% of the band) after 2,478. So the retention documented above is not only an RSS curve; it terminates in a wild pointer.

Filed as #8218 and fixed in #8219: refuse to mint past the edge (catchable RangeError, the #5067 trade) and make decode_proxy_id agree with addr_class about what a proxy id is. That makes the failure safe and named — it does not close this issue. The ceiling exists because slots are never reclaimed.

On the durable fix for (b). Having looked at what it takes: reclamation requires the collector to observe proxy-band payloads in traced slots, because a proxy handle is an id-band value rather than a heap object and there is no death signal today. The cheap-looking shortcuts do not work:

• Deciding an entry is dead from its target's or handler's reachability is circular — the entry is what keeps them alive.
• Backing a proxy with a shadow GC object gives it no referents, so it dies immediately; making the handle a real heap object drops the band's whole purpose (generic object dispatch would silently read fields off the proxy instead of trapping).
• Widening the band buys a linear factor against an unbounded leak, and HANDLE_BAND_MAX's own doc requires auditing every is_handle_band caller before it moves.

So it is observation or nothing, and observation has a correctness cliff: under-approximate the live set by one holder and a live proxy is reclaimed, after which lookup stops recognising it. That needs (i) hooks in each of the three funnels the collector actually visits slots through, (ii) pruning restricted to full traces, where the whole heap is scanned, and (iii) something covering a proxy id held only in a native frame across a trap call. Worth doing, but it is its own piece of work, not a follow-up commit — and it should land after #8208, so the box-cell share stops masking the measurement.

[proggeramlug]

The retention has a user-visible consequence, not just a footprint one — measured

From an in-flight 300-pass warm soak on the native backend, matched tree 183d30c53, default GC, no knobs (the #8163 close-condition run; 211/300 passes at time of writing). Posting now rather than at the end because it bears on whether this is scoped as a footprint defect.

It isn't. A warm process gets slow enough to fail requests outright.

Eight of the run's requests failed with UND_ERR_HEADERS_TIMEOUT — undici's client-side headers timeout, i.e. the caller gave up and got nothing. Not a wrong answer, not a slow-but-correct answer: a failed request from the client's point of view. The server's own log is clean at those moments (TypeError: value is not a function 0, E180 failed to pipe response 0, empty-body Unexpected end of JSON input 0 across 6,267 copying minors), so this is purely the process being unresponsive.

All eight are in a single pass, and that pass is an outlier by 7.7×:

pass 200:  84 minors
pass 201:  84 minors
pass 202:  84 minors
pass 203: 644 minors   <-- all 8 timeouts here
pass 204:  84 minors
pass 205:  84 minors

So the failure mode is a collection storm, not chronic slowness. One pass ran 644 copying minors and starved the event loop long enough that 8 of its 21 concurrent requests blew a default HTTP client timeout.

The steady state that produces it:

	at pass ~1	at pass ~203
live heap (post_in_use)	5.0 MB	82.7 MB
copying minors per pass	~0.2	84
RSS	—	545 MB

And the yield is the damning number: a representative minor at pass 203 reports pre_in_use=82,771,712 post_in_use=82,732,720 sweep_freed=38,992 — 39 KB reclaimed against 82 MB live, ~0.05%. The collector is running constantly and recovering essentially nothing, because the retained set is retained, not garbage. That is what turns a memory graph into a throughput collapse: every collection pays a full root-scan and mark cost over a live set that only grows.

Why the existing arms cannot see this. A 10-pass cold-start process never leaves the small-heap regime (post_in_use 5.2 → 5.6 MB, ~2 minors), so nothing in the fixture's default coverage — or in either transport's cold-start arms — reaches the knee. It requires one warm process past ~120 passes.

I'm not arguing this must block #8040; that's a scope call for whoever owns it. I am saying the grounds should be accurate: this is not "a footprint/throughput defect rather than a correctness one". Requests fail. If #8040's DoD says anything about hangs or served requests, this is inside it, and if it's tracked separately it should be tracked as a defect with user-visible failures rather than as an RSS number.

Final counts and the full cadence curve when the soak completes.