incremental compile: 44-module one-file edit spends ~8.1 s in link/archive after 43/44 object hits

[proggeramlug]

Finding

On a real 44-module / 10,478-LOC API, Perry's per-module object cache works, but a one-file edit still takes 10.16–11.31 s because link/archive processing takes about 8.12 s even when every object is already cached.

This re-measures the link-cache idea discussed in #131 on a substantially larger real application. The old estimate there was about 85 ms and was reasonably deemed marginal. On this target it is no longer marginal.

Reproduction

Host: Apple arm64, macOS 26.5, Perry d1d9929663e6b582e3a743a96f56f9e9a4f93aba.

Target: gscmaster API snapshot, entry src/main.ts, 44 TS modules, 10,478 LOC, 9 direct npm dependencies / 146 installed packages in the supplied graph. Perry selected bundled native shims for the reachable external packages.

Perry was built with the required static wrappers:

CARGO_TARGET_DIR=/Users/amlug/cargo-targets/wsizeaudit \
  cargo build --release -p perry -p perry-runtime-static -p perry-stdlib-static

All compiles used the resulting release directory as PERRY_RUNTIME_DIR, a dedicated PERRY_CACHE_DIR, and PERRY_CACHE_DEBUG_KEY=1 for cache evidence.

Case	N	Best–worst wall	Object result	Max RSS range
unchanged rebuild	5	0.28–0.29 s	whole-build-cache hit; 44 object mtimes unchanged	30.08–30.15 MB
edit one leaf module	3	10.56–11.31 s	43 reused, 1 stored each run	297.9–302.0 MB
edit db.ts (23 importers)	3	10.16–10.65 s	43 reused, 1 stored each run	297.6–303.6 MB
build cache disabled, all 44 objects hit, --no-link	1	1.47 s	44 reused	225.1 MB
same, but force link with PERRY_NO_LINK_CACHE=1	1	9.59 s	44 reused	297.1 MB

The isolated link/archive increment is 9.59 - 1.47 = 8.12 s, or 84.7% of the forced all-hit compile. A normal one-file edit cannot use the final-binary build cache because one object changes, so it pays that stage every time.

The dependency fan-out did not invalidate importers: both leaf and 23-importer edits regenerated exactly one object. This is not an object-cache invalidation bug.

Expected direction

Please profile and cache or avoid repeated work in the post-object link/archive path for one-object edits. Possible implementations include caching prepared/strip-deduplicated native archives by exact input fingerprint, or an incremental link strategy. The measurement gives an upper-bound win of about 8.12 s per edited build on this target (roughly 10.6 s down toward 2.5 s if the stage were nearly eliminated).

Acceptance should preserve the current byte-for-byte runtime correctness checks and must not trade runtime instruction count or RSS for build speed.

[proggeramlug]

Implemented in #8399.

Profile first

I instrumented the warm forced-link path before changing it. In a 10.13 s compile with all 44 object files reused and PERRY_NO_LINK_CACHE=1, the post-object work broke down as follows:

sub-stage	wall
well-known archive strip/dedup	8.474 s
other archive/command preparation	0.269 s
clang/ld64 invocation	0.193 s
post-link strip + cache manifest	0.043 s
measured post-object subtotal	8.979 s

Strip/dedup was 94.4% of the directly instrumented post-object subtotal. The existing final-link cache probe happens after that work; it can skip a wholly unchanged final binary, but it is both too late to avoid archive preparation and correctly misses when one application object changes.

Change

Cache the prepared well-known native archives under PERRY_CACHE_DIR/prepared-archives. A one-file edit still reaches the linker with its changed object, but unchanged wrapper archives no longer repeat extraction, symbol inspection, localization, and rebuilding.

The intermediate key includes the schema and Perry version, exact Perry executable bytes, target selector and effective target triple (or native host architecture/OS), sorted feature set, exact runtime and stdlib archive bytes, every ordered wrapper archive, and the exact resolved ar, nm, and objcopy executable bytes. Paths, sizes, and SHA-256 digests are framed into the key. Cached outputs have a manifest with their own sizes and SHA-256 digests and are rejected when missing or corrupt. A non-fatal strip/dedup fallback is deliberately not cached.

Application objects and linker flags cannot affect the prepared archive bytes, so they remain in the existing final-link cache key rather than this intermediate key. Either change still forces the final executable to relink. Empty wrapper sets bypass this cache entirely. PERRY_NO_ARCHIVE_CACHE=1 bypasses only this layer for diagnosis; --no-cache/PERRY_NO_CACHE=1 bypasses it too.

Fixture results

The before column is the #8396 audit's repeated baseline. The after column is this branch on the same copied rung4-api fixture and dedicated cache, with the required runtime/stdlib wrapper build and PERRY_RUNTIME_DIR pin.

case	before	after	cache evidence
unchanged rebuild	0.28–0.29 s	0.28–0.29 s	whole-build hit
edit leaf plans.ts	10.56–11.31 s	2.66, 2.73, 3.92 s	43 reused, 1 stored; archive HIT (third run was machine-contended)
edit db.ts	10.16–10.65 s	2.84–2.90 s	43 reused, 1 stored; archive HIT
all 44 hit, --no-link	1.47 s	1.01 s	all objects came from the populated cache
all 44 hit, forced link	9.59 s	1.83–1.92 s	44 reused; archive HIT; linker ran

The isolated archive/link increment falls from 8.12 s to 0.82–0.91 s, an 88.8–89.9% reduction. A final-build verification printed [archive-cache] MISS while populating key 969c7d…, then [archive-cache] HIT for the same key; the hit took 1.83 s with all 44 objects reused.

Compiler peak RSS on the profiled baseline forced link was 312.5 MB. Warm cache-hit forced links used 254.6–266.8 MB (266.3 MB in the final verification), so the cache does not add to the existing ~300 MB peak.

Hit/miss and correctness evidence

• Input object change: a temporary leaf edit produced 43 object hits + 1 miss/store, the archive cache hit, and the final-link report said linked: true, skipped: false.
• Linker flag change: PERRY_EXTRA_LINK_ARGS=-Wl,-no_uuid produced 44 object hits, an archive hit, and a real final relink; dwarfdump --uuid confirmed the flag affected the output.
• Runtime archive change: inserting a valid, unreferenced C object into the active runtime archive changed its SHA-256, produced an archive-cache MISS with a different key, and relinked. The original archive was restored and byte-verified afterward.
• Corrupt prepared output: the unit test populates an entry, corrupts its cached archive, and proves the next load rejects it.
• Key-test negative control: I temporarily removed the runtime archive hash from derivation and the key-coverage test failed at the runtime mutation assertion. Restoring the field made it pass.

The five prepared wrapper archives from the cache were byte-identical to freshly regenerated archives (SHA-256 matched individually). The final Mach-O files have the same 23,386,152-byte size and section layout, but not the same whole-file SHA-256: ld64 derived a different Mach-O UUID (and therefore ad-hoc signature) when identical archive bytes came from persistent cache paths instead of per-process strip-temp paths. Clean and cached programs had byte-identical stdout and stderr and the same exit status.

Because that metadata difference changes the emitted file, I also measured seven runs of each executable. Median instructions were 145,214,016 clean vs 145,316,582 cached (+0.071%); median peak RSS was 40,386,560 vs 40,419,328 bytes (+0.081%). Both are within run noise, with no runtime-cost trade.

Validation

• cargo build --release -p perry -p perry-runtime-static -p perry-stdlib-static: pass
• cargo test --release -p perry --bin perry: 1005 passed
• cargo test --release -p perry-runtime --lib: 2595 passed, 4 ignored
• bash scripts/run_lint_gates.sh: all 50 passed
• 19-program sweep: 19/19 compiled and byte-exact against every expected stdout
• Empty-wrapper regression guard: no archive-cache hashing/logging and byte-exact output