[jkelleyrtp]

Summary

Lowers rustd worker peak RSS on large Codex builds through codegen retention, one test graph, and sharing between normal and cfg(test) crate copies. The memory-retention changes preserve output bytes. Cfg sharing changes compilation identities while preserving per-program linking and runtime behavior. The store format is unchanged.

• DWARF (debuginfo.rs, debuginfo/split.rs): completed units are encoded as soon as they finish instead of keeping every gimli unit until the object is written. With split debuginfo, finished DWO units go to staged sidecar files. Fixture objects, DWOs and llvm-dwarfdump --debug-info --debug-line output are byte-identical before and after.
• Debug type sharing (debuginfo/types.rs, mod.rs): identical immutable Arc<[DebugType]> payloads are shared through a weak interner (SHA-256 candidate lookup plus exact equality). TypeSet Arc identity and links stay, so pointer-keyed users (Dwarf, KeptTypes) see no change. Before this, 2.0M TypeList Arcs held only 269k distinct contents.
• Test reachability (reach.rs): callee environments come from homes.home(db, callee, asked), the same key compile_ahead uses, so test builds stop recomputing codegen payloads under a second key. Reached functions and statics match for all 1,494 Codex tests, cold and edited.
• Store lookups (cached.rs, store.rs): retained maps use fixed-size SHA keys. Decoded entries are capped by a 4096-key FIFO, and concurrent reads of the same key are coalesced. Explicit serde keeps the stored bytes unchanged. The stable-name memo is bounded too.
• Symbol tables (symbols.rs): immutable normalized symbol records are shared between program tables. Per-table source metadata stays separate.
• One worker and one graph per test build (cargo/daemon.rs, cargo/state.rs, main.rs): cfg_test is no longer part of WorkerConfig. A cargo rustd test worker loads with set_test (whenever selection.dev_deps is set), and RA's SYSROOT_AS_BUILT graph then holds normal nodes plus unit-test nodes, with deps pointing at the normal ones. The normal and -unit checkouts share that worker, and whether a program is unit-test now travels with each request (unit_test in the sync line). Normal programs skip the tested copies of member libs/bins (unit_roots) but keep integration-test crates. cargo rustd build is unchanged.
• harness/cargo_rustd.py: adds a cross-crate generic fixture for incremental test selection, plus a test-graph fixture (unit + integration programs, a dev-dependency back-edge, an edit) that checks there is one worker and one workspace load.

Tried and dropped: a bounded describer channel (17% slower CLI build, <1% RSS saved), a shared name interner (0.14 GiB), and salsa LRU eviction (the features.rs edit went from 1 to 51 recompiled functions).

Codex numbers before cfg sharing

Codex rust-v0.99.0 (ec9f76ce), toolchain 1.99.0, CARGO_PROFILE_{DEV,TEST}_SPLIT_DEBUGINFO=unpacked, fresh daemon and store for each cold run, worker RSS sampled every 1 s (GiB), 31 GB machine.

Command	main (7d54230)	this branch (955c5c1)
cargo rustd build --bin codex, cold	300.0 s, worker 29.72 GiB, SIGKILL	201.8 s, worker 15.59 GiB, ok
one-line core/src/features.rs edit, same build	blocked (no cold build)	2.03 s, 1 of 353,520 fns compiled, spliced; 13.37 GiB
cargo rustd test -p codex-core --no-run, cold	533.2 s, worker 29.43 GiB, SIGKILL	408.0 s, one worker 20.67 GiB, ok
same test build after the features.rs edit	blocked	108.2 s, 17.88 GiB, daemon-served, 8,239 functions compiled
ruff 0.17.1 cargo rustd build --bin ruff, cold	75.9 s, 10.73 GiB (main ebcd234)	70.2 s, 8.85 GiB

The codex-core test build now completes. Before the single graph, the same branch ran two workers at 14.69 + 14.82 GiB (28.6 GiB together) and the cfg(test) one was SIGKILLed. One shared worker peaks at 20.67 GiB, 7.9 GiB less in total, but that is still above the 15 GiB target. The CLI rows above were measured before the single-graph change; rerun with it, the cold build is 201.0 s / 15.27 GiB and the edit is 2.03 s with 1 function spliced.

Cfg sharing

Lowering records cfg atoms and predicate results, including absent options and ancestor gates. Source twins match normal and test items through real macro expansions and hygiene; a source match alone does not prove that code can be shared. Eligibility follows item dependencies and rejects uncertain or cfg-dependent cases.

Fully shared test instances and types use normal-copy compilation identities before inference and MIR, while each program keeps its own link identities. Analysis-shared bodies reuse normal inference and raw MIR, remap per-copy references, then monomorphize in the test body's semantic environment. Statics and TLS remain separate. An ADT is unified only when its applicable impl sets are twin-equal and their items are fully shared; this restriction propagates to containing types and callers.

In a program containing both copies, reaching TypeId for a unified type discards tentative cache publication and retries unshared. Cached TypeId chains are compiled fresh for detection. A sharing-related compilation refusal also retries the whole program unshared once, with a logged counter. RUSTD_CFG_SHARING=0 disables sharing.

Same Codex tag, toolchain, unpacked debuginfo and RSS sampling as above. The cold A/B below was measured at 36840ea27d; the retained-prime/edit rows were measured at c4d3c094bc. They are separate runs, not a new final-commit cold A/B.

Test workload	Wall	Peak worker RSS	Backend functions compiled
Cold, sharing off	302.930 s	20.575 GiB	445,981
Cold, sharing on	329.457 s	19.181 GiB	375,536
Retained-store prime, sharing on after hydration fix	318.448 s	18.033 GiB	375,651
Shared-body edit in that warm daemon	18.080 s	15.994 GiB	2

The cold comparison saves 6.8% peak RSS and 15.8% backend compilations, with 8.8% longer wall time. After the hydration fix, all five edit programs splice; there is no whole-object relink. The previous retained-store edit compiled 101,894 backend functions on versus 8,244 off. The final edit emits two backend functions total; per-object patch counts include debug/redirect refresh and are not backend compile counts. No TypeId or fallback restart occurred in these Codex runs. The latest separate CLI edit took 1.249 s and spliced one function.

Correctness comparison with sharing on and off: codex_core has identical results, 939 passed, 0 failed, 4 ignored. The optional all comparison also has identical per-test statuses (37 passed, 497 failed, 13 ignored), but many failures are environmental because relocated binaries cannot find codex-linux-sandbox; this is not passing integration coverage. Harness fixtures cover both copies in one binary, distinct statics/TLS, cross-copy TypeId downcasts, impl identity, opaque/async layouts, N-only analysis and incremental edits.

Patchable builds no longer hydrate opaque checkout-owned store entries. Those entries do not carry the instance identities and dependency edges a patch base needs, so the first edit after a retained-store prime discarded its base. Dependency/library store reuse remains enabled. Checkout entries are still published for consumers that can hydrate them: the publication policy is independent of the patchable-build read policy. The trade-off is less checkout cache reuse on daemon startup in exchange for a usable incremental base; this does not change the store format. A 400-function fixture checks cold and retained primes, requires test-program splicing after a line-inserting body edit, and bounds sharing-on backend compiles by sharing-off plus two.

Where the bytes go

jemalloc heap profile of the normal-config worker on the branch, near its RSS peak in a sequential Codex test build. Rows are classified by allocation stack, not exact ownership (GiB):

Category	Near peak	Before all test target
infer / MIR / layout / solver memos	2.906	1.486
codegen symbols	2.361	1.880
codegen cache	1.462	0.144
macro expansions	1.454	1.374
codegen debuginfo (other)	1.275	0.255
syntax trees / parse	0.692	0.671
salsa dependency edges	0.664	0.599
lowered bodies	0.618	0.586
other declarations	0.521	0.501
type interners	0.473	0.366
def maps	0.441	0.416
source text	0.432	0.432
compiled-instance memos (all parts)	0.717	0.642
inlinable payloads + memos	0.561	0.526
object / module buffers	0.228	0.000
crate graph + item trees	0.081	0.079
other	0.181	0.176
Total	15.069	10.133

On main, codegen categories (DWARF most of all) were about 58% of the sampled live heap. The remaining big items in this pre-sharing profile are rust-analyzer memos (inference/MIR/expansions, ~5 GiB), which salsa LRU could not drop without breaking edit reuse, and duplicate cfg(test) analysis/codegen. The sharing changes above address that duplication (in codex-core, 1,929 of 3,070 bodies are outside #[cfg(test)]).

Checks

The cleanup passes the full sharing fixture, rustd tests (112), cfg tests (12), hir-def tests (541, one ignored), release hir-ty tests (70), splices (71/71) and breakers (175/175, 10 open), plus formatting and release Clippy with --cap-lints warn. The backend-count assertion now includes LLVM as well as Cranelift. Earlier full gates also included fuzz/edits.py smoke --backends cranelift (0 findings) and cargo_rustd.py --small, with only the two authorized local C-library encoding failures. This does not claim a warning-free default Clippy run.

Final CI on 21c9173ab8 is green: 13 jobs pass, none fail, deploy is skipped. This includes all platform harnesses and Linux x64 debuginfo. The store-full regression introduced by coupling checkout publication to hydration is fixed without changing its budgets or assertions; it passes locally under both backends and on Linux x64/ARM64 and Windows CI. The earlier macOS LLVM overlay output mismatch did not reproduce with Linux LLVM, including whole-object relinking; final macOS comparisons pass 71/71 under both backends, but the earlier mismatch's cause remains unestablished. The harness now prints both outputs on any mismatch.

Link to Devin session: https://dioxus.staging.devinenterprise.com/sessions/09412c48ac1f4d64a27913ba766cf477
Open in Devin Desktop: https://dioxus.staging.devinenterprise.com/desktop/session/09412c48ac1f4d64a27913ba766cf477?variant=devin-insiders
Requested by: @jkelleyrtp

[staging-devin-ai-integration]

I'll fix CI failures and address comments from users with write access that start with 'Devin'.

• Disable automatic comment, CI, and merge conflict monitoring