Add -Zrmeta-content-svh, -Zrmeta-strip-spans, -Zrmeta-normalize-src-hash: byte-stable .rmeta under non-interface edits

[Andrew Gazelka (andrewgazelka)]

Adds three opt-in -Z flags that make a library crate's .rmeta byte-identical across non-interface edits, so that content-addressed build systems (nix-cargo-unit derivations) get early cutoff: when a rebuilt .rmeta is byte-identical, dependents are not re-derived.

Flags

Flag	What it does	What it trades away
-Zrmeta-content-svh	Derives the SVH embedded in metadata from the encoded metadata bytes (plus session dep-tracking hash and stable crate id) instead of the HIR, so it is stable exactly when the rest of the metadata is. Rejected with -Cincremental.	Nothing user-visible. The local crate_hash query is untouched; the loader's link-time check keeps comparing encoded SVHs for equality.
-Zrmeta-strip-spans=non-exported|all	Replaces spans with DUMMY_SP at the single metadata span-encoding choke point. non-exported preserves spans in exported MIR bodies, hygiene data, and the definition spans of MIR-exported items; all strips everything and substitutes expansion-index-derived ExpnHashes. Proc-macro crates are exempt.	Dependent-crate diagnostics lose "defined here" notes into this crate (for all item spans under all; for items without exported MIR under non-exported). Under all, debuginfo a dependent emits for functions inlined from this crate resolves to the dependent's own first file at line 1 (verified in DWARF; call-site lines stay correct), and const-eval backtraces plus macro-backtrace notes into this crate degrade. Under non-exported, dependent DWARF was re-verified byte-equivalent to a no-flags baseline.
-Zrmeta-normalize-src-hash	Zeroes the per-file source content hashes (and optional -Zchecksum-hash-algorithm checksums) in the encoded source file table.	Dependent crates cannot render source snippets from this crate in diagnostics (the zero hash never verifies, so they degrade to file:line:col instead of risking quoting stale source), and debuginfo file checksums for this crate become zero.

All three are [TRACKED], default off. Flag-off encoding is unchanged: every new code path is gated on the flag values, and the flag-off byte-diff signature of a probe crate (rmeta size and per-edit-class differing byte counts) is identical to the pre-patch compiler's.

Per-edit-class byte-stability (stage1, small lib crate, --emit metadata,link -Copt-level=3 --remap-path-prefix, no incremental)

Edit class	flags off	content-svh + strip-spans=all + normalize-src-hash	same but strip-spans=non-exported	normalize-src-hash only
1. identical rebuild	IDENTICAL	IDENTICAL	IDENTICAL	IDENTICAL
2. comment edit (general)	differs (63 B)	IDENTICAL	differs (45 B)	differs (48 B)
2c. comment edit, same length/lines	differs (15 B)	IDENTICAL	IDENTICAL	IDENTICAL
2t. trailing comment at EOF	differs (261 B)	IDENTICAL	differs	differs
3. whitespace edit shifting all lines	differs (328 B, +1 size)	IDENTICAL	differs (299 B)	differs (312 B)
4. private non-generic #[inline(never)] body edit (same length)	differs (32 B)	IDENTICAL	IDENTICAL	differs (16 B)
control: generic fn body edit	differs (53 B)	differs (17 B)	differs (36 B)	differs (37 B)
control: signature change	differs (1373 B)	differs (1171 B)	differs (1300 B)	differs (1358 B)

The controls keep churning by design: exported MIR and interface data must stay visible to dependents. Isolation runs confirm each flag is necessary: with spans and src-hash handled but the SVH flag off, exactly the 16 SVH bytes still differ on every class.

non-exported's boundary is deliberate and now documented: its preserved spans are byte offsets resolved through the encoded source file length/line tables, so only byte-position-preserving edits stay identical (byte diff of case 2t shows the residual churn is exactly source_len and the line table of the SourceFile record). It is the zero-debuginfo-cost mode; all is the cutoff mode.

Verification history worth keeping (negative result)

The first version of non-exported stripped item definition spans while keeping MIR body spans. Independent DWARF inspection falsified its debuginfo claim: the consumer's DW_AT_decl_file/decl_line for a cross-crate-inlined function came out as the consumer's own first file, line 1, and the .debug_line rows bound this crate's real line numbers to the consumer's file, i.e. confidently wrong debuginfo, worse than all's honest no-line-info. Cause: the declaration file/line and the line-row file binding derive from the item's definition span, which is item metadata, not body metadata. Fixed by preserving def_span/def_ident_span for exactly the should_encode_mir set; re-verification shows consumer DWARF (DIEs and decoded line table) semantically identical to a no-flags baseline, with all unchanged.

End-to-end relink check (independently verified)

dep v1 vs v2 differ only in a private #[inline(never)] body (x + 1 vs x + 100). With the flags: the two .rmetas are byte-identical; the rlibs differ only in the codegen object member (the embedded .rmeta sections, extracted with objcopy --dump-section, are byte-identical to the standalone files). An app rlib compiled only against v1's .rmeta, linked into a binary against v2's rlib without recompiling app, links cleanly (no SVH mismatch) and prints v2's result (240); linked against v1's rlib as control it prints 42. That is precisely the cutoff-then-relink flow. Adding a new pub fn to dep changes the rmeta, i.e. the flags are not blanking metadata.

Tests

• New tests/run-make/rmeta-stability pins classes 1 to 4 plus 2c/2t, both controls, and the non-exported boundary (same-length edits identical, length-changing comment must differ): 1 passed.
• ./x test compiler/rustc_interface --stage 1 (includes the option dep-tracking tests with the three new tracked! entries): 17 passed, 0 failed.
• ./x test tests/incremental --stage 1 (flags off): 178 passed, 0 failed.
• ./x test tests/ui --stage 1 (flags off): 21689 passed, 0 failed, 238 ignored.
• ./x test tidy: clean.
• Smoke workspace (dep with derives, #[macro_export] macro, generics, #[inline] fns; app using all of it, compiled without flags against the flagged dep): builds, runs, --test binary 2 passed, 0 failed. A cross-crate E0308 against a dep type renders sanely; the visible degradation under all is exactly the "defined here" note losing its location (= note: function defined here instead of a snippet pointing into dep source). A proc-macro crate compiles under the flags (exempt, no ICE) and its consumer works.
• Edge crates (empty lib, re-export-only, pub const/pub static with a line shift) compile without ICE; the const/static crate's rmeta is byte-identical under the line shift with all.

Known limitations (deliberate or out of scope)

• Bodies whose MIR is exported keep affecting the rmeta: generics, #[inline], and small functions auto-selected for cross-crate inlining at -O (use #[inline(never)] or -Zcross-crate-inline-threshold=never to opt bodies out). This is required for correctness.
• Edits that add or remove nested definitions (closures, inline consts) or macro expansions shift DefIndex/ExpnId numbering and still churn the rmeta even for private bodies.
• Doc-comment edits legitimately churn (doc text is encoded for rustdoc).
• Under all, dependent-side debuginfo misattributes inlined-from-this-crate declaration DIEs to the consumer's first file, line 1 (documented in the unstable book).
• -Zrmeta-content-svh is rejected with -Cincremental (the green-reuse path never re-encodes metadata, so there are no fresh bytes to hash).

Upstream context

The unconditional SourceFile::src_hash encoding (16 bytes of MD5 churn on any file edit, even at -Cdebuginfo=0) appears to be an unreported gap upstream: the relink-dont-rebuild tracker (rust-lang#158844) does not list it, and checksum_hash right next to it is already optional behind -Zchecksum-hash-algorithm. Prior art: rust-lang#154724 (crate hash from the metadata encoding, the same direction as -Zrmeta-content-svh), rust-lang#151265 (-Zstable-crate-hash with a separate .spans file), rust-lang#143249 (wholesale span deletion; broke thiserror-dependent crates, which is why proc-macro crates are exempt here and span stripping is opt-in and graduated).

Note for index: rmeta byte-stability alone does not make the cutoff fire end to end. index still needs to split metadata and link outputs into separate derivations (today one unit derivation carries both rmeta and rlib, and the rlib legitimately changes on body edits), and the dep-info .d output with an unremapped store path must be fixed separately.

🤖 Generated with Claude Code

Note

Add -Zrmeta-content-svh, -Zrmeta-strip-spans, -Zrmeta-normalize-src-hash for byte-stable .rmeta under non-interface edits

• Adds three new unstable compiler flags to make .rmeta files byte-stable when only non-interface changes (e.g. function bodies, comments) are made, without affecting the crate's public interface hash.
• -Zrmeta-strip-spans (None/NonExported/All) replaces encoded spans with DUMMY_SP in metadata, with NonExported preserving spans for items whose MIR is exported.
• -Zrmeta-normalize-src-hash zeroes out per-file source content hashes in crate metadata so source file edits don't propagate into .rmeta bytes.
• -Zrmeta-content-svh derives the crate header SVH from encoded metadata bytes rather than the HIR hash; incompatible with -C incremental (rejected at session setup).
• A new run-make test verifies .rmeta byte stability across various edit/flag combinations.
• Behavioral Change: downstream crates reading stripped metadata will see DUMMY_SP for stripped spans, and expansion hashes in hygiene data become index-derived rather than content-derived when All mode is active.

^{Macroscope summarized 19bc529.}