Skip to content

rustc_monomorphize: shard volatile generic CGU buckets to reduce incremental invalidation scope - #162692

Draft
Trigodil wants to merge 4 commits into
rust-lang:mainfrom
Trigodil:fine-grained-generic-cgus
Draft

rustc_monomorphize: shard volatile generic CGU buckets to reduce incremental invalidation scope#162692
Trigodil wants to merge 4 commits into
rust-lang:mainfrom
Trigodil:fine-grained-generic-cgus

Conversation

@Trigodil

@Trigodil Trigodil commented Sep 12, 2026 •
edited by rustbot
Loading

Copy link
Copy Markdown

View all comments

Summary

The current CGU partitioner assigns every monomorphized instance of a generic
function from the same module into a single "volatile" bucket, keyed on
(module_def_id, volatile). This means editing the body of any instantiation
invalidates the entire bucket -- every other instantiation in that module gets
re-emitted on the next incremental build, even ones that share no code with the
change.

This PR adds -Z fine-grained-generic-cgus, which shards each volatile bucket
into a bounded number of sub-buckets by hashing the instantiation's fully-mangled
symbol name. An edit to one instantiation's code path now only invalidates the
~1/shard_count of instantiations that hash into the same shard.

Approach

Shard count is derived from -C codegen-units, floored at the host's available
parallelism and clamped to [4, 128]. This keeps the total CGU count in the same
ballpark as a normal build while ensuring the codegen backend always has enough
independent units to keep every core busy.

Unbounded splitting (one CGU per instantiation) was tested first and regresses
heavily on real crates: merge_codegen_units is intentionally skipped in
incremental mode, so hundreds of tiny object files pile up with no recombination,
and link cost dominates. Bounded sharding avoids this entirely.

Benchmarks (polars-core, ChunkedArray)

Configuration Cold Incremental
baseline 1.000x 1.000x
unbounded (1 CGU/instantiation) ~3.06x ~20.95x (regresses)
this patch (hash-sharded) 0.988x 0.961x
hash-sharded + -Z threads=4 0.657x 0.852x
hash-sharded + threads=4 + cranelift (cu=16) nil 0.521x (best)

Tested on polars-core which has high monomorphization volume
(ChunkedArray<T> instantiations survive merge_codegen_units and make the
volatile bucket expensive to invalidate). Methodology uses a body-edit approach

  • a trailing comment does not trigger CGU invalidation since rustc fingerprints
    by item content, not file bytes.

Notes

  • Little to no effect on non-incremental builds
  • Topology-aware clustering was also prototyped and benchmarked - It lost to hash-sharding due to lack of callees to cluster by

Disclosure

Claude Sonnet 4.6 was used to speed up compiler research and to rule out dead ends in the methodology used.

@rustbot rustbot added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue. labels Sep 12, 2026
@rust-log-analyzer

This comment has been minimized.

Sign in to view
@Trigodil
Trigodil force-pushed the fine-grained-generic-cgus branch from b2d8f24 to cfca780 Compare September 12, 2026 16:15
@rust-log-analyzer

This comment has been minimized.

Sign in to view
@panstromek

panstromek commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

Let's try this out.

@bors try @rust-timer queue

@rust-timer

This comment has been minimized.

Sign in to view
@rustbot rustbot added the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Sep 12, 2026
@rust-bors

This comment has been minimized.

Sign in to view
rust-bors Bot pushed a commit that referenced this pull request Sep 12, 2026
@bors
Auto merge of #162692 - Trigodil:fine-grained-generic-cgus, r=<try>
f35a77c 
rustc_monomorphize: shard volatile generic CGU buckets to reduce incremental invalidation scope
panstromek
panstromek reviewed Sep 12, 2026
View reviewed changes
Comment thread compiler/rustc_monomorphize/src/partitioning.rs Outdated
@panstromek

panstromek commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

@bors try cancel

@rust-bors

rust-bors Bot commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

Try build cancelled. Cancelled workflows:

Hint: if you want to run another try build, you do not need to manually cancel the previous one. Just run @bors try and bors will cancel the previous build automatically.

@Trigodil @panstromek
Update compiler/rustc_monomorphize/src/partitioning.rs
1ab4e4f 
Got it

Co-authored-by: Matyáš Racek <panstromek@seznam.cz>
@panstromek

panstromek commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

@bors try @rust-timer queue

@rust-timer

This comment has been minimized.

Sign in to view
@rust-bors

This comment has been minimized.

Sign in to view
rust-bors Bot pushed a commit that referenced this pull request Sep 12, 2026
@bors
Auto merge of #162692 - Trigodil:fine-grained-generic-cgus, r=<try>
6c4e7b9 
rustc_monomorphize: shard volatile generic CGU buckets to reduce incremental invalidation scope
@rust-log-analyzer

This comment has been minimized.

Sign in to view
@panstromek

panstromek commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

You can ignore tidy for experiments like this, it won't fail the try build.

@Trigodil

Trigodil commented Sep 12, 2026

Copy link
Copy Markdown
Author

You can ignore tidy for experiments like this, it won't fail the try build.

Gotcha, its just really annoying for me when one fails

@rust-log-analyzer

rust-log-analyzer commented Sep 12, 2026

Copy link
Copy Markdown
Collaborator

The job test-aarch64-gnu-llvm-21-1 failed! Check out the build log: (web) (plain enhanced) (plain)

Click to see the possible cause of the failure (guessed by this bot) 
test [codegen-units] tests/codegen-units/item-collection/unused-traits-and-generics.rs ... ok
test [codegen-units] tests/codegen-units/item-collection/unsizing.rs ... ok
test [codegen-units] tests/codegen-units/partitioning/incremental-merging.rs ... ok
test [codegen-units] tests/codegen-units/partitioning/inline-always.rs ... ok
FATAL: !(missing.is_empty() && unexpected.is_empty() && wrong_cgus.is_empty())
test [codegen-units] tests/codegen-units/partitioning/extern-generic.rs ... FAILED
FATAL: !(missing.is_empty() && unexpected.is_empty() && wrong_cgus.is_empty())
test [codegen-units] tests/codegen-units/partitioning/extern-drop-glue.rs ... FAILED
FATAL: !(missing.is_empty() && unexpected.is_empty() && wrong_cgus.is_empty())
test [codegen-units] tests/codegen-units/partitioning/local-drop-glue.rs ... FAILED
FATAL: !(missing.is_empty() && unexpected.is_empty() && wrong_cgus.is_empty())
test [codegen-units] tests/codegen-units/partitioning/local-generic.rs ... FAILED
test [codegen-units] tests/codegen-units/partitioning/local-transitive-inlining.rs ... ok
test [codegen-units] tests/codegen-units/partitioning/inlining-from-extern-crate.rs ... ok
FATAL: !(missing.is_empty() && unexpected.is_empty() && wrong_cgus.is_empty())
test [codegen-units] tests/codegen-units/partitioning/regular-modules.rs ... ok
test [codegen-units] tests/codegen-units/partitioning/methods-are-with-self-type.rs ... FAILED
test [codegen-units] tests/codegen-units/partitioning/statics.rs ... ok
FATAL: !(missing.is_empty() && unexpected.is_empty() && wrong_cgus.is_empty())
test [codegen-units] tests/codegen-units/partitioning/shared-generics.rs ... FAILED
FATAL: !(missing.is_empty() && unexpected.is_empty() && wrong_cgus.is_empty())
test [codegen-units] tests/codegen-units/partitioning/vtable-through-const.rs ... FAILED

failures:

---- [codegen-units] tests/codegen-units/partitioning/extern-generic.rs stdout ----

The following items were assigned to wrong codegen units:

fn cgu_generic_function::bar::<&str>
  expected: cgu_generic_function-in-extern_generic.volatile[External] 
  actual:   cgu_generic_function-in-extern_generic.volatile.shard012[External] 

fn cgu_generic_function::foo::<&str>
  expected: cgu_generic_function-in-extern_generic.volatile[External] 
  actual:   cgu_generic_function-in-extern_generic.volatile.shard001[External] 


thread '[codegen-units] tests/codegen-units/partitioning/extern-generic.rs' panicked at src/tools/compiletest/src/runtest/codegen_units.rs:109:13:
fatal error
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
---- [codegen-units] tests/codegen-units/partitioning/extern-generic.rs stdout end ----
---- [codegen-units] tests/codegen-units/partitioning/extern-drop-glue.rs stdout ----

The following items were assigned to wrong codegen units:

fn std::ptr::drop_glue::<LocalStruct> - shim(Some(LocalStruct))
  expected: extern_drop_glue-fallback.cgu[External] 
  actual:   extern_drop_glue-fallback.cgu.shard005[External] 

fn std::ptr::drop_glue::<cgu_extern_drop_glue::Struct> - shim(Some(cgu_extern_drop_glue::Struct))
  expected: extern_drop_glue-fallback.cgu[External] 
  actual:   extern_drop_glue-fallback.cgu.shard001[External] 

fn std::ptr::drop_glue::<mod1::LocalStruct> - shim(Some(mod1::LocalStruct))
  expected: extern_drop_glue-fallback.cgu[External] 
  actual:   extern_drop_glue-fallback.cgu.shard013[External] 


thread '[codegen-units] tests/codegen-units/partitioning/extern-drop-glue.rs' panicked at src/tools/compiletest/src/runtest/codegen_units.rs:109:13:
fatal error
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
---- [codegen-units] tests/codegen-units/partitioning/extern-drop-glue.rs stdout end ----
---- [codegen-units] tests/codegen-units/partitioning/local-drop-glue.rs stdout ----

The following items were assigned to wrong codegen units:

fn std::ptr::drop_glue::<(u32, Struct)> - shim(Some((u32, Struct)))
  expected: local_drop_glue-fallback.cgu[External] 
  actual:   local_drop_glue-fallback.cgu.shard003[External] 

fn std::ptr::drop_glue::<Outer> - shim(Some(Outer))
  expected: local_drop_glue-fallback.cgu[External] 
  actual:   local_drop_glue-fallback.cgu.shard015[External] 

fn std::ptr::drop_glue::<Struct> - shim(Some(Struct))
  expected: local_drop_glue-fallback.cgu[External] 
  actual:   local_drop_glue-fallback.cgu.shard000[External] 

fn std::ptr::drop_glue::<mod1::Struct2> - shim(Some(mod1::Struct2))
  expected: local_drop_glue-fallback.cgu[External] 
  actual:   local_drop_glue-fallback.cgu.shard013[External] 


thread '[codegen-units] tests/codegen-units/partitioning/local-drop-glue.rs' panicked at src/tools/compiletest/src/runtest/codegen_units.rs:109:13:
fatal error
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
---- [codegen-units] tests/codegen-units/partitioning/local-drop-glue.rs stdout end ----
---- [codegen-units] tests/codegen-units/partitioning/local-generic.rs stdout ----

The following items were assigned to wrong codegen units:

fn generic::<&str>
  expected: local_generic.volatile[External] 
  actual:   local_generic.volatile.shard002[External] 

fn generic::<char>
  expected: local_generic.volatile[External] 
  actual:   local_generic.volatile.shard003[External] 

fn generic::<u32>
  expected: local_generic.volatile[External] 
  actual:   local_generic.volatile.shard013[External] 

fn generic::<u64>
  expected: local_generic.volatile[External] 
  actual:   local_generic.volatile.shard008[External] 


thread '[codegen-units] tests/codegen-units/partitioning/local-generic.rs' panicked at src/tools/compiletest/src/runtest/codegen_units.rs:109:13:
fatal error
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
---- [codegen-units] tests/codegen-units/partitioning/local-generic.rs stdout end ----
---- [codegen-units] tests/codegen-units/partitioning/methods-are-with-self-type.rs stdout ----

The following items were assigned to wrong codegen units:

fn <type1::Struct as Trait>::default
  expected: methods_are_with_self_type-type1.volatile[External] 
  actual:   methods_are_with_self_type-type1.volatile.shard011[External] 

fn <type1::Struct as Trait>::foo
  expected: methods_are_with_self_type-type1.volatile[External] 
  actual:   methods_are_with_self_type-type1.volatile.shard011[External] 

fn <type2::Struct as Trait>::default
  expected: methods_are_with_self_type-type2.volatile[External] 
  actual:   methods_are_with_self_type-type2.volatile.shard011[External] 

fn <type2::Struct as Trait>::foo
  expected: methods_are_with_self_type-type2.volatile[External] 
  actual:   methods_are_with_self_type-type2.volatile.shard009[External] 

fn mod1::<impl SomeGenericType<char, &str>>::associated_fn
  expected: methods_are_with_self_type.volatile[External] 
  actual:   methods_are_with_self_type.volatile.shard013[External] 

fn mod1::<impl SomeGenericType<u32, u64>>::method
  expected: methods_are_with_self_type.volatile[External] 
  actual:   methods_are_with_self_type.volatile.shard015[External] 


thread '[codegen-units] tests/codegen-units/partitioning/methods-are-with-self-type.rs' panicked at src/tools/compiletest/src/runtest/codegen_units.rs:109:13:
fatal error
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
---- [codegen-units] tests/codegen-units/partitioning/methods-are-with-self-type.rs stdout end ----
---- [codegen-units] tests/codegen-units/partitioning/shared-generics.rs stdout ----

The following items were assigned to wrong codegen units:

fn shared_generics_aux::generic_fn::<u16>
  expected: shared_generics_aux-in-shared_generics.volatile[External] 
  actual:   shared_generics_aux-in-shared_generics.volatile.shard005[External] 


thread '[codegen-units] tests/codegen-units/partitioning/shared-generics.rs' panicked at src/tools/compiletest/src/runtest/codegen_units.rs:109:13:
fatal error
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
---- [codegen-units] tests/codegen-units/partitioning/shared-generics.rs stdout end ----
---- [codegen-units] tests/codegen-units/partitioning/vtable-through-const.rs stdout ----

The following items were assigned to wrong codegen units:

fn <mod1::NeedsDrop as mod1::Trait1>::do_something
  expected: vtable_through_const-mod1.volatile[External] 
  actual:   vtable_through_const-mod1.volatile.shard005[External] 

fn <mod1::NeedsDrop as mod1::Trait1>::do_something_else
  expected: vtable_through_const-mod1.volatile[External] 
  actual:   vtable_through_const-mod1.volatile.shard007[External] 

fn <mod1::NeedsDrop as mod1::Trait1Gen<u8>>::do_something
  expected: vtable_through_const-mod1.volatile[External] 
  actual:   vtable_through_const-mod1.volatile.shard014[External] 

fn <mod1::NeedsDrop as mod1::Trait1Gen<u8>>::do_something_else
  expected: vtable_through_const-mod1.volatile[External] 
  actual:   vtable_through_const-mod1.volatile.shard008[External] 

fn mod1::id::<char>
  expected: vtable_through_const-mod1.volatile[External] 
  actual:   vtable_through_const-mod1.volatile.shard004[External] 

fn std::ptr::drop_glue::<mod1::NeedsDrop> - shim(Some(mod1::NeedsDrop))
  expected: vtable_through_const-fallback.cgu[External] 
  actual:   vtable_through_const-fallback.cgu.shard012[External] 


thread '[codegen-units] tests/codegen-units/partitioning/vtable-through-const.rs' panicked at src/tools/compiletest/src/runtest/codegen_units.rs:109:13:
fatal error
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

@Trigodil

Trigodil commented Sep 12, 2026

Copy link
Copy Markdown
Author

It seems that forcing this on unconditionally breaks the tests, since they assert exact CGU names (e.g. local_generic.volatile) and now get a .shard0XX suffix appended everywhere

@rust-bors

rust-bors Bot commented Sep 12, 2026

Copy link
Copy Markdown
Contributor

☀️ Try build successful (CI)
Build commit: 6c4e7b9 (6c4e7b9b2b3bc7bc0abc4f3ee5402b3ecedd355e)
Base parent: 9c99d05 (9c99d05505bccb67912d68e05fe7fc7c58afcb41)

@rust-timer

This comment has been minimized.

Sign in to view
@rust-timer

rust-timer commented Sep 12, 2026

Copy link
Copy Markdown
Collaborator

Finished benchmarking commit (6c4e7b9): comparison URL.

Overall result: ❌✅ regressions and improvements - please read:

Benchmarking means the PR may be perf-sensitive. It's automatically marked not fit for rolling up. Overriding is possible but disadvised: it risks changing compiler perf.

Next, please: If you can, justify the regressions found in this try perf run in writing along with @rustbot label: +perf-regression-triaged. If not, fix the regressions and do another perf run. Neutral or positive results will clear the label automatically.

@bors rollup=never rustc-perf
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

mean range count
Regressions ❌
(primary)		 70.7% [0.1%, 1726.1%] 100
Regressions ❌
(secondary)		 52.5% [0.1%, 1727.0%] 69
Improvements ✅
(primary)		 -0.3% [-0.3%, -0.2%] 4
Improvements ✅
(secondary)		 -6.9% [-66.1%, -0.1%] 11
All ❌✅ (primary) 68.0% [-0.3%, 1726.1%] 104

Max RSS (memory usage)

Results (primary 2.4%, secondary 2.5%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

mean range count
Regressions ❌
(primary)		 9.3% [0.5%, 44.3%] 24
Regressions ❌
(secondary)		 9.0% [2.3%, 43.1%] 12
Improvements ✅
(primary)		 -7.5% [-24.4%, -0.4%] 17
Improvements ✅
(secondary)		 -8.7% [-17.4%, -4.3%] 7
All ❌✅ (primary) 2.4% [-24.4%, 44.3%] 41

Cycles

Results (primary 82.8%, secondary 63.6%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

mean range count
Regressions ❌
(primary)		 86.9% [0.4%, 1858.4%] 84
Regressions ❌
(secondary)		 72.4% [2.2%, 1857.6%] 53
Improvements ✅
(primary)		 -1.4% [-2.7%, -0.4%] 4
Improvements ✅
(secondary)		 -14.3% [-65.8%, -2.0%] 6
All ❌✅ (primary) 82.8% [-2.7%, 1858.4%] 88

Binary size

Results (primary 32.7%, secondary 42.8%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

mean range count
Regressions ❌
(primary)		 32.7% [1.9%, 81.9%] 102
Regressions ❌
(secondary)		 42.8% [0.1%, 118.5%] 48
Improvements ✅
(primary)		 - - 0
Improvements ✅
(secondary)		 - - 0
All ❌✅ (primary) 32.7% [1.9%, 81.9%] 102

Bootstrap: 497.802s -> 494.017s (-0.76%)
Artifact size: 406.93 MiB -> 406.67 MiB (-0.06%)

@rustbot rustbot added the perf-regression Performance regression. label Sep 12, 2026
@rustbot rustbot removed the S-waiting-on-perf Status: Waiting on a perf run to be completed. label Sep 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@panstromek panstromek panstromek left review comments
Reviewers whose approvals may not affect merge requirements

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

perf-regression Performance regression. S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. T-compiler Relevant to the compiler team, which will review and decide on the PR/issue.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@Trigodil @rust-log-analyzer @panstromek @rust-timer @rustbot