Skip to content

Fix missed embedded-payload niche enum optimization (size enums) - #2

Open
LusterSourav wants to merge 1 commit into
mainfrom
embedded-payload-niche
Open

Fix missed embedded-payload niche enum optimization (size enums)#2
LusterSourav wants to merge 1 commit into
mainfrom
embedded-payload-niche

Conversation

@LusterSourav

Copy link
Copy Markdown
Owner

Fixes rust-lang#160054. An enum whose last niche variant has a single scalar payload field at offset 0 can store that payload in the tag's remaining bits.

  • Adds embedded_payload: Option<(VariantIdx, u128)> to rustc_abi::TagEncoding::Niche
  • all_variants_fit skips the embedded variant's own layout; tag = niche_start + (i - start) + payload
  • Mirrors the field in rustc_public so serialized layouts stay stable
  • Widens remaining Niche matches to .. (const_eval, codegen_ssa, cranelift, llvm debuginfo, ty_utils, ty::layout)
  • Test: enum Unwind { Terminate(Reason), Cleanup(BasicBlock) } now 4 bytes (was 8), niche_start 0xFFFF_FF01, embedded_payload Some((2,2))

Local: rustc_abi full test suite passes (incl. embedded_payload_niche_layout). LLVM/Cranelift paths covered by CI.

When the last variant of a niche-encoded enum has exactly one scalar
field at offset 0, its payload can be stored in the otherwise unused
tag bits, shrinking enums such as:

    enum Unwind { Terminate(Reason), Cleanup(BasicBlock) }

from 8 bytes to 4 bytes.

`TagEncoding::Niche` gains `embedded_payload: Option<(VariantIdx, u128)>`
recording the embedded variant and how many distinct payload values fit
in the tag. The stable rustc_public mirror carries the same field so
serialized layouts round-trip. All existing consumers only pattern-match
the tag encoding with `..`.

Fixes rust-lang#160054.
@LusterSourav LusterSourav self-assigned this Aug 15, 2026
@LusterSourav LusterSourav added the invalid This doesn't seem right label Aug 15, 2026
@LusterSourav LusterSourav changed the title Fix missed embedded-payload niche enum optimization (size: enums) Fix missed embedded-payload niche enum optimization (size enums) Aug 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

invalid This doesn't seem right

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Missed enum layout optimization when Enum contains primitive with multiple niches and other variant have small data

1 participant