[experiment] - use DenseBitSetStorage to implement SparseBitMatrix without per-row allocation

[panstromek]

This is an experiment for the DenseBitSetStorage abstraction.

Based on cachegrind benchmarks, this has some impact, as SparseBitMatrix data usually has very few columns, so the empty row in the original implementation actually takes more memory than the non-empty row would allocate. Storing the data inline should take less memory.

This implementation doesn't make much sense though, because there's BitMatrix that's cleaner and works in pretty similar way.

If we proceed with this, it might make more sense to implement it differently (probably as growable BitMatrix).

[panstromek]

@bors try @rust-timer queue

[rust-log-analyzer]

The job aarch64-gnu-llvm-21-2 failed! Check out the build log: (web) (plain enhanced) (plain)

Click to see the possible cause of the failure (guessed by this bot)

    bit_set::tests::sparse_matrix_operations

test result: FAILED. 32 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 65.52ms

error: test failed, to rerun pass `-p rustc_index --lib`
Bootstrap failed while executing `--stage 2 test --skip tidy --skip intrinsic-test --skip src/tools/rust-analyzer --skip tests --skip library --skip tidyselftest`
Currently active steps:
test::CrateLibrustc { build_compiler: Compiler { stage: 1, host: aarch64-unknown-linux-gnu, forced_compiler: false }, target: aarch64-unknown-linux-gnu, crates: ["rustc-main", "rustc_abi", "rustc_arena", "rustc_ast", "rustc_ast_ir", "rustc_ast_lowering", "rustc_ast_passes", "rustc_ast_pretty", "rustc_attr_ir", "rustc_attr_parsing", "rustc_baked_icu_data", "rustc_borrowck", "rustc_builtin_macros", "rustc_codegen_llvm", "rustc_codegen_ssa", "rustc_const_eval", "rustc_crate_store", "rustc_data_structures", "rustc_driver", "rustc_driver_impl", "rustc_error_codes", "rustc_error_messages", "rustc_errors", "rustc_expand", "rustc_feature", "rustc_fs_util", "rustc_graphviz", "rustc_hashes", "rustc_hir", "rustc_hir_analysis", "rustc_hir_id", "rustc_hir_pretty", "rustc_hir_typeck", "rustc_incremental", "rustc_index", "rustc_index_macros", "rustc_infer", "rustc_interface", "rustc_lexer", "rustc_lint", "rustc_lint_defs", "rustc_llvm", "rustc_log", "rustc_macros", "rustc_metadata", "rustc_middle", "rustc_mir_build", "rustc_mir_dataflow", "rustc_mir_transform", "rustc_monomorphize", "rustc_next_trait_solver", "rustc_parse", "rustc_parse_format", "rustc_passes", "rustc_pattern_analysis", "rustc_privacy", "rustc_proc_macro", "rustc_public", "rustc_public_bridge", "rustc_query_impl", "rustc_resolve", "rustc_sanitizers", "rustc_serialize", "rustc_session", "rustc_span", "rustc_structures", "rustc_symbol_mangling", "rustc_target", "rustc_thread_pool", "rustc_trait_selection", "rustc_traits", "rustc_transmute", "rustc_ty_utils", "rustc_ty_walk", "rustc_type_ir", "rustc_type_ir_macros", "rustc_windows_rc"] } at src/bootstrap/src/core/build_steps/test.rs:3353
test::Crate { build_compiler: Compiler { stage: 1, host: aarch64-unknown-linux-gnu, forced_compiler: false }, target: aarch64-unknown-linux-gnu, mode: Rustc, crates: ["rustc-main", "rustc_abi", "rustc_arena", "rustc_ast", "rustc_ast_ir", "rustc_ast_lowering", "rustc_ast_passes", "rustc_ast_pretty", "rustc_attr_ir", "rustc_attr_parsing", "rustc_baked_icu_data", "rustc_borrowck", "rustc_builtin_macros", "rustc_codegen_llvm", "rustc_codegen_ssa", "rustc_const_eval", "rustc_crate_store", "rustc_data_structures", "rustc_driver", "rustc_driver_impl", "rustc_error_codes", "rustc_error_messages", "rustc_errors", "rustc_expand", "rustc_feature", "rustc_fs_util", "rustc_graphviz", "rustc_hashes", "rustc_hir", "rustc_hir_analysis", "rustc_hir_id", "rustc_hir_pretty", "rustc_hir_typeck", "rustc_incremental", "rustc_index", "rustc_index_macros", "rustc_infer", "rustc_interface", "rustc_lexer", "rustc_lint", "rustc_lint_defs", "rustc_llvm", "rustc_log", "rustc_macros", "rustc_metadata", "rustc_middle", "rustc_mir_build", "rustc_mir_dataflow", "rustc_mir_transform", "rustc_monomorphize", "rustc_next_trait_solver", "rustc_parse", "rustc_parse_format", "rustc_passes", "rustc_pattern_analysis", "rustc_privacy", "rustc_proc_macro", "rustc_public", "rustc_public_bridge", "rustc_query_impl", "rustc_resolve", "rustc_sanitizers", "rustc_serialize", "rustc_session", "rustc_span", "rustc_structures", "rustc_symbol_mangling", "rustc_target", "rustc_thread_pool", "rustc_trait_selection", "rustc_traits", "rustc_transmute", "rustc_ty_utils", "rustc_ty_walk", "rustc_type_ir", "rustc_type_ir_macros", "rustc_windows_rc"] } at src/bootstrap/src/core/build_steps/test.rs:3360
Build completed unsuccessfully in 0:29:51

[rust-bors]

☀️ Try build successful (CI)
Build commit: 2d9a55d (2d9a55de3590f93e20dc69685a11be3407316ca4)
Base parent: baf1f7c (baf1f7cc4faa79b938148ad52c53e96c0102cbc4)

[rust-timer]

Finished benchmarking commit (2d9a55d): comparison URL.

Overall result: ❌✅ regressions and improvements - please read:

Benchmarking means the PR may be perf-sensitive. It's automatically marked not fit for rolling up. Overriding is possible but disadvised: it risks changing compiler perf.

Next, please: If you can, justify the regressions found in this try perf run in writing along with @rustbot label: +perf-regression-triaged. If not, fix the regressions and do another perf run. Neutral or positive results will clear the label automatically.

@bors rollup=never rustc-perf
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	0.1%	[0.0%, 0.2%]	5
Improvements ✅ (primary)	-0.3%	[-0.3%, -0.3%]	2
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-0.3%	[-0.3%, -0.3%]	2

Max RSS (memory usage)

Results (primary -0.2%, secondary 5.1%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	2.2%	[2.2%, 2.3%]	2
Regressions ❌ (secondary)	11.0%	[4.8%, 14.4%]	5
Improvements ✅ (primary)	-2.6%	[-3.9%, -1.3%]	2
Improvements ✅ (secondary)	-2.2%	[-2.4%, -2.0%]	4
All ❌✅ (primary)	-0.2%	[-3.9%, 2.3%]	4

Cycles

Results (primary -2.3%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-2.3%	[-2.3%, -2.3%]	1
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-2.3%	[-2.3%, -2.3%]	1

Binary size

This perf run didn't have relevant results for this metric.

Bootstrap: 469.531s -> 484.971s (3.29%)
Artifact size: 400.26 MiB -> 400.18 MiB (-0.02%)

[Zalathar]

A related idea I had at one point is that if we declare these two concrete types:

pub(crate) struct RawBitSetRef<'a> {
    domain_size: usize,
    words: &'a [Word],
}

pub(crate) struct RawBitSetMut<'a> {
    domain_size: usize,
    words: &'a mut [Word],
}

Then the vast majority of bitset operations can be defined by projecting the underlying storage to a raw bitset and calling an underlying implementation on the raw bitset.

[panstromek]

I did try that with the storage abstraction (i.e. DenseBitSet<I, &mut [Word]>. The same thing, just expressed generically. The idea was that some places could use something like DenseBitSet<I, SmallVec<..>> on creation, project to DenseBitSet<I, &mut [Word]> and use that for the duration of the aglorithm to avoid per-access branching.

I tried this on a few hot bitsets, but so far haven't found a one where it would help. This makes me think that I probably misunderstand where the BitSet costs are coming from a bit.

(...and match stress regression is back, nice :D) Oh, nevermind, I'm on a different PR

[panstromek]

I removed the SparseBitMatrix usage in LiveLoans and now it's only used in region inference where this can potentially help more. Let's try again.

[edit] well... maybe I misremembered this but let's try anyway :D

@bors try @rust-timer queue

[rust-bors]

☀️ Try build successful (CI)
Build commit: 7b48929 (7b48929c0c311022bcf0f21ac2775a2138c1f8ae)
Base parent: 6b410a8 (6b410a8387fa41607d50b4f7da844f4d79a565bc)

[rust-timer]

Finished benchmarking commit (7b48929): comparison URL.

Overall result: ❌ regressions - please read:

Benchmarking means the PR may be perf-sensitive. It's automatically marked not fit for rolling up. Overriding is possible but disadvised: it risks changing compiler perf.

Next, please: If you can, justify the regressions found in this try perf run in writing along with @rustbot label: +perf-regression-triaged. If not, fix the regressions and do another perf run. Neutral or positive results will clear the label automatically.

@bors rollup=never rustc-perf
@rustbot label: -S-waiting-on-perf +perf-regression

Instruction count

Our most reliable metric. Used to determine the overall result above. However, even this metric can be noisy.

	mean	range	count
Regressions ❌ (primary)	0.2%	[0.1%, 0.3%]	3
Regressions ❌ (secondary)	0.2%	[0.1%, 0.2%]	5
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.2%	[0.1%, 0.3%]	3

Max RSS (memory usage)

Results (primary -0.7%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	-	-	0
Improvements ✅ (primary)	-0.7%	[-0.7%, -0.7%]	1
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-0.7%	[-0.7%, -0.7%]	1

Cycles

Results (primary 0.4%, secondary 2.4%)

A less reliable metric. May be of interest, but not used to determine the overall result above.

	mean	range	count
Regressions ❌ (primary)	1.3%	[0.7%, 2.4%]	5
Regressions ❌ (secondary)	2.4%	[2.2%, 2.6%]	2
Improvements ✅ (primary)	-0.6%	[-0.6%, -0.4%]	5
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	0.4%	[-0.6%, 2.4%]	10

Binary size

This perf run didn't have relevant results for this metric.

Bootstrap: 477.191s -> 477.8s (0.13%)
Artifact size: 403.62 MiB -> 403.64 MiB (0.00%)

[rust-bors]

☔ The latest upstream changes (presumably #161957) made this pull request unmergeable. Please resolve the merge conflicts by rebasing.