perf: parallel create/update bloomfilter

[wojiaodoubao]

Background

When training a Bloom filter on a dataset with 2 billion rows (string type), the task did not complete even after 3 hours.

---

Analysis

Lance currently supports two types of zone-based indexes:

• Bloom Filter
• ZoneMap

Their performance characteristics differ significantly:

• Bloom Filter training

  • CPU-intensive
  • Can benefit from parallelism

• ZoneMap construction

  • Not CPU-intensive
  • Bottleneck lies in I/O and memory bandwidth
  • Parallelism does not provide noticeable speedup

---

Experiments

1. Bloom Filter Training (Double, 20M rows, object storage)

We evaluated the impact of parallelism on Bloom filter training.

Parallelism	Time (s)
1	20.788
2	12.165
5	7.616
10	6.295

---

2. ZoneMap Index (Int, 200M rows)

We evaluated ZoneMap performance under different parallelism levels.

Parallelism	Time (s)
1	25
2	25
10	25

---

Proposed Changes

Update zoned.rs to introduce parallel execution support:

• Add:

  • train_parallel
  • rebuild_zones_parallel

• Rename existing methods:

  • train → train_serial
  • rebuild_zones → rebuild_zones_serial

Usage

• Bloom Filter → use *_parallel
• ZoneMap → use *_serial

---

• This pr is blocked by: fix: handle bloomfilter index overflow by multi-batch read&write #6522

[codecov]

Codecov Report

❌ Patch coverage is 85.78352% with 88 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance-index/src/scalar/zoned.rs	85.60%	37 Missing and 22 partials ⚠️
rust/lance-index/src/scalar/bloomfilter.rs	86.63%	15 Missing and 12 partials ⚠️
rust/lance-index/src/scalar/zonemap.rs	71.42%	1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!