perf: serialize updated_fragment_offsets as RoaringBitmap bytes in proto

[jerryjch]

Fixes: #7080

Summary

Follow-up to #6650. The updated_fragment_offsets field (proto field 9) stores per-fragment
matched row offsets as map<uint64, UInt32List> -- one uint32 per matched row. For dense
rewrites this produces multi-GB manifests (e.g. 86k matched rows x 4 bytes x many fragments).

This PR adds proto field 10 (map<uint64, bytes>) using portable RoaringBitmap serialization,
which typically compresses the same data to tens of bytes per fragment. Writers emit field 10
only; readers prefer field 10, falling back to field 9 for manifests written before this change.

Background

PR #6650 added updated_fragment_offsets to the Update transaction message so that
build_manifest can partially refresh _row_last_updated_at_version for matched rows only.
The encoding choice -- one uint32 per offset in a UInt32List -- was flagged post-merge as a
size regression for dense updates. The offsets are already stored internally as RoaringBitmap;
this PR aligns the proto encoding with that representation.

Changes

protos/transaction.proto

• Deprecate field 9 (map<uint64, UInt32List> updated_fragment_offsets) with a comment
  pointing to field 10.
• Add field 10: map<uint64, bytes> updated_fragment_offset_bitmaps with documentation of
  the dual-read strategy.

rust/lance/src/dataset/transaction.rs

Serialization (From<&Transaction> for pb::Transaction):

• Write field 10 only: RoaringBitmap::serialize_into produces portable bytes for each
  fragment's bitmap.
• Set field 9 to an empty HashMap (forward compat; old readers ignore unknown fields).

Deserialization (TryFrom<pb::Transaction> for Transaction):

• If field 10 is non-empty: deserialize each entry with RoaringBitmap::deserialize_from.
• Else if field 9 is non-empty: convert each UInt32List to RoaringBitmap::from_iter
  (legacy fallback).
• Same if !new_field.is_empty() { ... } else { ... } pattern used by the existing
  Rewrite.groups / Rewrite.old_fragments migration.

Invalid field 10 bytes fail deserialize with Error::invalid_input.

In-memory type unchanged: UpdatedFragmentOffsets(HashMap<u64, RoaringBitmap>).

Test plan

• test_proto_round_trip_field_10 -- write a transaction with field 10, read back, verify
  offsets match for two fragments.
• test_proto_legacy_field_9_read -- construct a proto with only field 9 populated
  (simulating an old writer), deserialize, verify offsets are correctly recovered.
• test_proto_field_10_takes_precedence_over_field_9 -- when both fields are present,
  field 10 values are used and field 9 is ignored.

Proto wire format change; team vote may be needed.

Backward compatibility

• Proto field numbers: field 9 is kept (deprecated, not removed). Field 10 is new. No field
  number reuse.
• Old readers: ignore unknown field 10; they only read field 9, which is now empty on new
  commits. Old Lance versions deserializing commits written by this PR will not recover
  offsets from the txn; that only affects audit/readTransaction() on historical
  commits, not table data or OCC.
• New readers: prefer field 10; fall back to field 9 for manifests written by older Lance
  versions that predate this change.
• No JNI or Java changes. The in-memory type (UpdatedFragmentOffsets) is unchanged; only
  the proto wire encoding changes.

Independent of #6748 and lance-spark #528 (JNI wiring). No mutual merge dependencies.

[github-actions]

Important

This PR touches the Lance format specification.

Substantive changes to the format specification — the .proto definitions
and the spec docs under docs/src/format/ — require a PMC vote before merge.
Minor edits such as typo fixes, wording, or formatting are excluded; use your
judgment.

If this is a meaningful format change:

• Start a vote following the Lance community voting process.
  Format specification modifications need 3 binding +1 votes (excluding the
  proposer), held on GitHub Discussions, with a minimum voting period of 1 week.
• Once the vote passes, link the completed vote in this PR. It should not be
  merged until the vote is linked.

[codecov]

Codecov Report

❌ Patch coverage is 93.84615% with 8 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/dataset/transaction.rs	93.84%	6 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

[jerryjch]

Hi, @wjones127 @pengw0048 I created a PR for #7080. Can you check if this is a good approach for the issue?

[pengw0048]

Thanks, @jerryjch . I'm not very familiar with the Lance internals and it would be best if a maintainer can review this PR.

[wjones127]

I have one question. Once that's addressed them I'm happy to open a PMC vote on this change.

[wjones127]

question(blocking): if an older reader sees just field 10, it will ignore it. What are the consequences of that?

[jerryjch]

@wjones127 Thanks for the review.
If an old reader deserializes a commit that only has field 10, it will not populate updated_fragment_offsets (field 9 is empty and field 10 is unknown to the old generated proto). That does not affect table data or OCC. Each writer—old or new—applies in-memory offsets in build_manifest at commit time and updates fragment.last_updated_at_version_meta in the manifest, which is what drives _row_last_updated_at_version at scan time. Both old and new readers read that from fragment metadata. The conflict resolver does not use updated_fragment_offsets. Field 9/10 is only the on-disk protobuf txn encoding.

The only consequence is on old Lance reading the stored txn (e.g. readTransaction()): offsets won’t round-trip for audit/history. Committed dataset versions remain correct. We chose single-write to field 10 to avoid persisting the large UInt32List on new commits. New readers can fall back to field 9 for txns written by older Lance. Old binaries need an upgrade to decode field 10 from txn bytes. This is pretty normal for the kind of proto migration.

[dshepelev15]

Hello @jerryjch, do you have any progress in the pr?

[jerryjch]

@dshepelev15, The PR is pending review and approval. cc @wjones127

[wjones127]

Thanks, I've opened a PMC vote thread here: #7705

That will close in one week, at which point we can merge this PR.

[pengw0048]

Hi @wjones127 , are we good with the vote?

[wjones127]

Hi @wjones127 , are we good with the vote?

Hi @pengw0048 I'm still trying to get votes. We need 3 +1 votes. We currently have zero. I'm asking PMC members to weigh in.