fix: keep row ids and addresses aligned in filter_deleted_ids

[chakshu-dhannawat]

Problem

optimize_indices fails with Invalid user input: batch.num_rows() != chunk.len() after deleting rows from a stable-row-id dataset that has an IVF_FLAT index. Reported in #7701.

Repro (from the issue):

import tempfile, numpy as np, pyarrow as pa, lance
tmp = tempfile.mkdtemp()
n, dim = 5000, 32
ids = np.arange(n)
vecs = np.sin(ids[:, None] * 0.1 + np.arange(dim)[None, :] * 0.3).astype(np.float32)
t = pa.table({
    "id": pa.array(ids, pa.int64()),
    "category": pa.array([["A","B","C","D","E"][i % 5] for i in range(n)]),
    "vec": pa.FixedSizeListArray.from_arrays(pa.array(vecs.ravel(), pa.float32()), dim),
})
ds = lance.write_dataset(t, tmp + "/t.lance", enable_stable_row_ids=True)
ds.create_index("vec", index_type="IVF_FLAT", num_partitions=16)
ds.delete("category = 'A'")
lance.dataset(tmp + "/t.lance").optimize.optimize_indices()  # raises

Root cause

When stable row ids are enabled, deleting rows also removes them from the row id index (decompose_segment_with_deletions skips deleted offsets), so those ids no longer resolve to an address.

filter_deleted_ids built its address list with a filter_map:

let addresses = ids.iter()
    .filter_map(|id| row_id_index.get(*id).map(|address| address.into()))
    .collect::<Vec<_>>();
self.filter_addr_or_ids(ids, &addresses).await

The filter_map silently drops the unresolvable ids, so addresses ends up shorter than ids. filter_addr_or_ids then receives the full ids slice alongside the shortened addresses slice, but it relies on the two being positionally aligned: it sorts the addresses, checks each one against its fragment's deletion vector, un-sorts, and zips the liveness result back against the input ids. Once the lengths diverge the zip pairs ids with the wrong verdict, so some already-deleted ids get reported as live. During an optimize_indices partition join, take_vectors asks take_rows for those ids, gets back fewer rows than the chunk it requested, and hits the batch.num_rows() != chunk.len() check.

Fix

Build ids and addresses together and unzip them, so the two slices stay the same length and positionally correspond. Ids that no longer resolve are dropped from both, which is correct since those rows are already gone.

Testing

Added optimize_after_delete_with_stable_row_ids_joins_partitions in rust/lance/src/index/vector/builder.rs, which ports the repro: stable row ids + IVF_FLAT (16 partitions) + delete + optimize_indices, then checks the surviving row count and that a search over the optimized index returns only live rows. It reproduces the exact original error (batch.num_rows() != chunk.len() (188 != 235)) on the unfixed code and passes with the fix. The existing builder (55) and rowids (16) test suites still pass; cargo fmt and cargo clippy -p lance --tests -- -D warnings are clean.

Summary by CodeRabbit

• Bug Fixes

  • Fixed filtering after deletions when stable row IDs are enabled, preventing valid records from being mismatched or incorrectly excluded.
  • Ensured vector indexes remain accurate after optimizing data with deleted rows.

• Tests

  • Added coverage for vector search and index optimization scenarios involving deleted records and stable row IDs.

[coderabbitai]

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 64a92ad8-fe99-48ee-9a4b-e910e83679f5

📥 Commits

Reviewing files that changed from the base of the PR and between bc2d137 and 46b0a0f.

📒 Files selected for processing (2)

• rust/lance/src/dataset.rs
• rust/lance/src/index/vector/builder.rs

 __________________________________________________
< Veni, Vidi, Validavi. I came, I saw, I reviewed. >
 --------------------------------------------------
  \
   \   \
        \ /\
        ( )
      .( o ).

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}