Expose the row address prefilter (Scanner::with_row_addr_prefilter) in the C/C++ API #95

Motivation

We use lance-c as the storage and search backend of a C++ search service. Several of our queries need to restrict a full-text or vector search to a set of rows we already know, before ranking:

• Compound FTS. Without a Boolean query binding (Compound FtsQuery C ABI surface — design preference (JSON vs typed handles)? #35), we compose MUST/SHOULD sub-queries from single-column Match scans. The MUST scans usually match a few hundred rows. A SHOULD scan with common terms, though, scores every row that matches any of its terms (up to 1.7M rows on our data) unless we can restrict it to the MUST candidates.
• Externally computed pre-filters. Some row sets come from our own in-memory bitmaps rather than from a filter expression Lance can evaluate.

Lance core already supports this through Scanner::with_row_addr_prefilter(RowAddrMask), and pylance exposes it as scanner(row_addr_allowlist=..., row_addr_blocklist=...). lance-c has no equivalent. The only option we have is a _rowid IN (...) filter (SQL, or via lance_scanner_additional_sql_filter). Lance evaluates that with a full _rowid read plus an IN-list parse, so it is slow even for a few ids.

Measured with pylance 11 on a 2.48M-row, 3-fragment dataset (stable row ids, 2 GB inverted index), restricting an OR Match scan to the candidate rows of a preceding AND Match scan (290 real queries, warm cache):

_rowid IN costs ~25–35 ms even for 3–5 ids. For a 1.5M-id set, planning the IN list alone took ~3 s and a filtered nearest search ~6.7 s, against 43 ms with row_addr_allowlist. Both restricted variants returned results identical to the unrestricted scan (same rows, same BM25 scores).

Requested API

A setter (or pair of setters) on LanceScanner that maps to Scanner::with_row_addr_prefilter, with C++ wrappers on lance::Scanner. Two shapes seem reasonable, and we're happy with either:

A. Row id arrays (in the style of lance_scanner_set_fragment_ids):

int32_t lance_scanner_set_row_addr_allowlist(LanceScanner* scanner, const uint64_t* row_ids, size_t len);
int32_t lance_scanner_set_row_addr_blocklist(LanceScanner* scanner, const uint64_t* row_ids, size_t len);

B. Serialized RowAddrTreeMap payloads (the same encoding pylance accepts), reusing RowAddrMask::from_serialized_parts:

int32_t lance_scanner_set_row_addr_prefilter(LanceScanner* scanner,
                                             const uint8_t* allowlist, size_t allowlist_len,
                                             const uint8_t* blocklist, size_t blocklist_len);

A is simpler for C callers. B is compact for large sets (a 1.5M-row mask is ~300 KiB against 12 MB of uint64_t) and matches pylance. With B, a helper that serializes a uint64_t array (like pylance's serialize_row_addrs) would spare C callers from reimplementing the treemap framing.

Expected semantics, matching pylance and core:

• Values are in the dataset's _rowid space: the stable row id when stable row ids are enabled, otherwise the row address.
• The mask applies before ranking. A nearest or full-text search, including one run through a prepared FTS query context, computes top-k over the allowed rows only, and scores are unchanged. A plain scan reads only the allowed rows.
• The result is allowed AND NOT blocked, ANDed with any filter expression.
• An empty allowlist selects no rows; not setting one means no restriction.
• Must be set before the scan starts.

Implementation notes

We tried this locally on main and hit two places where the mask can be silently lost:

1. Prepared FTS contexts. rewrite_prepared_fts_plan rebuilds each MatchQueryExec / PhraseQueryExec with new_with_segments(...).with_base_scorer(...). The rebuilt exec does not carry the original exec's external_mask, and core has no getter for it. A search with a prepared context therefore ignores the mask and returns every matching row. Re-applying .with_external_mask(...) on the rebuilt exec fixes it.
2. Segment-scoped scalar scans. PreparedScalarSegment::configure calls with_row_addr_prefilter itself, which would replace a caller's mask. It should either reject the combination or intersect the masks (RowAddrMask::intersect).

Tests worth having: a multi-fragment dataset with stable row ids (so _rowid ≠ row address), covering plain scans, plain and prepared FTS (AND and OR operators, scores equal to the unrestricted search), and nearest-neighbor top-k over the allowed rows.

Related: #35 (compound FTS queries). A native Boolean query would cover our compound-FTS case, but the row prefilter is independent and also serves vector search and externally computed row sets.