[Gabriel39]

lance_dataset_prewarm_index() can fail for a valid vector index when its name matches another schema column. The C++ Dataset::prewarm_index() wrapper consequently throws lance::Error. Resolve the indexed field from segment metadata for this collision so prewarming succeeds on a fresh session and subsequent queries reuse the warmed partitions.

Trigger and impact

A concrete reproducer is:

1. Create a dataset with id: Int32 and embedding: FixedSizeList<Float32, 8>.
2. Build an IVF_FLAT index on embedding, but name the index id. The regression uses two fragments and two index segments.
3. Open the dataset with a fresh session, before any vector query has warmed that session.
4. Call lance_dataset_prewarm_index(dataset, "id").

Before this fix, the call returns -1 with an error containing Data type is not a vector ... but Int32. Index construction and nearest-neighbor queries on embedding succeed for the same dataset. Consequently, callers cannot complete an explicit warmup before serving queries; applications that continue after the error still incur cold index reads. Applications that require warmup to succeed can reject an otherwise usable dataset. The demonstrated failure is in prewarm, without changing persisted data or index contents.

The name collision matters: an index name that is not a schema column already resolves through index metadata. The test uses a separate session for the cold control query, so that query cannot populate the session used to reproduce prewarm.

Cause and fix

Reproduced against lance-c 2089f6f (current main), which pins Lance SDK 68c12df. This is a correctness follow-up to #94.

The pinned Lance SDK passes the logical index name to open_generic_index(). Its vector field resolver checks for a schema column with that name before consulting the segment's field ID, so id incorrectly selects the scalar column instead of embedding.

For a name that matches a different schema field, the binding loads the named index's segment metadata and opens each affected vector segment using the indexed field path and segment UUID. It prewarms that opened index on the same dataset snapshot and session. The complete open-and-prewarm operation for each affected segment runs concurrently via try_join_all, preserving the SDK’s fail-fast error propagation and cancellation of pending segment futures. Other paths continue through the SDK, including missing-index errors and scalar/full-text prewarm with its aggregate residency validation.

This keeps the existing C/C++ API and dependency revision. It does not include the Doris-specific Foyer patch. Found while integrating upstream prewarm into apache/doris#69277.

Validation

Linux, Rust 1.97.0, using the unchanged lockfile and default features:

• Before the fix: the new cold-session collision test failed with -1 and the Int32/vector type error, after its independent cold-query control succeeded.
• cargo test --locked --test c_api_test test_prewarm_index: all 6 prewarm tests passed. The vector cases cover non-colliding names, names matching the indexed field, and names matching a different scalar field. Both segments return correct nearest neighbors with zero partition reloads after prewarm and a shared-session reopen.
• cargo test --locked --lib index::tests::: all 3 scheduling tests passed. Deterministic gates verify that all 8 segment opens start before any may finish, and all 8 prewarms start before any may finish. Separate checks verify immediate error propagation/cancellation during open or prewarm, and scalar/unresolved-segment fallback to the SDK. The concurrency tests failed against the serial implementation (1 segment started instead of 8); no timing thresholds or sleeps are used.
• Coverage also includes B-tree/FTS name collisions, invalid arguments, missing indices (including names matching an existing column), pinned snapshots, repeated calls, and dataset/session handle lifetimes.
• cargo test --locked: 492 passed, 0 failed; the 3 ignored C/C++ cases were run separately. Doc-tests also completed successfully.
• cargo test --locked --test compile_and_run_test -- --ignored --test-threads=1: all 3 C/C++ compilation/runtime cases passed.
• cargo clippy --locked --all-targets -- -D warnings: passed.
• cargo fmt -- --check and git diff --check: passed.

[yanghua]

Left one comment.

[yanghua]

This makes the collision path prewarm every vector segment sequentially. The SDK implementation uses try_join_all, so a logical index with many distributed-build segments currently warms concurrently. Since open_generic_index() and prewarm() can both perform substantial object-store I/O, the new path changes warmup latency from roughly the slowest segment to the sum of all segments. The two-segment local test cannot expose that regression. Please collect the colliding-vector futures and await them concurrently—ideally preserving the SDK path’s error/concurrency semantics—and add a deterministic multi-segment concurrency test.

[lance-gatekeeper]

✅ Gate recommendation: approve.

Resolving colliding vector segments by their stored field IDs remains a reasonable workaround for the pinned SDK. Scalar/FTS validation and snapshot/session behavior are preserved.

The revised path runs segment opens and prewarms concurrently and cancels pending work on error, addressing the concurrency concern. All three scheduling tests and all six C API prewarm tests pass, including cold-session collision handling and reuse of both warmed vector segments without partition reloads.

[yanghua]

LGTM