[python] Use persisted vector metrics for raw search and refinement

[TheR1sing3un]

Purpose

Vector search scores indexed rows using the metric persisted in the index file, but raw fallback and refinement infer their metric from query/table options and otherwise default to L2. Vindex writers default to inner product, and explicitly supplied build options need not remain in table options. Merging these scores can return the wrong top-k.

For example, index [2, 0] with the default IVF-FLAT metric, append [3, 0] without indexing it, and search for [1, 0] in full mode. The indexed row receives score 2.0, while the raw row previously received L2 score 0.2. The raw row now receives inner-product score 3.0 and ranks first.

Read the persisted metric from each opened index reader and use it for raw fallback and original-vector refinement in single, batch and primary-key searches. Persisted metadata takes precedence over table options, which may have changed since the index was built. Explicit query metric conflicts and incompatible metrics across searched index shards raise a ValueError instead of merging incomparable scores. Close readers on metadata/metric validation failures and reset the resolved metric for each read call.

For raw-only searches with a known vindex type and no configured metric, use the writer's inner-product default rather than L2 or a metric inferred from another column. Resolve the selected column's canonical fields.<column>.pk-vector.distance.metric before the vindex default. Raw searches without a known vindex type keep their existing fallback behavior.

Tests

The native mixed-search regression fails on the base revision with 3.0 != 0.2 and passes with this change.

python -m pytest pypaimon/tests/vector_metric_consistency_test.py pypaimon/tests/vector_search_filter_test.py pypaimon/tests/primary_key_global_index_golden_test.py -q

93 passed, 1 skipped, using Python 3.9.6, PyArrow 19.0.1 and paimon-vindex 0.4.0. The skip is the unrelated primary-key full-text fixture test because paimon-ftindex is unavailable; native vector tests and the Java-produced primary-key vector fixture ran.

• Real Paimon/Parquet and native IVF-FLAT coverage for default inner product and build-only inner-product/cosine/L2 settings, mixed indexed/raw results, single/batch refinement, changed table options, incompatible query/shard metrics, and top-k consistency before/after index construction.

• Metric-resolution checks without the native vindex dependency cover query option aliases, other-column settings, all five vindex defaults, reader closure on failures, and metric reset between read calls.

• Java-produced primary-key vector indexes retain identical refinement results when table metric options change.

• Changed files pass repository-configured flake8 and git diff --check.

• Raw-only primary-key regressions remove legacy metric aliases and verify L2, cosine and inner-product Top-K/score results against Java-produced data. Canonical PK query metric aliases are also checked for conflicts with persisted metrics.

[JingsongLi]

Reviewed c0dc74a. Requirement fit: supported; implementation has one correctness finding below.

Using the persisted metric fixes a real mixed indexed/raw ranking problem, but the new raw-only default regresses canonical primary-key metric options. The focused suite passed (89 tests, 1 unrelated full-text skip); an additional real-data primary-key reproduction exposes the gap.