[New feature] Add an adaptive filtered strategy PathSeer for Lucene filtered HNSW search

[lizhiyuell]

Description

This PR introduces PathSeer, an adaptive filtered HNSW search strategy, for Lucene's filtered HNSW search.

Motivation

Existing filtered HNSW strategies can perform very differently across workloads. In particular, the best strategy may change with filter selectivity and query-filter correlation, making it difficult for a fixed strategy to perform consistently well across different filtering regimes.

PathSeer aims to improve this adaptivity and provide a more robust recall-performance trade-off across workloads.

Approach

PathSeer combines distance-first-then-filter traversal with filter-first-then-distance traversal. The key idea is to preserve the connectivity and navigability of the search subgraph while opportunistically avoiding unnecessary vector similarity computations.

Compared with the original PathSeer design described in the paper, this PR adapts the algorithm for better compatibility with Lucene's existing HNSW index. It does not modify the graph structure or on-disk index format; instead, two-hop neighbors, similar to those used by ACORN, are used in place of the original expanding-zone neighbors.

Since two-hop neighbors may be relatively far from the query, this version only explores them while the candidate heap is not yet full. In addition, when traversing neighbors that do not satisfy the filter, PathSeer uses only filter-first-then-distance traversal for their subsequent expansion. This helps limit unnecessary distance computations in attribute-correlated workloads.

Benchmark

We evaluated PathSeer in two benchmark settings: the standard luceneutil Cohere v3 Wikipedia workload, covering different filter selectivities and search-effort configurations, and BEIR-based workloads, covering different filter selectivities and query-filter correlation settings. Selected results from the BEIR-based workloads are shown below, and the complete results are provided in the attached benchmark report.

The results show that PathSeer provides a more robust recall-performance trade-off across the tested filtering regimes. Across the Cohere benchmark, PathSeer improves average throughput over all baselines by 35.19% at the 90% recall target and 39.45% at the 95% recall target. Across the BEIR workloads, the corresponding average improvements are 96.60% and 87.34%.

Full benchmark report:
pathseer_lucene_benchmark_report.pdf

[msokolov]

results look very promising! thanks for sharing. I think it might take a little while for us to digest