Default Retriever to weighted hybrid search

[jioffe502]

Summary

This PR makes weighted hybrid retrieval the accuracy-oriented default for new NeMo Retriever indexes. Each query combines:

• dense semantic search;
• BM25 full-text search (FTS) for exact terms;
• weighted Reciprocal Rank Fusion (RRF), which merges the two ranked lists while favoring dense results.

The selected policy retrieves 50 candidates from each search leg, uses 0.8 dense / 0.2 FTS with RRF k=10, and returns the requested top results without a model reranker.

The key finding is that adding FTS is not automatically an accuracy win. LanceDB's equal-weight RRF can disrupt strong semantic rankings. Dense-favoring fusion preserves the semantic baseline while adding useful exact-match recall.

Evidence

We evaluated 16,283 queries across the full ViDoRe v3, BO767, Earnings/Consulting, and FinanceBench datasets. Every comparison reused the same extracted rows and index; no subsets or model reranker were used.

Evaluation	Queries	Equal RRF Δ Recall@5 / nDCG@10	Weighted RRF Δ Recall@5 / nDCG@10	Result
ViDoRe v3	14,514	-4.66 / -6.02 pp	+0.38 / +0.36 pp	Weighted recovers the regression
BO767	991	+0.10 / +0.29 pp	+1.11 / +1.52 pp	Weighted improves further
Earnings/Consulting	628	+1.75 / +1.25 pp	+1.91 / +1.41 pp	Weighted improves further
FinanceBench	150	-3.34 / -6.31 pp	-0.67 / -0.27 pp	Weighted is statistically tied with dense

Weighted hybrid improved both reported metrics on three evaluations and remained effectively dense-equivalent on FinanceBench. In contrast, equal-weight RRF regressed ViDoRe by 4.66 pp Recall@5 / 6.02 pp nDCG@10 and FinanceBench by 3.34 pp / 6.31 pp. The fusion policy—not merely the presence of FTS—is what makes hybrid a safe default.

Actual dense, equal-RRF, and weighted-RRF scores

Evaluation	Recall@5: dense / equal / weighted	nDCG@10: dense / equal / weighted
ViDoRe v3	0.4623 / 0.4157 / 0.4661	0.5161 / 0.4559 / 0.5197
BO767	0.8486 / 0.8496 / 0.8597	0.7480 / 0.7509 / 0.7632
Earnings/Consulting	0.6545 / 0.6720 / 0.6736	0.5666 / 0.5791 / 0.5807
FinanceBench	0.8467 / 0.8133 / 0.8400	0.7359 / 0.6728 / 0.7332

Why this configuration

The initial full-ViDoRe sweep evaluated 56 configurations over all 14,514 queries: candidate depths 10, 25, 50, and 100; dense weights from 0.5 through 0.975; and RRF k values 10 and 60.

• Leave-one-dataset-out selection chose the same 0.8 / 0.2, k=10 fusion on all eight held-out ViDoRe corpora.
• Depth 50 and depth 100 delivered nearly identical quality; depth 50 reduced live p95 from 166 ms to 110 ms.
• The selected policy was frozen before transfer to BO767, Earnings/Consulting, and FinanceBench. It was not tuned per dataset.

Selection gate and finalists

The selection gate required positive macro nDCG@10, non-negative macro Recall@10, a positive paired-confidence-interval lower bound for query-weighted nDCG@10, wins on at least six of eight ViDoRe corpora, and no corpus worse than -0.5 pp nDCG@10.

Candidate	Recall@5 vs dense	nDCG@10 vs dense	nDCG wins	Worst corpus	Live p95
depth 50, 0.8 / 0.2, k=10	+0.379 pp	+0.357 pp	6 / 8	-0.148 pp	110 ms
depth 100, 0.8 / 0.2, k=10	+0.369 pp	+0.360 pp	6 / 8	-0.117 pp	166 ms

Product behavior

• New or overwritten tables using index_mode=auto become hybrid.
• Appending with auto preserves the table's existing physical mode; persistent dense tables do not migrate silently.
• Explicit index_mode=hybrid upgrades an existing dense table by adding FTS.
• Explicit dense mode remains available for latency-sensitive deployments.
• Query retrieval_mode=auto continues to resolve from the table's physical vector and FTS capabilities.
• No query router, model reranker, approximate vector index, or runtime fusion tuning is introduced.

Implementation scope

• Share one index-mode policy across SDK planning, CLI ingestion, the VectorDB service, Helm, and canonical harness benchmarks.
• Make physical indexes authoritative over stale schema metadata.
• Create and wait for the service FTS index after the first write.
• Optimize incremental FTS state after 20 writes or 100,000 added rows and expose index state in health.
• Keep named dense benchmarks and resolved-mode baseline tagging for historical comparison.
• Add harness replay and ViDoRe extraction diagnostics against an existing hybrid-capable index.

Latency

Weighted hybrid adds 37-80 ms at p50 and 43-75 ms at p95 versus dense in these warmed sequential passes. The modes were not interleaved, so this result is directional; explicit dense mode remains the latency-oriented option.

Latency results

Recommendation

Adopt the fixed weighted-hybrid policy as the default for new indexes, preserve existing physical modes during append, and retain explicit dense mode. The evidence supports a robust cross-dataset default, not a claim that hybrid wins every query or dataset.

Validation

• uvx pre-commit run --all-files: all hooks passed.
• Focused retrieval and harness tests: 61 passed.
• Full post-rebase suite: 2,678 passed and 135 skipped; one Ray worker startup timed out and passed on isolated retry.
• Helm lint passed; auto, dense, and hybrid render into service configuration and deployment arguments, and invalid modes are rejected.
• All 16,283 queries completed in dense, equal-RRF, and weighted-RRF modes.
• All 74,016 positive ViDoRe qrels received an extraction-audit classification.

[greptile-apps]

Greptile Summary

This PR makes weighted hybrid retrieval the default for newly created indexes while preserving the physical mode of existing tables.

• Adds shared index-mode resolution and weighted-RRF fusion across library and service retrieval.
• Creates and maintains service-side FTS indexes, with optimization and health telemetry.
• Updates Helm configuration and benchmark tooling to distinguish dense and hybrid results.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains; the previously reported health-probe failure is addressed by converting transient index-inspection errors into an available HTTP 200 response with unknown index state.

Important Files Changed

Filename	Overview
nemo_retriever/src/nemo_retriever/service/vectordb_app.py	Adds service index-mode resolution, FTS creation and maintenance, weighted hybrid search, and exception-safe index health telemetry.
nemo_retriever/src/nemo_retriever/common/vdb/lancedb.py	Extends LanceDB retrieval to apply the shared weighted hybrid-fusion policy.
nemo_retriever/src/nemo_retriever/common/vdb/hybrid_fusion.py	Defines the shared weighted-RRF policy used by hybrid retrieval paths.
nemo_retriever/src/nemo_retriever/ingest/index_mode.py	Centralizes index-mode resolution for new, overwritten, and appended tables.
nemo_retriever/src/nemo_retriever/ingest/plan.py	Integrates automatic index-mode selection into local and batch ingest planning.
nemo_retriever/helm/templates/configmap.yaml	Validates and renders the configurable VectorDB index mode.
nemo_retriever/src/nemo_retriever/harness/retrieval_comparison.py	Adds replay tooling for comparing retrieval modes against an existing hybrid-capable index.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Ingest request] --> B{Requested index mode}
  B -->|auto + new table| C[Create vector table and FTS index]
  B -->|auto + existing table| D[Preserve physical index mode]
  B -->|explicit hybrid + dense table| E[Add FTS index]
  B -->|explicit dense| F[Use dense index]
  C --> G{Query capabilities}
  D --> G
  E --> G
  F --> G
  G -->|Vector + FTS| H[Dense and BM25 candidate retrieval]
  H --> I[Weighted RRF: 0.8 dense / 0.2 FTS]
  G -->|Vector only| J[Dense retrieval]

Loading

_{Reviews (6): Last reviewed commit: "Merge branch 'main' into jioffe502/hybri..." | Re-trigger Greptile}