fix: prevent HNSW node degree decay across merge generations by adding absolute M-based threshold

[waterWang]

Fix

This PR fixes a progressive recall degradation issue in IncrementalHnswGraphMerger when merging HNSW graphs across multiple merge generations.

Root Cause

In InitializedHnswGraphBuilder.copyGraphStructure(), a surviving node's neighbor list is copied verbatim minus its deleted neighbors. The node is flagged for repair only if it lost more than 15% of its neighbors in that single merge (DISCONNECTED_NODE_FACTOR = 0.85).

The threshold is relative to the node's current (potentially already thinned) degree at the start of each merge, not its target or original degree. Therefore, nodes that lose only a small fraction of their neighbors (<15%) stay above the threshold every time, and are never flagged and repaired, so their degree decays silently across merge generations with no re-diversification.

Fix

This fix adds an absolute M-based threshold as a second condition: if a node's remaining neighbor count is below M * DISCONNECTED_NODE_FACTOR, it is also flagged for repair regardless of the proportional loss. This prevents the silent progressive decay of connectivity across multiple merge generations.

Changes

• lucene/core/src/java/org/apache/lucene/util/hnsw/InitializedHnswGraphBuilder.java: Added absolute M-based threshold in copyGraphStructure() alongside the existing proportional threshold.

Related

Fixes #16552

[ethbak]

I don't think this will entirely fix #16552, and it would slow merges doing unnecessary work every time:

First, it uses M at all levels including level 0, but the L0 cap is 2M.

M is only the upper-layer cap.

This means newNeighbors.size() < M * DISCONNECTED_NODE_FACTOR undercounts on L0, which is the level the recorded drop in out-degree is on. If we do keep a cap-based floor, we should use newNeighbors.maxSize() - 1, which is M / 2M by level by construction.

Even with that fixed, a cap-based floor inherently has some issues:

By design the diversity heuristic (diversityCheck) stops adding neighbors once nothing left is closer to the node than to a neighbor it already picked, so plenty of nodes end up well under the cap.

Example: full-rebuild, GloVe-100, M=32, 90k docs, zero deletes:

Graph level=0 size=90000, Fanout min=1, mean=36.32, max=64

If we used 2M for L0, like we would need to, this would flag all nodes with under 64 * 0.85 = 54.4 neighbors, which is the vast majority of the nodes in the healthy graph even before any deletes (mean=36.32). Even if we continued using M on every level, plenty of healthy nodes have under 27.2 connections and would be flagged incorrectly every time.

Repair can't do anything for those nodes, since the heuristic just rejects the same candidates again, so they get re-searched and re-flagged every merge and accumulate a lot of wasted work.

In addition, we should benchmark any solution before thinking of merging

I still have the test harness I built to show the initial bug in #16552, so I'm happy to help re-run some tests to show how recall & performance are affected.

[msokolov]

I do appreciate the problem description, but also the description of the problems with this approach. I do think setting a minimum outdegree floor makes sense, and we should consider allowing it to override the diversification. And we definitely need benchmarks in order to settle on anything.