Add a scalar-quantized version of the de-duplicating vector format by kaivalnp · Pull Request #16506 · apache/lucene

kaivalnp · 2026-08-12T01:33:48Z

Description

Add a scalar-quantized version of the de-duplicating vector format DedupHnswScalarQuantizedVectorsFormat (follow-up to #15979), which stores each distinct vector once in both raw and scalar quantized form: searches score against the quantized vectors, while raw vectors remain available for exact reads and re-scoring (same as Lucene 10.4 OSQ).

De-duplication works like the raw format: vectors of the same dimension and encoding form a group, and each field maps its document ordinals onto shared group ordinals through a fieldOrdToGroupOrd translation.

Files:

.vdd: Raw vectors and per-field data (same as the flat de-duplicating format).
.vdqm (metadata): groups with their per-flavor block locations (each block records its own ScalarEncoding), followed by per-field entries. See below for "flavor".
.vdqd (quantized records): one block per FLOAT32 group and quantization flavor in use, one record per distinct vector (quantized bytes + corrective terms in-line with the Lucene 10.4 OSQ format).

Design choices:

Data-blind quantization: Vectors are quantized against a zero centroid instead of a per-field data-derived centroid.
1. A quantized record is then a pure function of the raw vector and its flavor (independent of the other vectors in the segment), allowing for sane de-duplication between fields containing overlapping vectors.
2. This also gives merges a fast path: records from source segments in this format are copied verbatim instead of re-reading raw vectors and re-running the interval optimization (re-quantization only happens on flush, or when a source has an incompatible encoding/flavor).
3. The tradeoff is accuracy on off-center data relative to centroid-centered quantization.
Per-similarity "flavors": One record can't serve all similarity functions (EUCLIDEAN needs the squared norm as its corrective term; COSINE needs write-time normalization).
1. Each group therefore stores one block per flavor in use — EUCLIDEAN, DOT_PRODUCT + MAXIMUM_INNER_PRODUCT, and COSINE, so fields whose similarity functions map to the same flavor share records, and every flavor scores through the stock Lucene104ScalarQuantizedVectorScorer with no custom score math.
2. The scorer is a small subclass of the raw format's scorer: doc operations on the field's values, vector operations delegated to the group view, ordinals translated once per comparison (same as the flat format).
Quantized records are 1:1 with group ordinals: Since quantization is lossy, distinct raw vectors can collapse to identical quantized bytes, so a second level of de-duplication is possible in principle.
1. This was deliberately left out: it would require maintaining a second ordinal mapping (groupOrd → quantizedOrd) off-heap, de-duplicating quantized bytes (hash + equality checks on collision), splitting corrective terms into a separate dense block, and writing temp files + read-back during merge for proper de-duplication.
2. Keeping records 1:1 with group ordinals shares the existing fieldOrdToGroupOrd map between raw and quantized data and keeps merges streaming, at the cost of occasionally storing byte-identical quantized records for distinct raw vectors.

DISCLAIMER: Much of this code was written by an AI, but I have reviewed and refined most of it!

kaivalnp · 2026-08-12T02:06:41Z

Benchmarks

Cohere v3 vectors, 100K docs, 10K queries, maxConn=64, beamWidth=250, topK=100, fanout=100, force merged to 1 segment.

main

quantized  visited  index(s)  index_docs/s  merge(s)  force_merge(s)  index_size(MB)         filterStrategy  recall  latency(ms)  netCPU  avgCpuCount
       no    12962     19.93       5016.55     22.12           20.83          403.01  query-time-pre-filter   0.495        4.301   4.300        1.000
       no     6198     21.58       4633.71     20.02           35.27          603.03      index-time-filter   0.995        1.941   1.940        1.000
   8 bits    12994     19.85       5038.04     23.42           28.17          502.19  query-time-pre-filter   0.495        5.060   5.059        1.000
   8 bits     6202     21.93       4560.38     23.92           47.01          751.75      index-time-filter   0.985        2.411   2.410        1.000
   4 bits    13136     20.35       4915.21     13.21           20.34          453.40  query-time-pre-filter   0.495        2.946   2.945        1.000
   4 bits     6279     22.57       4431.25     12.31           33.11          678.62      index-time-filter   0.931        1.288   1.287        0.999
   2 bits    13667     20.51       4875.43     13.71           20.74          429.31  query-time-pre-filter   0.489        2.387   2.386        1.000
   2 bits     6587     22.28       4487.73     12.31           32.34          642.48      index-time-filter   0.842        0.960   0.960        0.999
   1 bits    15227     19.60       5101.26     11.41           17.39          418.30  query-time-pre-filter   0.452        1.996   1.995        1.000
   1 bits     7412     22.28       4488.13     10.21           28.86          625.90      index-time-filter   0.726        0.893   0.892        0.999

This PR

quantized  visited  index(s)  index_docs/s  merge(s)  force_merge(s)  index_size(MB)         filterStrategy  recall  latency(ms)  netCPU  avgCpuCount
       no    12956     19.80       5051.53     18.92           20.13          403.38  query-time-pre-filter   0.495        4.481   4.480        1.000
       no     6192     24.71       4047.11     25.32           30.99          409.01      index-time-filter   0.995        2.062   2.062        1.000
   8 bits    12965     19.87       5031.95     22.82           24.53          502.57  query-time-pre-filter   0.495        5.129   5.128        1.000
   8 bits     6206     25.26       3958.51     30.33           37.65          508.19      index-time-filter   0.985        2.372   2.371        0.999
   4 bits    13167     20.06       4985.04     11.31           16.64          453.79  query-time-pre-filter   0.495        2.911   2.910        1.000
   4 bits     6279     25.18       3971.56     19.82           24.26          459.44      index-time-filter   0.922        1.288   1.286        0.999
   2 bits    13765     20.51       4875.91     11.21           17.33          429.54  query-time-pre-filter   0.481        2.325   2.323        1.000
   2 bits     6688     25.47       3926.50     18.92           26.52          435.27      index-time-filter   0.809        1.038   1.037        0.999
   1 bits    15405     19.88       5029.68     10.41           15.19          418.55  query-time-pre-filter   0.434        2.139   2.138        1.000
   1 bits     7590     24.74       4041.87     16.51           24.12          424.78      index-time-filter   0.687        0.952   0.951        0.999

Some observations:

recall is lower with the data-blind quantization, and drop increases with higher levels of quantization as expected (drop is <4% for this combination).
latency is higher with the de-duplication, and the increase as a percentage is higher with higher levels of quantization as expected (increase is <10% for this combination).
index_size(MB) drops with the de-duplicating vector format, and the index size is approximately the same b/w query-time-pre-filter and index-time-filter (differing in HNSW graph size).

msokolov · 2026-08-20T15:27:17Z

So I understand why recall might drop a bit due to less-optimal quantization (no centering -- is that the only change?), but I'm less clear why we expect an increase in latency when de-duplicating, especially as the increase correlating with reduced quantized bit size. I think I would expect a small increase, perhaps, due to the need for an indirection when looking up ordinals relative to a dense index, but this wouldn't vary with quantization.

kaivalnp · 2026-08-20T18:30:03Z

why we expect an increase in latency when de-duplicating, especially as the increase correlating with reduced quantized bit size

I think this is a combination of two things: the fixed cost per-node for the indirection (off-heap fieldOrdToGroupOrd lookup) will be a larger proportion of individual vector scoring with higher quantization (i.e. 1-bit vector computations are faster than 8-bit, so the additional latency from the indirection is a higher percentage increase) + more nodes being visited due to more approximate scores after loss of centering (also showing up as drop in recall).

msokolov

phew, a lot to cover. I had a few small things

msokolov · 2026-08-21T12:38:36Z

+  private static final FlatVectorsScorer FLAT_SCORER =
      FlatVectorScorerUtil.getLucene99FlatVectorsScorer();

+  private final FlatVectorsScorer scorer;


since this is a scorer that delegates can we name this the conventional way as either delegate or in?

Makes sense, changed to delegate!

msokolov · 2026-08-21T12:42:19Z

+    this.scorer = scorer;
+  }
+
+  /** Resolves the values to score; subclasses may unwrap composite values. */


Can we call it unwrap? I was confused about what this was doing when I saw it used below

Makes sense, changed!

msokolov · 2026-08-21T12:47:10Z

 * DedupMergeContext} accordingly.
 *
+ * <p>Also used by {@link DedupScalarQuantizedVectorsFormat} (with a non-null {@link
+ * DedupQuantizer}) to additionally write a quantized copy of each FLOAT32 group, into a separate


maybe in future they could be FLOAT16? Should we say each "full-precision" or "unqunatized" group?

Makes sense, changed each FLOAT32 group -> each applicable group (considering that we may be able to quantize FLOAT16 / BYTE vectors in the future)

msokolov · 2026-08-21T12:54:02Z

+  }
+
+  /** Supplies the distinct (raw) float vector at a group ordinal. */
+  interface FloatVectorSupplier {


what's the visibility of this? If we later want to quantize Float16 (ie short[]) or even byte[] will we need a breaking change?

This is package-private. When we enable quantization on float16[], it will likely need new methods inside this class and the de-duplicating vector writer -- but will not be breaking for end-users who can only consume the top-level format (i.e. DedupHnswScalarQuantizedVectorsFormat), which will be unchanged.

msokolov · 2026-08-21T13:01:44Z

+      if (flavorOrd < 0 || flavorOrd >= Flavor.values().length) {
+        throw new CorruptIndexException("Invalid flavor ordinal: " + flavorOrd, meta);
+      }
+      Flavor flavor = Flavor.values()[flavorOrd];


Lucene doesn't generally rely on Enum ordinals assigned by the JVM and rather will explicitly define codes to store in the index, that can later be mapped to enums as needed. This enables, for example, later removing support for an ordinal. EG see the way VectorSimlarityFunction is handled in Lucene94FieldInfosFormat, or indeed the way ScalarEncoding is being deserialized here using a custom function (fromWireNumber)

Makes sense, switched to explicit ordinals!

msokolov · 2026-08-21T13:14:33Z

+      OptimizedScalarQuantizer.QuantizationResult corrections)
+      throws IOException {
+    quantizedVectorData.writeBytes(packed, packed.length);
+    quantizedVectorData.writeInt(Float.floatToIntBits(corrections.lowerInterval()));


for flavors where the corrections are uniformly zero, can we skip writing them?

I think the additionalCorrection term is always zero for some flavors (DOT_PRODUCT and NORMALIZED, but not EUCLIDEAN).

Writing the (no-op) correction factor has some benefits: like parity with the non de-duplicating format (allowing re-use of the scorer and vector values classes), and aligning the quantized bytes to multiples of 4 for optimal scoring performance.

ok, the savings should be pretty small anyway

msokolov · 2026-08-21T13:18:57Z

-  void finish(IndexOutput meta, IndexOutput vectorData) throws IOException {
+  /**
+   * Merges each group's distinct vectors, followed by per-field metadata. When {@code quantizer} is
+   * non-null, a quantized copy of each FLOAT32 group is also written and its block location


each "full precision" group?

Makes sense, changed each FLOAT32 group -> each applicable group

msokolov · 2026-08-21T13:48:38Z

+ * quantizing.
+ *
+ * <p>Only {@link org.apache.lucene.index.VectorEncoding#FLOAT32} vectors are quantized; BYTE and
+ * FLOAT16 vectors are stored raw only, identical to {@link DedupFlatVectorsFormat}.


ah, okay, it's by design. But I think we would eventually want to be able to quantize FLOAT16 too?

Right, Lucene doesn't support quantizing FLOAT16 vectors today, we'll need to copy whatever the flat format does eventually.

msokolov · 2026-08-21T13:51:37Z

    @Override
    public FieldOrdToGroupOrd copy() {
-      return new FieldOrdToGroupOrdArrayList(fieldOrdToGroupOrd);
+      throw new UnsupportedOperationException("not meant for copying");


I guess we just never needed this?

Yes, this is used during flush, and was never copied -- so I threw an exception instead.

msokolov · 2026-08-21T14:00:55Z

+ * both raw and quantized form. General de-duplication behavior is covered by {@link
+ * TestDedupFlatVectorsFormat}; this test focuses on the quantized side.
+ */
+public class TestDedupScalarQuantizedVectorsFormat extends LuceneTestCase {


since we explicitly forbade copy() on some the vector values, let's add a test ensuring that those methods throw Unsupported exception

I'm not sure about how to reach the #copy method for those classes from public APIs, any suggestions to have a meaningful test? (or do you mean a simple one that creates a new FieldOrdToGroupOrdArrayList and expects an error on calling #copy)

yeah, I was thinking of a simple unit test

Also add CHANGES.txt entry

msokolov

thanks!

msokolov · 2026-08-24T20:58:15Z

+      float[] vector = vectors.get(ord);
+      if (flavor.normalized()) {
+        // normalize a copy: the source buffer is shared / owned by the group
+        System.arraycopy(vector, 0, normalized, 0, dimension);


Add a test to validate that on-heap field ord -> group ord mappings are not supposed to be copied.

kaivalnp · 2026-08-27T15:27:55Z

Thanks for the review @msokolov!

I'll merge this in a couple of days unless there are further comments.

Add a scalar-quantized version of the de-duplicating vector format

3710ac8

github-actions Bot added module:core/codecs module:sandbox labels Aug 12, 2026

This was referenced Aug 12, 2026

Add a de-duplicating flat vector format #15979

Merged

Support multiple HNSW graphs backed by the same vectors #14758

Open

Implement data-blind scalar quantization #16030

Open

kaivalnp linked an issue Aug 13, 2026 that may be closed by this pull request

Support multiple HNSW graphs backed by the same vectors #14758

Open

msokolov reviewed Aug 21, 2026

View reviewed changes

kaivalnp added 2 commits August 24, 2026 17:55

Merge branch 'main' into dedup-quantized-vectors

4fc8299

Address comments

452c6dd

Also add CHANGES.txt entry

kaivalnp marked this pull request as ready for review August 24, 2026 18:24

msokolov approved these changes Aug 24, 2026

View reviewed changes

kaivalnp added 2 commits August 27, 2026 15:23

Merge branch 'main' into dedup-quantized-vectors

5dd023b

Address comments

c3dd0ad

Add a test to validate that on-heap field ord -> group ord mappings are not supposed to be copied.

Uh oh!

Conversation

kaivalnp commented Aug 12, 2026

Description

Uh oh!

kaivalnp commented Aug 12, 2026

Benchmarks

Uh oh!

msokolov commented Aug 20, 2026

Uh oh!

kaivalnp commented Aug 20, 2026

Uh oh!

msokolov left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

msokolov left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kaivalnp commented Aug 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants