Implement data-blind scalar quantization

[mccullocht]

Add an option to the quantization format to enable or disable centering (enabled by default). When centering is disabled we also stop writing the float vectors which can lead to significant storage savings. Special handling is included during merges -- we check that all of the input is in the same encoding, and handle transcoding if some of the input is float vectors.

Large portions of this change were generated using claude code. I reviewed, tweaked, and tested the code before putting it up for review.

This change is being made as a new codec as the format changes to drop the center vector when centering is disabled. This is not strictly necessary as we could write a zero vector instead, but I have plans to make other format changes related to data blindness, see #16029.

luceneutil results -- 1M cohere vectors, 8 bit quantization.
before:

recall  latency(ms)  netCPU  avgCpuCount     nDoc  searchType  topK  fanout  resultSimilarity  decay  resultCount  maxConn  beamWidth  quantized  visited  index(s)  index_docs/s  force_merge(s)  num_segments  index_size(MB)  filterStrategy  filterSelectivity  overSample  vec_disk(MB)  vec_RAM(MB)  bp-reorder  indexType
 0.974        2.304   2.297        0.997  1000000         KNN   100     100               N/A    N/A      100.000       64        250     8 bits     8619    132.85       7527.40          235.00             1         5047.27            null                N/A       1.000      4898.071      991.821       false       HNSW

after

recall  latency(ms)  netCPU  avgCpuCount     nDoc  searchType  topK  fanout  resultSimilarity  decay  resultCount  maxConn  beamWidth  quantized  visited  index(s)  index_docs/s  force_merge(s)  num_segments  index_size(MB)  filterStrategy  filterSelectivity  overSample  vec_disk(MB)  vec_RAM(MB)  bp-reorder  indexType
 0.972        2.281   2.274        0.997  1000000         KNN   100     100               N/A    N/A      100.000       64        250     8 bits     8612    143.06       6990.07          160.33             1         1140.98            null                N/A       1.000      4898.071      991.821       false       HNSW

The harness extrapolates vector size from the input size so believe the on-disk index_size number -- this is about 4x smaller. Force merge is faster since we don't have to re-quantize vectors on merge. Recall is very similar but YMMV.

[github-actions]

This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the dev@lucene.apache.org list. Thank you for your contribution!

[github-actions]

This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the dev@lucene.apache.org list. Thank you for your contribution!

[kaivalnp]

Sorry for getting to this PR so late, but I realized that data-blind quantization would be super helpful for another use-case too: in an attempt to support multiple HNSW graphs backed by the same vector storage (#14758), we need to be able to effectively de-duplicate vectors across fields.

Lucene currently supports a centroid-centered quantization, which might cause the same raw vector to be quantized differently across different fields. If it is data-blind (i.e. centered on the zero vector), quantized bytes can be sanely de-duplicated.

See #16506.

[mikemccand]

+1, I love that we are making progress on data-blind quantization. This should work well when incoming vectors are isotropic (all dimensions behave the same i.e. the histograms of their per-dimension values approximate little baby gaussians with mean 0) and variance as required to be on unit sphere at that dimensionality.

But I think your average trained model won't just produce isotropic embeddings? For such cases (probably the common case? not sure), we have pre-conditioning / random Hadamard rotation (another PR in flight for this? -- yes #16092!) which should (usually? there are adversaries (intentional or otherwise) for any rotation matrix right?) scrub anisotropic vectors. Sort of like the record and record players in GEB -- thank you Kurt Gödel!).

These are all experimental vector codecs but I hope they eventually become default. If we always pre-condition then we almost always can do data blind quantization that is just as good as non-data-blind quantization.

Does this PR make any effort / at least javadocs to explain that you should ensure your incoming vectors are isotropic? Or to spot check if they really seem to be isotropic? luceneutil has all sorts of smell detection ("smelling pipeline" a recent genai model called it!) now to detect all sorts of problems your otherwise very-opaque-to-humans vectors might have.

[mccullocht]

I'll try to revive this over the next week or two.

This should work well when incoming vectors are isotropic (all dimensions behave the same i.e. the histograms of their per-dimension values approximate little baby gaussians with mean 0) and variance as required to be on unit sphere at that dimensionality.

IIRC the base quantizer should behave pretty well if the distribution is uncentered as well (this is what the lower/upper interval are for and why the dot product is unsigned), but if it's not ~Gaussian I would expect the error rate to be high.

RE: rotation -- I think we might want this to provide this but it should be layered just above Lucene. My concern is that if rotation is delegated to the segments during search costs will be very high -- I would expect O(1-2usecs) for a heavily optimized FWHT on good hardware, and you would have to multiply this cost by the number of segments. It'll look fine in a benchmark that's merged to one segment but poor in practice.

Does this PR make any effort / at least javadocs to explain that you should ensure your incoming vectors are isotropic? Or to spot check if they really seem to be isotropic? luceneutil has all sorts of smell detection ("smelling pipeline" a recent genai model called it!) now to detect all sorts of problems your otherwise very-opaque-to-humans vectors might have.

This is not documented, but is generally a constraint for most quantizers. It would be easy to document but I'm not sure what we would do if we detected your vectors don't quantize well. If we had an auto-quantization setting of some kind at that point you would just fall back to float32 or float16.

In general I'm annoyed by the amount of code I have to copy here, especially since all of the flat formats on the read side are just a fixed stride index read. Maybe the layer of abstraction should be closer to the vector layer (float[] -> byte[]) which would also be usable for unquantized float/byte inputs.

[github-actions]

This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the dev@lucene.apache.org list. Thank you for your contribution!