[python] Stream training samples into native vector index trainers

[TheR1sing3un]

Purpose

Vector index construction currently materializes the complete shard and training sample before calling the native trainer. Submit the same evenly spaced sample in bounded batches through VectorIndexTrainer.create, add_training_vectors, and finish_training. This removes the complete Python training matrix while preserving sample count/order, null-row handling, relative row IDs and automatic IVF corpus sizing.

Tests

• python -m pytest pypaimon/tests/vindex_training_test.py pypaimon/tests/global_index_build_test.py -q: 29 passed.
• Native byte-for-byte build comparisons cover IVF-FLAT, IVF-PQ, IVF-SQ, IVF-RQ and DiskANN at sample ratios 1.0 and 0.37 with batches crossing sample boundaries.
• Additional tests verify exact sample positions and bounded reads across batch sizes and sampling ratios, plus cleanup when adding training data or finishing training fails.
• Flake8, license headers and git diff --check passed.

Benchmark

Complete writer builds, including bounded source ingestion, training, adding all production vectors and writing the index. Each run uses a fresh process. macOS arm64, Python 3.9, paimon-vindex 0.4.0; 200,000 vectors x 128 dimensions, IVF-FLAT nlist=64, seed 42. Values are medians of three runs.

The sample-matrix ablation uses bounded reads but still allocates the complete training sample, separating the benefit of bounded input reads from native streaming.

Sample ratio	Variant	Peak RSS (MiB)	Finish (s)	Complete build (s)
1.0	Original one-shot	373.328	0.211	2.523
1.0	Bounded reads into sample matrix	372.297	0.209	2.509
1.0	Streaming trainer	280.531	0.197	2.475
0.1	Original one-shot	330.266	0.185	2.462
0.1	Bounded reads into sample matrix	239.125	0.178	2.452
0.1	Streaming trainer	237.078	0.180	2.441

All 18 runs produced identical index bytes within each sampling ratio. At ratio 1.0 the complete-build peak falls about 25%; at ratio 0.1 most of the memory reduction comes from bounding the initial read, with a smaller additional benefit from streaming. Build time is approximately unchanged. This bounds Python training buffers, not all native index-construction memory.

[JingsongLi]

Reviewed 50d1ec2. Requirement fit: supported; no actionable implementation finding.

This removes the full-shard read and full Python sample matrix from index training while preserving the evenly spaced non-null sample and production row IDs. The corpus size still reaches automatic IVF sizing, and training/add/write failures retain cleanup of the native trainer, result and temporary files.

The focused training/build suites passed locally with paimon-vindex 0.4.0: 29 tests, including byte-for-byte comparisons for IVF-FLAT/PQ/SQ/RQ and DiskANN. I also ran the three-mode complete-writer smoke benchmark at 10,000 × 32, sample ratios 1.0 and 0.1; index digests matched in every mode. Native training/build memory remains outside the Python-buffer bound, as documented.

[JingsongLi]

Remove benchmark.

[TheR1sing3un]

Remove benchmark.

done~

[JingsongLi]

Reviewed eeb57a4. Requirement fit: SUPPORTED. Implementation: CLEAN.

This avoids materializing the full vector corpus in Python for native index training while keeping the bounded training sample and all-vector ingestion in order. The native builder receives the same vectors and sampling behavior, and resources close on both success and failure. The benchmark addition has been removed as requested.

The focused suite passed 29 tests, including 37 subtests, with paimon_vindex 0.4.0; the checks include byte-for-byte index comparisons across the supported IVF and DiskANN cases. I have not independently measured production memory/throughput. Current CI is not fully green: the Python lane stopped on a Maven dependency download (commons-io, connection reset), rather than a reported Python test/lint regression. No blocking code issue found; that lane still needs a successful rerun.