feat: component-wise and sharded dataset sources (npy + parquet, LAION-400M)

[generall]

Lets bfb upload corpora published as separate files per component and as numbered parts — LAION-400M being the motivating case, which previously had no route in at all.

Three commits, each independently useful:

1. .npy dense vectors, .parquet payloads

The dataset layer assumed a dataset is a bundle (h5/tgz/csr: vectors, payloads and queries in one artifact). LAION ships img_emb_*.npy next to metadata_*.parquet. The only .npy reader was buried in the tar format keyed off the literal name vectors.npy, and there was no parquet reader.

Adds two component-scoped formats that a config pairs, row i of each landing on point i:

vectors:
  - size: 512
    source: { type: dataset, name: emb, format: npy, path: emb.npy }
payload:
  source:
    type: dataset
    dataset: { name: meta, format: parquet, path: meta.parquet, exclude: [exif] }

Parquet is read via the record API with arrow off — payload rows need no columnar machinery, and skipping it keeps the dependency tree an order smaller. Access is a streaming cursor plus a ring of recent rows rather than a decode-the-file cache: materializing a LAION metadata part up front would cost hundreds of MB before the first batch. Values with no JSON form (null, NaN, ±inf) leave the field absent; fill_null reproduces upload.py's df.fillna(0).

2. Sharded parts: datasets

parts:
  count: 410
  path: laion/img_emb_{i}.npy
  link: https://…/img_emb_{i}.npy

410 files become one row space, so point ids stay global. This also turns --offset from a hazard into a resume switch: a point's id is its dataset row, so per-part invocations used to run off the end of a part. -n is now capped by the rows remaining, and an offset past the end errors rather than uploading nothing.

Row counts are measured, never configured. A rows_per_part option was in the original design and got cut once the real files were probed — LAION has seven distinct part sizes, and the first short one is part 8, so a plausible-looking fixed size would silently pair payloads with the wrong vectors across ~98% of the corpus. Both formats keep their shape at a known end of the file (.npy header at the front, parquet footer at the back), so each part is sized with one ranged request and no download; results are cached in datasets/.parts-index/, keyed on the spec. A host that answers 200 to a Range: request is reported rather than silently streamed.

3. cache: evict — streaming a corpus larger than the disk

Each part is deleted once the reader passes it, and the next is prefetched in the background while the current one uploads. Peak disk ~4 GB instead of ~600 GB. Two safety rules: only parts bfb downloaded itself are ever deleted (a hand-staged file is never touched), and a part is unlinked only once no reader still references it — the parquet reader reopens its file by path on a rewind.

Verification against the published LAION files

• img_emb_0.npy and metadata_0.parquet both report 1,000,448 rows; payloads decode to the expected string/int/float fields.
• All 820 parts size in ~1.5 minutes over the network, downloading nothing; the vector and payload families agree exactly at 407,314,954 rows.
• Part size distribution: 404 × 1,000,448, one × 1,000,501, and short parts 8 (642,675), 107 (189,159), 220 (440,869), 319 (342,038), 409 (518,720).

cargo test 143 passing, clippy and cargo +nightly fmt --check clean.

Not included

Query/ground-truth component sources, so bfb search can report recall against full_scan.py output — the accuracy path still only understands bundle formats. Worth a follow-up.

🤖 Generated with Claude Code