Harden format readers against untrusted length fields (CWE-770 unbounded allocation)

[li-jin-quan]

Summary

All binary-format readers in this crate take lengths, counts and shapes from the (untrusted) embedding file and previously used them to drive up-front allocations — vec![0; n], Vec::with_capacity(n), Array1/Array2::zeros(shape) — before reading any payload byte. A file of a few dozen bytes declaring a large length could force a single allocation of up to 4 GiB (u32 fields) or effectively unbounded (u64 fields), enabling memory-exhaustion DoS (CWE-770 / CWE-400) in any service that loads embedding files from untrusted sources.

This PR makes every such reader grow its buffer from the bytes actually read, and fail with a format error if the stream is truncated.

Allocation sites fixed

Reader	Untrusted field	Before	After
Metadata::read_chunk	chunk_len (u64)	vec![0; chunk_len]	Take::read_to_end + truncation check
NdArray (read_ndarray_chunk)	rows (u64) × cols (u32)	Array2::zeros((rows, cols))	chunked incremental read → Array2::from_shape_vec
NdNorms::read_chunk	len (u64)	Array1::zeros((len,))	chunked incremental read → Array1::from
QuantizedArray::read_chunk	n_embeddings (usize)	Array1::zeros + Array2::zeros (u8)	chunked incremental reads (norms f32 + quantized u8)
read_string (shared by all vocab chunks)	string_len (u32)	vec![0; string_len]	Take::read_to_end + truncation check
read_vocab_items	vocab_len (u64)	Vec::with_capacity(vocab_len)	capped initial capacity, grows while parsing
Header::read_chunk	chunk_identifiers_len (u32)	Vec::with_capacity(n)	capped initial capacity
word2vec binary reader	n_words × embed_len (text header)	Array2::zeros before any read	per-word incremental read → from_shape_vec
text dims reader	n_words × dims	Vec::with_capacity(n_words * dims) (multiplication can overflow)	saturating_mul + capped capacity
fastText reader	size (u32)	Vec::with_capacity(size)	capped initial capacity
floret reader	n_buckets × embed_len	Vec::with_capacity(n_buckets * embed_len)	saturating_mul + capped capacity

Array reads use a shared helper (util::read_f32_vec / read_u8_vec) that allocates at most 64 Ki elements up front and grows in ≤64 Ki-element chunks, so peak memory is proportional to the data actually present in the stream.

Measured impact

With a counting global allocator, parsing the following malicious files:

• 20-byte file — header + Metadata chunk declaring chunk_len = 1 GiB:
  • before: single allocation of 1,073,741,824 bytes (before reading any TOML byte)
  • after: 124 bytes
• 70-byte file — header + one-word vocab + NdArray chunk declaring rows = 1, cols = 268,435,456:
  • before: single allocation of 1,073,741,824 bytes
  • after: 262,144 bytes

Testing

• New regression tests in tests/untrusted_alloc.rs assert that parsing the malicious files above never exceeds a 16 MiB single allocation (they fail on the pre-fix code with ~1 GiB).
• Full test suite: 86 passed, 0 failed.

Notes

• The mmap-based readers are unaffected: they map the file instead of allocating.
• The word2vec/text/fastText/floret readers parse third-party text formats where lengths come from the file header; the same unbounded-allocation pattern applied there, so they are hardened in the same pass for consistency.