Zero-vector batch rejection corrupts subsequent vector-index insertions #824

Summary

A batch containing a valid embedding followed by a zero vector returns -1 from cactus_index_add, but it has already modified the index files and document map. A subsequent successful insertion can return the rejected batch's content and vector, and the successfully inserted ID is missing after reopening the index.

Environment and validation scope

• Current upstream main: 2cfcdb8568eb9192323b27fd0012a834517b4419 (v2.2.2), fetched before reproducing.
• Host: Linux 6.19.14+kali-amd64, x86-64.
• Compiler: GCC/G++ 15.2.0, C++20, -O2.
• Index dimension: 4, fresh temporary directory; no model or model weights involved.
• Compiled the unmodified cactus-engine/src/index.cpp, index_ffi.cpp, and cactus-graph/src/last_error.cpp, exercising the real C index API and real mmap/file operations.
• This host cannot run the ordinary ARM64 build. The isolated build uses _Float16 for __fp16, skips the ARM-only threading header, and supplies scalar substitutes for the index's four arithmetic dependencies (FP32/FP16 conversion, FP16 dot product, sqrt/divide). The reproduction's basis vectors and zero vector have exact arithmetic. ARM64/SIMD integration has not been tested locally.
• The existing test_index.cpp suite passes all eight tests in this isolated configuration before any changes.

Minimal reproduction

Use a fresh index directory and the following C API sequence. Each document uses metadata "meta".

const float existing[] = {1, 0, 0, 0};
const float batch_good[] = {0, 1, 0, 0};
const float batch_zero[] = {0, 0, 0, 0};
const float later[] = {0, 0, 1, 0};

cactus_index_t index = cactus_index_init(fresh_directory, 4);

int id1 = 1;
const char* doc1 = "existing";
const char* meta = "meta";
const float* emb1 = existing;
cactus_index_add(index, &id1, &doc1, &meta, &emb1, 1, 4); // 0

int ids[] = {2, 3};
const char* docs[] = {"batch-good", "batch-zero"};
const char* metas[] = {"meta", "meta"};
const float* embeddings[] = {batch_good, batch_zero};
cactus_index_add(index, ids, docs, metas, embeddings, 2, 4); // -1
// cactus_get_last_error(): "Cannot normalize zero vector"

int id4 = 4;
const char* doc4 = "later-valid";
const float* emb4 = later;
cactus_index_add(index, &id4, &doc4, &meta, &emb4, 1, 4); // 0

// cactus_index_get(ID4) returns "batch-good" and [0,1,0,0], not ID4's data.
cactus_index_destroy(index);
index = cactus_index_init(fresh_directory, 4);
// cactus_index_get(ID2) succeeds despite its rejected batch.
// cactus_index_get(ID4) fails: "Document ID not found: 4".
cactus_index_destroy(index);

Actual behavior / reproduction evidence

Output from the fresh reproduction through the real C index API (expected missing-ID log lines omitted):

Before batch: index.bin=48 data.bin=24
Batch rc=-1 error=Cannot normalize zero vector
After rejection: index.bin=112 data.bin=60
ID1 rc=0 content=existing metadata=meta vector=[1,0,0,0]
ID2 rc=0 content=batch-good metadata=meta vector=[0,1,0,0]
ID3 rc=-1
ID4 rc=-1
Later insertion rc=0
After later insertion: index.bin=144 data.bin=79
ID1 rc=0 content=existing metadata=meta vector=[1,0,0,0]
ID2 rc=0 content=batch-good metadata=meta vector=[0,1,0,0]
ID3 rc=-1
ID4 rc=0 content=batch-good metadata=meta vector=[0,1,0,0]
After reopening:
ID1 rc=0 content=existing metadata=meta vector=[1,0,0,0]
ID2 rc=0 content=batch-good metadata=meta vector=[0,1,0,0]
ID3 rc=-1
ID4 rc=-1

Expected behavior

For validation/normalization failure, the entire batch should be rejected before mutating the index. After this failed batch:

• ID1 and both file contents/sizes should remain unchanged.
• IDs 2 and 3 should be absent.
• A subsequent successful insertion of ID4 should return "later-valid" and [0,0,1,0].
• Reopening should preserve IDs 1 and 4, with IDs 2 and 3 absent.

Why this matters

The caller receives an ordinary insertion error and can reasonably continue using the index. Continuing currently produces incorrect document/vector associations and loses an acknowledged successful insertion on reopen. This affects the shared native index behind the C API and language bindings.

Technical investigation

In Index::add_documents():

1. validate_documents() checks dimensions, sizes, and IDs, but does not normalize embeddings.
2. Both files are extended and remapped.
3. Each document's data/index records are written, its embedding is converted and normalized, and doc_id_map_ is updated inside the loop.
4. normalize() throws for the later zero embedding.
5. num_documents_ and the persisted header are incremented only after the complete loop.

Consequently, the physical append position/file sizes and the logical document count no longer agree. The next insertion appends after the enlarged file but assigns its map position from the unchanged count.

The existing test_errors() verifies that a zero embedding is rejected, but does not check state after rejection or a subsequent insertion/reopen.

Related upstream work / duplicate check

I searched all-state issues and PRs, conversation and inline review comments, discussions, current index history, and relevant audit branches using batch failure, partial insertion/mutation, zero vector/embedding, normalization failure, rejected batch, rollback, corruption, and persistence concepts.

• Cactus index #264 introduced the index. Its review comment concerns ftruncate/mmap failure and rollback.
• cactus-engine: runtime correctness & memory-safety #703 (81857ca) handles failed mmap state and query-dimension checks; its follow-up review concerns methods dereferencing failed mappings.
• Open fix(python): grow buffers instead of failing, guard optional ops, stop redundant FFI work #814 modifies Python binding buffers/validation, and Fix Android indexAdd input validation #788 modifies React Native Android input validation; neither changes this native normalization path.

This reproduction has successful file resizing/remapping and fails later on embedding normalization. I found no report or proposed patch for this exact failure mode.

Suggested direction

Validate and normalize the complete batch before mutating persistent index state, staging the normalized FP16 representation for the write phase; alternatively make the insertion transactional. This is a proposed direction, not an approved design. The focused requirement is atomicity for validation/normalization failures, without changing normalization behavior or the on-disk format.

I can contribute a focused fix and regressions covering both batch orderings, existing data, file integrity, subsequent insertion, vector retrieval, and reopening.