Clear the Vector Set index pointer on the diskless replication stream-in path

[tiagonapoli]

Problem

A Vector Set index record persists the native DiskANN handle IndexPtr in its value (offset 8 of the 56-byte VectorManager.Index). Diskless replication streams raw Tsavorite log records verbatim (ReplicationSnapshotIterator → DiskLogRecord.Serialize), so a replica taking a full sync adopts the primary's handle.

Nothing clears it on that path:

• GarnetRecordTriggers.OnDiskRead is the only hook that zeroes IndexPtr, and it only fires for records faulted in from disk. Streamed records land directly in memory, so it never runs.
• VectorManager.NeedsRecreate is exactly indexPtr == 0, so a foreign non-zero pointer is indistinguishable from a healthy local one and the lazy Service.RecreateIndex rebuild is skipped.
• The raw persisted value then reaches the P/Invoke unvalidated.

In production this faults the replica with SIGSEGV inside NativeDiskANNMethods.card, and the replica crash-loops so it never converges. Where the address happens to resolve to another live index instead of unmapped memory, the read succeeds and silently answers out of the wrong vector set.

This only affects the replication path. VADD/VREM replicate vector payloads, not native bytes, so a replica that is attached before the writes rebuilds its own index locally and is fine. It takes a full sync of an already-populated Vector Set to hit this.

Fix

Clear the pointer as the record enters the replica's store, in NetworkClusterSync — the single point where streamed records are ingested:

if (diskLogRecord.RecordType == VectorManager.RecordType)
    VectorManager.ClearIndexPointer(diskLogRecord.ValueSpan);

The index is then recreated on first touch. This is the same thing OnDiskRead already does for disk-faulted records, and SetContextForMigration already does for slot migration — diskless replication was the one path that carried the handle across nodes untouched.

VectorManager.ClearIndexPointer becomes public for this; Garnet.cluster already consumes the public ReadIndex / SetContextForMigration on the migration path in the same way.

Tests

New test/cluster/Garnet.test.cluster.vectorsets/VectorSets/ClusterVectorSetDisklessSyncTests.cs:

Test	Before fix	After fix
VectorSetReadableOnReplicaAfterDisklessFullSync	FAIL	PASS
VectorSetsStayPartitionedAcrossDisklessFullSync	FAIL	PASS
VectorSetReadableOnReplicaWithoutFullSync (control)	PASS	PASS

Existing cluster Vector Set coverage cannot catch this defect: those tests go through SimpleSetupClusterAsync, which never passes enableDisklessSync, and they attach replicas before writing, so each replica builds its own index from replicated VADD payloads.

Why the assertion is structural rather than behavioural. Cluster tests run every node inside a single process, so the primary's handle is still mapped and valid when the replica dereferences it. The replica quietly answers out of the primary's live index and every black-box read returns the correct answer — the SIGSEGV cannot be reproduced in-process, it needs two address spaces. The tests therefore assert the invariant that is actually violated: a node may only dereference a handle it allocated itself. Against unfixed main the replica holds the byte-identical pointer:

replica persisted the primary's DiskANN handle for '{vsdisk}solo' (0x2c28269b490)
Expected: not equal to 3034434876560
But was:  3034434876560

The record is read via Read_MainStore rather than VectorManager.ReadVectorIndex, because the latter consults NeedsRecreate and would rebuild the index, rewriting the field under test.

The tests also assert recoverFullSync:True from the sync metadata, so they cannot pass vacuously by silently degrading into an incremental AOF replay. The no-full-sync control passes throughout and pins the full sync down as the cause.

Garnet.test.cluster.vectorsets is added to InternalsVisibleTo so the test can read the persisted record.

Validation

• 3/3 new tests fail on unfixed main and pass with the fix.
• Garnet.test.cluster.vectorsets — 85/85 pass.
• Garnet.test.cluster.replication.disklesssync — 29/29 pass.
• Clean Release build across both target frameworks, 0 warnings.

Background

Found while reproducing a customer report on a live 1-shard / 3-replica cluster, where replicas were terminating with status=11/SEGV and crash-looping. The faulting managed stack from a captured minidump was NativeDiskANNMethods.card ← StorageSession.VectorSetInfo ← RespServerSession.NetworkVINFO.

The lifetime of a Vector Set index across primary, replica, AOF and full sync was then modelled in TLA+ to confirm the mechanism and to choose between candidate fixes. TLC reproduced the production crash as a 6-step counterexample (SyncReset → PrimaryCreate → SyncSnapshot → BeginRead → Deref) and, across seven scenarios, showed that:

• sanitizing on the stream-in path (this PR) is sufficient on its own;
• validating the pointer at the call site is also sufficient, but strictly more invasive;
• quiescing readers around the reset is not sufficient — the handle was never valid in this address space, so this is not a locking race. The per-key vectorSetLocks are already correct.

The three tests above are the executable form of those counterexamples.

[tiagonapoli]

reduce verbosity - too many comments
no need to say that was a regression test

[tiagonapoli]

Add RangeIndex management for this as well

[tiagonapoli]

Split into three stacked PRs, one per defect, each with its own tests: #1995 (diskless stream-in), #1996 (recovery re-entrancy), #1997 (AOF replay channel).