From 2af3d2ae3d44175c0f1ab75b0360248314c45c93 Mon Sep 17 00:00:00 2001 From: Badrish Chandramouli Date: Fri, 14 Aug 2026 15:47:43 -0700 Subject: [PATCH 01/27] Fix root causes behind recurring Garnet .NET CI flakes Triaged the last 60 "Garnet .NET CI" runs on main (39 failed, 68 failed-job logs) and bucketed every failure by root cause. CI runners have 2 cores, so each failure was reproduced locally under `taskset -c 0-1`; that is why these do not reproduce on high-core dev machines. Every change below addresses the underlying defect rather than relaxing a timeout, adding a retry, or weakening an assertion. Product fixes ------------- Vector Set contexts are allocated independently per node and index creation is not AOF-logged, so on slot migration the destination reserved a context from its own free list and stamped it into the AOF migrate records. Replay adopted that context verbatim, guarding only on IsMigrating and never on IsInUse. ContextMetadata.MarkInUse only asserted, which is compiled out in Release - the configuration the failing CI job runs - so two Vector Sets silently shared one namespace and destroyed each other's data. Migrated records are now remapped onto a context that is free locally, and the mapping is held for the duration of the migration. Element records are provably transmitted, applied, and acknowledged before the index key is sent (Task.WhenAll barrier in MigrateSessionSlots.CreateAndRunMigrateTasksAsync), so the mapping can be released when the index key arrives. A/B over 10 iterations: 12 context collisions, 748 records destroyed and 5 exceptions before, none after. A Vector Set's index record is written before the context metadata that reserves its context, and that metadata is only flushed once index creation succeeds. A checkpoint taken between the two captures a live index record whose context is not marked in use, so the free list hands the same context to the next Vector Set created and the two silently share a namespace - the same corruption class as the migration defect above. ReconcileRecoveredState now restores the reservation for every recovered index whose context is free. The hash slot the reservation needs is computed in RecoveredVectorSetIndexKey, which widens recoveredIndexes from byte to ushort to carry it. VectorSetRecoveredContextReservationTests covers this: with the fix the new Vector Set gets its own context, and without it recovery reports the context as free and the next VADD is handed the identical context, failing the test. The migration remap above is in-memory state keyed on contextMetadatas, and two paths rebuild that array underneath it: FLUSHDB/FLUSHALL through FlushGuard.Dispose, and recovery through ReconcileRecoveredState. Neither cleared the remap, so a cached entry could steer the remaining records of a migration into a context that had since been freed and handed to another Vector Set. Both paths now clear it. Recovery already treats a context still marked migrating as a failed migration and marks it for cleanup, so forcing the retried migration to re-resolve is the intended behaviour rather than only the safe one. Test fixes ---------- A thread parked in ExceptionInjectionHelper.ResetAndWaitAsync is only released by EnableException, but the cleanup a test runs on its way out is DisableException. Any test that leaves between a waiter arriving and being re-enabled - an assertion failing, or a wait for the arrival timing out on a slow 2-core runner - therefore strands that waiter permanently. The waiter is a server thread holding a pooled network buffer, so LimitedFixedBufferPool.Dispose spins forever on a reference that is never returned and the entire test process hangs: the job produces no results at all rather than one failing test, which is why these runs show up in CI as a bare timeout with no reportable failure. The cluster call sites already bound this with WaitAsync(timeout, token); the four Vector Set call sites use the unbounded synchronous ResetAndWait. GarnetServer now suspends parking for the duration of InternalDispose, which releases anyone already parked and stops anyone new from parking. Covering arrivals matters because disposal closes listeners before it drains handlers, so a request already in flight can reach a still-armed injection point after shutdown has begun and strand itself there; releasing only the waiters that were already parked leaves the identical hang one moment later. The suspension is a count rather than a flag so concurrent shutdowns are independent, and it is unwound in a finally so it cannot leak into a later test sharing this process-wide state - a leaked suspension would silently stop every subsequent injection point from pausing anything. No timeout is introduced. Both the helper methods and their call site are [Conditional("DEBUG")] and compile away in Release. ExceptionInjectionShutdownTests covers all three properties, and each fails when the corresponding behaviour is removed: disposal completes while a waiter is parked, a caller arriving after shutdown began does not park, and parking still pauses normally once the shutdown that suspended it has finished. StackExchange.Redis 2.12.8 pools SimpleResultBox in a [ThreadStatic] field and sets the exception outside the lock that ActivateContinuations pulses. When a server error faults the box before the caller reaches Monitor.Wait, the caller skips the wait and recycles the box while the reader thread still has a PulseAll outstanding. The next synchronous call on that thread consumes the stale pulse and returns with neither result nor exception, shifting every subsequent reply on that thread by one - a permanent lag that survives new multiplexers and new servers, because the box is thread-static and NUnit runs a fixture on one thread. The CI logs show this exactly: LuaScriptTests.Issue1079 loses its exception and Issue1235 later receives Issue1079's error, and ScriptLoadErrors pairs with Struct the same way, with counts matching 1:1. RespVectorSetTests.VADDErrors fails identically. Adds TestUtils.ThrowsRedisException, which runs the failing call on a dedicated thread so the poisoned box never lands on the shared test thread, and applies it to the two fixtures CI proves are affected. Garnet's wire output was confirmed byte-exact with a command/reply balance counter, so this corrects the client-side defect without hiding any server behaviour. DatabaseManagerBase.TakeCheckpointAsync runs compaction inline, and with the compactionMaxSegments: 1 used by VectorSetOverwriteTests the computed compactLength is zero, so untilAddress equals readOnlyAddress and every SAVE compacts the entire log. SETAsync was the only test in the fixture invoking the ~120 MB helper twice against one server; splitting the KEEPTTL case into its own test takes it from 13 s to 2 s. The 3-minute AsyncTimeout added previously is left in place as defence in depth. DisposeCluster disposed nodes in a bare loop, so one node throwing or stalling left the rest of the cluster alive and holding its ports; the next four tests then failed with "Failed to connect within 30 seconds" even though the test that actually broke had passed. Every node's dispose is now started before any of them is waited on, and each is then waited on under a bounded share of one budget. Starting them inside the wait loop would leave a single node that exhausts the budget with every later node still undisposed, which is the same abandoned port by another route. Each dispose gets a dedicated thread rather than a pool thread, because on a two-core runner the pool injects threads slowly enough that a queued dispose could otherwise sit unstarted behind the ones already blocked. Task.Wait(TimeSpan) rethrows a faulted dispose as an AggregateException rather than returning, so it is caught per node; without that the throw escaped mid-loop and abandoned every remaining node, which is the cascade this loop exists to prevent. A standalone probe confirmed it: 1 of 3 healthy nodes disposed before, 3 of 3 after. The wait is bounded only by the shared budget, with no per-node cap, so a merely slow node cannot fail a teardown that the pre-existing outer guard would have allowed. CLUSTER MEET returns before gossip propagates, and MIGRATE and SETSLOT are issued against the source node, so the source must know the target or it replies "ERR Unknown endpoint". Adds the missing convergence waits in ClusterMigrateTests, including one that waited in the wrong direction and one that had no wait at all. Slot re-assignment likewise settles asynchronously after CLUSTER FAILOVER FORCE, so ClusterDivergentReplicasTest now waits for the new primary to observe its own slot ownership before writing to it. Verification ------------ All runs pinned to 2 cores with `taskset -c 0-1`, Debug, net8.0: Garnet.test.vectorset 400/400 Garnet.test.scripting 619 passed, 30 skipped, 0 failed Garnet.test.cluster.vectorsets 84/84 Garnet.test.cluster.repl.vectorsets 7/7 Garnet.test.cluster 156/156 Garnet.test.cluster.replication 107/107 Garnet.test.cluster.migrate 56/56 Build is clean with 0 warnings, and `dotnet format` reports no changes for both Garnet.slnx and Tsavorite.slnx. Superseded upstream during rebase --------------------------------- Fixes from the original version of this branch were landed independently on main and have been dropped in favour of the upstream versions: - The missing VADDSetFlagsArg branch in HandleVectorSetAddReplication, and the drain that has to precede it, are both in #2080. That version is a superset: it also reads the index back and fails loudly rather than silently applying flags to a Vector Set that is not there. - UpgradeReplicasAsync racing a promoted primary that is still recovering is fixed in main with WaitForFailoverCompleted, which polls the failover state machine directly instead of inferring readiness from slot coverage. The WaitForAllSlotsServedAsync helper written for that call site is dropped with it rather than left unused. Deliberately unchanged ---------------------- ClusterSRAddReplicaAfterPrimaryCheckpoint and RespMemoryWriterOverflowTests.SortedSetAsync did not reproduce (3/3 and 6/6 locally), so no speculative change was made rather than risk masking a cause that has not been identified. Vector Set cleanup reclaims a whole context but decides a context is abandoned from the single key QueueCleanups happened to observe during compaction, so a context that lives on under a different key - after RENAME, or after being handed to the next Vector Set created - can be marked while live. Fixing that requires an allocation generation on the context so a stale request can be recognised, which is a design change to context ownership rather than a CI issue, and is being handled separately. Compensating guards were prototyped here and removed: they cannot cover the window in which a context is reserved before its index record is written, and detecting the damage after the fact only converts silent data loss into a different symptom. ClusterVectorSetTests.VectorSetMigrateManyBySlot: poll on any failed read ------------------------------------------------------------------------- This test failed three times across recent CI runs (windows net8.0 Debug and windows net10.0 Debug) with "Expected: 3, But was: 1". ClusterTestUtils.Execute catches every exception and returns the message as a single-element reply, so a failed read is indistinguishable from data by type. The retry loop that reads the migrated keys back from the new replica treated a reply as transient only when its text began with "Key has MOVED to ", and hit a hard ClassicAssert.AreEqual(3, length) for anything else - on the very first poll, before the replica had converged. Probing that window directly (reading the new replica immediately after MigrateSlots) shows three distinct transient replies, only the first of which the old guard recognised: "Key has MOVED to Endpoint ... but CommandFlags.NoRedirect was specified" "CLUSTERDOWN Hash slot not served" a nil reply, which casts to a null array and throws NullReferenceException Replaying the original guard in that window fails 2/2 tests with exactly the CI error; the new guard passes 5 runs of 2/2. A successful VSIM ... WITHSCORES WITHATTRIBS hit is always three elements, so a null or single-element reply always means the read did not land. The loop now treats all of them as "poll again", captures the last response for the failure message, and no longer asserts inside the loop - a caught NUnit assertion is still recorded against the test result even when the next poll would resolve it, which the single-slot variant of this test already documents. The element/score/attribute assertions are unchanged; they now run once after the loop, so a genuinely wrong reply still fails with the full NUnit diff, and a persistent error still fails with the response text instead of a bare length mismatch. The polling window matches the 15s the single-slot variant already uses for the same convergence event, and the previously absent sleep between polls stops the loop from spinning on the two cores it shares with the nodes it is waiting for. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 30f31d5b-9e53-4111-91a3-e16c94510a77 --- .../Testing/ExceptionInjectionHelper.cs | 56 +++- libs/host/GarnetServer.cs | 40 ++- .../Vector/VectorManager.ContextMetadata.cs | 4 + .../Resp/Vector/VectorManager.Replication.cs | 145 ++++++++--- libs/server/Resp/Vector/VectorManager.cs | 26 +- .../ClusterMigrateTests.cs | 11 + .../ClusterReplicationBaseTests.cs | 6 +- .../VectorSets/ClusterVectorSetTests.cs | 79 ++++-- .../Garnet.test.cluster/ClusterTestContext.cs | 98 ++++++- .../Garnet.test.scripting/LuaScriptTests.cs | 246 +++++++++--------- .../ExceptionInjectionShutdownTests.cs | 152 +++++++++++ .../RespVectorSetTests.cs | 110 ++++---- .../VectorSetOverwriteTests.cs | 15 +- ...ctorSetRecoveredContextReservationTests.cs | 137 ++++++++++ test/standalone/Garnet.test/TestUtils.cs | 58 +++++ 15 files changed, 915 insertions(+), 268 deletions(-) create mode 100644 test/standalone/Garnet.test.vectorset/ExceptionInjectionShutdownTests.cs create mode 100644 test/standalone/Garnet.test.vectorset/VectorSetRecoveredContextReservationTests.cs diff --git a/libs/common/Testing/ExceptionInjectionHelper.cs b/libs/common/Testing/ExceptionInjectionHelper.cs index 7498934a7bc..94a0affeba7 100644 --- a/libs/common/Testing/ExceptionInjectionHelper.cs +++ b/libs/common/Testing/ExceptionInjectionHelper.cs @@ -17,6 +17,12 @@ public static class ExceptionInjectionHelper static object @lock = new(); static TaskCompletionSource update = new(TaskCreationOptions.RunContinuationsAsynchronously); + /// + /// Non-zero while at least one scope is open. While it is non-zero + /// refuses to park, because nothing is left to signal it. + /// + static int parkingSuspensions; + /// /// Array of exception injection types /// @@ -128,6 +134,51 @@ public static bool TriggerCondition(ExceptionInjectionType exceptionType) #endif } + /// + /// Stops from parking, and releases anyone already parked. + /// + /// A parked waiter is only released by , but the cleanup a test runs on + /// its way out is . A test that leaves between a waiter arriving and + /// being re-enabled - an assertion failing, or a wait for the arrival timing out - therefore strands + /// that waiter permanently. The waiter is a server thread holding a pooled network buffer, so + /// LimitedFixedBufferPool.Dispose then spins forever waiting for a reference that is never + /// returned and the whole test process hangs rather than one test failing. + /// + /// Suspension covers arrivals as well as waiters already parked, so a request that reaches an + /// injection point after its owner has begun shutting down cannot re-create the same hang. + /// + /// Every call must be paired with . The count keeps concurrent shutdowns + /// independent, and ending every scope is what stops a suspension leaking into a later test sharing + /// this process-wide state. + /// + [Conditional("DEBUG")] + public static void SuspendParking() + { + TaskCompletionSource release; + + lock (@lock) + { + parkingSuspensions++; + release = update; + update = new(TaskCreationOptions.RunContinuationsAsynchronously); + } + + _ = release.TrySetResult(true); + } + + /// + /// Ends a scope. Parking resumes once every scope has ended. + /// + [Conditional("DEBUG")] + public static void ResumeParking() + { + lock (@lock) + { + Debug.Assert(parkingSuspensions > 0, "ResumeParking without a matching SuspendParking"); + parkingSuspensions--; + } + } + /// /// Wait on set condition /// @@ -149,7 +200,10 @@ public static async Task ResetAndWaitAsync(ExceptionInjectionType exceptionType) Task task; lock (@lock) { - if (IsEnabled(exceptionType)) + // Parking is suspended while a server is shutting down, because whoever armed this + // injection point is gone and will never re-enable it. Reading it under the lock is + // what makes a suspension raised at any point before here take effect. + if (IsEnabled(exceptionType) || parkingSuspensions > 0) break; task = update.Task; } diff --git a/libs/host/GarnetServer.cs b/libs/host/GarnetServer.cs index 8b872b40c73..b4fe319d3a1 100644 --- a/libs/host/GarnetServer.cs +++ b/libs/host/GarnetServer.cs @@ -564,23 +564,35 @@ public void Dispose(bool deleteDir = true) private void InternalDispose() { - // Phase 1: Stop listening on all servers to free ports immediately. - for (var i = 0; i < servers.Length; i++) - servers[i]?.Close(); + // A thread parked in a test injection point holds a pooled network buffer, and the drain below + // waits for every such buffer to come back. Suspending covers both waiters already parked and + // requests that reach an injection point while this runs, so shutdown cannot be blocked by an + // injection point whose owner is already gone. Compiled out in Release. + ExceptionInjectionHelper.SuspendParking(); + try + { + // Phase 1: Stop listening on all servers to free ports immediately. + for (var i = 0; i < servers.Length; i++) + servers[i]?.Close(); - // Phase 2: Drain active handlers and clean up remaining resources. - for (var i = 0; i < servers.Length; i++) - servers[i]?.Dispose(); + // Phase 2: Drain active handlers and clean up remaining resources. + for (var i = 0; i < servers.Length; i++) + servers[i]?.Dispose(); - // Phase 3: Dispose the provider (storage engine shutdown — may take time). - Provider?.Dispose(); + // Phase 3: Dispose the provider (storage engine shutdown — may take time). + Provider?.Dispose(); - subscribeBroker?.Dispose(); - storeEpoch?.Dispose(); - pubSubEpoch?.Dispose(); - opts.AuthSettings?.Dispose(); - if (disposeLoggerFactory) - loggerFactory?.Dispose(); + subscribeBroker?.Dispose(); + storeEpoch?.Dispose(); + pubSubEpoch?.Dispose(); + opts.AuthSettings?.Dispose(); + if (disposeLoggerFactory) + loggerFactory?.Dispose(); + } + finally + { + ExceptionInjectionHelper.ResumeParking(); + } } private static void DeleteDirectory(string path) diff --git a/libs/server/Resp/Vector/VectorManager.ContextMetadata.cs b/libs/server/Resp/Vector/VectorManager.ContextMetadata.cs index 91f85f88d32..af33eaeda27 100644 --- a/libs/server/Resp/Vector/VectorManager.ContextMetadata.cs +++ b/libs/server/Resp/Vector/VectorManager.ContextMetadata.cs @@ -448,6 +448,10 @@ public readonly void Dispose() manager.recoveredIndexes.Clear(); manager.recoveredMetadata.Clear(); + // Migration remappings name contexts in the array just replaced, so a surviving entry would + // steer the rest of that migration into a context this flush has already handed back + manager.ClearMigratedContextRemap(); + // Allow Vector Set operations again manager.vectorSetLocks.ReleaseLock(lockToken); diff --git a/libs/server/Resp/Vector/VectorManager.Replication.cs b/libs/server/Resp/Vector/VectorManager.Replication.cs index afcd435f146..7ceb513e83e 100644 --- a/libs/server/Resp/Vector/VectorManager.Replication.cs +++ b/libs/server/Resp/Vector/VectorManager.Replication.cs @@ -4,6 +4,7 @@ using System; using System.Buffers; using System.Buffers.Binary; +using System.Collections.Generic; using System.Diagnostics; using System.Linq; using System.Runtime.InteropServices; @@ -165,6 +166,87 @@ internal void ReplicateVectorSetSetAttribute(ReadOnlySpan key, ReadOnlySpa } } + /// + /// Maps a context chosen by a PRIMARY for an in-flight migration onto the context this node actually + /// stores that migrated Vector Set under. + /// + /// Guarded by lock (this), like the rest of the context metadata. + /// + private readonly Dictionary migratedContextRemap = new(); + + /// + /// Discard every in-flight migration remapping. + /// + /// Callers must already hold lock (this), which is what guards the map. + /// + private void ClearMigratedContextRemap() + { + Debug.Assert(Monitor.IsEntered(this), "Migration remap is guarded by lock (this)"); + + migratedContextRemap.Clear(); + } + + /// + /// Determine which context migrated data should be written to on this node. + /// + /// Contexts are assigned independently by each node, so a context that is free on the PRIMARY which chose it + /// can still be occupied here - most commonly by a deleted Vector Set whose element data has not finished being + /// cleaned up. Storing the migrated data there anyway would leave two Vector Sets sharing a context, corrupting + /// both, so in that case the migration is steered onto a context that is free locally. + /// + /// The mapping is remembered so that every record belonging to a migration resolves identically, and is + /// discarded once the index key for that migration arrives. + /// + private ulong ResolveMigratedContext(StorageSession currentSession, ulong migratedContext) + { + ulong localContext; + + lock (this) + { + if (migratedContextRemap.TryGetValue(migratedContext, out localContext)) + { + return localContext; + } + + var (contextIndex, contextValue) = ContextMetadata.DecomposeContext(migratedContext); + + // A context past the end of the local metadata has never been allocated on this node, + // so it cannot be adopted as-is + var contextKnownLocally = contextIndex < contextMetadatas.Length; + + if (contextKnownLocally && contextMetadatas[contextIndex].IsMigrating(contextIndex != 0, contextValue)) + { + // Already reserved for a migration, which is how a migration that was interrupted and resumed looks + migratedContextRemap[migratedContext] = migratedContext; + + return migratedContext; + } + + if (!contextKnownLocally || contextMetadatas[contextIndex].IsInUse(contextIndex != 0, contextValue)) + { + localContext = NextVectorSetContext(ushort.MaxValue); + + (contextIndex, contextValue) = ContextMetadata.DecomposeContext(localContext); + } + else + { + localContext = migratedContext; + + contextMetadatas[contextIndex].MarkInUse(contextIndex != 0, contextValue, ushort.MaxValue); + } + + contextMetadatas[contextIndex].MarkMigrating(contextIndex != 0, contextValue); + + migratedContextRemap[migratedContext] = localContext; + + _ = dirtyContextMetadatas.Add(contextIndex); + } + + UpdateContextMetadata(ref currentSession.vectorBasicContext); + + return localContext; + } + /// /// Vector Set adds are phrased as reads (once the index is created), so they require special handling. /// @@ -194,28 +276,21 @@ internal void HandleVectorSetAddReplication( ulong ns = BinaryPrimitives.ReadUInt32LittleEndian(elementNsBytes); // REPLICAs wouldn't have seen a reservation message, so allocate this on demand - var (contextIndex, contextValue) = ContextMetadata.DecomposeContext(ns & ~(ContextStep - 1)); - - var needsUpdate = false; - lock (this) - { - if (!contextMetadatas[contextIndex].IsMigrating(contextIndex != 0, contextValue)) - { - contextMetadatas[contextIndex].MarkInUse(contextIndex != 0, contextValue, ushort.MaxValue); - contextMetadatas[contextIndex].MarkMigrating(contextIndex != 0, contextValue); + var migratedContext = ns & ~(ContextStep - 1); + var localContext = ResolveMigratedContext(currentSession, migratedContext); - _ = dirtyContextMetadatas.Add(contextIndex); + scoped var localNamespaceBytes = elementNsBytes; - needsUpdate = true; - } - } - - if (needsUpdate) + Span remappedNamespaceBytes = stackalloc byte[sizeof(uint)]; + if (localContext != migratedContext) { - UpdateContextMetadata(ref currentSession.vectorBasicContext); + // Preserve the sub-namespace within the block, only the block itself moves + BinaryPrimitives.WriteUInt32LittleEndian(remappedNamespaceBytes, (uint)(localContext + (ns - migratedContext))); + + localNamespaceBytes = remappedNamespaceBytes; } - HandleMigratedElementKey(ref currentSession.stringBasicContext, ref currentSession.vectorBasicContext, elementNsBytes, elementKeyBytes, value); + HandleMigratedElementKey(ref currentSession.stringBasicContext, ref currentSession.vectorBasicContext, localNamespaceBytes, elementKeyBytes, value); return; } else if (input.arg1 == MigrateIndexKeyLogArg) @@ -230,36 +305,38 @@ internal void HandleVectorSetAddReplication( // but if you a migrate an EMPTY Vector Set that is not necessarily true // // So force reservation now - var (contextIndex, contextValue) = ContextMetadata.DecomposeContext(context & ~(ContextStep - 1)); - - var needsUpdate = false; - lock (this) - { - if (!contextMetadatas[contextIndex].IsMigrating(contextIndex != 0, contextValue)) - { - contextMetadatas[contextIndex].MarkInUse(contextIndex != 0, contextValue, ushort.MaxValue); - contextMetadatas[contextIndex].MarkMigrating(contextIndex != 0, contextValue); + var migratedContext = context & ~(ContextStep - 1); + var localContext = ResolveMigratedContext(currentSession, migratedContext); - _ = dirtyContextMetadatas.Add(contextIndex); + scoped var localIndexValue = value.ReadOnlySpan; - needsUpdate = true; - } - } - - if (needsUpdate) + Span remappedIndexValue = stackalloc byte[Index.Size]; + if (localContext != migratedContext) { - UpdateContextMetadata(ref currentSession.vectorBasicContext); + // The index records its own context, so it has to be rewritten to match where the data landed + localIndexValue.CopyTo(remappedIndexValue); + SetContextForMigration(remappedIndexValue, localContext + (context - migratedContext)); + + localIndexValue = remappedIndexValue; } ActiveThreadSession = currentSession; try { - HandleMigratedIndexKey(null, null, indexKey, value); + HandleMigratedIndexKey(null, null, indexKey, localIndexValue); } finally { ActiveThreadSession = null; } + + // Element records for a migration are all transmitted, applied, and acknowledged before its index + // key is sent, so no further record can name this context until a later migration reserves it + lock (this) + { + _ = migratedContextRemap.Remove(migratedContext); + } + return; } else if (input.arg1 == VectorManager.VADDSetFlagsArg) diff --git a/libs/server/Resp/Vector/VectorManager.cs b/libs/server/Resp/Vector/VectorManager.cs index d38fda784f3..095e0087759 100644 --- a/libs/server/Resp/Vector/VectorManager.cs +++ b/libs/server/Resp/Vector/VectorManager.cs @@ -172,7 +172,7 @@ private static void EnsureFilterBitmapSize(ref SpanByteAndMemory buffer, int res private readonly int dbId; - private ConcurrentDictionary recoveredIndexes; + private ConcurrentDictionary recoveredIndexes; private ConcurrentDictionary recoveredMetadata; public VectorManager(int dbId, GarnetServerOptions serverOptions, Func getTempSession, ILoggerFactory loggerFactory) @@ -308,6 +308,11 @@ public void ReconcileRecoveredState(bool requireNoReservedContexts = false) recoveredMetadata.Clear(); + // Rebuilding contextMetadatas invalidates any migration remapping built against the old array. + // An interrupted migration is treated as failed below - its context is marked for cleanup - so + // a surviving entry would steer the retried migration into a context being torn down. + ClearMigratedContextRemap(); + // If we come up and contexts are marked for migration, that means the migration FAILED // and we'd like those contexts back ASAP for (var i = 0; i < contextMetadatas.Length; i++) @@ -329,10 +334,22 @@ public void ReconcileRecoveredState(bool requireNoReservedContexts = false) } // Any non-deleted records we recovered for contexts being deleted, we need to undo that - foreach (var (context, _) in recoveredIndexes) + foreach (var (context, hashSlot) in recoveredIndexes) { var (contextIndex, contextValue) = ContextMetadata.DecomposeContext(context); + // The index record is written before the context metadata that reserves its context, so a + // recovery boundary between the two leaves a live index record pointing at a free context. + // Reserving it here keeps the context from being handed to a different Vector Set. + if (!contextMetadatas[contextIndex].IsInUse(contextIndex != 0, contextValue)) + { + contextMetadatas[contextIndex].MarkInUse(contextIndex != 0, contextValue, hashSlot); + + _ = dirtyContextMetadatas.Add(contextIndex); + + needsUpdated = true; + } + if (contextMetadatas[contextIndex].IsCleaningUp(contextIndex != 0, contextValue)) { contextMetadatas[contextIndex].ClearIsCleaningUp(contextIndex != 0, contextValue); @@ -399,7 +416,10 @@ public void RecoveredVectorSetIndexKey(ref TSourceLogRecord re } ReadIndex(record.ValueSpan, out var context, out _, out _, out _, out _, out _, out _, out _, out _); - recoveredIndexes[context] = 0; + + // The hash slot is needed to restore the context reservation in ReconcileRecoveredState, which + // has only this map to work from - the record itself is not retained past this call + recoveredIndexes[context] = HashSlotUtils.HashSlot(record.Key); } /// diff --git a/test/cluster/Garnet.test.cluster.migrate/ClusterMigrateTests.cs b/test/cluster/Garnet.test.cluster.migrate/ClusterMigrateTests.cs index 1c3f53e2c54..a8f70b00f63 100644 --- a/test/cluster/Garnet.test.cluster.migrate/ClusterMigrateTests.cs +++ b/test/cluster/Garnet.test.cluster.migrate/ClusterMigrateTests.cs @@ -1368,6 +1368,8 @@ public async Task ClusterSimpleMigrateContinuousReadWrite(CancellationToken canc context.clusterTestUtils.SetConfigEpoch(dstNodeIndex, dstNodeIndex + 2, logger: context.logger); context.clusterTestUtils.Meet(srcNodeIndex, dstNodeIndex, logger: context.logger); context.clusterTestUtils.WaitUntilNodeIsKnown(dstNodeIndex, srcNodeIndex, logger: context.logger); + // MIGRATE is issued against the source, so the source has to know the target endpoint + context.clusterTestUtils.WaitUntilNodeIsKnown(srcNodeIndex, dstNodeIndex, logger: context.logger); var migrateSlots = new List { 0, 10 }; // Start operations @@ -1748,6 +1750,9 @@ public void ClusterMigrateDataSlotsRange() context.clusterTestUtils.SetConfigEpoch(srcNodeIndex, srcNodeIndex + 1, logger: context.logger); context.clusterTestUtils.SetConfigEpoch(dstNodeIndex, dstNodeIndex + 2, logger: context.logger); context.clusterTestUtils.Meet(srcNodeIndex, dstNodeIndex, logger: context.logger); + context.clusterTestUtils.WaitUntilNodeIsKnown(dstNodeIndex, srcNodeIndex, logger: context.logger); + // MIGRATE is issued against the source, so the source has to know the target endpoint + context.clusterTestUtils.WaitUntilNodeIsKnown(srcNodeIndex, dstNodeIndex, logger: context.logger); var keySize = 16; var keyCount = 1024; @@ -2083,6 +2088,8 @@ public void ClusterMigrateWrite() context.clusterTestUtils.SetConfigEpoch(targetNodeIndex, targetNodeIndex + 1, logger: context.logger); context.clusterTestUtils.Meet(sourceNodeIndex, targetNodeIndex, logger: context.logger); context.clusterTestUtils.WaitUntilNodeIsKnown(targetNodeIndex, sourceNodeIndex, logger: context.logger); + // Slot state transitions are issued against the source, so the source has to know the target node id + context.clusterTestUtils.WaitUntilNodeIsKnown(sourceNodeIndex, targetNodeIndex, logger: context.logger); var sourceNodeId = context.clusterTestUtils.GetNodeIdFromNode(sourceNodeIndex, context.logger); var targetNodeId = context.clusterTestUtils.GetNodeIdFromNode(targetNodeIndex, context.logger); @@ -2169,6 +2176,8 @@ public void ClusterMigrateSetCopyUpdate(CancellationToken cancellationToken) context.clusterTestUtils.SetConfigEpoch(targetNodeIndex, targetNodeIndex + 1, logger: context.logger); context.clusterTestUtils.Meet(sourceNodeIndex, targetNodeIndex, logger: context.logger); context.clusterTestUtils.WaitUntilNodeIsKnown(targetNodeIndex, sourceNodeIndex, logger: context.logger); + // Slot state transitions are issued against the source, so the source has to know the target node id + context.clusterTestUtils.WaitUntilNodeIsKnown(sourceNodeIndex, targetNodeIndex, logger: context.logger); var sourceNodeId = context.clusterTestUtils.GetNodeIdFromNode(sourceNodeIndex, context.logger); var targetNodeId = context.clusterTestUtils.GetNodeIdFromNode(targetNodeIndex, context.logger); @@ -2242,6 +2251,8 @@ public void ClusterMigrateCustomProcDelRMW(CancellationToken cancellationToken) context.clusterTestUtils.SetConfigEpoch(targetNodeIndex, targetNodeIndex + 1, logger: context.logger); context.clusterTestUtils.Meet(sourceNodeIndex, targetNodeIndex, logger: context.logger); context.clusterTestUtils.WaitUntilNodeIsKnown(targetNodeIndex, sourceNodeIndex, logger: context.logger); + // Slot state transitions are issued against the source, so the source has to know the target node id + context.clusterTestUtils.WaitUntilNodeIsKnown(sourceNodeIndex, targetNodeIndex, logger: context.logger); var sourceNodeId = context.clusterTestUtils.GetNodeIdFromNode(sourceNodeIndex, context.logger); var targetNodeId = context.clusterTestUtils.GetNodeIdFromNode(targetNodeIndex, context.logger); diff --git a/test/cluster/Garnet.test.cluster.replication/ReplicationTests/ClusterReplicationBaseTests.cs b/test/cluster/Garnet.test.cluster.replication/ReplicationTests/ClusterReplicationBaseTests.cs index edff700fa70..33a41674a1d 100644 --- a/test/cluster/Garnet.test.cluster.replication/ReplicationTests/ClusterReplicationBaseTests.cs +++ b/test/cluster/Garnet.test.cluster.replication/ReplicationTests/ClusterReplicationBaseTests.cs @@ -1001,6 +1001,11 @@ void ClusterDivergentReplicasTest(bool performRMW, bool disableObjects, bool ckp _ = context.clusterTestUtils.AddDelSlotsRange(newPrimaryIndex, [(0, 16383)], addslot: true, context.logger); context.clusterTestUtils.BumpEpoch(newPrimaryIndex, logger: context.logger); + // Slot re-assignment settles asynchronously, so the new primary has to observe itself as the + // owner before it will serve writes instead of redirecting them + var newPrimaryId = context.clusterTestUtils.ClusterMyId(newPrimaryIndex, context.logger); + context.clusterTestUtils.WaitForSlotOwnership(newPrimaryIndex, newPrimaryId, [0, 16383], context.logger); + // New primary diverges to its own history by new random seed kvpairCount <<= 1; if (disableObjects) @@ -1013,7 +1018,6 @@ void ClusterDivergentReplicasTest(bool performRMW, bool disableObjects, bool ckp if (!ckptBeforeDivergence || multiCheckpointAfterDivergence) context.clusterTestUtils.Checkpoint(newPrimaryIndex, logger: context.logger); - var newPrimaryId = context.clusterTestUtils.ClusterMyId(newPrimaryIndex, context.logger); while (true) { var replicaConfig = context.clusterTestUtils.ClusterNodes(replicaIndex, context.logger); diff --git a/test/cluster/Garnet.test.cluster.vectorsets/VectorSets/ClusterVectorSetTests.cs b/test/cluster/Garnet.test.cluster.vectorsets/VectorSets/ClusterVectorSetTests.cs index 3bf35f61efd..624933e58cc 100644 --- a/test/cluster/Garnet.test.cluster.vectorsets/VectorSets/ClusterVectorSetTests.cs +++ b/test/cluster/Garnet.test.cluster.vectorsets/VectorSets/ClusterVectorSetTests.cs @@ -1337,10 +1337,14 @@ public void VectorSetMigrateManyBySlot() ClassicAssert.IsTrue(exc0.StartsWith("Key has MOVED to ")); } + // Same convergence window the single-slot variant of this test uses: the source replica keeps + // serving the migrated keys for a while, then passes through a window where the slot is not yet + // known-stable from its point of view, before it settles on redirecting to the new primary. var start = Stopwatch.GetTimestamp(); var success = false; - while (Stopwatch.GetElapsedTime(start) < TimeSpan.FromSeconds(5)) + string lastResponse = null; + while (Stopwatch.GetElapsedTime(start) < TimeSpan.FromSeconds(15)) { try { @@ -1348,29 +1352,30 @@ public void VectorSetMigrateManyBySlot() foreach (var (key, _, _, data, _) in primary0Keys.Concat(primary1Keys)) { var exc1 = (string)context.clusterTestUtils.Execute(secondary0, "VSIM", [key, "XB8", data, "WITHSCORES", "WITHATTRIBS"], flags: CommandFlags.NoRedirect); - if (!exc1.StartsWith("Key has MOVED to ")) + lastResponse = exc1; + if (exc1 is null || !exc1.StartsWith("Key has MOVED to ")) { migrationNotFinished = true; break; } } - if (migrationNotFinished) + if (!migrationNotFinished) { - continue; + success = true; + break; } - - success = true; - break; } - catch + catch (Exception ex) { - // Secondary can still have the key for a bit - Thread.Sleep(100); + // Secondary can still have the key for a bit (VSIM returns vector data, not a string) + lastResponse = ex.Message; } + + Thread.Sleep(100); } - ClassicAssert.IsTrue(success, "Original replica still has Vector Set long after primary has completed"); + ClassicAssert.IsTrue(success, $"Original replica still has Vector Set long after primary has completed; last response was '{lastResponse}'"); // Check available on new secondary var readonlyOnReplica1 = (string)context.clusterTestUtils.Execute(secondary1, "READONLY", [], flags: CommandFlags.NoRedirect); @@ -1379,37 +1384,67 @@ public void VectorSetMigrateManyBySlot() start = Stopwatch.GetTimestamp(); success = false; + lastResponse = null; - while (Stopwatch.GetElapsedTime(start) < TimeSpan.FromSeconds(5)) + // Poll until every migrated key is readable on the new replica, and only then assert on the + // contents. ClusterTestUtils.Execute turns any exception into a single-element reply holding the + // message, and a VSIM ... WITHSCORES WITHATTRIBS hit is always three elements, so a single element + // always means the read did not land: MOVED while this replica's view of slot ownership catches up, + // CLUSTERDOWN, or an SE.Redis timeout. Asserting inside the loop would record a failure on the test + // result even for a poll that the very next one resolves. + List<(string Key, byte[] Data, byte[][] Reply)> replicated = []; + + while (Stopwatch.GetElapsedTime(start) < TimeSpan.FromSeconds(15)) { success = true; + replicated.Clear(); foreach (var (key, _, _, data, _) in primary0Keys.Concat(primary1Keys)) { - var migrateSimRes = (byte[][])context.clusterTestUtils.Execute(secondary1, "VSIM", [key, "XB8", data, "WITHSCORES", "WITHATTRIBS"], flags: CommandFlags.NoRedirect); - - if (migrateSimRes.Length == 1 && Encoding.UTF8.GetString(migrateSimRes[0]).StartsWith("Key has MOVED to ")) + byte[][] migrateSimRes; + try { + migrateSimRes = (byte[][])context.clusterTestUtils.Execute(secondary1, "VSIM", [key, "XB8", data, "WITHSCORES", "WITHATTRIBS"], flags: CommandFlags.NoRedirect); + } + catch (Exception ex) + { + lastResponse = ex.Message; success = false; break; } - ClassicAssert.AreEqual(3, migrateSimRes.Length); - - var (elem, attr, score) = expected[(key, data)]; + // A nil reply casts to a null array, so null must be handled alongside the + // single-element error reply. + if (migrateSimRes is null || migrateSimRes.Length == 1) + { + lastResponse = migrateSimRes is null ? "" : Encoding.UTF8.GetString(migrateSimRes[0]); + success = false; + break; + } - ClassicAssert.IsTrue(elem.SequenceEqual(migrateSimRes[0])); - ClassicAssert.AreEqual(score, float.Parse(Encoding.ASCII.GetString(migrateSimRes[1]))); - ClassicAssert.IsTrue(attr.SequenceEqual(migrateSimRes[2])); + replicated.Add((key, data, migrateSimRes)); } if (success) { break; } + + Thread.Sleep(100); } - ClassicAssert.IsTrue(success, "New replica hasn't replicated Vector Set long after primary has received data"); + ClassicAssert.IsTrue(success, $"New replica hasn't replicated Vector Set long after primary has received data; last response was '{lastResponse}'"); + + foreach (var (key, data, migrateSimRes) in replicated) + { + ClassicAssert.AreEqual(3, migrateSimRes.Length, $"Unexpected VSIM reply length for key '{key}'"); + + var (elem, attr, score) = expected[(key, data)]; + + ClassicAssert.IsTrue(elem.SequenceEqual(migrateSimRes[0])); + ClassicAssert.AreEqual(score, float.Parse(Encoding.ASCII.GetString(migrateSimRes[1]))); + ClassicAssert.IsTrue(attr.SequenceEqual(migrateSimRes[2])); + } } [Test] diff --git a/test/cluster/Garnet.test.cluster/ClusterTestContext.cs b/test/cluster/Garnet.test.cluster/ClusterTestContext.cs index 6e145252533..d5e3e90da60 100644 --- a/test/cluster/Garnet.test.cluster/ClusterTestContext.cs +++ b/test/cluster/Garnet.test.cluster/ClusterTestContext.cs @@ -3,6 +3,7 @@ using System; using System.Collections.Generic; +using System.Diagnostics; using System.IO; using System.Linq; using System.Net; @@ -65,6 +66,14 @@ public class ClusterTestContext public CancellationTokenSource cts; + /// + /// Total time budget shared across all nodes in . Kept below the + /// teardown timeout so stalled nodes are reported precisely instead of tripping the outer guard. + /// No per-node cap is applied on top of this: a single slow node is only a problem once it + /// threatens the budget the remaining nodes still need to release their ports. + /// + private static readonly TimeSpan DisposeClusterBudget = TimeSpan.FromSeconds(45); + public void EnableGarnetLoggingEvents(GarnetTestLoggingEventType[] events) { foreach (var e in events) @@ -539,20 +548,89 @@ public GarnetServer CreateInstance( /// public void DisposeCluster() { - if (nodes != null) + if (nodes == null) + return; + + // Every node's dispose is started before any of them is waited on, and each is then waited on + // under a bounded share of one budget. GarnetServer.Dispose closes its listeners before draining + // handlers, so a node that stalls late in its own dispose cannot keep the others bound to their + // ports. Starting them inside the wait loop instead would mean a single node exhausting the + // budget left every later node undisposed, and aborting the loop on the first failure would do + // the same - either way the abandoned ports cascade into startup failures for subsequent tests + // in the same process. + var stalledNodes = new List(); + var disposeFailures = new List(); + var disposeTasks = new Task[nodes.Length]; + + for (var i = 0; i < nodes.Length; i++) + { + var node = nodes[i]; + if (node == null) + continue; + + nodes[i] = null; + logger.LogDebug("\t a. Before dispose node {i}{testName}", i, TestContext.CurrentContext.Test.Name); + + // Dispose blocks, so it gets a dedicated thread rather than a pool thread: on a two-core + // runner the pool injects threads slowly enough that a queued dispose could sit unstarted + // behind the ones already blocked, which is the same abandoned port by another route. + disposeTasks[i] = Task.Factory.StartNew(() => node.Dispose(true), TaskCreationOptions.LongRunning); + } + + var elapsed = Stopwatch.StartNew(); + + for (var i = 0; i < disposeTasks.Length; i++) { - for (var i = 0; i < nodes.Length; i++) + var disposeTask = disposeTasks[i]; + if (disposeTask == null) + continue; + + var wait = DisposeClusterBudget - elapsed.Elapsed; + + // Task.Wait rethrows a faulted dispose as an AggregateException. Letting it propagate + // would abandon every later node, which is the cascade this loop exists to prevent, so + // the fault is caught here and reported from disposeTask.Exception below. + bool completed; + try { - if (nodes[i] != null) - { - logger.LogDebug("\t a. Before dispose node {i}{testName}", i, TestContext.CurrentContext.Test.Name); - var node = nodes[i]; - nodes[i] = null; - node.Dispose(true); - logger.LogDebug("\t b. After dispose node {i}{testName}", i, TestContext.CurrentContext.Test.Name); - } + completed = wait > TimeSpan.Zero && disposeTask.Wait(wait); } + catch (AggregateException) + { + completed = true; + } + + // The dispose is already running on its own thread, so a node that outlives the budget + // still goes on to free its port; it is only the waiting that is given up here. + if (!completed) + { + stalledNodes.Add(i); + logger.LogError("\t !. Dispose stalled for node {i}{testName}", i, TestContext.CurrentContext.Test.Name); + continue; + } + + if (disposeTask.Exception is { } disposeException) + { + // Unwrapped so the reported failure names the original fault rather than the + // AggregateException the task wraps it in. + disposeFailures.Add(disposeException.InnerExceptions.Count == 1 + ? disposeException.InnerExceptions[0] + : disposeException); + logger.LogError(disposeException, "\t !. Dispose failed for node {i}{testName}", i, TestContext.CurrentContext.Test.Name); + continue; + } + + logger.LogDebug("\t b. After dispose node {i}{testName}", i, TestContext.CurrentContext.Test.Name); } + + if (stalledNodes.Count > 0) + disposeFailures.Add(new TimeoutException($"Dispose stalled for node(s): {string.Join(", ", stalledNodes)}")); + + if (disposeFailures.Count == 1) + throw disposeFailures[0]; + + if (disposeFailures.Count > 1) + throw new AggregateException("Dispose failed for multiple nodes", disposeFailures); } /// diff --git a/test/standalone/Garnet.test.scripting/LuaScriptTests.cs b/test/standalone/Garnet.test.scripting/LuaScriptTests.cs index e3620de7100..78398094bf2 100644 --- a/test/standalone/Garnet.test.scripting/LuaScriptTests.cs +++ b/test/standalone/Garnet.test.scripting/LuaScriptTests.cs @@ -192,7 +192,7 @@ public void GlobalsForbidden() var db = redis.GetDatabase(0); var globalFuncExc = - ClassicAssert.Throws( + TestUtils.ThrowsRedisException( () => { _ = db.ScriptEvaluate( @@ -204,13 +204,13 @@ public void GlobalsForbidden() ); ClassicAssert.IsTrue(globalFuncExc.Message.Contains("Attempt to modify a readonly table")); - var globalVar = ClassicAssert.Throws(() => db.ScriptEvaluate("global_var = 'hello'")); + var globalVar = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("global_var = 'hello'")); ClassicAssert.IsTrue(globalVar.Message.Contains("Attempt to modify a readonly table")); - var metatableUpdateOnGlobals = ClassicAssert.Throws(() => db.ScriptEvaluate("setmetatable(_G, nil)")); + var metatableUpdateOnGlobals = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("setmetatable(_G, nil)")); ClassicAssert.IsTrue(metatableUpdateOnGlobals.Message.Contains("Attempt to modify a readonly table")); - var rawSetG = ClassicAssert.Throws(() => db.ScriptEvaluate("rawset(_G, 'hello', 'world')")); + var rawSetG = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("rawset(_G, 'hello', 'world')")); ClassicAssert.IsTrue(globalVar.Message.Contains("Attempt to modify a readonly table")); } @@ -228,7 +228,7 @@ public void ReadOnlyGlobalTables() foreach (var illegal in illegalToModify) { - var exc = ClassicAssert.Throws(() => db.ScriptEvaluate($"table.insert({illegal}, 'foo')")); + var exc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate($"table.insert({illegal}, 'foo')")); ClassicAssert.IsTrue(exc.Message.Contains("Attempt to modify a readonly table")); } @@ -481,7 +481,7 @@ public void CanDoScriptFlush() server.ScriptFlush(); // Assert the script is not found - _ = Assert.Throws(() => db.ScriptEvaluate(scriptId)); + _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate(scriptId)); } [Test] @@ -551,11 +551,11 @@ public void FailureStatusReturn() var db = redis.GetDatabase(0); var statusReplyScript = "return redis.error_reply('GET')"; - var excReply = ClassicAssert.Throws(() => db.ScriptEvaluate(statusReplyScript)); + var excReply = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate(statusReplyScript)); ClassicAssert.AreEqual("ERR GET", excReply.Message); var directReplyScript = "return { err = 'Failure' }"; - var excDirect = ClassicAssert.Throws(() => db.ScriptEvaluate(directReplyScript)); + var excDirect = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate(directReplyScript)); ClassicAssert.AreEqual("Failure", excDirect.Message); } @@ -567,11 +567,11 @@ public void MiscMath() // ATan2 { - _ = ClassicAssert.Throws(() => db.ScriptEvaluate("math.atan2()")); - _ = ClassicAssert.Throws(() => db.ScriptEvaluate("math.atan2(1)")); - _ = ClassicAssert.Throws(() => db.ScriptEvaluate("math.atan2(1, 2, 3)")); - _ = ClassicAssert.Throws(() => db.ScriptEvaluate("math.atan2('a', 1)")); - _ = ClassicAssert.Throws(() => db.ScriptEvaluate("math.atan2(1, 'b')")); + _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("math.atan2()")); + _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("math.atan2(1)")); + _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("math.atan2(1, 2, 3)")); + _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("math.atan2('a', 1)")); + _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("math.atan2(1, 'b')")); var val = (string)db.ScriptEvaluate("return tostring(math.atan2(0.1, 0.2))"); ClassicAssert.AreEqual(Math.Round(Math.Atan2(0.1, 0.2), 14).ToString(), val); @@ -579,9 +579,9 @@ public void MiscMath() // Cosh { - _ = ClassicAssert.Throws(() => db.ScriptEvaluate("math.cosh()")); - _ = ClassicAssert.Throws(() => db.ScriptEvaluate("math.cosh('a')")); - _ = ClassicAssert.Throws(() => db.ScriptEvaluate("math.cosh(1, 2)")); + _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("math.cosh()")); + _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("math.cosh('a')")); + _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("math.cosh(1, 2)")); var val = (string)db.ScriptEvaluate("return tostring(math.cosh(0.1))"); ClassicAssert.AreEqual(Math.Round(Math.Cosh(0.1), 14).ToString(), val); @@ -589,9 +589,9 @@ public void MiscMath() // Log10 { - _ = ClassicAssert.Throws(() => db.ScriptEvaluate("math.log10()")); - _ = ClassicAssert.Throws(() => db.ScriptEvaluate("math.log10('a')")); - _ = ClassicAssert.Throws(() => db.ScriptEvaluate("math.log10(1, 2)")); + _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("math.log10()")); + _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("math.log10('a')")); + _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("math.log10(1, 2)")); var val = (string)db.ScriptEvaluate("return tostring(math.log10(0.1))"); ClassicAssert.AreEqual("-1.0", val); @@ -599,11 +599,11 @@ public void MiscMath() // Pow { - _ = ClassicAssert.Throws(() => db.ScriptEvaluate("math.pow()")); - _ = ClassicAssert.Throws(() => db.ScriptEvaluate("math.pow(1)")); - _ = ClassicAssert.Throws(() => db.ScriptEvaluate("math.pow(1, 2, 3)")); - _ = ClassicAssert.Throws(() => db.ScriptEvaluate("math.pow('a', 1)")); - _ = ClassicAssert.Throws(() => db.ScriptEvaluate("math.pow(1, 'b')")); + _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("math.pow()")); + _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("math.pow(1)")); + _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("math.pow(1, 2, 3)")); + _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("math.pow('a', 1)")); + _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("math.pow(1, 'b')")); var val = (string)db.ScriptEvaluate("return tostring(math.pow(0.1, 0.2))"); ClassicAssert.AreEqual(Math.Round(Math.Pow(0.1, 0.2), 14).ToString(), val); @@ -611,9 +611,9 @@ public void MiscMath() // Sinh { - _ = ClassicAssert.Throws(() => db.ScriptEvaluate("math.sinh()")); - _ = ClassicAssert.Throws(() => db.ScriptEvaluate("math.sinh('a')")); - _ = ClassicAssert.Throws(() => db.ScriptEvaluate("math.sinh(1, 2)")); + _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("math.sinh()")); + _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("math.sinh('a')")); + _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("math.sinh(1, 2)")); var val = (string)db.ScriptEvaluate("return tostring(math.sinh(0.1))"); ClassicAssert.AreEqual(Math.Round(Math.Sinh(0.1), 14).ToString(), val); @@ -621,9 +621,9 @@ public void MiscMath() // Tanh { - _ = ClassicAssert.Throws(() => db.ScriptEvaluate("math.tanh()")); - _ = ClassicAssert.Throws(() => db.ScriptEvaluate("math.tanh('a')")); - _ = ClassicAssert.Throws(() => db.ScriptEvaluate("math.tanh(1, 2)")); + _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("math.tanh()")); + _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("math.tanh('a')")); + _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("math.tanh(1, 2)")); var val = (string)db.ScriptEvaluate("return tostring(math.tanh(0.2))"); ClassicAssert.AreEqual(Math.Round(Math.Tanh(0.2), 14).ToString(), val); @@ -665,10 +665,10 @@ public void RedisSha1Hex() var resTable = (string)db.ScriptEvaluate("return redis.sha1hex({ 1234 })"); ClassicAssert.AreEqual("da39a3ee5e6b4b0d3255bfef95601890afd80709", resTable); - var excEmpty = ClassicAssert.Throws(() => db.ScriptEvaluate("return redis.sha1hex()")); + var excEmpty = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return redis.sha1hex()")); ClassicAssert.IsTrue(excEmpty.Message.StartsWith("ERR wrong number of arguments")); - var excTwo = ClassicAssert.Throws(() => db.ScriptEvaluate("return redis.sha1hex('a', 'b')")); + var excTwo = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return redis.sha1hex('a', 'b')")); ClassicAssert.IsTrue(excTwo.Message.StartsWith("ERR wrong number of arguments")); } @@ -678,16 +678,16 @@ public void RedisLog() using var redis = ConnectionMultiplexer.Connect(TestUtils.GetConfig()); var db = redis.GetDatabase(0); - var excZero = ClassicAssert.Throws(() => db.ScriptEvaluate("return redis.log()")); + var excZero = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return redis.log()")); ClassicAssert.IsTrue(excZero.Message.StartsWith("ERR redis.log() requires two arguments or more.")); - var excOne = ClassicAssert.Throws(() => db.ScriptEvaluate("return redis.log(redis.LOG_DEBUG)")); + var excOne = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return redis.log(redis.LOG_DEBUG)")); ClassicAssert.IsTrue(excOne.Message.StartsWith("ERR redis.log() requires two arguments or more.")); - var excBadLevelType = ClassicAssert.Throws(() => db.ScriptEvaluate("return redis.log('hello', 'world')")); + var excBadLevelType = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return redis.log('hello', 'world')")); ClassicAssert.IsTrue(excBadLevelType.Message.StartsWith("ERR First argument must be a number (log level).")); - var excBadLevelValue = ClassicAssert.Throws(() => db.ScriptEvaluate("return redis.log(-1, 'world')")); + var excBadLevelValue = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return redis.log(-1, 'world')")); ClassicAssert.IsTrue(excBadLevelValue.Message.StartsWith("ERR Invalid debug level.")); // Test logs at each level @@ -723,7 +723,7 @@ public void RedisSetRepl() using var redis = ConnectionMultiplexer.Connect(TestUtils.GetConfig()); var db = redis.GetDatabase(0); - var excNotSupported = ClassicAssert.Throws(() => db.ScriptEvaluate("return redis.set_repl(redis.REPL_ALL)")); + var excNotSupported = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return redis.set_repl(redis.REPL_ALL)")); ClassicAssert.IsTrue(excNotSupported.Message.StartsWith("ERR redis.set_repl is not supported in Garnet")); var constantsDefined = (int[])db.ScriptEvaluate("return {redis.REPL_ALL, redis.REPL_AOF, redis.REPL_REPLICA, redis.REPL_SLAVE, redis.REPL_NONE}"); @@ -747,10 +747,10 @@ public void RedisDebugAndBreakpoint() using var redis = ConnectionMultiplexer.Connect(TestUtils.GetConfig()); var db = redis.GetDatabase(0); - var excBreakpoint = ClassicAssert.Throws(() => db.ScriptEvaluate("redis.breakpoint()")); + var excBreakpoint = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("redis.breakpoint()")); ClassicAssert.IsTrue(excBreakpoint.Message.StartsWith("ERR redis.breakpoint is not supported in Garnet")); - var excDebug = ClassicAssert.Throws(() => db.ScriptEvaluate("redis.debug('hello')")); + var excDebug = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("redis.debug('hello')")); ClassicAssert.IsTrue(excDebug.Message.StartsWith("ERR redis.debug is not supported in Garnet")); } @@ -765,16 +765,16 @@ public void RedisAclCheckCmd() using var denyRedis = ConnectionMultiplexer.Connect(TestUtils.GetConfig(authUsername: "deny")); var denyDB = denyRedis.GetDatabase(0); - var noArgs = ClassicAssert.Throws(() => db.ScriptEvaluate("return redis.acl_check_cmd()")); + var noArgs = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return redis.acl_check_cmd()")); ClassicAssert.IsTrue(noArgs.Message.StartsWith("ERR Please specify at least one argument for this redis lib call")); - var invalidCmdArgType = ClassicAssert.Throws(() => db.ScriptEvaluate("return redis.acl_check_cmd({123})")); + var invalidCmdArgType = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return redis.acl_check_cmd({123})")); ClassicAssert.IsTrue(invalidCmdArgType.Message.StartsWith("ERR Lua redis lib command arguments must be strings or integers")); - var invalidCmd = ClassicAssert.Throws(() => db.ScriptEvaluate("return redis.acl_check_cmd('nope')")); + var invalidCmd = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return redis.acl_check_cmd('nope')")); ClassicAssert.IsTrue(invalidCmd.Message.StartsWith("ERR Invalid command passed to redis.acl_check_cmd()")); - var invalidArgType = ClassicAssert.Throws(() => db.ScriptEvaluate("return redis.acl_check_cmd('GET', {123})")); + var invalidArgType = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return redis.acl_check_cmd('GET', {123})")); ClassicAssert.IsTrue(invalidArgType.Message.StartsWith("ERR Lua redis lib command arguments must be strings or integers")); var canRun = (bool)db.ScriptEvaluate("return redis.acl_check_cmd('GET')"); @@ -814,13 +814,13 @@ public void RedisSetResp() using var redis = ConnectionMultiplexer.Connect(TestUtils.GetConfig()); var db = redis.GetDatabase(0); - var noArgs = ClassicAssert.Throws(() => db.ScriptEvaluate("redis.setresp()")); + var noArgs = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("redis.setresp()")); ClassicAssert.IsTrue(noArgs.Message.StartsWith("ERR redis.setresp() requires one argument.")); - var tooManyArgs = ClassicAssert.Throws(() => db.ScriptEvaluate("redis.setresp(1, 2)")); + var tooManyArgs = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("redis.setresp(1, 2)")); ClassicAssert.IsTrue(tooManyArgs.Message.StartsWith("ERR redis.setresp() requires one argument.")); - var badArg = ClassicAssert.Throws(() => db.ScriptEvaluate("redis.setresp({123})")); + var badArg = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("redis.setresp({123})")); ClassicAssert.IsTrue(badArg.Message.StartsWith("ERR RESP version must be 2 or 3.")); var resp2 = db.ScriptEvaluate("redis.setresp(2)"); @@ -829,7 +829,7 @@ public void RedisSetResp() var resp3 = db.ScriptEvaluate("redis.setresp(3)"); ClassicAssert.IsTrue(resp3.IsNull); - var badRespVersion = ClassicAssert.Throws(() => db.ScriptEvaluate("redis.setresp(1)")); + var badRespVersion = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("redis.setresp(1)")); ClassicAssert.IsTrue(badRespVersion.Message.StartsWith("ERR RESP version must be 2 or 3.")); } @@ -974,7 +974,7 @@ public void ScriptExistsErrors() using var redis = ConnectionMultiplexer.Connect(TestUtils.GetConfig()); var db = redis.GetDatabase(0); - var exc = ClassicAssert.Throws(() => db.Execute("SCRIPT", "EXISTS")); + var exc = TestUtils.ThrowsRedisException(() => db.Execute("SCRIPT", "EXISTS")); ClassicAssert.AreEqual("ERR wrong number of arguments for 'script|exists' command", exc.Message); } @@ -986,13 +986,13 @@ public void ScriptFlushErrors() // > 1 args { - var exc = ClassicAssert.Throws(() => db.Execute("SCRIPT", "FLUSH", "ASYNC", "BAR")); + var exc = TestUtils.ThrowsRedisException(() => db.Execute("SCRIPT", "FLUSH", "ASYNC", "BAR")); ClassicAssert.AreEqual("ERR SCRIPT FLUSH only support SYNC|ASYNC option", exc.Message); } // 1 arg, but not ASYNC or SYNC { - var exc = ClassicAssert.Throws(() => db.Execute("SCRIPT", "FLUSH", "NOW")); + var exc = TestUtils.ThrowsRedisException(() => db.Execute("SCRIPT", "FLUSH", "NOW")); ClassicAssert.AreEqual("ERR SCRIPT FLUSH only support SYNC|ASYNC option", exc.Message); } } @@ -1020,8 +1020,8 @@ public void MultiSessionScriptFlush() _ = db1.Execute("SCRIPT", "FLUSH", "SYNC"); - var exc1 = ClassicAssert.Throws(() => db1.Execute("EVALSHA", hash, "0")); - var exc2 = ClassicAssert.Throws(() => db2.Execute("EVALSHA", hash, "0")); + var exc1 = TestUtils.ThrowsRedisException(() => db1.Execute("EVALSHA", hash, "0")); + var exc2 = TestUtils.ThrowsRedisException(() => db2.Execute("EVALSHA", hash, "0")); ClassicAssert.True(exc1.Message.StartsWith("NOSCRIPT ")); ClassicAssert.True(exc2.Message.StartsWith("NOSCRIPT ")); @@ -1065,13 +1065,13 @@ public void ScriptLoadErrors() // 0 args { - var exc = ClassicAssert.Throws(() => db.Execute("SCRIPT", "LOAD")); + var exc = TestUtils.ThrowsRedisException(() => db.Execute("SCRIPT", "LOAD")); ClassicAssert.AreEqual("ERR wrong number of arguments for 'script|load' command", exc.Message); } // > 1 args { - var exc = ClassicAssert.Throws(() => db.Execute("SCRIPT", "LOAD", "return 'foo'", "return 'bar'")); + var exc = TestUtils.ThrowsRedisException(() => db.Execute("SCRIPT", "LOAD", "return 'foo'", "return 'bar'")); ClassicAssert.AreEqual("ERR wrong number of arguments for 'script|load' command", exc.Message); } } @@ -1111,7 +1111,7 @@ public void ScriptExistsMultiple() private static void DoErroneousRedisCall(IDatabase db, string[] args, string expectedError) { - var exc = Assert.Throws(() => db.ScriptEvaluate($"return redis.call({string.Join(',', args)})")); + var exc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate($"return redis.call({string.Join(',', args)})")); ClassicAssert.IsNotNull(exc); StringAssert.StartsWith(expectedError, exc!.Message); } @@ -1381,7 +1381,7 @@ public void IntentionalOOM() var loadedScriptOOM = scriptOOM.Load(redis.GetServers()[0]); // OOM actually happens and is reported - var exc = ClassicAssert.Throws(() => db.ScriptEvaluate(loadedScriptOOM, new { Ctrl = "OOM" })); + var exc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate(loadedScriptOOM, new { Ctrl = "OOM" })); ClassicAssert.AreEqual("ERR Lua encountered an error: not enough memory", exc.Message); // We can still run the script without issue (with non-crashing args) afterwards @@ -1472,7 +1472,7 @@ public void Issue939() } // Finally, check that nil is an illegal argument - var exc = ClassicAssert.Throws(() => db.ScriptEvaluate("return redis.call('GET', nil)")); + var exc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return redis.call('GET', nil)")); ClassicAssert.True(exc.Message.StartsWith("ERR Lua redis lib command arguments must be strings or integers")); } @@ -1489,10 +1489,10 @@ public void Issue1079() using var redis = ConnectionMultiplexer.Connect(TestUtils.GetConfig()); var db = redis.GetDatabase(); - var brokenExc1 = ClassicAssert.Throws(() => db.Execute("EVAL", BrokenScript, 0)); + var brokenExc1 = TestUtils.ThrowsRedisException(() => db.Execute("EVAL", BrokenScript, 0)); ClassicAssert.True(brokenExc1.Message.StartsWith("Compilation error: ")); - var brokenExc2 = ClassicAssert.Throws(() => db.Execute("EVAL", BrokenScript, 0)); + var brokenExc2 = TestUtils.ThrowsRedisException(() => db.Execute("EVAL", BrokenScript, 0)); ClassicAssert.AreEqual(brokenExc1.Message, brokenExc2.Message); var success = (string)db.Execute("EVAL", FixedScript, 0); @@ -1824,7 +1824,7 @@ public void Resp2ToLuaConversions() ClassicAssert.AreEqual("table", simpleStringRes[0]); ClassicAssert.AreEqual("PONG", simpleStringRes[1]); - var errExc = ClassicAssert.Throws(() => db.ScriptEvaluate("return { err = 'ERR mapped to ERR response' }")); + var errExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return { err = 'ERR mapped to ERR response' }")); ClassicAssert.AreEqual("ERR mapped to ERR response", errExc.Message); var nullBulkRes = (string[])db.ScriptEvaluate("local res = redis.call('GET', KEYS[1]); return { type(res), tostring(res) };", [(RedisKey)"not-set-ever"]); @@ -1856,14 +1856,14 @@ public void NoScriptCommandsForbidden() foreach (var (cmd, _) in fullCommands) { - var exc = ClassicAssert.Throws(() => db.ScriptEvaluate($"redis.call('{cmd}')")); + var exc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate($"redis.call('{cmd}')")); ClassicAssert.True(exc.Message.StartsWith("ERR This Redis command is not allowed from script"), $"Allowed NoScript command: {cmd}"); } var subCommands = allCommands.Where(static kv => (kv.Value.SubCommands?.Length ?? 0) > 0).SelectMany(static kv => kv.Value.SubCommands.Where(static t => t.Flags.HasFlag(RespCommandFlags.NoScript)).Select(t => (kv.Key, t.Name.Split('|')[1]))); foreach (var (cmd, subCmd) in subCommands) { - var exc = ClassicAssert.Throws(() => db.ScriptEvaluate($"redis.call('{cmd}', '{subCmd}')")); + var exc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate($"redis.call('{cmd}', '{subCmd}')")); ClassicAssert.True(exc.Message.StartsWith("ERR This Redis command is not allowed from script"), $"Allowed NoScript command: {cmd}|{subCmd}"); } } @@ -1884,20 +1884,20 @@ public void PermissionsEnforced() ClassicAssert.AreEqual("bar", allowRes[0]); // Not a lot of sub commands a non-admin can run, so use CLUSTER|MYID and check the exception - var allowSubExc = ClassicAssert.Throws(() => allowDb.ScriptEvaluate("return redis.call('CLUSTER', 'MYID')")); + var allowSubExc = TestUtils.ThrowsRedisException(() => allowDb.ScriptEvaluate("return redis.call('CLUSTER', 'MYID')")); ClassicAssert.False(allowSubExc.Message.Contains("NOPERM")); - var exc = ClassicAssert.Throws(() => denyDb.ScriptEvaluate("return redis.call('GET', 'foo')")); + var exc = TestUtils.ThrowsRedisException(() => denyDb.ScriptEvaluate("return redis.call('GET', 'foo')")); ClassicAssert.IsTrue(exc.Message.Contains("NOPERM")); - var excSub = ClassicAssert.Throws(() => denyDb.ScriptEvaluate("return redis.call('CLUSTER', 'MYID')")); + var excSub = TestUtils.ThrowsRedisException(() => denyDb.ScriptEvaluate("return redis.call('CLUSTER', 'MYID')")); ClassicAssert.IsTrue(excSub.Message.Contains("NOPERM")); // SET is dispatched through a separate fast path in LuaRunner, so cover it explicitly using var denySetRedis = ConnectionMultiplexer.Connect(TestUtils.GetConfig(authUsername: "denyset")); var denySetDb = denySetRedis.GetDatabase(); - var excSet = ClassicAssert.Throws(() => denySetDb.ScriptEvaluate("return redis.call('SET', 'foo', 'baz')")); + var excSet = TestUtils.ThrowsRedisException(() => denySetDb.ScriptEvaluate("return redis.call('SET', 'foo', 'baz')")); ClassicAssert.IsTrue(excSet.Message.Contains("NOPERM"), $"Expected NOPERM for denied SET, got: {excSet.Message}"); var allowSetRes = (string[])denySetDb.ScriptEvaluate("return redis.call('GET', 'foo')"); @@ -1932,7 +1932,7 @@ public void IntentionalTimeout() var loadedScriptTimeout = scriptTimeout.Load(redis.GetServers()[0]); // Timeout actually happens and is reported - var exc = ClassicAssert.Throws(() => db.ScriptEvaluate(loadedScriptTimeout, new { Ctrl = "Timeout" })); + var exc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate(loadedScriptTimeout, new { Ctrl = "Timeout" })); ClassicAssert.AreEqual("ERR Lua script exceeded configured timeout", exc.Message); // We can still run the script without issue (with non-crashing args) afterwards @@ -2043,12 +2043,12 @@ public void Bit() // tobit { - var noArgExc = ClassicAssert.Throws(() => db.ScriptEvaluate("return bit.tobit()")); + var noArgExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return bit.tobit()")); ClassicAssert.True(noArgExc.Message.Contains("bad argument") && noArgExc.Message.Contains("tobit")); // Extra arguments are legal, but ignored - var badTypeExc = ClassicAssert.Throws(() => db.ScriptEvaluate("return bit.tobit({})")); + var badTypeExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return bit.tobit({})")); ClassicAssert.True(badTypeExc.Message.Contains("bad argument") && badTypeExc.Message.Contains("tobit")); // Rules are suprisingly subtle, so test a bunch of tricky values @@ -2085,15 +2085,15 @@ public void Bit() // tohex { - var noArgExc = ClassicAssert.Throws(() => db.ScriptEvaluate("return bit.tohex()")); + var noArgExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return bit.tohex()")); ClassicAssert.True(noArgExc.Message.Contains("bad argument") && noArgExc.Message.Contains("tohex")); // Extra arguments are legal, but ignored - var badType1Exc = ClassicAssert.Throws(() => db.ScriptEvaluate("return bit.tohex({})")); + var badType1Exc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return bit.tohex({})")); ClassicAssert.True(badType1Exc.Message.Contains("bad argument") && badType1Exc.Message.Contains("tohex")); - var badType2Exc = ClassicAssert.Throws(() => db.ScriptEvaluate("return bit.tohex(1, {})")); + var badType2Exc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return bit.tohex(1, {})")); ClassicAssert.True(badType2Exc.Message.Contains("bad argument") && badType2Exc.Message.Contains("tohex")); // Make sure casing is handled correctly @@ -2135,12 +2135,12 @@ public void Bit() // bswap { - var noArgExc = ClassicAssert.Throws(() => db.ScriptEvaluate("return bit.bswap()")); + var noArgExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return bit.bswap()")); ClassicAssert.True(noArgExc.Message.Contains("bad argument") && noArgExc.Message.Contains("bswap")); // Extra arguments are legal, but ignored - var badTypeExc = ClassicAssert.Throws(() => db.ScriptEvaluate("return bit.bswap({})")); + var badTypeExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return bit.bswap({})")); ClassicAssert.True(badTypeExc.Message.Contains("bad argument") && badTypeExc.Message.Contains("bswap")); // Just brute force a bunch of trial values @@ -2165,12 +2165,12 @@ public void Bit() // bnot { - var noArgExc = ClassicAssert.Throws(() => db.ScriptEvaluate("return bit.bnot()")); + var noArgExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return bit.bnot()")); ClassicAssert.True(noArgExc.Message.Contains("bad argument") && noArgExc.Message.Contains("bnot")); // Extra arguments are legal, but ignored - var badTypeExc = ClassicAssert.Throws(() => db.ScriptEvaluate("return bit.bnot({})")); + var badTypeExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return bit.bnot({})")); ClassicAssert.True(badTypeExc.Message.Contains("bad argument") && badTypeExc.Message.Contains("bnot")); foreach (var input in new int[] { 0, 1, 2, 4, 8, 32, 64, 128, 256, 0x70F0_F0F0, 0x6BCD_EF01, int.MinValue, int.MaxValue, -1 }) @@ -2192,16 +2192,16 @@ public void Bit() foreach (var op in ops) { - var noArgExc = ClassicAssert.Throws(() => db.ScriptEvaluate($"return bit.{op.Name}()")); + var noArgExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate($"return bit.{op.Name}()")); ClassicAssert.True(noArgExc.Message.Contains("bad argument") && noArgExc.Message.Contains(op.Name)); - var badType1Exc = ClassicAssert.Throws(() => db.ScriptEvaluate($"return bit.{op.Name}({{}})")); + var badType1Exc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate($"return bit.{op.Name}({{}})")); ClassicAssert.True(badType1Exc.Message.Contains("bad argument") && badType1Exc.Message.Contains(op.Name)); - var badType2Exc = ClassicAssert.Throws(() => db.ScriptEvaluate($"return bit.{op.Name}(1, {{}})")); + var badType2Exc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate($"return bit.{op.Name}(1, {{}})")); ClassicAssert.True(badType2Exc.Message.Contains("bad argument") && badType2Exc.Message.Contains(op.Name)); - var badType3Exc = ClassicAssert.Throws(() => db.ScriptEvaluate($"return bit.{op.Name}(1, 2, {{}})")); + var badType3Exc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate($"return bit.{op.Name}(1, 2, {{}})")); ClassicAssert.True(badType3Exc.Message.Contains("bad argument") && badType3Exc.Message.Contains(op.Name)); // Gin up some unusual values and test them in different combinations @@ -2240,13 +2240,13 @@ public void Bit() foreach (var op in ops) { - var noArgExc = ClassicAssert.Throws(() => db.ScriptEvaluate($"return bit.{op.Name}()")); + var noArgExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate($"return bit.{op.Name}()")); ClassicAssert.True(noArgExc.Message.Contains("bad argument") && noArgExc.Message.Contains(op.Name)); - var badType1Exc = ClassicAssert.Throws(() => db.ScriptEvaluate($"return bit.{op.Name}({{}})")); + var badType1Exc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate($"return bit.{op.Name}({{}})")); ClassicAssert.True(badType1Exc.Message.Contains("bad argument") && badType1Exc.Message.Contains(op.Name)); - var badType2Exc = ClassicAssert.Throws(() => db.ScriptEvaluate($"return bit.{op.Name}(1, {{}})")); + var badType2Exc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate($"return bit.{op.Name}(1, {{}})")); ClassicAssert.True(badType2Exc.Message.Contains("bad argument") && badType2Exc.Message.Contains(op.Name)); // Extra args are allowed, but ignored @@ -2272,13 +2272,13 @@ public void CJson() // Encoding { - var noArgExc = ClassicAssert.Throws(() => db.ScriptEvaluate("return cjson.encode()")); + var noArgExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return cjson.encode()")); ClassicAssert.True(noArgExc.Message.Contains("bad argument") && noArgExc.Message.Contains("encode")); - var twoArgExc = ClassicAssert.Throws(() => db.ScriptEvaluate("return cjson.encode(1, 2)")); + var twoArgExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return cjson.encode(1, 2)")); ClassicAssert.True(twoArgExc.Message.Contains("bad argument") && twoArgExc.Message.Contains("encode")); - var badTypeExc = ClassicAssert.Throws(() => db.ScriptEvaluate("return cjson.encode((function() end))")); + var badTypeExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return cjson.encode((function() end))")); ClassicAssert.True(badTypeExc.Message.Contains("Cannot serialise")); var nilResp = (string)db.ScriptEvaluate("return cjson.encode(nil)"); @@ -2353,7 +2353,7 @@ public void CJson() ClassicAssert.AreEqual(new string('[', 1000) + 1 + new string(']', 1000), deeplyNestedButLegal); var deeplyNestedExc = - ClassicAssert.Throws( + TestUtils.ThrowsRedisException( () => db.ScriptEvaluate( @"local nested = 1 for x = 1, 1001 do @@ -2368,16 +2368,16 @@ public void CJson() // Decoding { - var noArgExc = ClassicAssert.Throws(() => db.ScriptEvaluate("return cjson.decode()")); + var noArgExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return cjson.decode()")); ClassicAssert.True(noArgExc.Message.Contains("bad argument") && noArgExc.Message.Contains("decode")); - var twoArgExc = ClassicAssert.Throws(() => db.ScriptEvaluate("return cjson.decode(1, 2)")); + var twoArgExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return cjson.decode(1, 2)")); ClassicAssert.True(twoArgExc.Message.Contains("bad argument") && twoArgExc.Message.Contains("decode")); - var badTypeExc = ClassicAssert.Throws(() => db.ScriptEvaluate("return cjson.decode({})")); + var badTypeExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return cjson.decode({})")); ClassicAssert.True(badTypeExc.Message.Contains("bad argument") && badTypeExc.Message.Contains("decode")); - var badFormatExc = ClassicAssert.Throws(() => db.ScriptEvaluate("return cjson.decode('hello world')")); + var badFormatExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return cjson.decode('hello world')")); ClassicAssert.True(badFormatExc.Message.Contains("Expected value but found invalid token")); var nullDecode = (string)db.ScriptEvaluate("return type(cjson.decode('null'))"); @@ -2429,7 +2429,7 @@ public void CJson() } ClassicAssert.AreEqual(0, deeplyNestedButLegalCur.Length); - var deeplyNestedExc = ClassicAssert.Throws(() => db.ScriptEvaluate($"return cjson.decode('{new string('[', 1001)}{new string(']', 1001)}')")); + var deeplyNestedExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate($"return cjson.decode('{new string('[', 1001)}{new string(']', 1001)}')")); ClassicAssert.True(deeplyNestedExc.Message.Contains("Found too many nested data structures")); } } @@ -2440,7 +2440,7 @@ public void CMsgPackPack() using var redis = ConnectionMultiplexer.Connect(TestUtils.GetConfig()); var db = redis.GetDatabase(); - var noArgExc = ClassicAssert.Throws(() => db.ScriptEvaluate("return cmsgpack.pack()")); + var noArgExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return cmsgpack.pack()")); ClassicAssert.True(noArgExc.Message.Contains("bad argument") && noArgExc.Message.Contains("pack")); // Multiple args are legal, and concat @@ -2634,13 +2634,13 @@ public void CMsgPackUnpack() using var redis = ConnectionMultiplexer.Connect(TestUtils.GetConfig()); var db = redis.GetDatabase(); - var noArgExc = ClassicAssert.Throws(() => db.ScriptEvaluate("return cmsgpack.unpack()")); + var noArgExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return cmsgpack.unpack()")); ClassicAssert.True(noArgExc.Message.Contains("bad argument") && noArgExc.Message.Contains("unpack")); // Multiple arguments are allowed, but ignored // Table ends before it should - var badDataExc = ClassicAssert.Throws(() => db.ScriptEvaluate("return cmsgpack.unpack('\\220\\0\\96')")); + var badDataExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return cmsgpack.unpack('\\220\\0\\96')")); ClassicAssert.True(badDataExc.Message.Contains("Missing bytes in input")); var nullResp = (string)db.ScriptEvaluate($"return type(cmsgpack.unpack({ToLuaString(0xC0)}))"); @@ -2948,31 +2948,31 @@ public void StructPackOnErrors() var db = redis.GetDatabase(); // Not power of two - var excNotPowerOfTwoArgs = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.pack('!9')")); + var excNotPowerOfTwoArgs = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.pack('!9')")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to format", excNotPowerOfTwoArgs.Message); // Invalid format - var excFormat = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.pack(123, 123)")); + var excFormat = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.pack(123, 123)")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to pack", excFormat.Message); // Format opt size exceed max int size 32 - var excOptSize = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.pack('I64', 512)")); + var excOptSize = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.pack('I64', 512)")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to format", excOptSize.Message); // Format opt size integeral size overflow - var excOptSizeOverflow = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.pack('I2147483648', 512)")); + var excOptSizeOverflow = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.pack('I2147483648', 512)")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to format", excOptSizeOverflow.Message); // String too short, expected at least {size} bytes but got {l} - var excArgPack = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.pack('c6', 'hello')")); + var excArgPack = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.pack('c6', 'hello')")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to pack", excArgPack.Message); // Invalid Control Options - var excControlOptions = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.pack('@I', 'hello')")); + var excControlOptions = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.pack('@I', 'hello')")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to format", excControlOptions.Message); // Invalid alignment - var excBadAlignment = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.pack('!9c1', 'Z')")); + var excBadAlignment = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.pack('!9c1', 'Z')")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to format", excBadAlignment.Message); } @@ -3105,55 +3105,55 @@ public void StructUnpackOnErrors() var db = redis.GetDatabase(); // Invalid format - var excFormat = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.unpack(123, 123)")); + var excFormat = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.unpack(123, 123)")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to unpack", excFormat.Message); // Format opt size exceed max int size 32 - var excOptSize = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.unpack('I64', '\\000\\000\\000\\000')")); + var excOptSize = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.unpack('I64', '\\000\\000\\000\\000')")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to format", excOptSize.Message); // Format opt size integeral size overflow - var excOptSizeOverflow = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.unpack('I2147483648', '\\000\\002')")); + var excOptSizeOverflow = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.unpack('I2147483648', '\\000\\002')")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to format", excOptSizeOverflow.Message); // String too short, expected at least {size} bytes but got {l} - var excArgPack = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.unpack('c6', 'hello')")); + var excArgPack = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.unpack('c6', 'hello')")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to unpack", excArgPack.Message); // Invalid Control Options - var excControlOptions = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.unpack('@I', 'hello')")); + var excControlOptions = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.unpack('@I', 'hello')")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to format", excControlOptions.Message); // Missing argument - var excMissingArgs = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.unpack('!8')")); + var excMissingArgs = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.unpack('!8')")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to unpack", excMissingArgs.Message); // Invalid number of arguments - var excBadAlignment = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.unpack('!5', '\\157\\255\\255\\255')")); + var excBadAlignment = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.unpack('!5', '\\157\\255\\255\\255')")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to format", excBadAlignment.Message); // Invalid Third argument - var excBadThirdArg = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.unpack('@I', 'hello', 'test')")); + var excBadThirdArg = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.unpack('@I', 'hello', 'test')")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to unpack", excBadThirdArg.Message); // Invalid Third argument - var excArgTooShort = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.unpack('c4', '\\065\\066')")); + var excArgTooShort = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.unpack('c4', '\\065\\066')")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to unpack", excArgTooShort.Message); // Missing character size - var excMissingCharacterSize = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.unpack('c0', 'dynamic')")); + var excMissingCharacterSize = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.unpack('c0', 'dynamic')")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to unpack", excMissingCharacterSize.Message); // Invalid character size - var excInvalidCharacterZeroSize = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.unpack('sc0', 'size\\000dynamic')")); + var excInvalidCharacterZeroSize = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.unpack('sc0', 'size\\000dynamic')")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to unpack", excInvalidCharacterZeroSize.Message); // Invalid String without terminator - var excBadString = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.unpack('s', 'hello')")); + var excBadString = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.unpack('s', 'hello')")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to unpack", excBadString.Message); // Third pos has to be greater than 0 - var unpackBadPosRes = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.unpack('s', 'hello', 0)")); + var unpackBadPosRes = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.unpack('s', 'hello', 0)")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to unpack", unpackBadPosRes.Message); } @@ -3197,16 +3197,16 @@ public void StructSizeOnErrors() using var redis = ConnectionMultiplexer.Connect(TestUtils.GetConfig()); var db = redis.GetDatabase(); - var excFormatString = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.size('s')")); + var excFormatString = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.size('s')")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to format", excFormatString.Message); - var excFormatCharacter = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.size('c0')")); + var excFormatCharacter = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.size('c0')")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to format", excFormatCharacter.Message); - var excBadControlOptions = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.size('@I')")); + var excBadControlOptions = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.size('@I')")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to format", excBadControlOptions.Message); - var excBadFormat = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.size(123)")); + var excBadFormat = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.size(123)")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to format", excBadFormat.Message); } @@ -3308,10 +3308,10 @@ public void LoadString() var db = redis.GetDatabase(); // load and loadstring are not part of the sandbox allowed functions - var loadExc = ClassicAssert.Throws(() => db.ScriptEvaluate("local x = load('return 123'); return x()")); + var loadExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("local x = load('return 123'); return x()")); ClassicAssert.True(loadExc.Message.Contains("attempt to call a nil value")); - var loadstringExc = ClassicAssert.Throws(() => db.ScriptEvaluate("local x = loadstring('return 123'); return x()")); + var loadstringExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("local x = loadstring('return 123'); return x()")); ClassicAssert.True(loadstringExc.Message.Contains("attempt to call a nil value")); } @@ -3404,7 +3404,7 @@ public void StressTimeouts() else { // Periodically cause a timeout - var exc = ClassicAssert.Throws(() => db.ScriptEvaluate(loadedScriptTimeout, timeout)); + var exc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate(loadedScriptTimeout, timeout)); ClassicAssert.AreEqual("ERR Lua script exceeded configured timeout", exc.Message); } } @@ -3588,7 +3588,7 @@ public void LuaSubscribe_StillNoScriptBlocked() using var redis = ConnectionMultiplexer.Connect(TestUtils.GetConfig()); var db = redis.GetDatabase(0); - var exc = ClassicAssert.Throws(() => + var exc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return redis.call('SUBSCRIBE', KEYS[1])", [new RedisKey("any_channel")])); ClassicAssert.IsTrue(exc.Message.Contains("not allowed from script"), diff --git a/test/standalone/Garnet.test.vectorset/ExceptionInjectionShutdownTests.cs b/test/standalone/Garnet.test.vectorset/ExceptionInjectionShutdownTests.cs new file mode 100644 index 00000000000..6598430876e --- /dev/null +++ b/test/standalone/Garnet.test.vectorset/ExceptionInjectionShutdownTests.cs @@ -0,0 +1,152 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT license. + +using System; +using System.Threading.Tasks; +using Garnet.common; +using NUnit.Framework; +using NUnit.Framework.Legacy; +using StackExchange.Redis; + +namespace Garnet.test +{ + /// + /// A thread parked in is only released by + /// , but the cleanup a test runs on its way out + /// is . A test that leaves between a waiter + /// arriving and being re-enabled strands that waiter, and because the waiter is a server thread holding + /// a pooled network buffer, LimitedFixedBufferPool.Dispose then spins forever on a reference that + /// is never returned - hanging the whole test process instead of failing one test. + /// + [TestFixture] + public class ExceptionInjectionShutdownTests : TestBase + { + private const ExceptionInjectionType Pause = ExceptionInjectionType.VectorSet_Pause_Before_Synthetic_Replication_Rmw; + + private global::Garnet.GarnetServer server; + + [SetUp] + public void Setup() + { + TestUtils.DeleteDirectory(TestUtils.MethodTestDir, wait: true); + } + + [TearDown] + public void TearDown() + { + ExceptionInjectionHelper.DisableException(Pause); + + server?.Dispose(); + server = null; + + TestUtils.DeleteDirectory(TestUtils.MethodTestDir); + TestUtils.OnTearDown(); + } + + [Test] + public void ServerDisposeCompletesWhileAnInjectionPointIsParked() + { + TestUtils.IgnoreIfExceptionInjectionDisabled(); + + server = TestUtils.CreateGarnetServer(TestUtils.MethodTestDir, enableVectorSetPreview: true); + server.Start(); + + using var redis = ConnectionMultiplexer.Connect(TestUtils.GetConfig()); + var db = redis.GetDatabase(0); + + ExceptionInjectionHelper.EnableException(Pause); + + // Parks inside ReplicateVectorSetAdd while holding a pooled network buffer. + _ = Task.Run(() => db.Execute("VADD", ["parked-vs", "VALUES", "3", "1", "2", "3", "elem"])); + + // ResetAndWaitAsync clears the flag when it arrives, so this is the arrival signal. + ClassicAssert.IsTrue( + ExceptionInjectionHelper.WaitOnClearAsync(Pause).Wait(TimeSpan.FromSeconds(30)), + "the server never reached the injection point"); + + // What a failing test leaves behind: the flag is cleared, so nothing will ever release the waiter. + ExceptionInjectionHelper.DisableException(Pause); + + // Hand ownership to the dispose task before starting it: if the waiter is stranded this call + // never returns, and TearDown must not go on to block the run on the same disposal. + var disposing = server; + server = null; + + var dispose = Task.Run(() => disposing.Dispose(deleteDir: false)); + + ClassicAssert.IsTrue( + dispose.Wait(TimeSpan.FromSeconds(30)), + "GarnetServer.Dispose() did not complete: a parked injection point kept a pooled buffer " + + "checked out, so the buffer pool drain never finished"); + } + + /// + /// Releasing only the waiters that are already parked leaves the same hang one moment later: + /// disposal closes listeners before it drains handlers, so a request that is already in flight can + /// reach a still-armed injection point after shutdown has begun and park there instead. Nothing + /// re-enables it, so it strands its pooled buffer exactly as before. + /// + [Test] + public void ArrivingAtAnInjectionPointDuringShutdownDoesNotPark() + { + TestUtils.IgnoreIfExceptionInjectionDisabled(); + + ExceptionInjectionHelper.EnableException(Pause); + ExceptionInjectionHelper.SuspendParking(); + + try + { + var arriving = Task.Run(() => ExceptionInjectionHelper.ResetAndWait(Pause)); + + ClassicAssert.IsTrue( + arriving.Wait(TimeSpan.FromSeconds(30)), + "a caller reaching an injection point after shutdown began parked instead of proceeding, " + + "so it would still be holding its pooled buffer when the drain runs"); + } + finally + { + ExceptionInjectionHelper.ResumeParking(); + ExceptionInjectionHelper.DisableException(Pause); + } + } + + /// + /// The suspension must not outlive the shutdown that raised it, or a later test in the same process + /// would find its injection points no longer pausing anything. + /// + [Test] + public void ParkingResumesAfterShutdownCompletes() + { + TestUtils.IgnoreIfExceptionInjectionDisabled(); + + ExceptionInjectionHelper.SuspendParking(); + ExceptionInjectionHelper.ResumeParking(); + + ExceptionInjectionHelper.EnableException(Pause); + + try + { + var parked = Task.Run(() => ExceptionInjectionHelper.ResetAndWait(Pause)); + + ClassicAssert.IsTrue( + ExceptionInjectionHelper.WaitOnClearAsync(Pause).Wait(TimeSpan.FromSeconds(30)), + "the caller never reached the injection point"); + + ClassicAssert.IsFalse( + parked.Wait(TimeSpan.FromSeconds(1)), + "the injection point did not pause: a suspension leaked past the shutdown that raised it"); + + // Releasing it the normal way proves the rendezvous is otherwise intact. + ExceptionInjectionHelper.EnableException(Pause); + + ClassicAssert.IsTrue( + parked.Wait(TimeSpan.FromSeconds(30)), + "re-enabling the injection point did not release the parked caller"); + } + finally + { + ExceptionInjectionHelper.DisableException(Pause); + } + } + } +} \ No newline at end of file diff --git a/test/standalone/Garnet.test.vectorset/RespVectorSetTests.cs b/test/standalone/Garnet.test.vectorset/RespVectorSetTests.cs index 2d0c279514e..31ad9fc36dd 100644 --- a/test/standalone/Garnet.test.vectorset/RespVectorSetTests.cs +++ b/test/standalone/Garnet.test.vectorset/RespVectorSetTests.cs @@ -71,7 +71,7 @@ public void DisabledWithFeatureFlag() foreach (var cmd in vectorSetCommands) { // Should all fault before any validation - var exc = ClassicAssert.Throws(() => db.Execute(cmd.ToString())); + var exc = TestUtils.ThrowsRedisException(() => db.Execute(cmd.ToString())); ClassicAssert.AreEqual("ERR Vector Set (preview) commands are not enabled", exc.Message); } } @@ -95,22 +95,22 @@ public void WrongTypeForVectorSetOpsOnNonVectorSetKeys() switch (cmd) { case RespCommand.VADD: - exc = ClassicAssert.Throws(() => db.Execute("VADD", ["foo", "REDUCE", "50", "VALUES", "75", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", new byte[] { 0, 0, 0, 0 }, "CAS", "NOQUANT", "EF", "16", "M", "32"])); + exc = TestUtils.ThrowsRedisException(() => db.Execute("VADD", ["foo", "REDUCE", "50", "VALUES", "75", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", new byte[] { 0, 0, 0, 0 }, "CAS", "NOQUANT", "EF", "16", "M", "32"])); break; case RespCommand.VCARD: // TODO: Implement when VCARD works continue; case RespCommand.VDIM: - exc = ClassicAssert.Throws(() => db.Execute("VDIM", ["foo"])); + exc = TestUtils.ThrowsRedisException(() => db.Execute("VDIM", ["foo"])); break; case RespCommand.VEMB: - exc = ClassicAssert.Throws(() => db.Execute("VEMB", ["foo", new byte[] { 0, 0, 0, 0 }])); + exc = TestUtils.ThrowsRedisException(() => db.Execute("VEMB", ["foo", new byte[] { 0, 0, 0, 0 }])); break; case RespCommand.VGETATTR: - exc = ClassicAssert.Throws(() => db.Execute("VGETATTR", ["foo", new byte[] { 0, 0, 0, 0 }])); + exc = TestUtils.ThrowsRedisException(() => db.Execute("VGETATTR", ["foo", new byte[] { 0, 0, 0, 0 }])); break; case RespCommand.VINFO: - exc = ClassicAssert.Throws(() => db.Execute("VINFO", ["foo"])); + exc = TestUtils.ThrowsRedisException(() => db.Execute("VINFO", ["foo"])); break; case RespCommand.VISMEMBER: // TODO: Implement when VISMEMBER works @@ -122,13 +122,13 @@ public void WrongTypeForVectorSetOpsOnNonVectorSetKeys() // TODO: Implement when VRANDMEMBER works continue; case RespCommand.VREM: - exc = ClassicAssert.Throws(() => db.Execute("VREM", ["foo", new byte[] { 0, 0, 0, 0 }])); + exc = TestUtils.ThrowsRedisException(() => db.Execute("VREM", ["foo", new byte[] { 0, 0, 0, 0 }])); break; case RespCommand.VSETATTR: // TODO: Implement when VSETATTR works continue; case RespCommand.VSIM: - exc = ClassicAssert.Throws(() => db.Execute("VSIM", ["foo", "VALUES", "75", "110.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "COUNT", "5", "EPSILON", "1.0", "EF", "40"])); + exc = TestUtils.ThrowsRedisException(() => db.Execute("VSIM", ["foo", "VALUES", "75", "110.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "COUNT", "5", "EPSILON", "1.0", "EF", "40"])); break; default: throw new InvalidOperationException($"Unexpected Vector Set command: {cmd}"); @@ -182,7 +182,7 @@ public void VADD() var res5 = db.Execute("VADD", ["fizz", "REDUCE", "50", "VALUES", "75", "150.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", new byte[] { 0, 0, 0, 0 }, "CAS", "NOQUANT", "EF", "16", "M", "32"]); ClassicAssert.AreEqual(1, (int)res5); - var exc1 = ClassicAssert.Throws(() => db.Execute("VADD", ["fizz", "VALUES", "4", "5.0", "6.0", "7.0", "8.0", new byte[] { 0, 0, 0, 1 }, "CAS", "NOQUANT", "EF", "16", "M", "32"])); + var exc1 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", ["fizz", "VALUES", "4", "5.0", "6.0", "7.0", "8.0", new byte[] { 0, 0, 0, 1 }, "CAS", "NOQUANT", "EF", "16", "M", "32"])); ClassicAssert.AreEqual("ERR Vector dimension mismatch - got 4 but set has 75", exc1.Message); // Add without specifying EF after first vector @@ -190,11 +190,11 @@ public void VADD() ClassicAssert.AreEqual(1, (int)res6); // Add without specifying M after first vector - var exc2 = ClassicAssert.Throws(() => db.Execute("VADD", ["fizz", "REDUCE", "50", "VALUES", "75", "180.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", new byte[] { 0, 0, 0, 4 }, "CAS", "NOQUANT", "EF", "16"])); + var exc2 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", ["fizz", "REDUCE", "50", "VALUES", "75", "180.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", new byte[] { 0, 0, 0, 4 }, "CAS", "NOQUANT", "EF", "16"])); ClassicAssert.AreEqual("ERR asked M value mismatch with existing vector set", exc2.Message); // Mismatch vector size for projection - var exc3 = ClassicAssert.Throws(() => db.Execute("VADD", ["fizz", "REDUCE", "50", "VALUES", "5", "1.0", "2.0", "3.0", "4.0", "5.0", new byte[] { 0, 0, 0, 0 }, "CAS", "NOQUANT", "EF", "16", "M", "32"])); + var exc3 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", ["fizz", "REDUCE", "50", "VALUES", "5", "1.0", "2.0", "3.0", "4.0", "5.0", new byte[] { 0, 0, 0, 0 }, "CAS", "NOQUANT", "EF", "16", "M", "32"])); ClassicAssert.AreEqual("ERR REDUCE dimension must be <= vector dimensions", exc3.Message); } @@ -258,7 +258,7 @@ public void VADDXPREQB8() } // REDUCE not allowed with XPREQ8 - var exc1 = ClassicAssert.Throws(() => db.Execute("VADD", ["fizz", "REDUCE", "2", "XB8", smallVectorData, new byte[] { 0, 0, 0, 0 }, "XPREQ8"])); + var exc1 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", ["fizz", "REDUCE", "2", "XB8", smallVectorData, new byte[] { 0, 0, 0, 0 }, "XPREQ8"])); ClassicAssert.AreEqual("ERR asked quantization mismatch with existing vector set", exc1.Message); // Create a vector set with XB8 + XPREQ8 @@ -287,61 +287,61 @@ public void VADDErrors() var vectorSetKey = $"{nameof(VADDErrors)}_{Guid.NewGuid()}"; // Bad arity - var exc1 = ClassicAssert.Throws(() => db.Execute("VADD")); + var exc1 = TestUtils.ThrowsRedisException(() => db.Execute("VADD")); ClassicAssert.AreEqual("ERR wrong number of arguments for 'VADD' command", exc1.Message); - var exc2 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey])); + var exc2 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey])); ClassicAssert.AreEqual("ERR wrong number of arguments for 'VADD' command", exc2.Message); - var exc3 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "FP32"])); + var exc3 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "FP32"])); ClassicAssert.AreEqual("ERR wrong number of arguments for 'VADD' command", exc3.Message); - var exc4 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES"])); + var exc4 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES"])); ClassicAssert.AreEqual("ERR wrong number of arguments for 'VADD' command", exc4.Message); - var exc5 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1"])); + var exc5 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1"])); ClassicAssert.AreEqual("ERR wrong number of arguments for 'VADD' command", exc5.Message); - var exc6 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "1.0"])); + var exc6 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "1.0"])); ClassicAssert.AreEqual("ERR wrong number of arguments for 'VADD' command", exc6.Message); // Reduce after vector - var exc7 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "2", "1.0", "2.0", "bar", "REDUCE", "1"])); + var exc7 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "2", "1.0", "2.0", "bar", "REDUCE", "1"])); ClassicAssert.AreEqual("ERR invalid option after element", exc7.Message); // Duplicate flags // TODO: Redis doesn't error on these which seems... wrong, confirm with them - //var exc8 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "CAS", "CAS"])); - //var exc9 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "NOQUANT", "Q8"])); - //var exc10 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "EF", "1", "EF", "1"])); - //var exc11 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "SETATTR", "abc", "SETATTR", "abc"])); - //var exc12 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "M", "5", "M", "5"])); + //var exc8 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "CAS", "CAS"])); + //var exc9 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "NOQUANT", "Q8"])); + //var exc10 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "EF", "1", "EF", "1"])); + //var exc11 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "SETATTR", "abc", "SETATTR", "abc"])); + //var exc12 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "M", "5", "M", "5"])); // M out of range (Redis imposes M >= 4 and m <= 4096 - var exc13 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "M", "1"])); + var exc13 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "M", "1"])); ClassicAssert.AreEqual("ERR M must be an integer between 4 and 4096", exc13.Message); - var exc14 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "M", "10000"])); + var exc14 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "M", "10000"])); ClassicAssert.AreEqual("ERR M must be an integer between 4 and 4096", exc14.Message); // Missing/bad option value - var exc20 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "EF"])); + var exc20 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "EF"])); ClassicAssert.AreEqual("ERR invalid option after element", exc20.Message); - var exc21 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "EF", "0"])); + var exc21 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "EF", "0"])); ClassicAssert.AreEqual("ERR EF must be an integer between 1 and 1000000", exc21.Message); - var exc22 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "SETATTR"])); + var exc22 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "SETATTR"])); ClassicAssert.AreEqual("ERR invalid option after element", exc22.Message); - var exc23 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "M"])); + var exc23 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "M"])); ClassicAssert.AreEqual("ERR invalid option after element", exc23.Message); - var exc24 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "2", "2.0", "bar"])); + var exc24 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "2", "2.0", "bar"])); ClassicAssert.AreEqual("ERR invalid vector specification", exc24.Message); - var exc25 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "0", "bar"])); + var exc25 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "0", "bar"])); ClassicAssert.AreEqual("ERR invalid vector specification", exc25.Message); - var exc26 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "fizz", "bar"])); + var exc26 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "fizz", "bar"])); ClassicAssert.AreEqual("ERR invalid vector specification", exc26.Message); // Unknown option - var exc27 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "FOO"])); + var exc27 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "FOO"])); ClassicAssert.AreEqual("ERR invalid option after element", exc27.Message); // Malformed FP32 var binary = new float[] { 1, 2, 3 }; var blob = MemoryMarshal.Cast(binary)[..^1].ToArray(); - var exc15 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "FP32", blob, "bar"])); + var exc15 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "FP32", blob, "bar"])); ClassicAssert.AreEqual("ERR invalid vector specification", exc15.Message); // Mismatch after creating a vector set @@ -349,41 +349,41 @@ public void VADDErrors() _ = db.Execute("VADD", [vectorSetKey, "VALUES", "75", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", new byte[] { 0, 0, 1, 0 }, "NOQUANT", "EF", "6", "M", "10", "XDISTANCE_METRIC", "L2"]); - var exc16 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "2", "1.0", "2.0", "fizz", "NOQUANT", "EF", "6", "M", "10"])); + var exc16 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "2", "1.0", "2.0", "fizz", "NOQUANT", "EF", "6", "M", "10"])); ClassicAssert.AreEqual("ERR Vector dimension mismatch - got 2 but set has 75", exc16.Message); - var exc17 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "75", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "fizz", "XPREQ8", "EF", "6", "M", "10"])); + var exc17 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "75", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "fizz", "XPREQ8", "EF", "6", "M", "10"])); ClassicAssert.AreEqual("ERR asked quantization mismatch with existing vector set", exc17.Message); - var exc18 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "75", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "fizz", "NOQUANT", "EF", "12", "M", "20"])); + var exc18 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "75", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "fizz", "NOQUANT", "EF", "12", "M", "20"])); ClassicAssert.AreEqual("ERR asked M value mismatch with existing vector set", exc18.Message); // TODO: Redis doesn't appear to validate attributes... so that's weird // Empty Vector Set keys are forbidden (TODO: Remove this constraint) - var exc19 = ClassicAssert.Throws(() => db.Execute("VADD", ["", "VALUES", "75", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", new byte[] { 0, 0, 0, 0 }, "XPREQ8"])); + var exc19 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", ["", "VALUES", "75", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", new byte[] { 0, 0, 0, 0 }, "XPREQ8"])); ClassicAssert.AreEqual("ERR Vector Set key cannot be empty", exc19.Message); // Malformed XDISTANCE_METRIC - var exc31 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "75", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "bar", "NOQUANT", "XDISTANCE_METRIC"])); + var exc31 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "75", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "bar", "NOQUANT", "XDISTANCE_METRIC"])); ClassicAssert.AreEqual("ERR invalid option after element", exc31.Message); - var exc32 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "75", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "bar", "NOQUANT", "XDISTANCE_METRIC", "FOO"])); + var exc32 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "75", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "bar", "NOQUANT", "XDISTANCE_METRIC", "FOO"])); ClassicAssert.AreEqual("ERR invalid XDISTANCE_METRIC", exc32.Message); // Invalid vector type keyword (not FP32, VALUES, or XB8) - var exc40 = ClassicAssert.Throws(() => db.Execute("VADD", ["mykey", "GARBAGE", "data", "elem1"])); + var exc40 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", ["mykey", "GARBAGE", "data", "elem1"])); ClassicAssert.AreEqual("ERR invalid vector specification", exc40.Message); // VALUES count exceeding MaxVectorDimensions (65536) must be rejected - var exc41 = ClassicAssert.Throws(() => db.Execute("VADD", ["foo", "VALUES", "100000", "1.0", "elem"])); + var exc41 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", ["foo", "VALUES", "100000", "1.0", "elem"])); ClassicAssert.IsTrue(exc41.Message.Contains("maximum"), $"Expected dimension limit error, got: {exc41.Message}"); // EF exceeding MaxExplorationFactor (1,000,000) must be rejected - var exc42 = ClassicAssert.Throws(() => db.Execute("VADD", ["foo", "VALUES", "3", "1.0", "2.0", "3.0", new byte[] { 0, 0, 0, 0 }, "CAS", "NOQUANT", "EF", "2000000000", "M", "32"])); + var exc42 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", ["foo", "VALUES", "3", "1.0", "2.0", "3.0", new byte[] { 0, 0, 0, 0 }, "CAS", "NOQUANT", "EF", "2000000000", "M", "32"])); ClassicAssert.IsTrue(exc42.Message.Contains("EF must be an integer between"), $"Expected EF validation error, got: {exc42.Message}"); // REDUCE dim exceeding vector dimensions must be rejected - var exc43 = ClassicAssert.Throws(() => db.Execute("VADD", ["foo", "REDUCE", "100000", "VALUES", "3", "1.0", "2.0", "3.0", new byte[] { 0, 0, 0, 0 }, "CAS", "NOQUANT", "EF", "16", "M", "32"])); + var exc43 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", ["foo", "REDUCE", "100000", "VALUES", "3", "1.0", "2.0", "3.0", new byte[] { 0, 0, 0, 0 }, "CAS", "NOQUANT", "EF", "16", "M", "32"])); ClassicAssert.IsTrue(exc43.Message.Contains("REDUCE dimension must be <= vector dimensions"), $"Expected REDUCE dimension limit error, got: {exc43.Message}"); - var exc33 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "75", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "bar", "NOQUANT", "XDISTANCE_METRIC", "XCOSINE_NORMALIZED"])); + var exc33 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "75", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "bar", "NOQUANT", "XDISTANCE_METRIC", "XCOSINE_NORMALIZED"])); ClassicAssert.AreEqual("ERR Distance metric mismatch - got XCosine_Normalized but set has L2", exc33.Message); } @@ -437,7 +437,7 @@ public void VectorSetOpacity() var res1 = db.Execute("VADD", ["foo", "REDUCE", "50", "VALUES", "75", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", new byte[] { 0, 0, 0, 0 }, "CAS", "NOQUANT", "EF", "16", "M", "32"]); ClassicAssert.AreEqual(1, (int)res1); - var res2 = ClassicAssert.Throws(() => db.StringGet("foo")); + var res2 = TestUtils.ThrowsRedisException(() => db.StringGet("foo")); ClassicAssert.True(res2.Message.Contains("WRONGTYPE")); } @@ -793,7 +793,7 @@ public void VDIM() var res4 = db.Execute("VDIM", "bar"); ClassicAssert.AreEqual(75, (int)res4); - var exc1 = ClassicAssert.Throws(() => db.Execute("VDIM", "fizz")); + var exc1 = TestUtils.ThrowsRedisException(() => db.Execute("VDIM", "fizz")); ClassicAssert.IsTrue(exc1.Message.Contains("Key not found")); // TODO: Add WRONGTYPE behavior check once implemented @@ -1057,7 +1057,7 @@ public void VSIMBadFilters() foreach (var (filter, why) in badFilters) { - var exc = ClassicAssert.Throws( + var exc = TestUtils.ThrowsRedisException( () => db.Execute("VSIM", [VectorSet, "VALUES", "3", "0.0", "0.0", "0.0", "FILTER", filter, "COUNT", "10"]), $"Expected compile failure for filter '{filter}' ({why})"); ClassicAssert.AreEqual(CompileErr, exc.Message, $"Wrong error message for filter '{filter}' ({why})"); @@ -1246,19 +1246,19 @@ public void VSIMErrors() ClassicAssert.AreEqual(1, (int)res1); // FILTER-EF exceeding MaxFilteringScaleFactor must be rejected - var exc1 = ClassicAssert.Throws(() => db.Execute("VSIM", ["foo", "VALUES", "3", "0.0", "0.0", "0.0", "FILTER", ".year > 1950", "FILTER-EF", "999999999", "COUNT", "3", "WITHATTRIBS"])); + var exc1 = TestUtils.ThrowsRedisException(() => db.Execute("VSIM", ["foo", "VALUES", "3", "0.0", "0.0", "0.0", "FILTER", ".year > 1950", "FILTER-EF", "999999999", "COUNT", "3", "WITHATTRIBS"])); ClassicAssert.AreEqual("ERR FILTER-EF must be an integer between 4 and 256", exc1.Message); // COUNT exceeding MaxRetrieveCount must be rejected - var exc2 = ClassicAssert.Throws(() => db.Execute("VSIM", ["foo", "VALUES", "3", "0.0", "0.0", "0.0", "COUNT", "999999999"])); + var exc2 = TestUtils.ThrowsRedisException(() => db.Execute("VSIM", ["foo", "VALUES", "3", "0.0", "0.0", "0.0", "COUNT", "999999999"])); ClassicAssert.AreEqual("ERR COUNT must be an integer between 0 and 100000000", exc2.Message); // VALUES count exceeding MaxVectorDimensions (65536) must be rejected - var exc3 = ClassicAssert.Throws(() => db.Execute("VSIM", ["foo", "VALUES", "100000", "1.0"])); + var exc3 = TestUtils.ThrowsRedisException(() => db.Execute("VSIM", ["foo", "VALUES", "100000", "1.0"])); ClassicAssert.AreEqual("ERR vector exceeds maximum of 65536 dimensions", exc3.Message); // EF exceeding MaxExplorationFactor (1,000,000) must be rejected - var exc4 = ClassicAssert.Throws(() => db.Execute("VSIM", ["foo", "VALUES", "3", "0.0", "0.0", "0.0", "EF", "2000000000"])); + var exc4 = TestUtils.ThrowsRedisException(() => db.Execute("VSIM", ["foo", "VALUES", "3", "0.0", "0.0", "0.0", "EF", "2000000000"])); ClassicAssert.AreEqual("ERR EF must be an integer between 1 and 1000000", exc4.Message); } @@ -1699,7 +1699,7 @@ public void RepeatedVectorSetDeletes() var addRes2 = (int)db.Execute("VADD", ["foo", "XB8", bytes2, new byte[] { 0, 0, 0, 1 }, "XPREQ8"]); ClassicAssert.AreEqual(1, addRes2); - var readExc = ClassicAssert.Throws(() => db.Execute("GET", ["foo"])); + var readExc = TestUtils.ThrowsRedisException(() => db.Execute("GET", ["foo"])); ClassicAssert.IsTrue(readExc.Message.Equals("WRONGTYPE Operation against a key holding the wrong kind of value."), $"In iteration: {i}"); } @@ -2798,7 +2798,7 @@ public void VREM() // Remove on non-vector set fails // TODO: test against Redis, how do they respond (I expect WRONGTYPE, but needs verification) //_ = db.StringSet("fizz", "buzz"); - //var exc1 = ClassicAssert.Throws(() => db.Execute("VREM", "fizz", new byte[] { 0, 0, 0, 0 })); + //var exc1 = TestUtils.ThrowsRedisException(() => db.Execute("VREM", "fizz", new byte[] { 0, 0, 0, 0 })); //ClassicAssert.AreEqual("", exc1.Message); // Remove exists diff --git a/test/standalone/Garnet.test.vectorset/VectorSetOverwriteTests.cs b/test/standalone/Garnet.test.vectorset/VectorSetOverwriteTests.cs index 0a79d160cd4..8fa557ecbf1 100644 --- a/test/standalone/Garnet.test.vectorset/VectorSetOverwriteTests.cs +++ b/test/standalone/Garnet.test.vectorset/VectorSetOverwriteTests.cs @@ -136,12 +136,11 @@ static async Task RunCommandAsync(IDatabaseAsync executeDB, IDatabaseAsync readD } [Test] - public async Task SETAsync() + public Task SETAsync() { - await TestVectorSetOverwrittenCommandAsync(RunCommandPlainAsync).ConfigureAwait(false); - await TestVectorSetOverwrittenCommandAsync(RunCommandEXAsync).ConfigureAwait(false); + return TestVectorSetOverwrittenCommandAsync(RunCommandAsync); - static async Task RunCommandPlainAsync(IDatabaseAsync executeDB, IDatabaseAsync readDB, RedisKey againstKey) + static async Task RunCommandAsync(IDatabaseAsync executeDB, IDatabaseAsync readDB, RedisKey againstKey) { var res = await executeDB.StringSetAsync(againstKey, "foo").ConfigureAwait(false); ClassicAssert.IsTrue(res); @@ -149,8 +148,14 @@ static async Task RunCommandPlainAsync(IDatabaseAsync executeDB, IDatabaseAsync var finalValue = await readDB.StringGetAsync(againstKey).ConfigureAwait(false); ClassicAssert.AreEqual("foo", finalValue); } + } + + [Test] + public Task SETKeepTtlAsync() + { + return TestVectorSetOverwrittenCommandAsync(RunCommandAsync); - static async Task RunCommandEXAsync(IDatabaseAsync executeDB, IDatabaseAsync readDB, RedisKey againstKey) + static async Task RunCommandAsync(IDatabaseAsync executeDB, IDatabaseAsync readDB, RedisKey againstKey) { var res = (string)await executeDB.ExecuteAsync("SET", againstKey, "foo", "KEEPTTL").ConfigureAwait(false); ClassicAssert.AreEqual("OK", res); diff --git a/test/standalone/Garnet.test.vectorset/VectorSetRecoveredContextReservationTests.cs b/test/standalone/Garnet.test.vectorset/VectorSetRecoveredContextReservationTests.cs new file mode 100644 index 00000000000..b12056bdfbb --- /dev/null +++ b/test/standalone/Garnet.test.vectorset/VectorSetRecoveredContextReservationTests.cs @@ -0,0 +1,137 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT license. + +using System.Collections.Generic; +using System.Linq; +using System.Threading.Tasks; +using NUnit.Framework; +using NUnit.Framework.Legacy; +using StackExchange.Redis; + +namespace Garnet.test +{ + /// + /// A Vector Set's index record is written before the context metadata that reserves its context, and + /// the metadata is only flushed once index creation succeeds. A checkpoint taken between the two + /// captures a live index record whose context is not marked in use, so recovery has to restore that + /// reservation from the index records themselves - otherwise the free-list hands the same context to + /// the next Vector Set created and the two share a namespace. + /// + [TestFixture] + public class VectorSetRecoveredContextReservationTests : TestBase + { + private global::Garnet.GarnetServer server; + + [SetUp] + public void Setup() + { + TestUtils.DeleteDirectory(TestUtils.MethodTestDir, wait: true); + } + + [TearDown] + public void TearDown() + { + server?.Dispose(); + server = null; + + TestUtils.DeleteDirectory(TestUtils.MethodTestDir); + TestUtils.OnTearDown(); + } + + private void StartServer(bool tryRecover) + { + server = TestUtils.CreateGarnetServer( + TestUtils.MethodTestDir, + memorySize: "8m", + pageSize: "16k", + enableAOF: true, + aofMemorySize: "2g", + tryRecover: tryRecover, + enableVectorSetPreview: true); + server.Start(); + } + + [Test] + public async Task RecoveredVectorSetDoesNotShareContextWithNewVectorSetAsync() + { + const string Recovered = "recovered-vs"; + const string Fresh = "fresh-vs"; + const int Elements = 500; + const int Dim = 32; + + StartServer(tryRecover: false); + + using (var redis = ConnectionMultiplexer.Connect(TestUtils.GetConfig(allowAdmin: true))) + { + var db = redis.GetDatabase(0); + + for (var i = 0; i < Elements; i++) + { + ClassicAssert.AreEqual(1, (int)db.Execute("VADD", BuildVaddArgs(Recovered, Dim, i, $"recovered_{i}"))); + } + +#pragma warning disable CS0618 // ForegroundSave is obsolete but is what the recovery tests use + redis.GetServers().Single().Save(SaveType.ForegroundSave); +#pragma warning restore CS0618 + + var committed = await server.Store.WaitForCommitAsync(); + ClassicAssert.IsTrue(committed, "checkpoint commit did not complete"); + } + + server.Dispose(deleteDir: false); + StartServer(tryRecover: true); + + using (var redis = ConnectionMultiplexer.Connect(TestUtils.GetConfig(allowAdmin: true))) + { + var db = redis.GetDatabase(0); + + ClassicAssert.AreEqual(Elements, (int)db.Execute("VCARD", [Recovered]), + "the recovered Vector Set lost elements during recovery"); + + // Creating a Vector Set now must not be handed the recovered set's context. If it is, the two + // share a namespace and writes to one are visible in - and destroy - the other. + for (var i = 0; i < Elements; i++) + { + ClassicAssert.AreEqual(1, (int)db.Execute("VADD", BuildVaddArgs(Fresh, Dim, i + Elements, $"fresh_{i}"))); + } + + ClassicAssert.AreEqual(Elements, (int)db.Execute("VCARD", [Fresh]), + "the newly created Vector Set did not receive all of its elements"); + + ClassicAssert.AreEqual(Elements, (int)db.Execute("VCARD", [Recovered]), + "creating a new Vector Set after recovery changed the recovered Vector Set's cardinality, " + + "so the two were handed the same context"); + + var simArgs = new List { Recovered, "VALUES", $"{Dim}" }; + for (var d = 0; d < Dim; d++) + { + simArgs.Add($"{((0 % 7) + 1) * (d + 1) % 13}"); + } + simArgs.Add("COUNT"); + simArgs.Add($"{Elements}"); + + var recoveredMembers = ((RedisResult[])db.Execute("VSIM", simArgs.ToArray())) + .Select(static r => (string)r) + .ToArray(); + + CollectionAssert.IsNotEmpty(recoveredMembers); + CollectionAssert.IsSubsetOf( + recoveredMembers, + Enumerable.Range(0, Elements).Select(static i => $"recovered_{i}").ToArray(), + "the recovered Vector Set returned elements belonging to the Vector Set created after recovery"); + } + } + + private static object[] BuildVaddArgs(string key, int dim, int seed, string element) + { + var args = new List { key, "VALUES", $"{dim}" }; + for (var d = 0; d < dim; d++) + { + args.Add($"{((seed % 7) + 1) * (d + 1) % 13}"); + } + args.Add(element); + + return args.ToArray(); + } + } +} \ No newline at end of file diff --git a/test/standalone/Garnet.test/TestUtils.cs b/test/standalone/Garnet.test/TestUtils.cs index 203c3ea7a4a..54427dc6546 100644 --- a/test/standalone/Garnet.test/TestUtils.cs +++ b/test/standalone/Garnet.test/TestUtils.cs @@ -12,6 +12,7 @@ using System.Net.Security; using System.Net.Sockets; using System.Runtime.CompilerServices; +using System.Runtime.ExceptionServices; using System.Security.Cryptography; using System.Security.Cryptography.X509Certificates; using System.Text; @@ -279,6 +280,63 @@ public static void WaitUntilNextSecond(IDatabase db, long baseSeconds) } } + /// + /// Asserts that throws , running it on a dedicated thread. + /// + /// + /// StackExchange.Redis' synchronous request path (ConnectionMultiplexer.ExecuteSyncImpl) parks the caller on a + /// [ThreadStatic] result box. A server error reply faults that box from the connection's reader thread, and when + /// the fault lands before the caller reaches Monitor.Wait the caller skips the wait and recycles the box while the + /// reader thread still has a Monitor.PulseAll outstanding on it. The next synchronous call on the same thread + /// takes the recycled box, consumes that stale pulse, and returns with neither a result nor an exception, shifting every + /// subsequent reply on that thread by one. Running calls that are expected to fail on their own thread keeps the + /// recycled box off the shared test thread. + /// + /// Expected exception type. + /// Delegate performing the call that is expected to fail. + /// The exception thrown by . + public static TActual ThrowsRedisException(TestDelegate code) where TActual : Exception + => ClassicAssert.Throws(CaptureOnDedicatedThread(code)); + + /// + /// Variant of that reports on failure. + /// + /// Expected exception type. + /// Delegate performing the call that is expected to fail. + /// Message reported when no matching exception is thrown. + /// Format arguments for . + /// The exception thrown by . + public static TActual ThrowsRedisException(TestDelegate code, string message, params object[] args) where TActual : Exception + => ClassicAssert.Throws(CaptureOnDedicatedThread(code), message, args); + + /// + /// Runs to completion on a dedicated thread, returning a delegate that replays whatever it threw. + /// + /// Delegate to run. + /// Delegate that rethrows the captured exception, preserving its original stack trace. + private static TestDelegate CaptureOnDedicatedThread(TestDelegate code) + { + ExceptionDispatchInfo captured = null; + + var thread = new Thread(() => + { + try + { + code(); + } + catch (Exception ex) + { + captured = ExceptionDispatchInfo.Capture(ex); + } + }) + { IsBackground = true }; + + thread.Start(); + thread.Join(); + + return () => captured?.Throw(); + } + /// /// Create GarnetServer /// From fd34fb2923441dedba7d7cff76076337d35b82bf Mon Sep 17 00:00:00 2001 From: Badrish Chandramouli Date: Tue, 8 Sep 2026 19:29:12 -0700 Subject: [PATCH 02/27] Fix replication teardown races that hang server dispose and poison CI fixtures AofSyncDriver.Dispose() disposed each AOF sync task's GarnetClientSession while its worker task was still running. GarnetClientSession documents that it "expects mono-threaded client access, i.e., no concurrent invocations of API by client", so this is a contract violation, and it is the dominant source of flakiness in the cluster suites. When the disposing thread wins the race, the worker has just published its send frame (SendResponse nulls responseObject before sending) and has not yet reacquired one. The disposer's ReturnResponseObject() therefore returns nothing, and the worker then calls GetResponseObject(): LightConcurrentStack.TryPop samples disposed==false on an empty stack, releases the latch, and the caller allocates a fresh GarnetSaeaBuffer renting from the replication pool. The disposer disposes the stack in that gap, the worker dies at its next Throttle() check without returning the frame, and its second Dispose() returns immediately on the disposed counter. The buffer is stranded. LimitedFixedBufferPool.Dispose() then spin-waits forever for totalReferences to reach zero, so ReplicationManager.Dispose() never returns, the node never releases its port, and every later test in the fixture fails on "Failed to connect within 30 seconds". CI shows exactly this: one unreturned 64 MB Replication SaeaSendBuffer (2 << AofPageSizeBits, i.e. an AOF sync buffer) logged immediately before "Timed out waiting for DisposeCluster", followed by 26 consecutive failures in the same job. Fix the ordering rather than making the sender thread-safe, which would paper over the contract violation and add synchronization to the send path: - AofSyncTask.RunAofSyncTaskAsync disposes its client before leaving the active worker monitor, so a drained monitor implies every client was torn down by the one thread that used it and its buffer is back in the pool. - AofSyncDriver.Dispose() breaks the connections instead of disposing the sessions, waits for the workers to quiesce, and only then disposes the tasks. The task dispose is kept so a client connected without a running worker still returns its buffer, and it also makes the iterator disposal race-free. - GarnetClientSession.CloseConnection() closes only the socket, which is what unblocks a worker parked on a send without touching the buffer it owns. This is confined to teardown; no hot path changes. Verified by modelling the acquire-then-abandon sequence directly against GarnetTcpNetworkSender: when a foreign thread disposes the sender while the worker holds a rented frame, LimitedFixedBufferPool.Dispose() never completes; when the owning thread returns its own frame, the pool drains immediately. Also fix RespTests.ClientKillTestAsync, which asserted IsConnected was false after a single fire-and-forget Ping. Socket.Connected reflects the last completed I/O, and TCP guarantees the first send after a remote FIN succeeds into the kernel buffer, so the assertion could only pass by winning a race with the client's receive loop. It now pings until the disconnect surfaces, bounded at 10s, so a CLIENT KILL that genuinely failed still fails the test. Validated on 2 cores (taskset -c 0-1) to match the CI runners: Garnet.test.cluster.multilog 105 passed / 12 skipped / 0 failed, with no DisposeCluster timeout, no buffer pool dispose diagnostic and no leaked epochs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 30f31d5b-9e53-4111-91a3-e16c94510a77 --- .../ClientSession/GarnetClientSession.cs | 9 ++++++++ .../PrimaryOps/AofOperations/AofSyncDriver.cs | 14 ++++++++++--- .../PrimaryOps/AofOperations/AofSyncTask.cs | 18 +++++++++++++++- test/standalone/Garnet.test/RespTests.cs | 21 +++++++++++++++---- 4 files changed, 54 insertions(+), 8 deletions(-) diff --git a/libs/client/ClientSession/GarnetClientSession.cs b/libs/client/ClientSession/GarnetClientSession.cs index 7a1e988da0a..1cb7b9ee442 100644 --- a/libs/client/ClientSession/GarnetClientSession.cs +++ b/libs/client/ClientSession/GarnetClientSession.cs @@ -407,6 +407,15 @@ public Task ReconnectAsync(int timeoutMs = 0, CancellationToken token = default) return ConnectAsync(timeoutMs, token); } + /// + /// Closes the underlying connection so a thread blocked on network I/O through this session + /// fails fast, without disposing the session itself. The session expects mono-threaded access, + /// so a caller that is not the thread using the session must not dispose it: doing so races + /// that thread's use of the send buffer it has rented. Closing the connection is safe because + /// it only invalidates the socket, leaving the buffer owned by the thread that rented it. + /// + public void CloseConnection() => socket?.Dispose(); + /// /// Dispose instance /// diff --git a/libs/cluster/Server/Replication/PrimaryOps/AofOperations/AofSyncDriver.cs b/libs/cluster/Server/Replication/PrimaryOps/AofOperations/AofSyncDriver.cs index abe18580540..8fb702ee452 100644 --- a/libs/cluster/Server/Replication/PrimaryOps/AofOperations/AofSyncDriver.cs +++ b/libs/cluster/Server/Replication/PrimaryOps/AofOperations/AofSyncDriver.cs @@ -121,13 +121,21 @@ public void Dispose() // Cancel cts cts?.Cancel(); - // Dispose sync tasks + // Break the connections so a task blocked on a send fails fast. The client sessions are + // deliberately not disposed here: they expect mono-threaded access, and a task may still + // be writing through one. Disposing it from this thread races that write and can strand + // the send buffer the task rented, which leaves the replication buffer pool's dispose + // spinning forever and the whole server stuck in shutdown holding its port. foreach (var aofSyncTask in aofSyncTasks) - aofSyncTask?.Dispose(); + aofSyncTask?.CloseConnection(); - // Wait for tasks to exit + // Wait for tasks to exit; each disposes its own client before leaving the monitor activeWorkerMonitor.Dispose(); + // Dispose sync tasks, now that no task is running + foreach (var aofSyncTask in aofSyncTasks) + aofSyncTask?.Dispose(); + // Finally, dispose the cts cts?.Dispose(); } diff --git a/libs/cluster/Server/Replication/PrimaryOps/AofOperations/AofSyncTask.cs b/libs/cluster/Server/Replication/PrimaryOps/AofOperations/AofSyncTask.cs index 9e8915007db..be464afea0f 100644 --- a/libs/cluster/Server/Replication/PrimaryOps/AofOperations/AofSyncTask.cs +++ b/libs/cluster/Server/Replication/PrimaryOps/AofOperations/AofSyncTask.cs @@ -143,6 +143,19 @@ public AofSyncTask( this.logger = logger; } + /// + /// Closes the network connection so a task blocked on a send fails fast, without + /// disposing the client session the task may still be writing through. + /// + public void CloseConnection() + { + try + { + garnetClient?.CloseConnection(); + } + catch { } + } + public void Dispose() { try @@ -344,9 +357,12 @@ await iter.BulkConsumeAllAsync( } finally { + // The client is disposed before leaving the monitor so that a drained monitor + // means every client has already been torn down by the one thread that was + // using it, and the send buffer it rented is back in the replication pool. + garnetClient?.Dispose(); if (enteredMonitor) _ = aofSyncDriver.activeWorkerMonitor.Exit(); - garnetClient?.Dispose(); } [Conditional("DEBUG")] diff --git a/test/standalone/Garnet.test/RespTests.cs b/test/standalone/Garnet.test/RespTests.cs index ed1b2c60576..d7cac3cb456 100644 --- a/test/standalone/Garnet.test/RespTests.cs +++ b/test/standalone/Garnet.test/RespTests.cs @@ -4657,12 +4657,25 @@ static async Task ConnectAsync() // Check that we really killed the connection backing a GarnetClient static void AssertNotConnected(GarnetClient client) { - // Force the issue by attempting a command - try + // IsConnected reads Socket.Connected, which reports the state as of the last completed + // I/O. Once the server closes the connection the first send still succeeds locally and + // only a following one observes the reset, so a single ping cannot establish that the + // connection is gone. Keep issuing pings until the disconnect surfaces; a connection + // that was not killed keeps answering them and still fails the assert below. + var elapsed = Stopwatch.StartNew(); + while (client.IsConnected && elapsed.Elapsed < TimeSpan.FromSeconds(10)) { - client.Ping(static (_, __) => { }); + try + { + client.Ping(static (_, __) => { }); + } + catch + { + break; + } + + Thread.Sleep(10); } - catch { } ClassicAssert.IsFalse(client.IsConnected); } From 12b02303b2259742c6fe31b158e3a98c05b8d9c1 Mon Sep 17 00:00:00 2001 From: Badrish Chandramouli Date: Tue, 8 Sep 2026 20:13:16 -0700 Subject: [PATCH 03/27] Close the connect-time window where a connection break is lost AofSyncDriver.Dispose() relies on CloseConnection() to fail a worker's pending network I/O so the worker exits and the driver's wait for it completes. That signal could be lost: CloseConnection() reads the socket field, and a worker inside ConnectAsync has not published it yet. The window is not narrow. The field is written in the continuation of `socket = await ConnectSendSocketAsync(...)`, which on a loaded 2-core runner can be delayed arbitrarily. ConnectSendSocketAsync honours the token, but nothing after it does on the non-TLS path: NetworkHandler.StartAsync returns immediately when TLS is off, and the AUTH, CLIENT SETINFO and CLIENT SETNAME exchanges that follow await a reply with no cancellation token. A worker that lost the break therefore rents its send buffer and parks forever on a peer that is also shutting down, so the driver's wait never completes and the pool dispose spins. Make it a handshake: CloseConnection() records the request before closing, and the connecting thread rechecks it immediately after publishing the socket, so either the closer observes the socket or the connector observes the request. The recheck sits before the network handler is constructed and before the send buffer is rented, so the throw path allocates nothing. Both sides need a StoreLoad fence for the handshake to hold. Publishing the socket and reading the request are a store then a load of a different location, which the acquire load does not order, so without the barrier both sides can read stale values and the break is lost anyway. Verified on 2 cores (taskset -c 0-1): Garnet.test.cluster.multilog 105 passed / 12 skipped / 0 failed, Garnet.test.cluster.replication 107/107, with no DisposeCluster timeout and no buffer pool dispose diagnostic. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 30f31d5b-9e53-4111-91a3-e16c94510a77 --- .../ClientSession/GarnetClientSession.cs | 35 ++++++++++++++++++- 1 file changed, 34 insertions(+), 1 deletion(-) diff --git a/libs/client/ClientSession/GarnetClientSession.cs b/libs/client/ClientSession/GarnetClientSession.cs index 1cb7b9ee442..17e70b709e9 100644 --- a/libs/client/ClientSession/GarnetClientSession.cs +++ b/libs/client/ClientSession/GarnetClientSession.cs @@ -42,6 +42,8 @@ public sealed partial class GarnetClientSession : IServerHook, IMessageConsumer static readonly Exception disposeException = new GarnetClientDisposedException(); Socket socket; + + int closeRequested; int disposed; // Send @@ -147,6 +149,7 @@ public GarnetClientSession( public unsafe void Connect(CancellationToken token = default) { socket = ConnectSendSocket(); + ThrowIfCloseRequested(); networkHandler = new GarnetClientSessionTcpNetworkHandler( this, socket, @@ -199,6 +202,7 @@ public unsafe void Connect(CancellationToken token = default) public async Task ConnectAsync(int timeoutMs = 0, CancellationToken token = default) { socket = await ConnectSendSocketAsync(timeoutMs, token).ConfigureAwait(false); + ThrowIfCloseRequested(); networkHandler = new GarnetClientSessionTcpNetworkHandler( this, socket, @@ -413,8 +417,37 @@ public Task ReconnectAsync(int timeoutMs = 0, CancellationToken token = default) /// so a caller that is not the thread using the session must not dispose it: doing so races /// that thread's use of the send buffer it has rented. Closing the connection is safe because /// it only invalidates the socket, leaving the buffer owned by the thread that rented it. + /// + /// The session is not reusable afterwards: the request is sticky, so a later + /// or throws. + /// + public void CloseConnection() + { + _ = Interlocked.Exchange(ref closeRequested, 1); + Volatile.Read(ref socket)?.Dispose(); + } + + /// + /// A close that lands while the connection is still being established would otherwise be lost: + /// the socket is not yet published for to find, and the exchanges + /// that follow await a reply with no cancellation token, so the connecting thread would park + /// indefinitely on a peer that is shutting down. Checking here after the socket is published + /// means either the closer observes the socket or the connecting thread observes the request. /// - public void CloseConnection() => socket?.Dispose(); + void ThrowIfCloseRequested() + { + // Publishing the socket and reading the request are a store followed by a load of a + // different location, so without a StoreLoad fence both sides can miss: this thread's + // store is still buffered while it reads a stale request, and the closer reads a stale + // socket. The closer's Interlocked.Exchange fences its own side; this fences ours. + Interlocked.MemoryBarrier(); + + if (Volatile.Read(ref closeRequested) == 0) + return; + + socket?.Dispose(); + throw new ObjectDisposedException(nameof(GarnetClientSession), "Connection was closed while connecting"); + } /// /// Dispose instance From fb400bfbbb07190a4b42d793152e42141faf98fc Mon Sep 17 00:00:00 2001 From: Badrish Chandramouli Date: Wed, 9 Sep 2026 15:22:56 -0700 Subject: [PATCH 04/27] Keep the worker monitor exit unconditional in the AOF sync task Disposing the client before leaving the monitor means the monitor exit is now downstream of a call that can throw. A throw there would leave the worker counted as active forever, so AofSyncDriver.Dispose would block indefinitely waiting for the monitor to drain -- turning a client teardown failure into a hang of the whole replication dispose path. Wrap the client dispose so the exit always runs, and log the failure. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 30f31d5b-9e53-4111-91a3-e16c94510a77 --- .../PrimaryOps/AofOperations/AofSyncTask.cs | 24 ++++++++++++++----- 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/libs/cluster/Server/Replication/PrimaryOps/AofOperations/AofSyncTask.cs b/libs/cluster/Server/Replication/PrimaryOps/AofOperations/AofSyncTask.cs index be464afea0f..6b2f2217510 100644 --- a/libs/cluster/Server/Replication/PrimaryOps/AofOperations/AofSyncTask.cs +++ b/libs/cluster/Server/Replication/PrimaryOps/AofOperations/AofSyncTask.cs @@ -357,12 +357,24 @@ await iter.BulkConsumeAllAsync( } finally { - // The client is disposed before leaving the monitor so that a drained monitor - // means every client has already been torn down by the one thread that was - // using it, and the send buffer it rented is back in the replication pool. - garnetClient?.Dispose(); - if (enteredMonitor) - _ = aofSyncDriver.activeWorkerMonitor.Exit(); + try + { + // The client is disposed before leaving the monitor so that a drained monitor + // means every client has already been torn down by the one thread that was + // using it, and the send buffer it rented is back in the replication pool. + garnetClient?.Dispose(); + } + catch (Exception ex) + { + // Leaving the monitor is what releases the dispose that is waiting on this + // task, so it has to happen even if tearing the client down fails. + logger?.LogError(ex, "[{sublogIdx}]({method}) failed to dispose client", physicalSublogIdx, nameof(RunAofSyncTaskAsync)); + } + finally + { + if (enteredMonitor) + _ = aofSyncDriver.activeWorkerMonitor.Exit(); + } } [Conditional("DEBUG")] From 4a1857230b99d2d0c352fab8e7b7d11c5a10054b Mon Sep 17 00:00:00 2001 From: Badrish Chandramouli Date: Wed, 9 Sep 2026 17:28:55 -0700 Subject: [PATCH 05/27] Stop logging routine background-task cancellation as Error/Critical CommitTaskAsync and CompactionTaskAsync (StoreWrapper) and MainMonitorTaskAsync (GarnetServerMonitor) caught every exception and logged it at Error/Critical, including the OperationCanceledException that is raised as part of normal operation when their own token is cancelled. Their sibling tasks, AutoCheckpointTask and ObjectCollectTaskAsync, already suppress exactly this case. The cancellation is expected on two routine paths: store wrapper disposal, and SuspendPrimaryOnlyTasksAsync when a node transitions to a replica via CLUSTER REPLICATE. Cluster tests therefore emitted a burst of Error/Critical entries on every replica transition and every server teardown. Measured over the PrimaryUnavailableRecoveryAsync fixture (6 cases, 2 cores): 42 "CompactionTask exception received" plus 36 "MainMonitorTask exception" entries per run before, 0 after, with identical pass rate and timings. Since cluster tests log at LogLevel.Error, these were most of what the CI logs contained, which made genuine failures in this suite very hard to diagnose. The catch filter is guarded on token.IsCancellationRequested, so a cancellation originating from any other source is still reported. --- libs/server/Metrics/GarnetServerMonitor.cs | 4 ++++ libs/server/StoreWrapper.cs | 10 ++++++++++ 2 files changed, 14 insertions(+) diff --git a/libs/server/Metrics/GarnetServerMonitor.cs b/libs/server/Metrics/GarnetServerMonitor.cs index 4f03adcb9c0..c6e39d5ad83 100644 --- a/libs/server/Metrics/GarnetServerMonitor.cs +++ b/libs/server/Metrics/GarnetServerMonitor.cs @@ -309,6 +309,10 @@ private async Task MainMonitorTaskAsync(CancellationToken token) CleanupGlobalLatencyMetrics(); } } + catch (OperationCanceledException) when (token.IsCancellationRequested) + { + // Suppress the exception if the task was cancelled because the monitor was disposed + } catch (Exception ex) { logger?.LogCritical(ex, "MainMonitorTask exception"); diff --git a/libs/server/StoreWrapper.cs b/libs/server/StoreWrapper.cs index 5e81d8be9fc..b7aa164b98d 100644 --- a/libs/server/StoreWrapper.cs +++ b/libs/server/StoreWrapper.cs @@ -688,6 +688,11 @@ async Task CommitTaskAsync(int commitFrequencyMs, CancellationToken token = defa } } } + catch (OperationCanceledException) when (token.IsCancellationRequested) + { + // Suppress the exception if the task was cancelled because the store wrapper was disposed + // or because the node transitioned to a replica and primary-only tasks were suspended + } catch (Exception ex) { logger?.LogError(ex, "CommitTask exception received."); @@ -713,6 +718,11 @@ async Task CompactionTaskAsync(int compactionFrequencySecs, CancellationToken to await Task.Delay(compactionFrequencySecs * 1000, token).ConfigureAwait(false); } } + catch (OperationCanceledException) when (token.IsCancellationRequested) + { + // Suppress the exception if the task was cancelled because the store wrapper was disposed + // or because the node transitioned to a replica and primary-only tasks were suspended + } catch (Exception ex) { logger?.LogError(ex, "CompactionTask exception received."); From aee38ffd0444e914e078b2eb029f4138e3d07715 Mon Sep 17 00:00:00 2001 From: Badrish Chandramouli Date: Wed, 9 Sep 2026 18:33:06 -0700 Subject: [PATCH 06/27] Keep a node gossiping after a single gossip round fails GossipMainAsync ran InitConnectionsAsync and the configuration broadcast directly inside the loop's try block, so any exception raised by a round left the loop, disposed the cluster connection store and decremented the active task count. TryStartGossipTasks is called once, from ClusterManager.Start, and nothing supervises or restarts the task, so the node stopped gossiping for the remainder of the process. The node keeps answering gossip it receives, so it still looks healthy, and the only trace is one Warning. Cluster tests log at LogLevel.Error, so in CI this is completely silent. What follows is a cluster that can no longer converge on any subsequent configuration change: WaitForSyncAsync polls until the test token expires and reports "Cancellation Requested", with nothing in the log between the last setup message and teardown. A round failing is transient by nature: it covers connection setup, including the TLS handshake, and a broadcast to every peer, each with its own timeout. Losing gossip permanently is not a reasonable response to one of those failing. Failures raised while the round runs are now logged and the loop continues at the next gossip interval. Cancellation still leaves the loop, and is matched on OperationCanceledException rather than only TaskCanceledException, so the cancellation raised while disposing banned worker connections is no longer reported as an error during shutdown. ClusterGossipSurvivesFailedRound covers this: it introduces the nodes, fails one round on every node, and then assigns slots, which can only propagate if the loops are still running. Reverting to the previous behaviour fails the test after 30s with "node 0 never converged after a gossip round failed". --- libs/cluster/Server/Gossip/Gossip.cs | 26 +++++--- libs/common/Testing/ExceptionInjectionType.cs | 5 ++ .../ClusterNegativeTests.cs | 62 +++++++++++++++++++ 3 files changed, 86 insertions(+), 7 deletions(-) diff --git a/libs/cluster/Server/Gossip/Gossip.cs b/libs/cluster/Server/Gossip/Gossip.cs index 130d2e99154..a08ff85dc17 100644 --- a/libs/cluster/Server/Gossip/Gossip.cs +++ b/libs/cluster/Server/Gossip/Gossip.cs @@ -343,18 +343,30 @@ async Task GossipMainAsync() while (true) { ctsGossip.Token.ThrowIfCancellationRequested(); - await InitConnectionsAsync().ConfigureAwait(false); - // Choose between full broadcast or sample gossip to few nodes - if (GossipSamplePercent == 100) - await BroadcastGossipSendAsync().ConfigureAwait(false); - else - await GossipSampleSendAsync().ConfigureAwait(false); + try + { + ExceptionInjectionHelper.TriggerException(ExceptionInjectionType.Cluster_Gossip_Round_Fail); + await InitConnectionsAsync().ConfigureAwait(false); + + // Choose between full broadcast or sample gossip to few nodes + if (GossipSamplePercent == 100) + await BroadcastGossipSendAsync().ConfigureAwait(false); + else + await GossipSampleSendAsync().ConfigureAwait(false); + } + catch (Exception ex) when (!ctsGossip.Token.IsCancellationRequested) + { + // Nothing restarts this task, so letting a single failed round leave the loop would stop + // this node gossiping for the rest of the process. Its view then stops being offered to + // the rest of the cluster and no further configuration change can ever converge. + logger?.LogWarning("Gossip round failed {msg}", ex.Message); + } await Task.Delay(gossipDelay, ctsGossip.Token).ConfigureAwait(false); } } - catch (TaskCanceledException) when (ctsGossip.Token.IsCancellationRequested) + catch (OperationCanceledException) when (ctsGossip.Token.IsCancellationRequested) { // Suppress the exception if the task was cancelled because of store wrapper disposal } diff --git a/libs/common/Testing/ExceptionInjectionType.cs b/libs/common/Testing/ExceptionInjectionType.cs index dda1fa7e4c7..1bee94961e9 100644 --- a/libs/common/Testing/ExceptionInjectionType.cs +++ b/libs/common/Testing/ExceptionInjectionType.cs @@ -125,5 +125,10 @@ public enum ExceptionInjectionType /// Vector Set: pause in the index-recreate window before RMW write refreshed index pointer back to the main store. /// VectorSet_Pause_Before_Recreate_Rmw, + /// + /// Cluster gossip: fail a single round of the main gossip loop, to exercise recovery from a transient + /// error raised while connections are initialized or the configuration is broadcast. + /// + Cluster_Gossip_Round_Fail, } } \ No newline at end of file diff --git a/test/cluster/Garnet.test.cluster/ClusterNegativeTests.cs b/test/cluster/Garnet.test.cluster/ClusterNegativeTests.cs index e0033c27c54..d2eebb697ac 100644 --- a/test/cluster/Garnet.test.cluster/ClusterNegativeTests.cs +++ b/test/cluster/Garnet.test.cluster/ClusterNegativeTests.cs @@ -840,6 +840,68 @@ public async Task ClusterFailoverResetsPrimaryOnTakeOverFailureAsync(Cancellatio // Verify data is still accessible from the primary context.ValidateKVCollectionAgainstReplica(ref context.kvPairs, primaryIndex); } + + [Test, Order(17), CancelAfter(testTimeout)] + [Category("CLUSTER")] + public void ClusterGossipSurvivesFailedRound() + { + const int nodeCount = 3; + List<(int, int)>[] ranges = [[(0, 5460)], [(5461, 10921)], [(10922, 16383)]]; + + context.CreateInstances(nodeCount, timeout: timeout); + context.CreateConnection(); + + // Introduce the nodes to each other. This runs on the meet task rather than the main gossip loop, + // so it still completes while the loop below is failing. + for (var i = 0; i < nodeCount; i++) + context.clusterTestUtils.SetConfigEpoch(i, i + 1, logger: context.logger); + for (var i = 1; i < nodeCount; i++) + { + context.clusterTestUtils.Meet(0, i, logger: context.logger); + context.clusterTestUtils.WaitUntilNodeIsKnown(0, i, logger: context.logger); + } + for (var i = 0; i < nodeCount; i++) + context.clusterTestUtils.WaitClusterNodesSync(i, nodeCount, context.logger); + + // Fail a round on every node's gossip loop. Nothing restarts that loop, so a round that is allowed + // to terminate it leaves the node silent for the rest of the process. + ExceptionInjectionHelper.EnableException(ExceptionInjectionType.Cluster_Gossip_Round_Fail); + try + { + // Longer than the gossip interval, so every node has run at least one failing round. + Thread.Sleep(TimeSpan.FromSeconds(7)); + } + finally + { + ExceptionInjectionHelper.DisableException(ExceptionInjectionType.Cluster_Gossip_Round_Fail); + } + + // Nothing is failing any more, so the assignment made now has to reach every node. + for (var i = 0; i < nodeCount; i++) + _ = context.clusterTestUtils.AddDelSlotsRange(i, ranges[i], addslot: true, logger: context.logger); + + var deadline = DateTime.UtcNow.AddSeconds(30); + while (true) + { + var pending = -1; + for (var i = 0; i < nodeCount; i++) + { + var slots = context.clusterTestUtils.ClusterSlots(i, context.logger); + if (slots == null || slots.Count != nodeCount) + { + pending = i; + break; + } + } + + if (pending < 0) + break; + + ClassicAssert.Less(DateTime.UtcNow, deadline, + $"node {pending} never converged after a gossip round failed"); + Thread.Sleep(100); + } + } #endif } } \ No newline at end of file From e30e7406eea146519841029d528ef5c60e158230 Mon Sep 17 00:00:00 2001 From: Badrish Chandramouli Date: Wed, 9 Sep 2026 19:51:38 -0700 Subject: [PATCH 07/27] Reconnect promptly after a test restarts a node The shared StackExchange.Redis configuration used by every Garnet test set ReconnectRetryPolicy to LinearRetry(10s). That policy gates how soon the multiplexer may retry a dropped connection, so the first command issued after a test restarts a node blocks until the gate opens. Measured on ClusterManagementTests.PrimaryUnavailableRecoveryAsync, which restarts two replicas and then talks to them: a bare PING to the first restarted node took 10250-10656ms, while the second node - restarted at the same moment and reached once the gate had already opened - answered in 0-2ms. The server was serving the whole time; only the client was waiting. That put the test at 24-25s against its 30s budget on an idle two-core machine, with a hard-coded 10s workload window accounting for most of the rest. The remaining slack was smaller than normal CI jitter, which is why it failed intermittently in two different places depending on where the clock ran out. Shortening the gate to 250ms drops the same PING to 292-501ms and the test to 13-16s. No timeout was raised, no assertion relaxed, and no retry added; the dead time was removed. Every test that restarts a node benefits. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 30f31d5b-9e53-4111-91a3-e16c94510a77 --- test/standalone/Garnet.test/TestUtils.cs | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/test/standalone/Garnet.test/TestUtils.cs b/test/standalone/Garnet.test/TestUtils.cs index 54427dc6546..ead5679f5d1 100644 --- a/test/standalone/Garnet.test/TestUtils.cs +++ b/test/standalone/Garnet.test/TestUtils.cs @@ -1026,7 +1026,10 @@ public static ConfigurationOptions GetConfig( SyncTimeout = (int)TimeSpan.FromSeconds(30).TotalMilliseconds, AsyncTimeout = (int)TimeSpan.FromSeconds(30).TotalMilliseconds, AllowAdmin = allowAdmin, - ReconnectRetryPolicy = new LinearRetry((int)TimeSpan.FromSeconds(10).TotalMilliseconds), + // Gates how often the multiplexer may retry a dropped connection. Tests restart nodes + // routinely, and the first command issued afterwards blocks until the next retry is + // allowed, so a long interval is dead time added to every such test. + ReconnectRetryPolicy = new LinearRetry((int)TimeSpan.FromMilliseconds(250).TotalMilliseconds), ConnectRetry = 5, IncludeDetailInExceptions = true, AbortOnConnectFail = true, From d9173d95b33b1d9d35cf52d376734299f2b8560e Mon Sep 17 00:00:00 2001 From: Badrish Chandramouli Date: Thu, 10 Sep 2026 09:33:49 -0700 Subject: [PATCH 08/27] Complete the TLS handshake off the accept loop GarnetServerTcp ran the server-side TLS handshake inline in HandleNewConnection, which executes on the accept loop. The handshake is several network round trips driven by the peer and has no deadline, so the loop stayed parked in it: while one peer negotiated, no other connection was accepted, and a peer that connected without ever sending a ClientHello stopped the listener permanently. That serialization is why TLS clusters are the ones that stall in CI. Every node opens gossip and replication connections to every other node, so one slow handshake on a small runner holds up the rest and the cluster never reaches the configuration the test is waiting for - with nothing logged, because the accept loop is not failing, only waiting. Start the handler asynchronously instead. Socket setup still happens synchronously on the accept loop; only the handshake moves to the thread pool, and its outcome is observed so a failure is logged and the handler disposed exactly as before. --- libs/server/Servers/GarnetServerTcp.cs | 29 ++++++++++++++++++++- test/standalone/Garnet.test/RespTlsTests.cs | 29 +++++++++++++++++++++ 2 files changed, 57 insertions(+), 1 deletion(-) diff --git a/libs/server/Servers/GarnetServerTcp.cs b/libs/server/Servers/GarnetServerTcp.cs index dc95044145d..f29f7faa9ed 100644 --- a/libs/server/Servers/GarnetServerTcp.cs +++ b/libs/server/Servers/GarnetServerTcp.cs @@ -7,6 +7,7 @@ using System.Net; using System.Net.Sockets; using System.Threading; +using System.Threading.Tasks; using Garnet.common; using Garnet.networking; using Garnet.server.TLS; @@ -287,7 +288,14 @@ private unsafe bool HandleNewConnection(SocketAsyncEventArgs e) { IncrementConnectionsReceived(); ExceptionInjectionHelper.TriggerException(ExceptionInjectionType.Dispose_After_Handler_Registered_Before_Start); - handler.Start(tlsOptions?.TlsServerOptions, remoteEndpointName); + + // A TLS handshake takes several network round trips, driven by the peer. Completing it here + // would hold the accept loop for its whole duration, so a peer that connects and negotiates + // slowly - or never - stops this server from accepting any other connection. StartAsync + // performs the socket setup synchronously and leaves only the handshake to the thread pool. + var startTask = handler.StartAsync(tlsOptions?.TlsServerOptions, remoteEndpointName); + if (!startTask.IsCompletedSuccessfully) + _ = ObserveHandlerStartAsync(startTask, handler); } catch (Exception ex) { @@ -310,6 +318,25 @@ private unsafe bool HandleNewConnection(SocketAsyncEventArgs e) return true; } + /// + /// Observes the result of an in-flight handler start so a failed handshake is reported and cleaned up + /// the same way a synchronous failure is, without keeping the accept loop waiting for it. + /// + /// Task returned by the handler's asynchronous start. + /// Handler the task belongs to. + private async Task ObserveHandlerStartAsync(Task startTask, ServerTcpNetworkHandler handler) + { + try + { + await startTask.ConfigureAwait(false); + } + catch (Exception ex) + { + logger?.LogError(ex, "Error calling Start on network handler"); + handler.Dispose(); + } + } + /// /// Create session (message consumer) given incoming bytes /// diff --git a/test/standalone/Garnet.test/RespTlsTests.cs b/test/standalone/Garnet.test/RespTlsTests.cs index 87d3ce31b61..bbe226e8a62 100644 --- a/test/standalone/Garnet.test/RespTlsTests.cs +++ b/test/standalone/Garnet.test/RespTlsTests.cs @@ -4,6 +4,9 @@ using System; using System.Collections.Generic; using System.Linq; +using System.Net; +using System.Net.Security; +using System.Net.Sockets; using System.Text; using System.Threading; using System.Threading.Tasks; @@ -523,5 +526,31 @@ public void TlsCanSetGetCommandsChunks(int bytesSent) response = lightClientRequest.Execute($"MGET{sb}", expectedResponse.Length, bytesSent); ClassicAssert.AreEqual(expectedResponse, response); } + + [Test] + public async Task PeerThatNeverNegotiatesDoesNotBlockOtherConnections() + { + // A TLS handshake is driven by the peer and takes several network round trips. If the server + // completes it on the accept loop, a peer that connects without negotiating holds that loop and + // no other connection is ever accepted. + var endpoint = new IPEndPoint(IPAddress.Loopback, TestUtils.TestPort); + + ClassicAssert.IsTrue(await TryHandshakeAsync(endpoint), "TLS handshake failed before opening the silent connection"); + + using var silent = new TcpClient(); + await silent.ConnectAsync(endpoint); + + ClassicAssert.IsTrue(await TryHandshakeAsync(endpoint), "TLS handshake did not complete while a peer that never negotiates was connected"); + } + + static async Task TryHandshakeAsync(IPEndPoint endpoint) + { + using var client = new TcpClient(); + await client.ConnectAsync(endpoint); + using var sslStream = new SslStream(client.GetStream(), leaveInnerStreamOpen: false, (_, _, _, _) => true); + var handshake = sslStream.AuthenticateAsClientAsync("GarnetTest"); + return await Task.WhenAny(handshake, Task.Delay(TimeSpan.FromSeconds(15))).ConfigureAwait(false) == handshake + && handshake.IsCompletedSuccessfully; + } } } \ No newline at end of file From 0f19bd5ba4e1781f06d0c4b93a5a4d5aa5cd5824 Mon Sep 17 00:00:00 2001 From: Badrish Chandramouli Date: Wed, 16 Sep 2026 10:27:58 -0700 Subject: [PATCH 09/27] Report which node disagreed when a cluster never converges WaitForSyncAsync polled until its deadline and then reported only "Cancellation Requested", which says nothing about what it was waiting for. Describe which of the expected nodes the server had not agreed on, and how, so a cluster that fails to converge is diagnosable from the CI log alone. The description is built solely on the failure path, so the polling loop is unchanged. The MergeSlotMap fix this originally accompanied landed upstream in #2089, which restructures the same guard and adds wider regression coverage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 30f31d5b-9e53-4111-91a3-e16c94510a77 --- .../Garnet.test.cluster/ClusterTestUtils.cs | 26 ++++++++++++++++++- 1 file changed, 25 insertions(+), 1 deletion(-) diff --git a/test/cluster/Garnet.test.cluster/ClusterTestUtils.cs b/test/cluster/Garnet.test.cluster/ClusterTestUtils.cs index 06ea1e14812..3be0afc4fc5 100644 --- a/test/cluster/Garnet.test.cluster/ClusterTestUtils.cs +++ b/test/cluster/Garnet.test.cluster/ClusterTestUtils.cs @@ -408,7 +408,10 @@ private async Task WaitForSyncAsync(ClientClusterConfig clusterConfig) if (count > 0) { - await BackOffAsync(cancellationToken: context.cts.Token).ConfigureAwait(false); + var msg = context.cts.IsCancellationRequested + ? DescribeSyncMismatch(server.EndPoint, expectedConfig, nodes) + : null; + await BackOffAsync(cancellationToken: context.cts.Token, msg: msg).ConfigureAwait(false); goto retry; } } @@ -416,6 +419,27 @@ private async Task WaitForSyncAsync(ClientClusterConfig clusterConfig) } } + /// + /// Describes which of the expected nodes a server has not yet agreed on, so a cluster that never + /// converges reports what it was waiting for instead of only that it ran out of time. + /// + /// Server whose view was inspected. + /// Expected cluster configuration, keyed by node id. + /// Nodes the server reported, or null if it reported none. + private static string DescribeSyncMismatch(EndPoint endPoint, Dictionary expectedConfig, IEnumerable nodes) + { + var sb = new StringBuilder($"Cluster did not converge; {endPoint} disagrees on:"); + foreach (var (nodeId, raw) in expectedConfig) + { + var node = nodes?.FirstOrDefault(n => n.NodeId == nodeId); + if (node == null) + sb.Append($"\n {nodeId}: not known to this node. Expected: {raw.Trim()}"); + else if (!NodesEqual(new ClientClusterNode(raw.Trim()), node)) + sb.Append($"\n {nodeId}: expected [{raw.Trim()}] actual [{node.Raw?.Trim()}]"); + } + return sb.ToString(); + } + public (List, List) SimpleSetupCluster( int primary_count = -1, int replica_count = -1, From c823f3a1daf22c98ca7360d87f144688eb11d2ed Mon Sep 17 00:00:00 2001 From: Badrish Chandramouli Date: Thu, 10 Sep 2026 11:12:31 -0700 Subject: [PATCH 10/27] Ship the object log up to the address the transferred main log needs GetLogFileSize reports the byte ranges that cluster full sync copies to a replica. The main log range ends at flushedLogicalAddress, captured at PERSISTENCE_CALLBACK, but the main object log range ended at snapshotStartObjectLogTail, captured earlier at WAIT_FLUSH. The two are not a matching pair. ShiftReadOnlyAddress defers its flush through epoch.BumpCurrentEpoch, so the drain action runs on whichever thread next observes the epoch as safe. When that lands between WAIT_FLUSH and PERSISTENCE_CALLBACK, flushedLogicalAddress and hlogEndObjectLogTail both advance while snapshotStartObjectLogTail does not. The replica then receives main log records whose object bytes were never sent, ObjectLogReader.Read returns zero and deserialization throws EndOfStreamException. Local recovery of the same checkpoint succeeds, because the files there are complete, which is why only replication saw this. Recovery installs hlogEndObjectLogTail as the object log tail, so that is the address the transferred range has to reach. TransferredSnapshotCheckpointRecoversAllObjects copies exactly the reported ranges and recovers from them, with a page aligned ShiftReadOnlyAddress injected into the checkpoint window. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 30f31d5b-9e53-4111-91a3-e16c94510a77 --- .../cs/src/core/Index/Recovery/Recovery.cs | 4 +- .../ObjectLogCheckpointTransferTests.cs | 206 ++++++++++++++++++ 2 files changed, 208 insertions(+), 2 deletions(-) create mode 100644 libs/storage/Tsavorite/cs/test/test.recovery/ObjectLogCheckpointTransferTests.cs diff --git a/libs/storage/Tsavorite/cs/src/core/Index/Recovery/Recovery.cs b/libs/storage/Tsavorite/cs/src/core/Index/Recovery/Recovery.cs index 4f0199ecae3..14fa9419caa 100644 --- a/libs/storage/Tsavorite/cs/src/core/Index/Recovery/Recovery.cs +++ b/libs/storage/Tsavorite/cs/src/core/Index/Recovery/Recovery.cs @@ -157,7 +157,7 @@ public struct LogFileInfo /// Address of ; the start of the lowest object log segment /// in use by the hybrid log at snapshot PREPARE time public long hybridLogObjectFileStartAddress; - /// The objectLogTail taken at the start of WAIT_FLUSH, corresponding to the hlog's FlushedUntilAddress at that point + /// The hlogEndObjectLogTail taken at PERSISTENCE_CALLBACK, corresponding to the hlog's FlushedUntilAddress at that point public long hybridLogObjectFileEndAddress; /// The snapshotEndObjectLogTail taken at PERSISTENCE_CALLBACK, which corresponds to the object log position for the final TailAddress /// written by the checkpoint. (Start address is always 0.) @@ -253,7 +253,7 @@ public LogFileInfo GetLogFileSize(Guid token) // a main-log Flush. However the snapshot does cause object-log segments for the mutable range to be written. hasSnapshotObjects = hasSnapshotObjects, hybridLogObjectFileStartAddress = hasSnapshotObjects ? (long)current.info.beginAddressObjectLogSegment << current.info.hlogEndObjectLogTail.SegmentSizeBits : 0, - hybridLogObjectFileEndAddress = hasSnapshotObjects ? (long)current.info.snapshotStartObjectLogTail.CurrentAddress : 0, + hybridLogObjectFileEndAddress = hasSnapshotObjects ? (long)current.info.hlogEndObjectLogTail.CurrentAddress : 0, snapshotObjectFileEndAddress = hasSnapshotObjects ? (long)current.info.snapshotEndObjectLogTail.CurrentAddress : 0, }; diff --git a/libs/storage/Tsavorite/cs/test/test.recovery/ObjectLogCheckpointTransferTests.cs b/libs/storage/Tsavorite/cs/test/test.recovery/ObjectLogCheckpointTransferTests.cs new file mode 100644 index 00000000000..6841f9699aa --- /dev/null +++ b/libs/storage/Tsavorite/cs/test/test.recovery/ObjectLogCheckpointTransferTests.cs @@ -0,0 +1,206 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT license. + +using System; +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using Garnet.test; +using NUnit.Framework; +using NUnit.Framework.Legacy; +using Tsavorite.core; +using static Tsavorite.test.TestUtils; + +namespace Tsavorite.test.recovery.objects +{ + using ClassAllocator = ObjectAllocator>; + using ClassStoreFunctions = StoreFunctions; + + /// + /// Verifies that the byte ranges reported by + /// are sufficient to recover a Snapshot checkpoint on a different node. Cluster replication full sync transfers only + /// those ranges of the main log, the main object log, the snapshot and the snapshot object log; the target node then + /// recovers from them, so a range that stops short of the data referenced by the transferred main log leaves the + /// target unable to deserialize its objects. + /// + [TestFixture] + public class ObjectLogCheckpointTransferTests : TestBase + { + const int NumRecords = 6000; + const long MainLogSegmentSize = 1L << 20; + const long ObjectLogSegmentSize = 1L << 30; + const int ThrottleCheckpointFlushDelayMs = 20; + const string LogBaseName = "xfer.log"; + const string ObjectLogBaseName = "xfer.obj.log"; + const string CheckpointDirName = "check-points"; + + string sourceDir; + string targetDir; + + [SetUp] + public void Setup() + { + RecreateDirectory(MethodTestDir); + sourceDir = Path.Combine(MethodTestDir, "source"); + targetDir = Path.Combine(MethodTestDir, "target"); + } + + [TearDown] + public void TearDown() => TestUtils.OnTearDown(); + + // flushMainLogDuringCheckpoint models a concurrent main-log flush (as issued by LogSizeTracker eviction) that + // starts after the checkpoint has captured its snapshot-start object-log position: it advances both + // FlushedUntilAddress and the main object-log tail before the checkpoint records them at PERSISTENCE_CALLBACK. + [Test] + [Category("TsavoriteKV"), Category("CheckpointRestore")] + public async Task TransferredSnapshotCheckpointRecoversAllObjects([Values] bool flushMainLogDuringCheckpoint) + { + Prepare(sourceDir, out var log, out var objlog, out var store); + Guid token; + LogFileInfo logFileInfo; + try + { + using (var session = store.NewSession(new TestObjectFunctions())) + { + var bContext = session.BasicContext; + for (var ii = 0; ii < NumRecords; ii++) + _ = bContext.Upsert(new TestObjectKey { key = ii }, new TestObjectValue { value = ii }); + } + + ClassicAssert.IsTrue(store.TryInitiateHybridLogCheckpoint(out token, CheckpointType.Snapshot)); + + Task concurrentFlush = null; + if (flushMainLogDuringCheckpoint) + { + concurrentFlush = Task.Run(() => + { + // The flush buffers are created immediately after the snapshot-start object-log position is captured, + // so this waits until the checkpoint is inside its (throttled) snapshot flush. + while (store._hybridLogCheckpoint.objectLogFlushBuffers is null) + Thread.Yield(); + // Page-aligned, as produced by AllocatorBase.CalculateReadOnlyAddress for automatic and size-tracker shifts. + var readOnlyAddress = store.Log.TailAddress & ~(MinKvLogPageSize - 1); + store.Log.ShiftReadOnlyAddress(readOnlyAddress, wait: true); + }); + } + + await store.CompleteCheckpointAsync().AsTask().ConfigureAwait(false); + if (concurrentFlush is not null) + await concurrentFlush.ConfigureAwait(false); + + logFileInfo = store.GetLogFileSize(token); + } + finally + { + Destroy(log, objlog, store); + } + + TransferCheckpoint(token, logFileInfo); + + Prepare(targetDir, out log, out objlog, out store); + try + { + _ = await store.RecoverAsync(default, token).ConfigureAwait(false); + + using var session = store.NewSession(new TestObjectFunctions()); + var bContext = session.BasicContext; + for (var ii = 0; ii < NumRecords; ii++) + { + var key = new TestObjectKey { key = ii }; + TestObjectInput input = default; + TestObjectOutput output = new(); + var status = bContext.Read(key, ref input, ref output); + if (status.IsPending) + { + Assert.That(bContext.CompletePendingWithOutputs(out var completedOutputs, wait: true), Is.True); + (status, output) = GetSinglePendingResult(completedOutputs); + } + + ClassicAssert.IsTrue(status.Found, $"key {ii} not found on the transfer target"); + ClassicAssert.AreEqual(ii, output.value.value, $"key {ii} has the wrong value on the transfer target"); + } + } + finally + { + Destroy(log, objlog, store); + } + } + + // Copies exactly the file ranges that TsavoriteSnapshotReader sends to a replica during full sync. + private void TransferCheckpoint(Guid token, LogFileInfo logFileInfo) + { + var sourceCheckpointDir = Path.Combine(sourceDir, CheckpointDirName, "cpr-checkpoints", token.ToString()); + var targetCheckpointDir = Path.Combine(targetDir, CheckpointDirName, "cpr-checkpoints", token.ToString()); + _ = Directory.CreateDirectory(targetCheckpointDir); + foreach (var metadataFile in Directory.GetFiles(sourceCheckpointDir, "info.dat*")) + File.Copy(metadataFile, Path.Combine(targetCheckpointDir, Path.GetFileName(metadataFile))); + + if (logFileInfo.hybridLogFileEndAddress > PageHeader.Size) + { + CopyRange(Path.Combine(sourceDir, LogBaseName), Path.Combine(targetDir, LogBaseName), + logFileInfo.hybridLogFileStartAddress, logFileInfo.hybridLogFileEndAddress, MainLogSegmentSize); + if (logFileInfo.hasSnapshotObjects) + CopyRange(Path.Combine(sourceDir, ObjectLogBaseName), Path.Combine(targetDir, ObjectLogBaseName), + logFileInfo.hybridLogObjectFileStartAddress, logFileInfo.hybridLogObjectFileEndAddress, ObjectLogSegmentSize); + } + + if (logFileInfo.snapshotFileEndAddress > PageHeader.Size) + { + CopyRange(Path.Combine(sourceCheckpointDir, "snapshot.dat"), Path.Combine(targetCheckpointDir, "snapshot.dat"), + 0, logFileInfo.snapshotFileEndAddress, MainLogSegmentSize); + if (logFileInfo.hasSnapshotObjects) + CopyRange(Path.Combine(sourceCheckpointDir, "snapshot.obj.dat"), Path.Combine(targetCheckpointDir, "snapshot.obj.dat"), + 0, logFileInfo.snapshotObjectFileEndAddress, ObjectLogSegmentSize); + } + } + + private static void CopyRange(string sourceBaseName, string targetBaseName, long startAddress, long endAddress, long segmentSize) + { + for (var segment = startAddress / segmentSize; startAddress < endAddress && segment <= (endAddress - 1) / segmentSize; segment++) + { + var segmentStartAddress = segment * segmentSize; + var from = Math.Max(startAddress, segmentStartAddress) - segmentStartAddress; + var to = Math.Min(endAddress, segmentStartAddress + segmentSize) - segmentStartAddress; + + var buffer = new byte[to - from]; + using (var sourceStream = File.OpenRead($"{sourceBaseName}.{segment}")) + { + _ = sourceStream.Seek(from, SeekOrigin.Begin); + sourceStream.ReadExactly(buffer); + } + + using var targetStream = new FileStream($"{targetBaseName}.{segment}", FileMode.Create, FileAccess.Write); + _ = targetStream.Seek(from, SeekOrigin.Begin); + targetStream.Write(buffer); + } + } + + private static void Prepare(string dir, out IDevice log, out IDevice objlog, out TsavoriteKV store) + { + _ = Directory.CreateDirectory(dir); + log = Devices.CreateLogDevice(Path.Combine(dir, LogBaseName)); + objlog = Devices.CreateLogDevice(Path.Combine(dir, ObjectLogBaseName)); + store = new(new() + { + IndexSize = 1L << 22, + LogDevice = log, + ObjectLogDevice = objlog, + SegmentSize = MainLogSegmentSize, + ObjectLogSegmentSize = ObjectLogSegmentSize, + LogMemorySize = 32 * MinKvLogPageSize, + PageSize = MinKvLogPageSize, + ThrottleCheckpointFlushDelayMs = ThrottleCheckpointFlushDelayMs, + CheckpointDir = Path.Combine(dir, CheckpointDirName) + }, StoreFunctions.Create(new TestObjectKey.Comparer(), () => new TestObjectValue.Serializer()) + , (allocatorSettings, storeFunctions) => new(allocatorSettings, storeFunctions) + ); + } + + private static void Destroy(IDevice log, IDevice objlog, TsavoriteKV store) + { + store.Dispose(); + log.Dispose(); + objlog.Dispose(); + } + } +} \ No newline at end of file From 60977e1b1b6778750cf5b02772fca1205f477216 Mon Sep 17 00:00:00 2001 From: Badrish Chandramouli Date: Thu, 10 Sep 2026 11:41:19 -0700 Subject: [PATCH 11/27] Pulse the replica in the time domain its replay side actually uses The primary emits a tail witness pulse whenever multi-log is on, which is either several physical sublogs or several replay tasks. It sourced the pulse value from the sequence number generator, but that generator is only created when the log is sharded across several physical sublogs. A single physical sublog replayed by several tasks therefore dereferenced a null generator, and the NullReferenceException took down the AOF sync task roughly a tail witness period after it started. Every write after that point stayed on the primary. Allocating a generator for that configuration would have replaced the crash with something worse. Records on a single physical sublog carry no sequence number; the replay side times its virtual sublogs by log address, publishing the address it has reached. A generator value is a stopwatch reading, so feeding one into that comparison would push the read consistency frontier arbitrarily far ahead of the data and hold it there. The pulse now carries the observed tail, which is the address replay itself publishes once it has consumed everything at or below it. The window is why this only ever surfaced under load. The pulse fires once the first witness period elapses, so a primary that ships its whole payload sooner never reaches it, and a slower one dies with the replica still empty. ClusterReplicationSinglePhysicalLogTailWitnessPulseTest leaves the sync task idle past a witness period before writing anything. Without the change it reproduces the CI failure exactly, the replica pinned at 64 while the primary moves on; with it the fixture passes 106 of 106 pinned to two cores. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 30f31d5b-9e53-4111-91a3-e16c94510a77 --- .../PrimaryOps/AofOperations/AofSyncTask.cs | 15 +++++- .../ClusterReplicationShardedLog.cs | 54 +++++++++++++++++++ 2 files changed, 68 insertions(+), 1 deletion(-) diff --git a/libs/cluster/Server/Replication/PrimaryOps/AofOperations/AofSyncTask.cs b/libs/cluster/Server/Replication/PrimaryOps/AofOperations/AofSyncTask.cs index 6b2f2217510..78ef037d893 100644 --- a/libs/cluster/Server/Replication/PrimaryOps/AofOperations/AofSyncTask.cs +++ b/libs/cluster/Server/Replication/PrimaryOps/AofOperations/AofSyncTask.cs @@ -29,6 +29,7 @@ public class AofSyncTask : IBulkLogEntryConsumer, IDisposable long previousAddress; readonly bool timePulseEnabled; + readonly bool pulseUsesSequenceNumbers; readonly GarnetAppendOnlyFile appendOnlyFile; readonly TsavoriteLog physicalSublog; readonly long[] pulseTailSnapshot; @@ -117,6 +118,10 @@ public AofSyncTask( timePulseEnabled = clusterProvider.serverOptions.MultiLogEnabled; if (timePulseEnabled) { + // Records carry a generator sequence number only when the log is sharded across + // several physical sublogs; a single physical sublog times its virtual sublogs by + // log address instead, and has no generator to read from. + pulseUsesSequenceNumbers = clusterProvider.serverOptions.AofPhysicalSublogCount > 1; physicalSublog = appendOnlyFile.Log.GetSubLog(physicalSublogIdx); pulseTailSnapshot = new long[clusterProvider.serverOptions.AofPhysicalSublogCount]; pulseTailScratch = new long[clusterProvider.serverOptions.AofPhysicalSublogCount]; @@ -295,7 +300,15 @@ void SendAdvanceTimePulse() } } - var sequenceNumber = appendOnlyFile.GetLargerThanMaximumSequenceNumber(); + // The pulse has to carry a value in whatever domain the replay side measures virtual + // sublog time in. Sharded across several physical sublogs that is a generator sequence + // number, because that is what each record carries. On a single physical sublog records + // carry no sequence number and replay publishes the log address it has reached, so the + // pulse is the observed tail: the address replay itself would publish once it has + // consumed everything at or below it. + var sequenceNumber = pulseUsesSequenceNumbers + ? appendOnlyFile.GetLargerThanMaximumSequenceNumber() + : pulseTailScratch[physicalSublogIdx]; if (iter.NextAddress < physicalSublog.TailAddress) { lastAdvanceTimePulse = now; diff --git a/test/cluster/Garnet.test.cluster.multilog/MultiLogTests/ClusterReplicationShardedLog.cs b/test/cluster/Garnet.test.cluster.multilog/MultiLogTests/ClusterReplicationShardedLog.cs index 2be80e64df1..7fe5a1c8844 100644 --- a/test/cluster/Garnet.test.cluster.multilog/MultiLogTests/ClusterReplicationShardedLog.cs +++ b/test/cluster/Garnet.test.cluster.multilog/MultiLogTests/ClusterReplicationShardedLog.cs @@ -462,5 +462,59 @@ public async Task ClusterAofUpgradeSLtoSLMRRecoverAsync([Values] bool useStoredP ClassicAssert.AreEqual(values, result); } } + + [Test, Order(6)] + [Category("REPLICATION")] + public async Task ClusterReplicationSinglePhysicalLogTailWitnessPulseTest() + { + var primaryNodeIndex = 0; + var replicaNodeIndex = 1; + + // One physical sublog replayed by several tasks. Multi-log is on, so the primary emits + // tail witness pulses, but records on a single physical sublog carry no generator + // sequence number and the replay side times its virtual sublogs by log address. + context.CreateInstances(2, + disableObjects: false, + enableAOF: true, + useTLS: useTLS, + asyncReplay: asyncReplay, + sublogCount: 1, + replayTaskCount: TestReplayTaskCount); + context.CreateConnection(useTLS: useTLS); + + var resp = context.clusterTestUtils.AddDelSlotsRange(primaryNodeIndex, [(0, 16383)], addslot: true, logger: context.logger); + ClassicAssert.AreEqual("OK", resp); + + context.clusterTestUtils.SetConfigEpoch(primaryNodeIndex, primaryNodeIndex + 1, logger: context.logger); + context.clusterTestUtils.SetConfigEpoch(replicaNodeIndex, replicaNodeIndex + 1, logger: context.logger); + context.clusterTestUtils.Meet(primaryNodeIndex, replicaNodeIndex, logger: context.logger); + context.clusterTestUtils.WaitUntilNodeIsKnown(primaryNodeIndex, replicaNodeIndex, logger: context.logger); + context.clusterTestUtils.WaitUntilNodeIsKnown(replicaNodeIndex, primaryNodeIndex, logger: context.logger); + + resp = context.clusterTestUtils.ClusterReplicate(replicaNodeIndex, primaryNodeIndex, logger: context.logger); + ClassicAssert.AreEqual("OK", resp); + context.clusterTestUtils.WaitForReplicaRecovery(replicaNodeIndex, logger: context.logger); + + // Leave the AOF sync task idle for longer than AofTailWitnessFreqMs (100 ms by default, + // and the harness does not override it) so it emits at least one pulse before anything is + // written. Everything below this point depends on the sync task having survived that pulse. + await Task.Delay(TimeSpan.FromSeconds(1)).ConfigureAwait(false); + + string[] keys = ["{_}a", "{_}b", "{_}c"]; + string[] values = ["10", "15", "20"]; + for (var i = 0; i < keys.Length; i++) + { + var state = context.clusterTestUtils.SetKey(primaryNodeIndex, Encoding.ASCII.GetBytes(keys[i]), Encoding.ASCII.GetBytes(values[i]), out _, out _, logger: context.logger); + ClassicAssert.AreEqual(ResponseState.OK, state); + } + + context.clusterTestUtils.WaitForReplicaAofSync(primaryNodeIndex, replicaNodeIndex, context.logger); + + for (var i = 0; i < keys.Length; i++) + { + var value = context.clusterTestUtils.GetKey(replicaNodeIndex, Encoding.ASCII.GetBytes(keys[i]), out _, out _, out _); + ClassicAssert.AreEqual(values[i], value, $"At replica, key {keys[i]}"); + } + } } } \ No newline at end of file From d6a3c271b67fd6abbcb49ce25139646da6c3461b Mon Sep 17 00:00:00 2001 From: Badrish Chandramouli Date: Thu, 10 Sep 2026 13:33:27 -0700 Subject: [PATCH 12/27] Cover slot ownership across every gossip ordering The merge rule that lets a replica take a slot it only relays needs the receiver to hear from the replica before it hears from the owning primary, which is a question of ordering rather than of any single merge call. Drive the real merge and serialization code through the layout the cluster tests build, over many orderings of the gossip exchanges. A configuration only goes on the wire when it changed since the last send on that connection, so the simulation carries the same per-connection caches the server keeps. That is what makes a divergence permanent rather than transient: a view the receiver refused is never offered again, and the cluster sits quiescent exchanging empty pings. Without the merge fix the simulation stops converging within a handful of trials, with the true owner holding no slots while its replica holds them all. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 30f31d5b-9e53-4111-91a3-e16c94510a77 --- .../Garnet.test.cluster/ClusterConfigTests.cs | 195 ++++++++++++++++++ 1 file changed, 195 insertions(+) diff --git a/test/cluster/Garnet.test.cluster/ClusterConfigTests.cs b/test/cluster/Garnet.test.cluster/ClusterConfigTests.cs index e4b11afa9b3..f3df00446bd 100644 --- a/test/cluster/Garnet.test.cluster/ClusterConfigTests.cs +++ b/test/cluster/Garnet.test.cluster/ClusterConfigTests.cs @@ -1,5 +1,7 @@ // Copyright (c) Microsoft Corporation. // Licensed under the MIT license. +using System; +using System.Collections.Concurrent; using System.Collections.Generic; using System.Linq; using System.Net; @@ -503,5 +505,198 @@ public void ClusterConfigMergeSlotMapReplicaHandoffDoesNotLeakToLaterSlotsTest() Assert.That(merged.GetWorkerIdFromSlot(UnownedSlot), Is.EqualTo(ClusterConfig.RESERVED_WORKER_ID), "the later unowned slot must not inherit the correction computed for the earlier slot"); } + + /// + /// Models one node of a gossiping cluster: its published configuration plus the two caches that + /// decide whether a configuration is actually put on the wire. + /// + private sealed class GossipNode + { + public string NodeId; + + /// + /// ClusterManager.currentConfig. + /// + public ClusterConfig Config; + + /// + /// GarnetServerNode.lastConfig, one per outgoing gossip connection. + /// + public readonly Dictionary LastConfigSentTo = []; + + /// + /// ClusterCommands.lastSentConfig, one per inbound gossip session. + /// + public readonly Dictionary LastConfigRepliedTo = []; + } + + private static readonly ConcurrentDictionary EmptyBanList = new(); + + /// + /// Models ClusterManager.TryMerge. + /// + private static void GossipTryMerge(GossipNode node, ClusterConfig senderConfig) + { + var currentCopy = node.Config.Copy(); + var next = currentCopy.Merge(senderConfig, EmptyBanList).HandleConfigEpochCollision(senderConfig); + if (currentCopy != next) + node.Config = next; + } + + /// + /// Models one gossip exchange: GarnetServerNode.TryGossip sends a configuration only when the local + /// configuration object changed since the last send on that connection, the receiver merges it and + /// replies with its own pre-merge configuration under the same rule, and the sender merges the reply. + /// + private static void GossipExchange(GossipNode from, GossipNode to, bool withMeet = false) + { + byte[] request; + if (!from.LastConfigSentTo.TryGetValue(to.NodeId, out var lastSent) || lastSent != from.Config) + { + from.LastConfigSentTo[to.NodeId] = from.Config; + request = from.Config.ToByteArray(); + } + else + { + request = []; + } + + var receiverConfig = to.Config; + if (request.Length > 0) + { + var senderConfig = ClusterConfig.FromByteArray(request); + if (withMeet || receiverConfig.IsKnown(senderConfig.LocalNodeId)) + GossipTryMerge(to, senderConfig); + } + + byte[] reply; + if (withMeet || !to.LastConfigRepliedTo.TryGetValue(from.NodeId, out var lastReplied) || lastReplied != receiverConfig) + { + to.LastConfigRepliedTo[from.NodeId] = receiverConfig; + reply = receiverConfig.ToByteArray(); + } + else + { + reply = []; + } + + if (reply.Length > 0) + { + var replyConfig = ClusterConfig.FromByteArray(reply); + + // A MEET response is merged unconditionally, the peer is trusted because an admin issued the meet. + if (withMeet || from.Config.IsKnown(replyConfig.LocalNodeId)) + GossipTryMerge(from, replyConfig); + } + } + + /// + /// Replays what SimpleSetupCluster does: assign slots to the two primaries, set config epochs, then + /// issue every CLUSTER MEET from the first node. + /// + private static List BuildGossipCluster() + { + var nodes = new List(); + for (var i = 0; i < 4; i++) + { + var nodeId = Generator.CreateHexId(); + nodes.Add(new GossipNode + { + NodeId = nodeId, + Config = new ClusterConfig().InitializeLocalWorker( + nodeId, "127.0.0.1", ClusterTestContext.Port + i, configEpoch: 0, + Garnet.cluster.NodeRole.PRIMARY, null, "") + }); + } + + nodes[0].Config = nodes[0].Config.AssignSlots([.. Enumerable.Range(0, 8192)], ClusterConfig.LOCAL_WORKER_ID, SlotState.STABLE); + nodes[1].Config = nodes[1].Config.AssignSlots([.. Enumerable.Range(8192, 8192)], ClusterConfig.LOCAL_WORKER_ID, SlotState.STABLE); + + for (var i = 0; i < nodes.Count; i++) + nodes[i].Config = nodes[i].Config.SetLocalWorkerConfigEpoch(i + 1); + + for (var i = 1; i < nodes.Count; i++) + GossipExchange(nodes[0], nodes[i], withMeet: true); + + return nodes; + } + + /// + /// The state WaitForSyncAsync waits for: every node agrees that the two primaries own their halves of + /// the slot space and that the two replicas own nothing and are flagged as replicas. + /// + private static bool GossipClusterConverged(List nodes) + { + foreach (var node in nodes) + { + for (var slot = 0; slot < 8192; slot++) + if (node.Config.GetNodeIdFromSlot((ushort)slot) != nodes[0].NodeId) return false; + + for (var slot = 8192; slot < 16384; slot++) + if (node.Config.GetNodeIdFromSlot((ushort)slot) != nodes[1].NodeId) return false; + + if (node.Config.GetNodeRoleFromNodeId(nodes[2].NodeId) != Garnet.cluster.NodeRole.REPLICA) return false; + if (node.Config.GetNodeRoleFromNodeId(nodes[3].NodeId) != Garnet.cluster.NodeRole.REPLICA) return false; + } + return true; + } + + private static string DescribeGossipCluster(List nodes) + { + var sb = new StringBuilder(); + foreach (var node in nodes) + { + var info = node.Config.GetClusterInfo(null); + for (var i = 0; i < nodes.Count; i++) + info = info.Replace(nodes[i].NodeId, $"n{i}"); + sb.Append(info).Append('\n'); + } + return sb.ToString(); + } + + /// + /// Drives the real merge code through the cluster layout SimpleSetupCluster builds, over many gossip + /// orderings. A configuration is only put on the wire when the sender's configuration object changed + /// since its last send on that connection, so a view that a merge refuses to take is never offered + /// again and any divergence the merge rules allow is permanent rather than transient. + /// + [Test, Order(13)] + [Category("CLUSTER-CONFIG"), CancelAfter(60_000)] + public void ClusterConfigGossipConvergesForEveryOrderingTest() + { + const int Trials = 200; + const int RoundsPerTrial = 60; + + var pairs = new List<(int From, int To)>(); + for (var from = 0; from < 4; from++) + for (var to = 0; to < 4; to++) + if (from != to) pairs.Add((from, to)); + + var random = new Random(12345); + for (var trial = 0; trial < Trials; trial++) + { + var nodes = BuildGossipCluster(); + + // CLUSTER REPLICATE bumps the local config epoch so the role change can propagate. Gossip runs + // between the two calls because the test waits for the first primary to see its replica. + nodes[2].Config = nodes[2].Config.MakeReplicaOf(nodes[0].NodeId).BumpLocalNodeConfigEpoch(); + var interleaved = random.Next(0, 8); + for (var i = 0; i < interleaved; i++) + { + var (from, to) = pairs[random.Next(pairs.Count)]; + GossipExchange(nodes[from], nodes[to]); + } + nodes[3].Config = nodes[3].Config.MakeReplicaOf(nodes[1].NodeId).BumpLocalNodeConfigEpoch(); + + for (var round = 0; round < RoundsPerTrial && !GossipClusterConverged(nodes); round++) + { + foreach (var (from, to) in pairs.OrderBy(_ => random.Next())) + GossipExchange(nodes[from], nodes[to]); + } + + Assert.That(GossipClusterConverged(nodes), Is.True, + $"cluster did not converge after {RoundsPerTrial} gossip rounds in trial {trial}:\n{DescribeGossipCluster(nodes)}"); + } + } } } \ No newline at end of file From 1846778e6b979fcbcd7c28fb3a39e6127f78e9e2 Mon Sep 17 00:00:00 2001 From: Badrish Chandramouli Date: Thu, 10 Sep 2026 13:33:43 -0700 Subject: [PATCH 13/27] Fail a replica sync whose checkpoint recovery failed Recovering the checkpoint a primary ships is the step that gives a replica the primary's data. When it threw, the error was logged and swallowed unless FailOnRecoveryError was set, and the sync carried on to initialize the AOF and adopt the replication offset the primary sent. The replica then advertised being caught up while holding none of the primary's data, and nothing anywhere reported a problem. FailOnRecoveryError governs whether a standalone server starts despite a bad local checkpoint, which is a different question from whether a replica may silently diverge, so the replica path now propagates regardless of it. The caller already handles this: the sync aborts, the error travels back over CLUSTER REPLICATE, and the operator sees why. MultiDatabaseManager already throws unconditionally on this path and needs no change. ClusterReplicaCheckpointRecoveryFailureAbortsSyncTest fails the recovery through the existing DEBUG-only injection points. Without the change the replica answers OK to a sync that recovered nothing, which is the silent data loss itself. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 30f31d5b-9e53-4111-91a3-e16c94510a77 --- libs/common/Testing/ExceptionInjectionType.cs | 4 ++ .../server/Databases/SingleDatabaseManager.cs | 7 ++- .../ClusterNegativeTests.cs | 46 +++++++++++++++++++ 3 files changed, 56 insertions(+), 1 deletion(-) diff --git a/libs/common/Testing/ExceptionInjectionType.cs b/libs/common/Testing/ExceptionInjectionType.cs index 1bee94961e9..e9745056d02 100644 --- a/libs/common/Testing/ExceptionInjectionType.cs +++ b/libs/common/Testing/ExceptionInjectionType.cs @@ -130,5 +130,9 @@ public enum ExceptionInjectionType /// error raised while connections are initialized or the configuration is broadcast. /// Cluster_Gossip_Round_Fail, + /// + /// Replica full sync: fail recovery of the checkpoint received from the primary, before any of it is applied. + /// + Replication_Fail_Replica_Checkpoint_Recovery, } } \ No newline at end of file diff --git a/libs/server/Databases/SingleDatabaseManager.cs b/libs/server/Databases/SingleDatabaseManager.cs index 420fbe96578..e0a28f8190c 100644 --- a/libs/server/Databases/SingleDatabaseManager.cs +++ b/libs/server/Databases/SingleDatabaseManager.cs @@ -61,6 +61,8 @@ public override async ValueTask RecoverCheckpointAsync(bool replicaRecover = fal { if (replicaRecover) { + ExceptionInjectionHelper.TriggerException(ExceptionInjectionType.Replication_Fail_Replica_Checkpoint_Recovery); + // Note: Since replicaRecover only pertains to cluster-mode, we can use the default store pointers (since multi-db mode is disabled in cluster-mode) if (metadata!.storeIndexToken != default && metadata.storeHlogToken != default) { @@ -88,7 +90,10 @@ public override async ValueTask RecoverCheckpointAsync(bool replicaRecover = fal // be visible at the default log level. Logger?.LogError(ex, "Error during recovery of store; storeVersion = {storeVersion};", storeVersion); - if (StoreWrapper.serverOptions.FailOnRecoveryError) + // A replica that continues here would hold an incomplete store while still advertising the + // replication offset the primary sent it, diverging from the primary with nothing to signal it. + // Fail the sync instead, independently of FailOnRecoveryError, which governs standalone startup. + if (replicaRecover || StoreWrapper.serverOptions.FailOnRecoveryError) throw; } diff --git a/test/cluster/Garnet.test.cluster/ClusterNegativeTests.cs b/test/cluster/Garnet.test.cluster/ClusterNegativeTests.cs index d2eebb697ac..b58f56b9fa1 100644 --- a/test/cluster/Garnet.test.cluster/ClusterNegativeTests.cs +++ b/test/cluster/Garnet.test.cluster/ClusterNegativeTests.cs @@ -471,6 +471,52 @@ public void ClusterReplicaSyncTimeoutTest() context.ValidateKVCollectionAgainstReplica(ref context.kvPairs, replicaIndex); } + + [Test, Order(6), CancelAfter(testTimeout)] + public void ClusterReplicaCheckpointRecoveryFailureAbortsSyncTest() + { + var primaryIndex = 0; + var replicaIndex = 1; + var nodes_count = 2; + context.CreateInstances(nodes_count, disableObjects: false, enableAOF: true, timeout: timeout); + context.CreateConnection(); + + _ = context.clusterTestUtils.AddDelSlotsRange(primaryIndex, [(0, 16383)], addslot: true, logger: context.logger); + context.clusterTestUtils.SetConfigEpoch(primaryIndex, primaryIndex + 1, logger: context.logger); + context.clusterTestUtils.SetConfigEpoch(replicaIndex, replicaIndex + 1, logger: context.logger); + context.clusterTestUtils.Meet(primaryIndex, replicaIndex, logger: context.logger); + + var keyLength = 32; + var kvpairCount = 32; + context.kvPairs = []; + context.PopulatePrimary(ref context.kvPairs, keyLength, kvpairCount, primaryIndex, null); + + // Take a checkpoint so the replica performs a disk-based full sync and recovers from the shipped token + var primaryLastSaveTime = context.clusterTestUtils.LastSave(primaryIndex, logger: context.logger); + context.clusterTestUtils.WaitUntilNextSecond(primaryIndex, primaryLastSaveTime); + context.clusterTestUtils.Checkpoint(primaryIndex, logger: context.logger); + context.clusterTestUtils.WaitCheckpoint(primaryIndex, primaryLastSaveTime, logger: context.logger); + + try + { + ExceptionInjectionHelper.EnableException(ExceptionInjectionType.Replication_Fail_Replica_Checkpoint_Recovery); + + var respReplicate = context.clusterTestUtils.ClusterReplicate(replicaNodeIndex: replicaIndex, primaryNodeIndex: primaryIndex, failEx: false, logger: context.logger); + + // Recovery failed before applying any of the checkpoint, so the replica holds none of the primary's + // data. Reporting success here is what lets a replica advertise the primary's replication offset + // over an empty store, so the sync must fail and carry the underlying error. + ClassicAssert.AreEqual(0, context.clusterTestUtils.DBSize(replicaIndex, logger: context.logger), + "recovery must have left the replica store empty for this scenario to be under test"); + + Assert.That(respReplicate, Does.Contain(nameof(ExceptionInjectionType.Replication_Fail_Replica_Checkpoint_Recovery)), + "a failed replica checkpoint recovery must abort the sync and surface the underlying error"); + } + finally + { + ExceptionInjectionHelper.DisableException(ExceptionInjectionType.Replication_Fail_Replica_Checkpoint_Recovery); + } + } #endif [Test, Order(10), CancelAfter(60_000)] From ad8c1afc89a91c29bd137afadfc9cde57e93b0af Mon Sep 17 00:00:00 2001 From: Badrish Chandramouli Date: Thu, 10 Sep 2026 13:38:26 -0700 Subject: [PATCH 14/27] Give the testhost longer to connect back to vstest vstest.console launches a testhost process and waits for it to connect back. The default ninety seconds is occasionally not enough on a loaded two-core runner, and when it elapses the job fails having run no tests at all, with nothing to attribute it to beyond "machine slowness". This governs process startup only. A test that hangs once it is running is still caught by the blame hang timeout, so nothing about a real stall is hidden by the longer wait. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 30f31d5b-9e53-4111-91a3-e16c94510a77 --- .github/workflows/ci.yml | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index aaeb906f4ae..cb90be7f159 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -13,6 +13,11 @@ on: env: DOTNET_SKIP_FIRST_TIME_EXPERIENCE: 1 DOTNET_NOLOGO: true + # vstest.console waits this many seconds for the testhost process it just launched to connect back + # to it. The default of 90 is occasionally not enough on a loaded two-core runner, and when it + # elapses the job fails without having run a single test. This governs process startup only; a test + # that hangs is still caught by --blame-hang-timeout. + VSTEST_CONNECTION_TIMEOUT: 180 permissions: contents: read From ed1724c8a7bde60282e820b031f4b1343a4e39f7 Mon Sep 17 00:00:00 2001 From: Badrish Chandramouli Date: Thu, 10 Sep 2026 20:29:30 -0700 Subject: [PATCH 15/27] Drop the dedicated-thread workaround for SE.Redis result boxes StackExchange.Redis 2.x parked a synchronous caller on a [ThreadStatic] result box. A server error reply faulted that box from the connection's reader thread, and when the fault landed before the caller reached Monitor.Wait the caller skipped the wait and recycled the box while the reader thread still had a PulseAll outstanding on it. The next synchronous call on that thread took the recycled box, consumed the stale pulse, and returned with neither a result nor an exception, shifting every subsequent reply on that thread by one. TestUtils.ThrowsRedisException sidestepped that by running each expected-to-fail call on its own thread, keeping the recycled box off the shared test thread. The bump to 3.2.0 fixes the box handling upstream, so the workaround now only obscures what the tests assert. Reverts all 178 call sites to ClassicAssert.Throws and deletes the helper along with the using it needed. LuaScriptTests.cs and RespVectorSetTests.cs are byte-identical to their pre-workaround state. Verified rather than assumed: with the helper gone, LuaScriptTests (123 of the reverted sites) ran 4x at 435/435 and RespVectorSetTests (the other 55) 164/164, all under taskset -c 0-1 to match the two-core CI runners - 1,904 executions with no shifted replies. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 30f31d5b-9e53-4111-91a3-e16c94510a77 --- .../Garnet.test.scripting/LuaScriptTests.cs | 246 +++++++++--------- .../RespVectorSetTests.cs | 110 ++++---- test/standalone/Garnet.test/TestUtils.cs | 58 ----- 3 files changed, 178 insertions(+), 236 deletions(-) diff --git a/test/standalone/Garnet.test.scripting/LuaScriptTests.cs b/test/standalone/Garnet.test.scripting/LuaScriptTests.cs index 78398094bf2..e3620de7100 100644 --- a/test/standalone/Garnet.test.scripting/LuaScriptTests.cs +++ b/test/standalone/Garnet.test.scripting/LuaScriptTests.cs @@ -192,7 +192,7 @@ public void GlobalsForbidden() var db = redis.GetDatabase(0); var globalFuncExc = - TestUtils.ThrowsRedisException( + ClassicAssert.Throws( () => { _ = db.ScriptEvaluate( @@ -204,13 +204,13 @@ public void GlobalsForbidden() ); ClassicAssert.IsTrue(globalFuncExc.Message.Contains("Attempt to modify a readonly table")); - var globalVar = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("global_var = 'hello'")); + var globalVar = ClassicAssert.Throws(() => db.ScriptEvaluate("global_var = 'hello'")); ClassicAssert.IsTrue(globalVar.Message.Contains("Attempt to modify a readonly table")); - var metatableUpdateOnGlobals = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("setmetatable(_G, nil)")); + var metatableUpdateOnGlobals = ClassicAssert.Throws(() => db.ScriptEvaluate("setmetatable(_G, nil)")); ClassicAssert.IsTrue(metatableUpdateOnGlobals.Message.Contains("Attempt to modify a readonly table")); - var rawSetG = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("rawset(_G, 'hello', 'world')")); + var rawSetG = ClassicAssert.Throws(() => db.ScriptEvaluate("rawset(_G, 'hello', 'world')")); ClassicAssert.IsTrue(globalVar.Message.Contains("Attempt to modify a readonly table")); } @@ -228,7 +228,7 @@ public void ReadOnlyGlobalTables() foreach (var illegal in illegalToModify) { - var exc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate($"table.insert({illegal}, 'foo')")); + var exc = ClassicAssert.Throws(() => db.ScriptEvaluate($"table.insert({illegal}, 'foo')")); ClassicAssert.IsTrue(exc.Message.Contains("Attempt to modify a readonly table")); } @@ -481,7 +481,7 @@ public void CanDoScriptFlush() server.ScriptFlush(); // Assert the script is not found - _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate(scriptId)); + _ = Assert.Throws(() => db.ScriptEvaluate(scriptId)); } [Test] @@ -551,11 +551,11 @@ public void FailureStatusReturn() var db = redis.GetDatabase(0); var statusReplyScript = "return redis.error_reply('GET')"; - var excReply = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate(statusReplyScript)); + var excReply = ClassicAssert.Throws(() => db.ScriptEvaluate(statusReplyScript)); ClassicAssert.AreEqual("ERR GET", excReply.Message); var directReplyScript = "return { err = 'Failure' }"; - var excDirect = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate(directReplyScript)); + var excDirect = ClassicAssert.Throws(() => db.ScriptEvaluate(directReplyScript)); ClassicAssert.AreEqual("Failure", excDirect.Message); } @@ -567,11 +567,11 @@ public void MiscMath() // ATan2 { - _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("math.atan2()")); - _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("math.atan2(1)")); - _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("math.atan2(1, 2, 3)")); - _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("math.atan2('a', 1)")); - _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("math.atan2(1, 'b')")); + _ = ClassicAssert.Throws(() => db.ScriptEvaluate("math.atan2()")); + _ = ClassicAssert.Throws(() => db.ScriptEvaluate("math.atan2(1)")); + _ = ClassicAssert.Throws(() => db.ScriptEvaluate("math.atan2(1, 2, 3)")); + _ = ClassicAssert.Throws(() => db.ScriptEvaluate("math.atan2('a', 1)")); + _ = ClassicAssert.Throws(() => db.ScriptEvaluate("math.atan2(1, 'b')")); var val = (string)db.ScriptEvaluate("return tostring(math.atan2(0.1, 0.2))"); ClassicAssert.AreEqual(Math.Round(Math.Atan2(0.1, 0.2), 14).ToString(), val); @@ -579,9 +579,9 @@ public void MiscMath() // Cosh { - _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("math.cosh()")); - _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("math.cosh('a')")); - _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("math.cosh(1, 2)")); + _ = ClassicAssert.Throws(() => db.ScriptEvaluate("math.cosh()")); + _ = ClassicAssert.Throws(() => db.ScriptEvaluate("math.cosh('a')")); + _ = ClassicAssert.Throws(() => db.ScriptEvaluate("math.cosh(1, 2)")); var val = (string)db.ScriptEvaluate("return tostring(math.cosh(0.1))"); ClassicAssert.AreEqual(Math.Round(Math.Cosh(0.1), 14).ToString(), val); @@ -589,9 +589,9 @@ public void MiscMath() // Log10 { - _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("math.log10()")); - _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("math.log10('a')")); - _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("math.log10(1, 2)")); + _ = ClassicAssert.Throws(() => db.ScriptEvaluate("math.log10()")); + _ = ClassicAssert.Throws(() => db.ScriptEvaluate("math.log10('a')")); + _ = ClassicAssert.Throws(() => db.ScriptEvaluate("math.log10(1, 2)")); var val = (string)db.ScriptEvaluate("return tostring(math.log10(0.1))"); ClassicAssert.AreEqual("-1.0", val); @@ -599,11 +599,11 @@ public void MiscMath() // Pow { - _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("math.pow()")); - _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("math.pow(1)")); - _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("math.pow(1, 2, 3)")); - _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("math.pow('a', 1)")); - _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("math.pow(1, 'b')")); + _ = ClassicAssert.Throws(() => db.ScriptEvaluate("math.pow()")); + _ = ClassicAssert.Throws(() => db.ScriptEvaluate("math.pow(1)")); + _ = ClassicAssert.Throws(() => db.ScriptEvaluate("math.pow(1, 2, 3)")); + _ = ClassicAssert.Throws(() => db.ScriptEvaluate("math.pow('a', 1)")); + _ = ClassicAssert.Throws(() => db.ScriptEvaluate("math.pow(1, 'b')")); var val = (string)db.ScriptEvaluate("return tostring(math.pow(0.1, 0.2))"); ClassicAssert.AreEqual(Math.Round(Math.Pow(0.1, 0.2), 14).ToString(), val); @@ -611,9 +611,9 @@ public void MiscMath() // Sinh { - _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("math.sinh()")); - _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("math.sinh('a')")); - _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("math.sinh(1, 2)")); + _ = ClassicAssert.Throws(() => db.ScriptEvaluate("math.sinh()")); + _ = ClassicAssert.Throws(() => db.ScriptEvaluate("math.sinh('a')")); + _ = ClassicAssert.Throws(() => db.ScriptEvaluate("math.sinh(1, 2)")); var val = (string)db.ScriptEvaluate("return tostring(math.sinh(0.1))"); ClassicAssert.AreEqual(Math.Round(Math.Sinh(0.1), 14).ToString(), val); @@ -621,9 +621,9 @@ public void MiscMath() // Tanh { - _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("math.tanh()")); - _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("math.tanh('a')")); - _ = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("math.tanh(1, 2)")); + _ = ClassicAssert.Throws(() => db.ScriptEvaluate("math.tanh()")); + _ = ClassicAssert.Throws(() => db.ScriptEvaluate("math.tanh('a')")); + _ = ClassicAssert.Throws(() => db.ScriptEvaluate("math.tanh(1, 2)")); var val = (string)db.ScriptEvaluate("return tostring(math.tanh(0.2))"); ClassicAssert.AreEqual(Math.Round(Math.Tanh(0.2), 14).ToString(), val); @@ -665,10 +665,10 @@ public void RedisSha1Hex() var resTable = (string)db.ScriptEvaluate("return redis.sha1hex({ 1234 })"); ClassicAssert.AreEqual("da39a3ee5e6b4b0d3255bfef95601890afd80709", resTable); - var excEmpty = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return redis.sha1hex()")); + var excEmpty = ClassicAssert.Throws(() => db.ScriptEvaluate("return redis.sha1hex()")); ClassicAssert.IsTrue(excEmpty.Message.StartsWith("ERR wrong number of arguments")); - var excTwo = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return redis.sha1hex('a', 'b')")); + var excTwo = ClassicAssert.Throws(() => db.ScriptEvaluate("return redis.sha1hex('a', 'b')")); ClassicAssert.IsTrue(excTwo.Message.StartsWith("ERR wrong number of arguments")); } @@ -678,16 +678,16 @@ public void RedisLog() using var redis = ConnectionMultiplexer.Connect(TestUtils.GetConfig()); var db = redis.GetDatabase(0); - var excZero = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return redis.log()")); + var excZero = ClassicAssert.Throws(() => db.ScriptEvaluate("return redis.log()")); ClassicAssert.IsTrue(excZero.Message.StartsWith("ERR redis.log() requires two arguments or more.")); - var excOne = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return redis.log(redis.LOG_DEBUG)")); + var excOne = ClassicAssert.Throws(() => db.ScriptEvaluate("return redis.log(redis.LOG_DEBUG)")); ClassicAssert.IsTrue(excOne.Message.StartsWith("ERR redis.log() requires two arguments or more.")); - var excBadLevelType = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return redis.log('hello', 'world')")); + var excBadLevelType = ClassicAssert.Throws(() => db.ScriptEvaluate("return redis.log('hello', 'world')")); ClassicAssert.IsTrue(excBadLevelType.Message.StartsWith("ERR First argument must be a number (log level).")); - var excBadLevelValue = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return redis.log(-1, 'world')")); + var excBadLevelValue = ClassicAssert.Throws(() => db.ScriptEvaluate("return redis.log(-1, 'world')")); ClassicAssert.IsTrue(excBadLevelValue.Message.StartsWith("ERR Invalid debug level.")); // Test logs at each level @@ -723,7 +723,7 @@ public void RedisSetRepl() using var redis = ConnectionMultiplexer.Connect(TestUtils.GetConfig()); var db = redis.GetDatabase(0); - var excNotSupported = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return redis.set_repl(redis.REPL_ALL)")); + var excNotSupported = ClassicAssert.Throws(() => db.ScriptEvaluate("return redis.set_repl(redis.REPL_ALL)")); ClassicAssert.IsTrue(excNotSupported.Message.StartsWith("ERR redis.set_repl is not supported in Garnet")); var constantsDefined = (int[])db.ScriptEvaluate("return {redis.REPL_ALL, redis.REPL_AOF, redis.REPL_REPLICA, redis.REPL_SLAVE, redis.REPL_NONE}"); @@ -747,10 +747,10 @@ public void RedisDebugAndBreakpoint() using var redis = ConnectionMultiplexer.Connect(TestUtils.GetConfig()); var db = redis.GetDatabase(0); - var excBreakpoint = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("redis.breakpoint()")); + var excBreakpoint = ClassicAssert.Throws(() => db.ScriptEvaluate("redis.breakpoint()")); ClassicAssert.IsTrue(excBreakpoint.Message.StartsWith("ERR redis.breakpoint is not supported in Garnet")); - var excDebug = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("redis.debug('hello')")); + var excDebug = ClassicAssert.Throws(() => db.ScriptEvaluate("redis.debug('hello')")); ClassicAssert.IsTrue(excDebug.Message.StartsWith("ERR redis.debug is not supported in Garnet")); } @@ -765,16 +765,16 @@ public void RedisAclCheckCmd() using var denyRedis = ConnectionMultiplexer.Connect(TestUtils.GetConfig(authUsername: "deny")); var denyDB = denyRedis.GetDatabase(0); - var noArgs = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return redis.acl_check_cmd()")); + var noArgs = ClassicAssert.Throws(() => db.ScriptEvaluate("return redis.acl_check_cmd()")); ClassicAssert.IsTrue(noArgs.Message.StartsWith("ERR Please specify at least one argument for this redis lib call")); - var invalidCmdArgType = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return redis.acl_check_cmd({123})")); + var invalidCmdArgType = ClassicAssert.Throws(() => db.ScriptEvaluate("return redis.acl_check_cmd({123})")); ClassicAssert.IsTrue(invalidCmdArgType.Message.StartsWith("ERR Lua redis lib command arguments must be strings or integers")); - var invalidCmd = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return redis.acl_check_cmd('nope')")); + var invalidCmd = ClassicAssert.Throws(() => db.ScriptEvaluate("return redis.acl_check_cmd('nope')")); ClassicAssert.IsTrue(invalidCmd.Message.StartsWith("ERR Invalid command passed to redis.acl_check_cmd()")); - var invalidArgType = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return redis.acl_check_cmd('GET', {123})")); + var invalidArgType = ClassicAssert.Throws(() => db.ScriptEvaluate("return redis.acl_check_cmd('GET', {123})")); ClassicAssert.IsTrue(invalidArgType.Message.StartsWith("ERR Lua redis lib command arguments must be strings or integers")); var canRun = (bool)db.ScriptEvaluate("return redis.acl_check_cmd('GET')"); @@ -814,13 +814,13 @@ public void RedisSetResp() using var redis = ConnectionMultiplexer.Connect(TestUtils.GetConfig()); var db = redis.GetDatabase(0); - var noArgs = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("redis.setresp()")); + var noArgs = ClassicAssert.Throws(() => db.ScriptEvaluate("redis.setresp()")); ClassicAssert.IsTrue(noArgs.Message.StartsWith("ERR redis.setresp() requires one argument.")); - var tooManyArgs = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("redis.setresp(1, 2)")); + var tooManyArgs = ClassicAssert.Throws(() => db.ScriptEvaluate("redis.setresp(1, 2)")); ClassicAssert.IsTrue(tooManyArgs.Message.StartsWith("ERR redis.setresp() requires one argument.")); - var badArg = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("redis.setresp({123})")); + var badArg = ClassicAssert.Throws(() => db.ScriptEvaluate("redis.setresp({123})")); ClassicAssert.IsTrue(badArg.Message.StartsWith("ERR RESP version must be 2 or 3.")); var resp2 = db.ScriptEvaluate("redis.setresp(2)"); @@ -829,7 +829,7 @@ public void RedisSetResp() var resp3 = db.ScriptEvaluate("redis.setresp(3)"); ClassicAssert.IsTrue(resp3.IsNull); - var badRespVersion = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("redis.setresp(1)")); + var badRespVersion = ClassicAssert.Throws(() => db.ScriptEvaluate("redis.setresp(1)")); ClassicAssert.IsTrue(badRespVersion.Message.StartsWith("ERR RESP version must be 2 or 3.")); } @@ -974,7 +974,7 @@ public void ScriptExistsErrors() using var redis = ConnectionMultiplexer.Connect(TestUtils.GetConfig()); var db = redis.GetDatabase(0); - var exc = TestUtils.ThrowsRedisException(() => db.Execute("SCRIPT", "EXISTS")); + var exc = ClassicAssert.Throws(() => db.Execute("SCRIPT", "EXISTS")); ClassicAssert.AreEqual("ERR wrong number of arguments for 'script|exists' command", exc.Message); } @@ -986,13 +986,13 @@ public void ScriptFlushErrors() // > 1 args { - var exc = TestUtils.ThrowsRedisException(() => db.Execute("SCRIPT", "FLUSH", "ASYNC", "BAR")); + var exc = ClassicAssert.Throws(() => db.Execute("SCRIPT", "FLUSH", "ASYNC", "BAR")); ClassicAssert.AreEqual("ERR SCRIPT FLUSH only support SYNC|ASYNC option", exc.Message); } // 1 arg, but not ASYNC or SYNC { - var exc = TestUtils.ThrowsRedisException(() => db.Execute("SCRIPT", "FLUSH", "NOW")); + var exc = ClassicAssert.Throws(() => db.Execute("SCRIPT", "FLUSH", "NOW")); ClassicAssert.AreEqual("ERR SCRIPT FLUSH only support SYNC|ASYNC option", exc.Message); } } @@ -1020,8 +1020,8 @@ public void MultiSessionScriptFlush() _ = db1.Execute("SCRIPT", "FLUSH", "SYNC"); - var exc1 = TestUtils.ThrowsRedisException(() => db1.Execute("EVALSHA", hash, "0")); - var exc2 = TestUtils.ThrowsRedisException(() => db2.Execute("EVALSHA", hash, "0")); + var exc1 = ClassicAssert.Throws(() => db1.Execute("EVALSHA", hash, "0")); + var exc2 = ClassicAssert.Throws(() => db2.Execute("EVALSHA", hash, "0")); ClassicAssert.True(exc1.Message.StartsWith("NOSCRIPT ")); ClassicAssert.True(exc2.Message.StartsWith("NOSCRIPT ")); @@ -1065,13 +1065,13 @@ public void ScriptLoadErrors() // 0 args { - var exc = TestUtils.ThrowsRedisException(() => db.Execute("SCRIPT", "LOAD")); + var exc = ClassicAssert.Throws(() => db.Execute("SCRIPT", "LOAD")); ClassicAssert.AreEqual("ERR wrong number of arguments for 'script|load' command", exc.Message); } // > 1 args { - var exc = TestUtils.ThrowsRedisException(() => db.Execute("SCRIPT", "LOAD", "return 'foo'", "return 'bar'")); + var exc = ClassicAssert.Throws(() => db.Execute("SCRIPT", "LOAD", "return 'foo'", "return 'bar'")); ClassicAssert.AreEqual("ERR wrong number of arguments for 'script|load' command", exc.Message); } } @@ -1111,7 +1111,7 @@ public void ScriptExistsMultiple() private static void DoErroneousRedisCall(IDatabase db, string[] args, string expectedError) { - var exc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate($"return redis.call({string.Join(',', args)})")); + var exc = Assert.Throws(() => db.ScriptEvaluate($"return redis.call({string.Join(',', args)})")); ClassicAssert.IsNotNull(exc); StringAssert.StartsWith(expectedError, exc!.Message); } @@ -1381,7 +1381,7 @@ public void IntentionalOOM() var loadedScriptOOM = scriptOOM.Load(redis.GetServers()[0]); // OOM actually happens and is reported - var exc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate(loadedScriptOOM, new { Ctrl = "OOM" })); + var exc = ClassicAssert.Throws(() => db.ScriptEvaluate(loadedScriptOOM, new { Ctrl = "OOM" })); ClassicAssert.AreEqual("ERR Lua encountered an error: not enough memory", exc.Message); // We can still run the script without issue (with non-crashing args) afterwards @@ -1472,7 +1472,7 @@ public void Issue939() } // Finally, check that nil is an illegal argument - var exc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return redis.call('GET', nil)")); + var exc = ClassicAssert.Throws(() => db.ScriptEvaluate("return redis.call('GET', nil)")); ClassicAssert.True(exc.Message.StartsWith("ERR Lua redis lib command arguments must be strings or integers")); } @@ -1489,10 +1489,10 @@ public void Issue1079() using var redis = ConnectionMultiplexer.Connect(TestUtils.GetConfig()); var db = redis.GetDatabase(); - var brokenExc1 = TestUtils.ThrowsRedisException(() => db.Execute("EVAL", BrokenScript, 0)); + var brokenExc1 = ClassicAssert.Throws(() => db.Execute("EVAL", BrokenScript, 0)); ClassicAssert.True(brokenExc1.Message.StartsWith("Compilation error: ")); - var brokenExc2 = TestUtils.ThrowsRedisException(() => db.Execute("EVAL", BrokenScript, 0)); + var brokenExc2 = ClassicAssert.Throws(() => db.Execute("EVAL", BrokenScript, 0)); ClassicAssert.AreEqual(brokenExc1.Message, brokenExc2.Message); var success = (string)db.Execute("EVAL", FixedScript, 0); @@ -1824,7 +1824,7 @@ public void Resp2ToLuaConversions() ClassicAssert.AreEqual("table", simpleStringRes[0]); ClassicAssert.AreEqual("PONG", simpleStringRes[1]); - var errExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return { err = 'ERR mapped to ERR response' }")); + var errExc = ClassicAssert.Throws(() => db.ScriptEvaluate("return { err = 'ERR mapped to ERR response' }")); ClassicAssert.AreEqual("ERR mapped to ERR response", errExc.Message); var nullBulkRes = (string[])db.ScriptEvaluate("local res = redis.call('GET', KEYS[1]); return { type(res), tostring(res) };", [(RedisKey)"not-set-ever"]); @@ -1856,14 +1856,14 @@ public void NoScriptCommandsForbidden() foreach (var (cmd, _) in fullCommands) { - var exc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate($"redis.call('{cmd}')")); + var exc = ClassicAssert.Throws(() => db.ScriptEvaluate($"redis.call('{cmd}')")); ClassicAssert.True(exc.Message.StartsWith("ERR This Redis command is not allowed from script"), $"Allowed NoScript command: {cmd}"); } var subCommands = allCommands.Where(static kv => (kv.Value.SubCommands?.Length ?? 0) > 0).SelectMany(static kv => kv.Value.SubCommands.Where(static t => t.Flags.HasFlag(RespCommandFlags.NoScript)).Select(t => (kv.Key, t.Name.Split('|')[1]))); foreach (var (cmd, subCmd) in subCommands) { - var exc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate($"redis.call('{cmd}', '{subCmd}')")); + var exc = ClassicAssert.Throws(() => db.ScriptEvaluate($"redis.call('{cmd}', '{subCmd}')")); ClassicAssert.True(exc.Message.StartsWith("ERR This Redis command is not allowed from script"), $"Allowed NoScript command: {cmd}|{subCmd}"); } } @@ -1884,20 +1884,20 @@ public void PermissionsEnforced() ClassicAssert.AreEqual("bar", allowRes[0]); // Not a lot of sub commands a non-admin can run, so use CLUSTER|MYID and check the exception - var allowSubExc = TestUtils.ThrowsRedisException(() => allowDb.ScriptEvaluate("return redis.call('CLUSTER', 'MYID')")); + var allowSubExc = ClassicAssert.Throws(() => allowDb.ScriptEvaluate("return redis.call('CLUSTER', 'MYID')")); ClassicAssert.False(allowSubExc.Message.Contains("NOPERM")); - var exc = TestUtils.ThrowsRedisException(() => denyDb.ScriptEvaluate("return redis.call('GET', 'foo')")); + var exc = ClassicAssert.Throws(() => denyDb.ScriptEvaluate("return redis.call('GET', 'foo')")); ClassicAssert.IsTrue(exc.Message.Contains("NOPERM")); - var excSub = TestUtils.ThrowsRedisException(() => denyDb.ScriptEvaluate("return redis.call('CLUSTER', 'MYID')")); + var excSub = ClassicAssert.Throws(() => denyDb.ScriptEvaluate("return redis.call('CLUSTER', 'MYID')")); ClassicAssert.IsTrue(excSub.Message.Contains("NOPERM")); // SET is dispatched through a separate fast path in LuaRunner, so cover it explicitly using var denySetRedis = ConnectionMultiplexer.Connect(TestUtils.GetConfig(authUsername: "denyset")); var denySetDb = denySetRedis.GetDatabase(); - var excSet = TestUtils.ThrowsRedisException(() => denySetDb.ScriptEvaluate("return redis.call('SET', 'foo', 'baz')")); + var excSet = ClassicAssert.Throws(() => denySetDb.ScriptEvaluate("return redis.call('SET', 'foo', 'baz')")); ClassicAssert.IsTrue(excSet.Message.Contains("NOPERM"), $"Expected NOPERM for denied SET, got: {excSet.Message}"); var allowSetRes = (string[])denySetDb.ScriptEvaluate("return redis.call('GET', 'foo')"); @@ -1932,7 +1932,7 @@ public void IntentionalTimeout() var loadedScriptTimeout = scriptTimeout.Load(redis.GetServers()[0]); // Timeout actually happens and is reported - var exc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate(loadedScriptTimeout, new { Ctrl = "Timeout" })); + var exc = ClassicAssert.Throws(() => db.ScriptEvaluate(loadedScriptTimeout, new { Ctrl = "Timeout" })); ClassicAssert.AreEqual("ERR Lua script exceeded configured timeout", exc.Message); // We can still run the script without issue (with non-crashing args) afterwards @@ -2043,12 +2043,12 @@ public void Bit() // tobit { - var noArgExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return bit.tobit()")); + var noArgExc = ClassicAssert.Throws(() => db.ScriptEvaluate("return bit.tobit()")); ClassicAssert.True(noArgExc.Message.Contains("bad argument") && noArgExc.Message.Contains("tobit")); // Extra arguments are legal, but ignored - var badTypeExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return bit.tobit({})")); + var badTypeExc = ClassicAssert.Throws(() => db.ScriptEvaluate("return bit.tobit({})")); ClassicAssert.True(badTypeExc.Message.Contains("bad argument") && badTypeExc.Message.Contains("tobit")); // Rules are suprisingly subtle, so test a bunch of tricky values @@ -2085,15 +2085,15 @@ public void Bit() // tohex { - var noArgExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return bit.tohex()")); + var noArgExc = ClassicAssert.Throws(() => db.ScriptEvaluate("return bit.tohex()")); ClassicAssert.True(noArgExc.Message.Contains("bad argument") && noArgExc.Message.Contains("tohex")); // Extra arguments are legal, but ignored - var badType1Exc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return bit.tohex({})")); + var badType1Exc = ClassicAssert.Throws(() => db.ScriptEvaluate("return bit.tohex({})")); ClassicAssert.True(badType1Exc.Message.Contains("bad argument") && badType1Exc.Message.Contains("tohex")); - var badType2Exc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return bit.tohex(1, {})")); + var badType2Exc = ClassicAssert.Throws(() => db.ScriptEvaluate("return bit.tohex(1, {})")); ClassicAssert.True(badType2Exc.Message.Contains("bad argument") && badType2Exc.Message.Contains("tohex")); // Make sure casing is handled correctly @@ -2135,12 +2135,12 @@ public void Bit() // bswap { - var noArgExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return bit.bswap()")); + var noArgExc = ClassicAssert.Throws(() => db.ScriptEvaluate("return bit.bswap()")); ClassicAssert.True(noArgExc.Message.Contains("bad argument") && noArgExc.Message.Contains("bswap")); // Extra arguments are legal, but ignored - var badTypeExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return bit.bswap({})")); + var badTypeExc = ClassicAssert.Throws(() => db.ScriptEvaluate("return bit.bswap({})")); ClassicAssert.True(badTypeExc.Message.Contains("bad argument") && badTypeExc.Message.Contains("bswap")); // Just brute force a bunch of trial values @@ -2165,12 +2165,12 @@ public void Bit() // bnot { - var noArgExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return bit.bnot()")); + var noArgExc = ClassicAssert.Throws(() => db.ScriptEvaluate("return bit.bnot()")); ClassicAssert.True(noArgExc.Message.Contains("bad argument") && noArgExc.Message.Contains("bnot")); // Extra arguments are legal, but ignored - var badTypeExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return bit.bnot({})")); + var badTypeExc = ClassicAssert.Throws(() => db.ScriptEvaluate("return bit.bnot({})")); ClassicAssert.True(badTypeExc.Message.Contains("bad argument") && badTypeExc.Message.Contains("bnot")); foreach (var input in new int[] { 0, 1, 2, 4, 8, 32, 64, 128, 256, 0x70F0_F0F0, 0x6BCD_EF01, int.MinValue, int.MaxValue, -1 }) @@ -2192,16 +2192,16 @@ public void Bit() foreach (var op in ops) { - var noArgExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate($"return bit.{op.Name}()")); + var noArgExc = ClassicAssert.Throws(() => db.ScriptEvaluate($"return bit.{op.Name}()")); ClassicAssert.True(noArgExc.Message.Contains("bad argument") && noArgExc.Message.Contains(op.Name)); - var badType1Exc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate($"return bit.{op.Name}({{}})")); + var badType1Exc = ClassicAssert.Throws(() => db.ScriptEvaluate($"return bit.{op.Name}({{}})")); ClassicAssert.True(badType1Exc.Message.Contains("bad argument") && badType1Exc.Message.Contains(op.Name)); - var badType2Exc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate($"return bit.{op.Name}(1, {{}})")); + var badType2Exc = ClassicAssert.Throws(() => db.ScriptEvaluate($"return bit.{op.Name}(1, {{}})")); ClassicAssert.True(badType2Exc.Message.Contains("bad argument") && badType2Exc.Message.Contains(op.Name)); - var badType3Exc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate($"return bit.{op.Name}(1, 2, {{}})")); + var badType3Exc = ClassicAssert.Throws(() => db.ScriptEvaluate($"return bit.{op.Name}(1, 2, {{}})")); ClassicAssert.True(badType3Exc.Message.Contains("bad argument") && badType3Exc.Message.Contains(op.Name)); // Gin up some unusual values and test them in different combinations @@ -2240,13 +2240,13 @@ public void Bit() foreach (var op in ops) { - var noArgExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate($"return bit.{op.Name}()")); + var noArgExc = ClassicAssert.Throws(() => db.ScriptEvaluate($"return bit.{op.Name}()")); ClassicAssert.True(noArgExc.Message.Contains("bad argument") && noArgExc.Message.Contains(op.Name)); - var badType1Exc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate($"return bit.{op.Name}({{}})")); + var badType1Exc = ClassicAssert.Throws(() => db.ScriptEvaluate($"return bit.{op.Name}({{}})")); ClassicAssert.True(badType1Exc.Message.Contains("bad argument") && badType1Exc.Message.Contains(op.Name)); - var badType2Exc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate($"return bit.{op.Name}(1, {{}})")); + var badType2Exc = ClassicAssert.Throws(() => db.ScriptEvaluate($"return bit.{op.Name}(1, {{}})")); ClassicAssert.True(badType2Exc.Message.Contains("bad argument") && badType2Exc.Message.Contains(op.Name)); // Extra args are allowed, but ignored @@ -2272,13 +2272,13 @@ public void CJson() // Encoding { - var noArgExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return cjson.encode()")); + var noArgExc = ClassicAssert.Throws(() => db.ScriptEvaluate("return cjson.encode()")); ClassicAssert.True(noArgExc.Message.Contains("bad argument") && noArgExc.Message.Contains("encode")); - var twoArgExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return cjson.encode(1, 2)")); + var twoArgExc = ClassicAssert.Throws(() => db.ScriptEvaluate("return cjson.encode(1, 2)")); ClassicAssert.True(twoArgExc.Message.Contains("bad argument") && twoArgExc.Message.Contains("encode")); - var badTypeExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return cjson.encode((function() end))")); + var badTypeExc = ClassicAssert.Throws(() => db.ScriptEvaluate("return cjson.encode((function() end))")); ClassicAssert.True(badTypeExc.Message.Contains("Cannot serialise")); var nilResp = (string)db.ScriptEvaluate("return cjson.encode(nil)"); @@ -2353,7 +2353,7 @@ public void CJson() ClassicAssert.AreEqual(new string('[', 1000) + 1 + new string(']', 1000), deeplyNestedButLegal); var deeplyNestedExc = - TestUtils.ThrowsRedisException( + ClassicAssert.Throws( () => db.ScriptEvaluate( @"local nested = 1 for x = 1, 1001 do @@ -2368,16 +2368,16 @@ public void CJson() // Decoding { - var noArgExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return cjson.decode()")); + var noArgExc = ClassicAssert.Throws(() => db.ScriptEvaluate("return cjson.decode()")); ClassicAssert.True(noArgExc.Message.Contains("bad argument") && noArgExc.Message.Contains("decode")); - var twoArgExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return cjson.decode(1, 2)")); + var twoArgExc = ClassicAssert.Throws(() => db.ScriptEvaluate("return cjson.decode(1, 2)")); ClassicAssert.True(twoArgExc.Message.Contains("bad argument") && twoArgExc.Message.Contains("decode")); - var badTypeExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return cjson.decode({})")); + var badTypeExc = ClassicAssert.Throws(() => db.ScriptEvaluate("return cjson.decode({})")); ClassicAssert.True(badTypeExc.Message.Contains("bad argument") && badTypeExc.Message.Contains("decode")); - var badFormatExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return cjson.decode('hello world')")); + var badFormatExc = ClassicAssert.Throws(() => db.ScriptEvaluate("return cjson.decode('hello world')")); ClassicAssert.True(badFormatExc.Message.Contains("Expected value but found invalid token")); var nullDecode = (string)db.ScriptEvaluate("return type(cjson.decode('null'))"); @@ -2429,7 +2429,7 @@ public void CJson() } ClassicAssert.AreEqual(0, deeplyNestedButLegalCur.Length); - var deeplyNestedExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate($"return cjson.decode('{new string('[', 1001)}{new string(']', 1001)}')")); + var deeplyNestedExc = ClassicAssert.Throws(() => db.ScriptEvaluate($"return cjson.decode('{new string('[', 1001)}{new string(']', 1001)}')")); ClassicAssert.True(deeplyNestedExc.Message.Contains("Found too many nested data structures")); } } @@ -2440,7 +2440,7 @@ public void CMsgPackPack() using var redis = ConnectionMultiplexer.Connect(TestUtils.GetConfig()); var db = redis.GetDatabase(); - var noArgExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return cmsgpack.pack()")); + var noArgExc = ClassicAssert.Throws(() => db.ScriptEvaluate("return cmsgpack.pack()")); ClassicAssert.True(noArgExc.Message.Contains("bad argument") && noArgExc.Message.Contains("pack")); // Multiple args are legal, and concat @@ -2634,13 +2634,13 @@ public void CMsgPackUnpack() using var redis = ConnectionMultiplexer.Connect(TestUtils.GetConfig()); var db = redis.GetDatabase(); - var noArgExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return cmsgpack.unpack()")); + var noArgExc = ClassicAssert.Throws(() => db.ScriptEvaluate("return cmsgpack.unpack()")); ClassicAssert.True(noArgExc.Message.Contains("bad argument") && noArgExc.Message.Contains("unpack")); // Multiple arguments are allowed, but ignored // Table ends before it should - var badDataExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return cmsgpack.unpack('\\220\\0\\96')")); + var badDataExc = ClassicAssert.Throws(() => db.ScriptEvaluate("return cmsgpack.unpack('\\220\\0\\96')")); ClassicAssert.True(badDataExc.Message.Contains("Missing bytes in input")); var nullResp = (string)db.ScriptEvaluate($"return type(cmsgpack.unpack({ToLuaString(0xC0)}))"); @@ -2948,31 +2948,31 @@ public void StructPackOnErrors() var db = redis.GetDatabase(); // Not power of two - var excNotPowerOfTwoArgs = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.pack('!9')")); + var excNotPowerOfTwoArgs = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.pack('!9')")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to format", excNotPowerOfTwoArgs.Message); // Invalid format - var excFormat = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.pack(123, 123)")); + var excFormat = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.pack(123, 123)")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to pack", excFormat.Message); // Format opt size exceed max int size 32 - var excOptSize = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.pack('I64', 512)")); + var excOptSize = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.pack('I64', 512)")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to format", excOptSize.Message); // Format opt size integeral size overflow - var excOptSizeOverflow = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.pack('I2147483648', 512)")); + var excOptSizeOverflow = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.pack('I2147483648', 512)")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to format", excOptSizeOverflow.Message); // String too short, expected at least {size} bytes but got {l} - var excArgPack = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.pack('c6', 'hello')")); + var excArgPack = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.pack('c6', 'hello')")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to pack", excArgPack.Message); // Invalid Control Options - var excControlOptions = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.pack('@I', 'hello')")); + var excControlOptions = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.pack('@I', 'hello')")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to format", excControlOptions.Message); // Invalid alignment - var excBadAlignment = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.pack('!9c1', 'Z')")); + var excBadAlignment = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.pack('!9c1', 'Z')")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to format", excBadAlignment.Message); } @@ -3105,55 +3105,55 @@ public void StructUnpackOnErrors() var db = redis.GetDatabase(); // Invalid format - var excFormat = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.unpack(123, 123)")); + var excFormat = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.unpack(123, 123)")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to unpack", excFormat.Message); // Format opt size exceed max int size 32 - var excOptSize = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.unpack('I64', '\\000\\000\\000\\000')")); + var excOptSize = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.unpack('I64', '\\000\\000\\000\\000')")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to format", excOptSize.Message); // Format opt size integeral size overflow - var excOptSizeOverflow = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.unpack('I2147483648', '\\000\\002')")); + var excOptSizeOverflow = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.unpack('I2147483648', '\\000\\002')")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to format", excOptSizeOverflow.Message); // String too short, expected at least {size} bytes but got {l} - var excArgPack = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.unpack('c6', 'hello')")); + var excArgPack = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.unpack('c6', 'hello')")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to unpack", excArgPack.Message); // Invalid Control Options - var excControlOptions = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.unpack('@I', 'hello')")); + var excControlOptions = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.unpack('@I', 'hello')")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to format", excControlOptions.Message); // Missing argument - var excMissingArgs = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.unpack('!8')")); + var excMissingArgs = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.unpack('!8')")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to unpack", excMissingArgs.Message); // Invalid number of arguments - var excBadAlignment = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.unpack('!5', '\\157\\255\\255\\255')")); + var excBadAlignment = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.unpack('!5', '\\157\\255\\255\\255')")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to format", excBadAlignment.Message); // Invalid Third argument - var excBadThirdArg = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.unpack('@I', 'hello', 'test')")); + var excBadThirdArg = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.unpack('@I', 'hello', 'test')")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to unpack", excBadThirdArg.Message); // Invalid Third argument - var excArgTooShort = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.unpack('c4', '\\065\\066')")); + var excArgTooShort = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.unpack('c4', '\\065\\066')")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to unpack", excArgTooShort.Message); // Missing character size - var excMissingCharacterSize = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.unpack('c0', 'dynamic')")); + var excMissingCharacterSize = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.unpack('c0', 'dynamic')")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to unpack", excMissingCharacterSize.Message); // Invalid character size - var excInvalidCharacterZeroSize = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.unpack('sc0', 'size\\000dynamic')")); + var excInvalidCharacterZeroSize = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.unpack('sc0', 'size\\000dynamic')")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to unpack", excInvalidCharacterZeroSize.Message); // Invalid String without terminator - var excBadString = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.unpack('s', 'hello')")); + var excBadString = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.unpack('s', 'hello')")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to unpack", excBadString.Message); // Third pos has to be greater than 0 - var unpackBadPosRes = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.unpack('s', 'hello', 0)")); + var unpackBadPosRes = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.unpack('s', 'hello', 0)")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to unpack", unpackBadPosRes.Message); } @@ -3197,16 +3197,16 @@ public void StructSizeOnErrors() using var redis = ConnectionMultiplexer.Connect(TestUtils.GetConfig()); var db = redis.GetDatabase(); - var excFormatString = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.size('s')")); + var excFormatString = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.size('s')")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to format", excFormatString.Message); - var excFormatCharacter = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.size('c0')")); + var excFormatCharacter = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.size('c0')")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to format", excFormatCharacter.Message); - var excBadControlOptions = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.size('@I')")); + var excBadControlOptions = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.size('@I')")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to format", excBadControlOptions.Message); - var excBadFormat = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("return struct.size(123)")); + var excBadFormat = ClassicAssert.Throws(() => db.ScriptEvaluate("return struct.size(123)")); ClassicAssert.AreEqual("ERR Lua encountered an error: bad argument to format", excBadFormat.Message); } @@ -3308,10 +3308,10 @@ public void LoadString() var db = redis.GetDatabase(); // load and loadstring are not part of the sandbox allowed functions - var loadExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("local x = load('return 123'); return x()")); + var loadExc = ClassicAssert.Throws(() => db.ScriptEvaluate("local x = load('return 123'); return x()")); ClassicAssert.True(loadExc.Message.Contains("attempt to call a nil value")); - var loadstringExc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate("local x = loadstring('return 123'); return x()")); + var loadstringExc = ClassicAssert.Throws(() => db.ScriptEvaluate("local x = loadstring('return 123'); return x()")); ClassicAssert.True(loadstringExc.Message.Contains("attempt to call a nil value")); } @@ -3404,7 +3404,7 @@ public void StressTimeouts() else { // Periodically cause a timeout - var exc = TestUtils.ThrowsRedisException(() => db.ScriptEvaluate(loadedScriptTimeout, timeout)); + var exc = ClassicAssert.Throws(() => db.ScriptEvaluate(loadedScriptTimeout, timeout)); ClassicAssert.AreEqual("ERR Lua script exceeded configured timeout", exc.Message); } } @@ -3588,7 +3588,7 @@ public void LuaSubscribe_StillNoScriptBlocked() using var redis = ConnectionMultiplexer.Connect(TestUtils.GetConfig()); var db = redis.GetDatabase(0); - var exc = TestUtils.ThrowsRedisException(() => + var exc = ClassicAssert.Throws(() => db.ScriptEvaluate("return redis.call('SUBSCRIBE', KEYS[1])", [new RedisKey("any_channel")])); ClassicAssert.IsTrue(exc.Message.Contains("not allowed from script"), diff --git a/test/standalone/Garnet.test.vectorset/RespVectorSetTests.cs b/test/standalone/Garnet.test.vectorset/RespVectorSetTests.cs index 31ad9fc36dd..2d0c279514e 100644 --- a/test/standalone/Garnet.test.vectorset/RespVectorSetTests.cs +++ b/test/standalone/Garnet.test.vectorset/RespVectorSetTests.cs @@ -71,7 +71,7 @@ public void DisabledWithFeatureFlag() foreach (var cmd in vectorSetCommands) { // Should all fault before any validation - var exc = TestUtils.ThrowsRedisException(() => db.Execute(cmd.ToString())); + var exc = ClassicAssert.Throws(() => db.Execute(cmd.ToString())); ClassicAssert.AreEqual("ERR Vector Set (preview) commands are not enabled", exc.Message); } } @@ -95,22 +95,22 @@ public void WrongTypeForVectorSetOpsOnNonVectorSetKeys() switch (cmd) { case RespCommand.VADD: - exc = TestUtils.ThrowsRedisException(() => db.Execute("VADD", ["foo", "REDUCE", "50", "VALUES", "75", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", new byte[] { 0, 0, 0, 0 }, "CAS", "NOQUANT", "EF", "16", "M", "32"])); + exc = ClassicAssert.Throws(() => db.Execute("VADD", ["foo", "REDUCE", "50", "VALUES", "75", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", new byte[] { 0, 0, 0, 0 }, "CAS", "NOQUANT", "EF", "16", "M", "32"])); break; case RespCommand.VCARD: // TODO: Implement when VCARD works continue; case RespCommand.VDIM: - exc = TestUtils.ThrowsRedisException(() => db.Execute("VDIM", ["foo"])); + exc = ClassicAssert.Throws(() => db.Execute("VDIM", ["foo"])); break; case RespCommand.VEMB: - exc = TestUtils.ThrowsRedisException(() => db.Execute("VEMB", ["foo", new byte[] { 0, 0, 0, 0 }])); + exc = ClassicAssert.Throws(() => db.Execute("VEMB", ["foo", new byte[] { 0, 0, 0, 0 }])); break; case RespCommand.VGETATTR: - exc = TestUtils.ThrowsRedisException(() => db.Execute("VGETATTR", ["foo", new byte[] { 0, 0, 0, 0 }])); + exc = ClassicAssert.Throws(() => db.Execute("VGETATTR", ["foo", new byte[] { 0, 0, 0, 0 }])); break; case RespCommand.VINFO: - exc = TestUtils.ThrowsRedisException(() => db.Execute("VINFO", ["foo"])); + exc = ClassicAssert.Throws(() => db.Execute("VINFO", ["foo"])); break; case RespCommand.VISMEMBER: // TODO: Implement when VISMEMBER works @@ -122,13 +122,13 @@ public void WrongTypeForVectorSetOpsOnNonVectorSetKeys() // TODO: Implement when VRANDMEMBER works continue; case RespCommand.VREM: - exc = TestUtils.ThrowsRedisException(() => db.Execute("VREM", ["foo", new byte[] { 0, 0, 0, 0 }])); + exc = ClassicAssert.Throws(() => db.Execute("VREM", ["foo", new byte[] { 0, 0, 0, 0 }])); break; case RespCommand.VSETATTR: // TODO: Implement when VSETATTR works continue; case RespCommand.VSIM: - exc = TestUtils.ThrowsRedisException(() => db.Execute("VSIM", ["foo", "VALUES", "75", "110.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "COUNT", "5", "EPSILON", "1.0", "EF", "40"])); + exc = ClassicAssert.Throws(() => db.Execute("VSIM", ["foo", "VALUES", "75", "110.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "COUNT", "5", "EPSILON", "1.0", "EF", "40"])); break; default: throw new InvalidOperationException($"Unexpected Vector Set command: {cmd}"); @@ -182,7 +182,7 @@ public void VADD() var res5 = db.Execute("VADD", ["fizz", "REDUCE", "50", "VALUES", "75", "150.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", new byte[] { 0, 0, 0, 0 }, "CAS", "NOQUANT", "EF", "16", "M", "32"]); ClassicAssert.AreEqual(1, (int)res5); - var exc1 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", ["fizz", "VALUES", "4", "5.0", "6.0", "7.0", "8.0", new byte[] { 0, 0, 0, 1 }, "CAS", "NOQUANT", "EF", "16", "M", "32"])); + var exc1 = ClassicAssert.Throws(() => db.Execute("VADD", ["fizz", "VALUES", "4", "5.0", "6.0", "7.0", "8.0", new byte[] { 0, 0, 0, 1 }, "CAS", "NOQUANT", "EF", "16", "M", "32"])); ClassicAssert.AreEqual("ERR Vector dimension mismatch - got 4 but set has 75", exc1.Message); // Add without specifying EF after first vector @@ -190,11 +190,11 @@ public void VADD() ClassicAssert.AreEqual(1, (int)res6); // Add without specifying M after first vector - var exc2 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", ["fizz", "REDUCE", "50", "VALUES", "75", "180.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", new byte[] { 0, 0, 0, 4 }, "CAS", "NOQUANT", "EF", "16"])); + var exc2 = ClassicAssert.Throws(() => db.Execute("VADD", ["fizz", "REDUCE", "50", "VALUES", "75", "180.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", new byte[] { 0, 0, 0, 4 }, "CAS", "NOQUANT", "EF", "16"])); ClassicAssert.AreEqual("ERR asked M value mismatch with existing vector set", exc2.Message); // Mismatch vector size for projection - var exc3 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", ["fizz", "REDUCE", "50", "VALUES", "5", "1.0", "2.0", "3.0", "4.0", "5.0", new byte[] { 0, 0, 0, 0 }, "CAS", "NOQUANT", "EF", "16", "M", "32"])); + var exc3 = ClassicAssert.Throws(() => db.Execute("VADD", ["fizz", "REDUCE", "50", "VALUES", "5", "1.0", "2.0", "3.0", "4.0", "5.0", new byte[] { 0, 0, 0, 0 }, "CAS", "NOQUANT", "EF", "16", "M", "32"])); ClassicAssert.AreEqual("ERR REDUCE dimension must be <= vector dimensions", exc3.Message); } @@ -258,7 +258,7 @@ public void VADDXPREQB8() } // REDUCE not allowed with XPREQ8 - var exc1 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", ["fizz", "REDUCE", "2", "XB8", smallVectorData, new byte[] { 0, 0, 0, 0 }, "XPREQ8"])); + var exc1 = ClassicAssert.Throws(() => db.Execute("VADD", ["fizz", "REDUCE", "2", "XB8", smallVectorData, new byte[] { 0, 0, 0, 0 }, "XPREQ8"])); ClassicAssert.AreEqual("ERR asked quantization mismatch with existing vector set", exc1.Message); // Create a vector set with XB8 + XPREQ8 @@ -287,61 +287,61 @@ public void VADDErrors() var vectorSetKey = $"{nameof(VADDErrors)}_{Guid.NewGuid()}"; // Bad arity - var exc1 = TestUtils.ThrowsRedisException(() => db.Execute("VADD")); + var exc1 = ClassicAssert.Throws(() => db.Execute("VADD")); ClassicAssert.AreEqual("ERR wrong number of arguments for 'VADD' command", exc1.Message); - var exc2 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey])); + var exc2 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey])); ClassicAssert.AreEqual("ERR wrong number of arguments for 'VADD' command", exc2.Message); - var exc3 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "FP32"])); + var exc3 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "FP32"])); ClassicAssert.AreEqual("ERR wrong number of arguments for 'VADD' command", exc3.Message); - var exc4 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES"])); + var exc4 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES"])); ClassicAssert.AreEqual("ERR wrong number of arguments for 'VADD' command", exc4.Message); - var exc5 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1"])); + var exc5 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1"])); ClassicAssert.AreEqual("ERR wrong number of arguments for 'VADD' command", exc5.Message); - var exc6 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "1.0"])); + var exc6 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "1.0"])); ClassicAssert.AreEqual("ERR wrong number of arguments for 'VADD' command", exc6.Message); // Reduce after vector - var exc7 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "2", "1.0", "2.0", "bar", "REDUCE", "1"])); + var exc7 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "2", "1.0", "2.0", "bar", "REDUCE", "1"])); ClassicAssert.AreEqual("ERR invalid option after element", exc7.Message); // Duplicate flags // TODO: Redis doesn't error on these which seems... wrong, confirm with them - //var exc8 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "CAS", "CAS"])); - //var exc9 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "NOQUANT", "Q8"])); - //var exc10 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "EF", "1", "EF", "1"])); - //var exc11 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "SETATTR", "abc", "SETATTR", "abc"])); - //var exc12 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "M", "5", "M", "5"])); + //var exc8 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "CAS", "CAS"])); + //var exc9 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "NOQUANT", "Q8"])); + //var exc10 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "EF", "1", "EF", "1"])); + //var exc11 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "SETATTR", "abc", "SETATTR", "abc"])); + //var exc12 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "M", "5", "M", "5"])); // M out of range (Redis imposes M >= 4 and m <= 4096 - var exc13 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "M", "1"])); + var exc13 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "M", "1"])); ClassicAssert.AreEqual("ERR M must be an integer between 4 and 4096", exc13.Message); - var exc14 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "M", "10000"])); + var exc14 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "M", "10000"])); ClassicAssert.AreEqual("ERR M must be an integer between 4 and 4096", exc14.Message); // Missing/bad option value - var exc20 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "EF"])); + var exc20 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "EF"])); ClassicAssert.AreEqual("ERR invalid option after element", exc20.Message); - var exc21 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "EF", "0"])); + var exc21 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "EF", "0"])); ClassicAssert.AreEqual("ERR EF must be an integer between 1 and 1000000", exc21.Message); - var exc22 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "SETATTR"])); + var exc22 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "SETATTR"])); ClassicAssert.AreEqual("ERR invalid option after element", exc22.Message); - var exc23 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "M"])); + var exc23 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "M"])); ClassicAssert.AreEqual("ERR invalid option after element", exc23.Message); - var exc24 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "2", "2.0", "bar"])); + var exc24 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "2", "2.0", "bar"])); ClassicAssert.AreEqual("ERR invalid vector specification", exc24.Message); - var exc25 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "0", "bar"])); + var exc25 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "0", "bar"])); ClassicAssert.AreEqual("ERR invalid vector specification", exc25.Message); - var exc26 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "fizz", "bar"])); + var exc26 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "fizz", "bar"])); ClassicAssert.AreEqual("ERR invalid vector specification", exc26.Message); // Unknown option - var exc27 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "FOO"])); + var exc27 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "1", "2.0", "bar", "FOO"])); ClassicAssert.AreEqual("ERR invalid option after element", exc27.Message); // Malformed FP32 var binary = new float[] { 1, 2, 3 }; var blob = MemoryMarshal.Cast(binary)[..^1].ToArray(); - var exc15 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "FP32", blob, "bar"])); + var exc15 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "FP32", blob, "bar"])); ClassicAssert.AreEqual("ERR invalid vector specification", exc15.Message); // Mismatch after creating a vector set @@ -349,41 +349,41 @@ public void VADDErrors() _ = db.Execute("VADD", [vectorSetKey, "VALUES", "75", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", new byte[] { 0, 0, 1, 0 }, "NOQUANT", "EF", "6", "M", "10", "XDISTANCE_METRIC", "L2"]); - var exc16 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "2", "1.0", "2.0", "fizz", "NOQUANT", "EF", "6", "M", "10"])); + var exc16 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "2", "1.0", "2.0", "fizz", "NOQUANT", "EF", "6", "M", "10"])); ClassicAssert.AreEqual("ERR Vector dimension mismatch - got 2 but set has 75", exc16.Message); - var exc17 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "75", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "fizz", "XPREQ8", "EF", "6", "M", "10"])); + var exc17 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "75", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "fizz", "XPREQ8", "EF", "6", "M", "10"])); ClassicAssert.AreEqual("ERR asked quantization mismatch with existing vector set", exc17.Message); - var exc18 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "75", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "fizz", "NOQUANT", "EF", "12", "M", "20"])); + var exc18 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "75", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "fizz", "NOQUANT", "EF", "12", "M", "20"])); ClassicAssert.AreEqual("ERR asked M value mismatch with existing vector set", exc18.Message); // TODO: Redis doesn't appear to validate attributes... so that's weird // Empty Vector Set keys are forbidden (TODO: Remove this constraint) - var exc19 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", ["", "VALUES", "75", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", new byte[] { 0, 0, 0, 0 }, "XPREQ8"])); + var exc19 = ClassicAssert.Throws(() => db.Execute("VADD", ["", "VALUES", "75", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", new byte[] { 0, 0, 0, 0 }, "XPREQ8"])); ClassicAssert.AreEqual("ERR Vector Set key cannot be empty", exc19.Message); // Malformed XDISTANCE_METRIC - var exc31 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "75", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "bar", "NOQUANT", "XDISTANCE_METRIC"])); + var exc31 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "75", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "bar", "NOQUANT", "XDISTANCE_METRIC"])); ClassicAssert.AreEqual("ERR invalid option after element", exc31.Message); - var exc32 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "75", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "bar", "NOQUANT", "XDISTANCE_METRIC", "FOO"])); + var exc32 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "75", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "bar", "NOQUANT", "XDISTANCE_METRIC", "FOO"])); ClassicAssert.AreEqual("ERR invalid XDISTANCE_METRIC", exc32.Message); // Invalid vector type keyword (not FP32, VALUES, or XB8) - var exc40 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", ["mykey", "GARBAGE", "data", "elem1"])); + var exc40 = ClassicAssert.Throws(() => db.Execute("VADD", ["mykey", "GARBAGE", "data", "elem1"])); ClassicAssert.AreEqual("ERR invalid vector specification", exc40.Message); // VALUES count exceeding MaxVectorDimensions (65536) must be rejected - var exc41 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", ["foo", "VALUES", "100000", "1.0", "elem"])); + var exc41 = ClassicAssert.Throws(() => db.Execute("VADD", ["foo", "VALUES", "100000", "1.0", "elem"])); ClassicAssert.IsTrue(exc41.Message.Contains("maximum"), $"Expected dimension limit error, got: {exc41.Message}"); // EF exceeding MaxExplorationFactor (1,000,000) must be rejected - var exc42 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", ["foo", "VALUES", "3", "1.0", "2.0", "3.0", new byte[] { 0, 0, 0, 0 }, "CAS", "NOQUANT", "EF", "2000000000", "M", "32"])); + var exc42 = ClassicAssert.Throws(() => db.Execute("VADD", ["foo", "VALUES", "3", "1.0", "2.0", "3.0", new byte[] { 0, 0, 0, 0 }, "CAS", "NOQUANT", "EF", "2000000000", "M", "32"])); ClassicAssert.IsTrue(exc42.Message.Contains("EF must be an integer between"), $"Expected EF validation error, got: {exc42.Message}"); // REDUCE dim exceeding vector dimensions must be rejected - var exc43 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", ["foo", "REDUCE", "100000", "VALUES", "3", "1.0", "2.0", "3.0", new byte[] { 0, 0, 0, 0 }, "CAS", "NOQUANT", "EF", "16", "M", "32"])); + var exc43 = ClassicAssert.Throws(() => db.Execute("VADD", ["foo", "REDUCE", "100000", "VALUES", "3", "1.0", "2.0", "3.0", new byte[] { 0, 0, 0, 0 }, "CAS", "NOQUANT", "EF", "16", "M", "32"])); ClassicAssert.IsTrue(exc43.Message.Contains("REDUCE dimension must be <= vector dimensions"), $"Expected REDUCE dimension limit error, got: {exc43.Message}"); - var exc33 = TestUtils.ThrowsRedisException(() => db.Execute("VADD", [vectorSetKey, "VALUES", "75", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "bar", "NOQUANT", "XDISTANCE_METRIC", "XCOSINE_NORMALIZED"])); + var exc33 = ClassicAssert.Throws(() => db.Execute("VADD", [vectorSetKey, "VALUES", "75", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "bar", "NOQUANT", "XDISTANCE_METRIC", "XCOSINE_NORMALIZED"])); ClassicAssert.AreEqual("ERR Distance metric mismatch - got XCosine_Normalized but set has L2", exc33.Message); } @@ -437,7 +437,7 @@ public void VectorSetOpacity() var res1 = db.Execute("VADD", ["foo", "REDUCE", "50", "VALUES", "75", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", "4.0", "1.0", "2.0", "3.0", new byte[] { 0, 0, 0, 0 }, "CAS", "NOQUANT", "EF", "16", "M", "32"]); ClassicAssert.AreEqual(1, (int)res1); - var res2 = TestUtils.ThrowsRedisException(() => db.StringGet("foo")); + var res2 = ClassicAssert.Throws(() => db.StringGet("foo")); ClassicAssert.True(res2.Message.Contains("WRONGTYPE")); } @@ -793,7 +793,7 @@ public void VDIM() var res4 = db.Execute("VDIM", "bar"); ClassicAssert.AreEqual(75, (int)res4); - var exc1 = TestUtils.ThrowsRedisException(() => db.Execute("VDIM", "fizz")); + var exc1 = ClassicAssert.Throws(() => db.Execute("VDIM", "fizz")); ClassicAssert.IsTrue(exc1.Message.Contains("Key not found")); // TODO: Add WRONGTYPE behavior check once implemented @@ -1057,7 +1057,7 @@ public void VSIMBadFilters() foreach (var (filter, why) in badFilters) { - var exc = TestUtils.ThrowsRedisException( + var exc = ClassicAssert.Throws( () => db.Execute("VSIM", [VectorSet, "VALUES", "3", "0.0", "0.0", "0.0", "FILTER", filter, "COUNT", "10"]), $"Expected compile failure for filter '{filter}' ({why})"); ClassicAssert.AreEqual(CompileErr, exc.Message, $"Wrong error message for filter '{filter}' ({why})"); @@ -1246,19 +1246,19 @@ public void VSIMErrors() ClassicAssert.AreEqual(1, (int)res1); // FILTER-EF exceeding MaxFilteringScaleFactor must be rejected - var exc1 = TestUtils.ThrowsRedisException(() => db.Execute("VSIM", ["foo", "VALUES", "3", "0.0", "0.0", "0.0", "FILTER", ".year > 1950", "FILTER-EF", "999999999", "COUNT", "3", "WITHATTRIBS"])); + var exc1 = ClassicAssert.Throws(() => db.Execute("VSIM", ["foo", "VALUES", "3", "0.0", "0.0", "0.0", "FILTER", ".year > 1950", "FILTER-EF", "999999999", "COUNT", "3", "WITHATTRIBS"])); ClassicAssert.AreEqual("ERR FILTER-EF must be an integer between 4 and 256", exc1.Message); // COUNT exceeding MaxRetrieveCount must be rejected - var exc2 = TestUtils.ThrowsRedisException(() => db.Execute("VSIM", ["foo", "VALUES", "3", "0.0", "0.0", "0.0", "COUNT", "999999999"])); + var exc2 = ClassicAssert.Throws(() => db.Execute("VSIM", ["foo", "VALUES", "3", "0.0", "0.0", "0.0", "COUNT", "999999999"])); ClassicAssert.AreEqual("ERR COUNT must be an integer between 0 and 100000000", exc2.Message); // VALUES count exceeding MaxVectorDimensions (65536) must be rejected - var exc3 = TestUtils.ThrowsRedisException(() => db.Execute("VSIM", ["foo", "VALUES", "100000", "1.0"])); + var exc3 = ClassicAssert.Throws(() => db.Execute("VSIM", ["foo", "VALUES", "100000", "1.0"])); ClassicAssert.AreEqual("ERR vector exceeds maximum of 65536 dimensions", exc3.Message); // EF exceeding MaxExplorationFactor (1,000,000) must be rejected - var exc4 = TestUtils.ThrowsRedisException(() => db.Execute("VSIM", ["foo", "VALUES", "3", "0.0", "0.0", "0.0", "EF", "2000000000"])); + var exc4 = ClassicAssert.Throws(() => db.Execute("VSIM", ["foo", "VALUES", "3", "0.0", "0.0", "0.0", "EF", "2000000000"])); ClassicAssert.AreEqual("ERR EF must be an integer between 1 and 1000000", exc4.Message); } @@ -1699,7 +1699,7 @@ public void RepeatedVectorSetDeletes() var addRes2 = (int)db.Execute("VADD", ["foo", "XB8", bytes2, new byte[] { 0, 0, 0, 1 }, "XPREQ8"]); ClassicAssert.AreEqual(1, addRes2); - var readExc = TestUtils.ThrowsRedisException(() => db.Execute("GET", ["foo"])); + var readExc = ClassicAssert.Throws(() => db.Execute("GET", ["foo"])); ClassicAssert.IsTrue(readExc.Message.Equals("WRONGTYPE Operation against a key holding the wrong kind of value."), $"In iteration: {i}"); } @@ -2798,7 +2798,7 @@ public void VREM() // Remove on non-vector set fails // TODO: test against Redis, how do they respond (I expect WRONGTYPE, but needs verification) //_ = db.StringSet("fizz", "buzz"); - //var exc1 = TestUtils.ThrowsRedisException(() => db.Execute("VREM", "fizz", new byte[] { 0, 0, 0, 0 })); + //var exc1 = ClassicAssert.Throws(() => db.Execute("VREM", "fizz", new byte[] { 0, 0, 0, 0 })); //ClassicAssert.AreEqual("", exc1.Message); // Remove exists diff --git a/test/standalone/Garnet.test/TestUtils.cs b/test/standalone/Garnet.test/TestUtils.cs index ead5679f5d1..886e732a36f 100644 --- a/test/standalone/Garnet.test/TestUtils.cs +++ b/test/standalone/Garnet.test/TestUtils.cs @@ -12,7 +12,6 @@ using System.Net.Security; using System.Net.Sockets; using System.Runtime.CompilerServices; -using System.Runtime.ExceptionServices; using System.Security.Cryptography; using System.Security.Cryptography.X509Certificates; using System.Text; @@ -280,63 +279,6 @@ public static void WaitUntilNextSecond(IDatabase db, long baseSeconds) } } - /// - /// Asserts that throws , running it on a dedicated thread. - /// - /// - /// StackExchange.Redis' synchronous request path (ConnectionMultiplexer.ExecuteSyncImpl) parks the caller on a - /// [ThreadStatic] result box. A server error reply faults that box from the connection's reader thread, and when - /// the fault lands before the caller reaches Monitor.Wait the caller skips the wait and recycles the box while the - /// reader thread still has a Monitor.PulseAll outstanding on it. The next synchronous call on the same thread - /// takes the recycled box, consumes that stale pulse, and returns with neither a result nor an exception, shifting every - /// subsequent reply on that thread by one. Running calls that are expected to fail on their own thread keeps the - /// recycled box off the shared test thread. - /// - /// Expected exception type. - /// Delegate performing the call that is expected to fail. - /// The exception thrown by . - public static TActual ThrowsRedisException(TestDelegate code) where TActual : Exception - => ClassicAssert.Throws(CaptureOnDedicatedThread(code)); - - /// - /// Variant of that reports on failure. - /// - /// Expected exception type. - /// Delegate performing the call that is expected to fail. - /// Message reported when no matching exception is thrown. - /// Format arguments for . - /// The exception thrown by . - public static TActual ThrowsRedisException(TestDelegate code, string message, params object[] args) where TActual : Exception - => ClassicAssert.Throws(CaptureOnDedicatedThread(code), message, args); - - /// - /// Runs to completion on a dedicated thread, returning a delegate that replays whatever it threw. - /// - /// Delegate to run. - /// Delegate that rethrows the captured exception, preserving its original stack trace. - private static TestDelegate CaptureOnDedicatedThread(TestDelegate code) - { - ExceptionDispatchInfo captured = null; - - var thread = new Thread(() => - { - try - { - code(); - } - catch (Exception ex) - { - captured = ExceptionDispatchInfo.Capture(ex); - } - }) - { IsBackground = true }; - - thread.Start(); - thread.Join(); - - return () => captured?.Throw(); - } - /// /// Create GarnetServer /// From c0d6d9ceee8199972f56cef570d1e8cb78201a35 Mon Sep 17 00:00:00 2001 From: Badrish Chandramouli Date: Thu, 10 Sep 2026 21:02:03 -0700 Subject: [PATCH 16/27] Recover Vector Sets from the whole log, not just the snapshot Two independent recovery defects, both of which predate this branch. A live Vector Set could lose every one of its element records. ReconcileRecoveredState decides liveness by absence: a context marked in use in ContextMetadata but missing from recoveredIndexes is treated as abandoned, and the cleanup task then deletes the set's element records. But recoveredIndexes was populated only from OnRecoverySnapshotRead, which the store raises only for records inside the checkpoint snapshot's address range - ClearBitsOnPage gates on snapshotFromAddress > 0, and the main-log pass supplies 0. An index record flushed to the main log before the checkpoint is therefore recovered but never reported, so its set is judged abandoned. The stub record survives, which is why the key still exists and VDIM still answers while VEMB returns nothing at all. Completing the census from the recovered store is what fixes it, so liveness is decided against every index record rather than the subset the snapshot happened to cover. The array of ContextMetadata now spans recovered metadata and recovered index contexts both, because the metadata record holding a context can be missing when its index record is not. Recovery could also abort outright. A snapshot covers a range, so it legitimately presents several versions of one metadata record when an update lands in the checkpoint's fuzzy region and is copied to the tail. RecoveredContextMetadata assumed at most one and threw. Recovery swallows that, leaving the server up on a partially recovered store. ContextMetadata.Version already ordered these versions and simply was not consulted; newest now wins. Empty versions are kept until ReconcileRecoveredState trims them, so a newest version that released its last context still supersedes older non-empty ones. Both regression tests are deterministic rather than timing luck. The first fills pages through the product's own flush path and guards on FlushedUntilAddress > BeginAddress so it cannot silently stop covering the case. The second choreographs two [Conditional("DEBUG")] barriers to release a metadata RMW into the new version while the checkpoint is parked before it captures finalLogicalAddress. Ablation, taskset -c 0-1: with the fix reverted all four executions fail, two with "Expected: 8 But was: 0" and two with "Recovered multiple instances of the same ContextMetadata"; with it applied all four pass. Full suite 404/404. No hot path is touched. Both sites run only during recovery, and the added scan uses the same IterateLookupSnapshot facility the cleanup task already runs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 30f31d5b-9e53-4111-91a3-e16c94510a77 --- libs/common/Testing/ExceptionInjectionType.cs | 11 ++ .../Vector/VectorManager.ContextMetadata.cs | 2 + libs/server/Resp/Vector/VectorManager.cs | 93 +++++++++-- .../Storage/Functions/GarnetRecordTriggers.cs | 2 + .../RespVectorSetTests.cs | 157 ++++++++++++++++++ 5 files changed, 251 insertions(+), 14 deletions(-) diff --git a/libs/common/Testing/ExceptionInjectionType.cs b/libs/common/Testing/ExceptionInjectionType.cs index e9745056d02..d3539b42097 100644 --- a/libs/common/Testing/ExceptionInjectionType.cs +++ b/libs/common/Testing/ExceptionInjectionType.cs @@ -125,6 +125,17 @@ public enum ExceptionInjectionType /// Vector Set: pause in the index-recreate window before RMW write refreshed index pointer back to the main store. /// VectorSet_Pause_Before_Recreate_Rmw, + + /// + /// Pause before the RMW that persists a ContextMetadata record. + /// + VectorSet_Pause_Before_Context_Metadata_Rmw, + + /// + /// Pause when a checkpoint reaches the end of its fuzzy region, before it captures the + /// last address the snapshot covers. + /// + Checkpoint_Pause_At_Flush_Begin, /// /// Cluster gossip: fail a single round of the main gossip loop, to exercise recovery from a transient /// error raised while connections are initialized or the configuration is broadcast. diff --git a/libs/server/Resp/Vector/VectorManager.ContextMetadata.cs b/libs/server/Resp/Vector/VectorManager.ContextMetadata.cs index af33eaeda27..5657a5ee3b1 100644 --- a/libs/server/Resp/Vector/VectorManager.ContextMetadata.cs +++ b/libs/server/Resp/Vector/VectorManager.ContextMetadata.cs @@ -661,6 +661,8 @@ private void UpdateContextMetadata(ref VectorBasicContext ctx) input.CallbackContext = (nint)Unsafe.AsPointer(ref MemoryMarshal.GetReference(dataSpan)); } + ExceptionInjectionHelper.ResetAndWait(ExceptionInjectionType.VectorSet_Pause_Before_Context_Metadata_Rmw); + var status = ctx.RMW(key, ref input); if (status.IsPending) diff --git a/libs/server/Resp/Vector/VectorManager.cs b/libs/server/Resp/Vector/VectorManager.cs index 095e0087759..043f07bdd8b 100644 --- a/libs/server/Resp/Vector/VectorManager.cs +++ b/libs/server/Resp/Vector/VectorManager.cs @@ -6,7 +6,6 @@ using System.Buffers.Binary; using System.Collections.Concurrent; using System.Diagnostics; -using System.Linq; using System.Runtime.InteropServices; using System.Text; using System.Threading.Channels; @@ -275,6 +274,14 @@ public void ReconcileRecoveredState(bool requireNoReservedContexts = false) ref var ctx = ref session.storageSession.vectorBasicContext; + // recoveredIndexes only holds the index records the store reported while ingesting a checkpoint + // snapshot, which covers a single range of the log - anything already flushed to the main log + // before the checkpoint is recovered without being reported. Liveness below is decided by + // absence, so completing the set from the recovered store is what keeps a live Vector Set from + // being mistaken for an abandoned context and having its element records deleted. + RecoveredIndexScanFunctions indexScan = new(this); + _ = session.storageSession.stringBasicContext.Session.IterateLookupSnapshot(ref indexScan); + var needsUpdated = false; lock (this) @@ -292,14 +299,35 @@ public void ReconcileRecoveredState(bool requireNoReservedContexts = false) } // Any ContextMetadatas we found need to be restored - if (!recoveredMetadata.IsEmpty) + // + // The array must also span every recovered index, because the reservation for a context is + // restored below through contextMetadatas[contextIndex] - the metadata record holding that + // context can be missing when the index record is not. + var maxIndex = -1; + foreach (var (index, metadata) in recoveredMetadata) + { + if (!metadata.IsEmpty && index > maxIndex) + { + maxIndex = index; + } + } + + foreach (var context in recoveredIndexes.Keys) + { + var (contextIndex, _) = ContextMetadata.DecomposeContext(context); + if (contextIndex > maxIndex) + { + maxIndex = contextIndex; + } + } + + if (maxIndex >= 0) { - var maxContext = recoveredMetadata.Keys.Max(); - contextMetadatas = new ContextMetadata[maxContext + 1]; + contextMetadatas = new ContextMetadata[maxIndex + 1]; for (var i = 0; i < contextMetadatas.Length; i++) { - if (!recoveredMetadata.TryGetValue(i, out contextMetadatas[i])) + if (!recoveredMetadata.TryGetValue(i, out contextMetadatas[i]) || contextMetadatas[i].IsEmpty) { contextMetadatas[i] = new(); } @@ -405,6 +433,41 @@ public void ReconcileRecoveredState(bool requireNoReservedContexts = false) _ = cleanupTaskChannel.TryPublish(); } + /// + /// Collects every live Vector Set index record in the store into . + /// + private sealed class RecoveredIndexScanFunctions : IScanIteratorFunctions + { + private readonly VectorManager manager; + + internal RecoveredIndexScanFunctions(VectorManager manager) + { + this.manager = manager; + } + + public void OnException(Exception exception, long numberOfRecords) { } + public bool OnStart(long beginAddress, long endAddress) => true; + public void OnStop(bool completed, long numberOfRecords) { } + + /// + public bool Reader(in TSourceLogRecord logRecord, RecordMetadata recordMetadata, long numberOfRecords, out CursorRecordResult cursorRecordResult) + where TSourceLogRecord : ISourceLogRecord + { + cursorRecordResult = CursorRecordResult.Skip; + + if (logRecord.HasNamespace || logRecord.RecordType != RecordType || logRecord.ValueSpan.Length != IndexSize) + { + return true; + } + + ReadIndex(logRecord.ValueSpan, out var context, out _, out _, out _, out _, out _, out _, out _, out _); + manager.recoveredIndexes[context] = HashSlotUtils.HashSlot(logRecord.Key); + + cursorRecordResult = CursorRecordResult.Accept; + return true; + } + } + /// /// Called during recovery for each Vector Set index key. /// @@ -435,17 +498,19 @@ public void RecoveredContextMetadata(ref TSourceLogRecord reco var index = BinaryPrimitives.ReadInt32LittleEndian(record.Key); var metadata = MemoryMarshal.Cast(record.ValueSpan)[0]; - // During recovery, we can trim off empty ContextMetadata + // A snapshot covers a range of the log, so it holds every version of a record written in that + // range - the same index is legitimately presented more than once, newest version last only if + // each update was copied to the tail. ContextMetadata.Version orders them, and only the newest + // describes the state to restore. // - // ReconcileRecoveredState will fill in any gaps this causes - if (metadata.IsEmpty) - { - return; - } - - if (!recoveredMetadata.TryAdd(index, metadata)) + // Empty entries are kept here and trimmed in ReconcileRecoveredState, so that a newest version + // which released its last context still supersedes the older non-empty versions. + lock (recoveredMetadata) { - throw new GarnetException($"Recovered multiple instances of the same ContextMetadata: {index}"); + if (!recoveredMetadata.TryGetValue(index, out var existing) || metadata.Version >= existing.Version) + { + recoveredMetadata[index] = metadata; + } } } diff --git a/libs/server/Storage/Functions/GarnetRecordTriggers.cs b/libs/server/Storage/Functions/GarnetRecordTriggers.cs index d9e121e9f99..2fd9c54ce44 100644 --- a/libs/server/Storage/Functions/GarnetRecordTriggers.cs +++ b/libs/server/Storage/Functions/GarnetRecordTriggers.cs @@ -2,6 +2,7 @@ // Licensed under the MIT license. using System; +using Garnet.common; using Tsavorite.core; namespace Garnet.server @@ -180,6 +181,7 @@ public readonly void OnCheckpoint(CheckpointTrigger trigger, Guid checkpointToke case CheckpointTrigger.FlushBegin: rangeIndexManager?.SnapshotAllTreesForCheckpoint(checkpointToken); rangeIndexManager?.ClearCheckpointBarrier(); + ExceptionInjectionHelper.ResetAndWait(ExceptionInjectionType.Checkpoint_Pause_At_Flush_Begin); break; case CheckpointTrigger.CheckpointCompleted: vectorManager?.CheckpointCompleted(); diff --git a/test/standalone/Garnet.test.vectorset/RespVectorSetTests.cs b/test/standalone/Garnet.test.vectorset/RespVectorSetTests.cs index 2d0c279514e..57b2e690b09 100644 --- a/test/standalone/Garnet.test.vectorset/RespVectorSetTests.cs +++ b/test/standalone/Garnet.test.vectorset/RespVectorSetTests.cs @@ -4295,6 +4295,163 @@ public async Task VSETATTRAsync() ClassicAssert.IsNull(res8); } + /// + /// A Vector Set whose index record was flushed to the main log before the checkpoint was taken must + /// survive recovery: the checkpoint snapshot only reports records in its own range, so the index + /// record is recovered from the main log without ever being reported, and the recovered context must + /// not be mistaken for an abandoned one and swept by the cleanup task. + /// + [Test] + public async Task RecoverVectorSetFlushedBeforeCheckpointAsync() + { + // This test drives the log through a real flush, so it needs its own small-page server. + server.Dispose(deleteDir: true); + TestUtils.DeleteDirectory(TestUtils.MethodTestDir, wait: true); + server = CreateSmallPageGarnetServer(tryRecover: false); + server.Start(); + server.Provider.StoreWrapper.DefaultDatabase.VectorManager.AllocateTestContexts(preAllocatedContexts); + + var vector = Enumerable.Range(0, 8).Select(static x => (byte)x).ToArray(); + var element = new byte[] { 0, 0, 0, 0 }; + + using (var redis = ConnectionMultiplexer.Connect(TestUtils.GetConfig(allowAdmin: true))) + { + var db = redis.GetDatabase(0); + + ClassicAssert.IsTrue(db.StringSet("control", "controlvalue")); + _ = db.Execute("VADD", ["flushed", "XB8", vector, element, "CAS", "NOQUANT", "EF", "16", "M", "32"]); + + // Fill past the tail page so the product's own page-fill path moves the read-only boundary and + // flushes "flushed" to the main log well before the checkpoint below starts. + for (var i = 0; i < 4_000; i++) + { + ClassicAssert.IsTrue(db.StringSet($"filler:{i}", new string('x', 64))); + } + + var log = server.Provider.StoreWrapper.DefaultDatabase.Store.Log; + ClassicAssert.Greater(log.FlushedUntilAddress, log.BeginAddress, "the filler writes did not flush any of the log, so this test would not cover the flushed-before-checkpoint case"); + + _ = db.Execute("VADD", ["snapshotted", "XB8", vector, element, "CAS", "NOQUANT", "EF", "16", "M", "32"]); + +#pragma warning disable CS0618 // ForegroundSave is deprecated in the client, but is what this test needs + redis.GetServers()[0].Save(SaveType.ForegroundSave); +#pragma warning restore CS0618 + _ = await server.Store.WaitForCommitAsync(); + } + + server.Dispose(deleteDir: false); + server = CreateSmallPageGarnetServer(tryRecover: true); + server.Start(); + + using (var redis = ConnectionMultiplexer.Connect(TestUtils.GetConfig())) + { + var db = redis.GetDatabase(0); + + // Plain keys prove recovery itself worked, isolating any Vector Set loss below. + ClassicAssert.AreEqual("controlvalue", (string)db.StringGet("control")); + ClassicAssert.AreEqual(new string('x', 64), (string)db.StringGet("filler:0")); + + foreach (var key in new[] { "flushed", "snapshotted" }) + { + var embedding = (string[])db.Execute("VEMB", [key, element]); + ClassicAssert.AreEqual(8, embedding?.Length ?? 0, $"VEMB {key} did not return the recovered embedding"); + } + } + } + +#if DEBUG + /// + /// A ContextMetadata update that lands inside a checkpoint's fuzzy region is copied to the tail, so the + /// snapshot holds both the old and the new version of that record. Recovery must take the newest + /// version rather than treating the pair as corruption and aborting. + /// + [Test] + public async Task RecoverContextMetadataUpdatedInsideFuzzyRegionAsync() + { + TestUtils.IgnoreIfExceptionInjectionDisabled(); + + const ExceptionInjectionType MetadataPause = ExceptionInjectionType.VectorSet_Pause_Before_Context_Metadata_Rmw; + const ExceptionInjectionType CheckpointPause = ExceptionInjectionType.Checkpoint_Pause_At_Flush_Begin; + + var vector = Enumerable.Range(0, 8).Select(static x => (byte)x).ToArray(); + var element = new byte[] { 0, 0, 0, 0 }; + + using (var redis = ConnectionMultiplexer.Connect(TestUtils.GetConfig(allowAdmin: true))) + { + var db = redis.GetDatabase(0); + + // First version of the ContextMetadata record. + _ = db.Execute("VADD", ["before", "XB8", vector, element, "CAS", "NOQUANT", "EF", "16", "M", "32"]); + + ExceptionInjectionHelper.EnableException(MetadataPause); + try + { + // Park a second VADD just before it persists its ContextMetadata update. + var vaddTask = Task.Run(() => + { + using var vaddRedis = ConnectionMultiplexer.Connect(TestUtils.GetConfig()); + return vaddRedis.GetDatabase(0).Execute("VADD", ["during", "XB8", vector, element, "CAS", "NOQUANT", "EF", "16", "M", "32"]); + }); + await ExceptionInjectionHelper.WaitOnClearAsync(MetadataPause).WaitAsync(TimeSpan.FromSeconds(30)); + + // Park a checkpoint at the end of its fuzzy region, before it captures the last address the + // snapshot covers. + ExceptionInjectionHelper.EnableException(CheckpointPause); + try + { +#pragma warning disable CS0618 // ForegroundSave is deprecated in the client, but is what this test needs + var saveTask = Task.Run(() => redis.GetServers()[0].Save(SaveType.ForegroundSave)); +#pragma warning restore CS0618 + await ExceptionInjectionHelper.WaitOnClearAsync(CheckpointPause).WaitAsync(TimeSpan.FromSeconds(30)); + + // Release the metadata update into the new version, forcing it to be copied to the tail + // while the old version stays in the snapshot's range. + ExceptionInjectionHelper.EnableException(MetadataPause); + _ = await vaddTask.WaitAsync(TimeSpan.FromSeconds(30)); + ExceptionInjectionHelper.DisableException(MetadataPause); + + ExceptionInjectionHelper.EnableException(CheckpointPause); + await saveTask.WaitAsync(TimeSpan.FromSeconds(60)); + } + finally + { + ExceptionInjectionHelper.DisableException(CheckpointPause); + } + } + finally + { + ExceptionInjectionHelper.DisableException(MetadataPause); + } + + _ = await server.Store.WaitForCommitAsync(); + } + + server.Dispose(deleteDir: false); + + // failOnRecoveryError surfaces a recovery failure that the server would otherwise log and continue + // past, leaving a partially recovered store behind. + server = TestUtils.CreateGarnetServer(TestUtils.MethodTestDir, enableAOF: true, tryRecover: true, aofMemorySize: DefaultAOFMemorySize, enableVectorSetPreview: true, failOnRecoveryError: true); + server.Start(); + + using (var redis = ConnectionMultiplexer.Connect(TestUtils.GetConfig())) + { + var db = redis.GetDatabase(0); + + foreach (var key in new[] { "before", "during" }) + { + var embedding = (string[])db.Execute("VEMB", [key, element]); + ClassicAssert.AreEqual(8, embedding?.Length ?? 0, $"VEMB {key} did not return the recovered embedding"); + } + } + } +#endif + + /// + /// Create a new GarnetServer instance with a small enough log that ordinary writes flush pages. + /// + private static GarnetServer CreateSmallPageGarnetServer(bool tryRecover) + => TestUtils.CreateGarnetServer(TestUtils.MethodTestDir, enableAOF: true, tryRecover: tryRecover, aofMemorySize: DefaultAOFMemorySize, enableVectorSetPreview: true, lowMemory: true, memorySize: "64m", pageSize: "16k", failOnRecoveryError: true); + /// /// Create a new GarnetServer instance with common parameters. /// From 718616160440911fed1a6dc74187f9025402f449 Mon Sep 17 00:00:00 2001 From: Badrish Chandramouli Date: Fri, 11 Sep 2026 00:55:20 -0700 Subject: [PATCH 17/27] Fix hash field expiration on copy update, and make expiry/blocking tests deterministic HashObject: initialize the expiration span lookup in the copy constructor HasExpirableItems tests expirationTimes, but IsExpired(ReadOnlySpan) reads expirationTimeSpanLookup. InitializeExpirationStructures and CleanupExpirationStructuresIfEmpty keep the two in sync; the copy constructor behind Clone() set only expirationTimes, leaving the alternate lookup as a default struct whose inner dictionary is null. Clone() is the copy-update path, so any RMW against a hash that carries field expirations and whose record is immutable - in the read-only region, or read back from disk - dereferenced a null dictionary and terminated the session. CanDoHashExpireLTM reproduces this: without the fix it fails with SocketClosed after the server throws NullReferenceException from HashObject.ContainsKey via PostCopyUpdater. The path exists only on .NET 9 and above. Tests: express member expiration as absolute deadlines Hash field and sorted set member expiry tests set short relative TTLs and then asserted that a member was still alive, which holds only if less wall-clock time elapses between two commands than the TTL. On a loaded two-core runner that bound is not guaranteed, and the member expires before the assertion runs. MemberExpiry supplies Pending() for members that must stay alive and Imminent() paired with WaitUntilPast() for members that must expire. Waiting for a deadline to pass is a lower bound and is unaffected by a slow runner. Assertions are unchanged or tightened: HEXPIRETIME and HPEXPIRETIME now compare exactly against the deadline that was set, where before they bracketed a range. The suites also no longer sleep for fixed multi-second intervals. CanDoHashCollect additionally sets its pending expirations before arming the imminent one. HashExpire purges already-expired fields as its first action, so a stall between the two commands would have dropped the fields that HCOLLECT is meant to reclaim. Tests: stop starving the thread pool in MultipleClientsUnblockAndAddTest The test dispatched 23 work items through Task.Run, each making a blocking StackExchange.Redis call. With ThreadPool.MinThreads equal to the two available cores, the pool injected replacement threads at roughly one per second, so the last task did not start for seven seconds and the commands reached the server after the ten second BLMPOP deadline. Issuing them through the async API lets them all reach the server promptly, which is also the interleaving the test exists to exercise. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 30f31d5b-9e53-4111-91a3-e16c94510a77 --- libs/server/Objects/Hash/HashObject.cs | 2 + .../Garnet.test.collections/MemberExpiry.cs | 66 +++++ .../Garnet.test.collections/RespHashTests.cs | 237 +++++++++++++----- .../RespSortedSetTests.cs | 143 +++++++---- test/standalone/Garnet.test/RespTests.cs | 17 +- 5 files changed, 341 insertions(+), 124 deletions(-) create mode 100644 test/standalone/Garnet.test.collections/MemberExpiry.cs diff --git a/libs/server/Objects/Hash/HashObject.cs b/libs/server/Objects/Hash/HashObject.cs index f88f365a48c..2357be3f6e8 100644 --- a/libs/server/Objects/Hash/HashObject.cs +++ b/libs/server/Objects/Hash/HashObject.cs @@ -156,6 +156,8 @@ public HashObject(Dictionary hash, Dictionary expi this.expirationQueue = expirationQueue; #if NET9_0_OR_GREATER hashSpanLookup = hash.GetAlternateLookup>(); + if (expirationTimes is not null) + expirationTimeSpanLookup = expirationTimes.GetAlternateLookup>(); #endif } diff --git a/test/standalone/Garnet.test.collections/MemberExpiry.cs b/test/standalone/Garnet.test.collections/MemberExpiry.cs new file mode 100644 index 00000000000..97a0f6cd038 --- /dev/null +++ b/test/standalone/Garnet.test.collections/MemberExpiry.cs @@ -0,0 +1,66 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT license. + +using System; +using System.Threading; +using System.Threading.Tasks; + +namespace Garnet.test +{ + /// + /// Absolute expiration deadlines for tests that exercise member-level expiration of hash fields + /// and sorted set members. Deadlines are absolute rather than relative so that no assertion + /// depends on an upper bound on the wall-clock time that elapses between commands. + /// + internal static class MemberExpiry + { + /// + /// Milliseconds by which leads the current time. The deadline has to be + /// in the future when the server processes the command: a deadline that has already passed + /// makes the server drop the member outright instead of letting it expire lazily. + /// + private const int ImminentLeadMs = 200; + + /// + /// Time to live, in milliseconds, of a deadline returned by . + /// + public const long PendingTtlMs = 60 * 60 * 1000; + + /// + /// An absolute deadline, in unix milliseconds, that is not reached while a test runs. A member + /// carrying it has a pending expiration but always reads back as live. + /// + public static long Pending() => DateTimeOffset.UtcNow.ToUnixTimeMilliseconds() + PendingTtlMs; + + /// + /// An absolute deadline, in unix milliseconds, shortly in the future. Pair it with + /// or to drive a member into the + /// expired state through the lazy expiration path. + /// + public static long Imminent() => DateTimeOffset.UtcNow.ToUnixTimeMilliseconds() + ImminentLeadMs; + + /// + /// Blocks until the wall clock is strictly past , which is the + /// condition under which the server reports a member as expired. + /// + /// Absolute deadline in unix milliseconds. + public static void WaitUntilPast(long deadlineMs) + { + for (var remaining = RemainingMs(deadlineMs); remaining >= 0; remaining = RemainingMs(deadlineMs)) + Thread.Sleep((int)remaining + 1); + } + + /// + /// Awaits until the wall clock is strictly past , which is the + /// condition under which the server reports a member as expired. + /// + /// Absolute deadline in unix milliseconds. + public static async Task WaitUntilPastAsync(long deadlineMs) + { + for (var remaining = RemainingMs(deadlineMs); remaining >= 0; remaining = RemainingMs(deadlineMs)) + await Task.Delay((int)remaining + 1).ConfigureAwait(false); + } + + private static long RemainingMs(long deadlineMs) => deadlineMs - DateTimeOffset.UtcNow.ToUnixTimeMilliseconds(); + } +} \ No newline at end of file diff --git a/test/standalone/Garnet.test.collections/RespHashTests.cs b/test/standalone/Garnet.test.collections/RespHashTests.cs index a565955c95e..e6ba50fcb8f 100644 --- a/test/standalone/Garnet.test.collections/RespHashTests.cs +++ b/test/standalone/Garnet.test.collections/RespHashTests.cs @@ -4,7 +4,6 @@ using System; using System.Linq; using System.Text; -using System.Threading; using System.Threading.Tasks; using Garnet.server; using NUnit.Framework; @@ -64,10 +63,17 @@ public async Task CanSetAndGetOnePairWithExpire() using var redis = ConnectionMultiplexer.Connect(TestUtils.GetConfig()); var db = redis.GetDatabase(0); db.HashSet("user:user1", [new HashEntry("Title", "Tsavorite")]); - db.HashFieldExpire("user:user1", ["Title"], TimeSpan.FromMilliseconds(100)); + + // Give the field a pending expiration that is not reached while the test runs + db.Execute("HPEXPIREAT", "user:user1", MemberExpiry.Pending(), "FIELDS", "1", "Title"); string r = db.HashGet("user:user1", "Title"); ClassicAssert.AreEqual("Tsavorite", r); - await Task.Delay(200).ConfigureAwait(false); + + // Expire the field + var deadline = MemberExpiry.Imminent(); + db.Execute("HPEXPIREAT", "user:user1", deadline, "FIELDS", "1", "Title"); + await MemberExpiry.WaitUntilPastAsync(deadline).ConfigureAwait(false); + r = db.HashGet("user:user1", "Title"); ClassicAssert.IsNull(r); } @@ -164,7 +170,7 @@ public void CanDelWithExpire() using var redis = ConnectionMultiplexer.Connect(TestUtils.GetConfig()); var db = redis.GetDatabase(0); db.HashSet("user:user1", [new HashEntry("Title", "Tsavorite"), new HashEntry("Year", "2021")]); - db.HashFieldExpire("user:user1", ["Title"], TimeSpan.FromMilliseconds(100)); + db.HashFieldExpire("user:user1", ["Title"], TimeSpan.FromMilliseconds(MemberExpiry.PendingTtlMs)); var result = db.HashDelete(new RedisKey("user:user1"), new RedisValue("Title")); ClassicAssert.AreEqual(true, result); string resultGet = db.HashGet("user:user1", "Year"); @@ -214,10 +220,17 @@ public async Task CanDoHLenWithExpire() using var redis = ConnectionMultiplexer.Connect(TestUtils.GetConfig()); var db = redis.GetDatabase(0); db.HashSet("user:user1", [new HashEntry("Title", "Tsavorite"), new HashEntry("Year", "2021"), new HashEntry("Company", "Acme")]); - db.HashFieldExpire("user:user1", ["Title"], TimeSpan.FromMilliseconds(100)); + + // Give the field a pending expiration that is not reached while the test runs + db.Execute("HPEXPIREAT", "user:user1", MemberExpiry.Pending(), "FIELDS", "1", "Title"); var result = db.HashLength("user:user1"); ClassicAssert.AreEqual(3, result); - await Task.Delay(150).ConfigureAwait(false); + + // Expire the field + var deadline = MemberExpiry.Imminent(); + db.Execute("HPEXPIREAT", "user:user1", deadline, "FIELDS", "1", "Title"); + await MemberExpiry.WaitUntilPastAsync(deadline).ConfigureAwait(false); + result = db.HashLength("user:user1"); ClassicAssert.AreEqual(2, result); db.HashSet("user:user1", [new HashEntry("Year", "new2021")]); // Trigger deletion of expired field @@ -247,14 +260,19 @@ public async Task CanDoGetAllWithExpire() HashEntry[] hashEntries = [new HashEntry("Title", "Tsavorite"), new HashEntry("Year", "2021"), new HashEntry("Company", "Acme")]; db.HashSet("user:user1", hashEntries); - db.HashFieldExpire("user:user1", ["Title"], TimeSpan.FromMilliseconds(100)); + + // Give the field a pending expiration that is not reached while the test runs + db.Execute("HPEXPIREAT", "user:user1", MemberExpiry.Pending(), "FIELDS", "1", "Title"); var result = db.HashGetAll("user:user1"); ClassicAssert.AreEqual(hashEntries.Length, result.Length); ClassicAssert.AreEqual(hashEntries.Length, result.Select(r => r.Name).Distinct().Count()); ClassicAssert.IsTrue(hashEntries.OrderBy(e => e.Name).SequenceEqual(result.OrderBy(r => r.Name))); - await Task.Delay(200).ConfigureAwait(false); + // Expire the field + var deadline = MemberExpiry.Imminent(); + db.Execute("HPEXPIREAT", "user:user1", deadline, "FIELDS", "1", "Title"); + await MemberExpiry.WaitUntilPastAsync(deadline).ConfigureAwait(false); result = db.HashGetAll("user:user1"); ClassicAssert.AreEqual(hashEntries.Length - 1, result.Length); @@ -285,12 +303,17 @@ public async Task CanDoHExistsWithExpire() using var redis = ConnectionMultiplexer.Connect(TestUtils.GetConfig()); var db = redis.GetDatabase(0); db.HashSet("user:user1", [new HashEntry("Title", "Tsavorite"), new HashEntry("Year", "2021"), new HashEntry("Company", "Acme")]); - db.HashFieldExpire("user:user1", ["Title"], TimeSpan.FromMilliseconds(100)); + + // Give the field a pending expiration that is not reached while the test runs + db.Execute("HPEXPIREAT", "user:user1", MemberExpiry.Pending(), "FIELDS", "1", "Title"); var result = db.HashExists(new RedisKey("user:user1"), "Title"); ClassicAssert.IsTrue(result); - await Task.Delay(200).ConfigureAwait(false); + // Expire the field + var deadline = MemberExpiry.Imminent(); + db.Execute("HPEXPIREAT", "user:user1", deadline, "FIELDS", "1", "Title"); + await MemberExpiry.WaitUntilPastAsync(deadline).ConfigureAwait(false); result = db.HashExists(new RedisKey("user:user1"), "Title"); ClassicAssert.IsFalse(result); @@ -320,12 +343,17 @@ public async Task CanDoHStrLenWithExire() using var redis = ConnectionMultiplexer.Connect(TestUtils.GetConfig()); var db = redis.GetDatabase(0); db.HashSet("user:user1", [new HashEntry("Title", "Tsavorite")]); - db.HashFieldExpire("user:user1", ["Title"], TimeSpan.FromMilliseconds(100)); + + // Give the field a pending expiration that is not reached while the test runs + db.Execute("HPEXPIREAT", "user:user1", MemberExpiry.Pending(), "FIELDS", "1", "Title"); long r = db.HashStringLength("user:user1", "Title"); ClassicAssert.AreEqual(9, r); - await Task.Delay(200).ConfigureAwait(false); + // Expire the field + var deadline = MemberExpiry.Imminent(); + db.Execute("HPEXPIREAT", "user:user1", deadline, "FIELDS", "1", "Title"); + await MemberExpiry.WaitUntilPastAsync(deadline).ConfigureAwait(false); r = db.HashStringLength("user:user1", "Title"); ClassicAssert.AreEqual(0, r); @@ -355,14 +383,19 @@ public async Task CanDoHKeysWithExpire() using var redis = ConnectionMultiplexer.Connect(TestUtils.GetConfig()); var db = redis.GetDatabase(0); db.HashSet("user:user1", [new HashEntry("Title", "Tsavorite"), new HashEntry("Year", "2021"), new HashEntry("Company", "Acme")]); - db.HashFieldExpire("user:user1", ["Title"], TimeSpan.FromMilliseconds(100)); + + // Give the field a pending expiration that is not reached while the test runs + db.Execute("HPEXPIREAT", "user:user1", MemberExpiry.Pending(), "FIELDS", "1", "Title"); var result = db.HashKeys("user:user1"); ClassicAssert.AreEqual(3, result.Length); ClassicAssert.IsTrue(Array.Exists(result, t => t.Equals("Title"))); ClassicAssert.IsTrue(Array.Exists(result, t => t.Equals("Year"))); ClassicAssert.IsTrue(Array.Exists(result, t => t.Equals("Company"))); - await Task.Delay(200).ConfigureAwait(false); + // Expire the field + var deadline = MemberExpiry.Imminent(); + db.Execute("HPEXPIREAT", "user:user1", deadline, "FIELDS", "1", "Title"); + await MemberExpiry.WaitUntilPastAsync(deadline).ConfigureAwait(false); result = db.HashKeys("user:user1"); ClassicAssert.AreEqual(2, result.Length); @@ -398,14 +431,19 @@ public async Task CanDoHValsWithExpire() using var redis = ConnectionMultiplexer.Connect(TestUtils.GetConfig()); var db = redis.GetDatabase(0); db.HashSet("user:user1", [new HashEntry("Title", "Tsavorite"), new HashEntry("Year", "2021"), new HashEntry("Company", "Acme")]); - db.HashFieldExpire("user:user1", ["Title"], TimeSpan.FromMilliseconds(100)); + + // Give the field a pending expiration that is not reached while the test runs + db.Execute("HPEXPIREAT", "user:user1", MemberExpiry.Pending(), "FIELDS", "1", "Title"); var result = db.HashValues("user:user1"); ClassicAssert.AreEqual(3, result.Length); ClassicAssert.IsTrue(Array.Exists(result, t => t.Equals("Tsavorite"))); ClassicAssert.IsTrue(Array.Exists(result, t => t.Equals("2021"))); ClassicAssert.IsTrue(Array.Exists(result, t => t.Equals("Acme"))); - await Task.Delay(200).ConfigureAwait(false); + // Expire the field + var deadline = MemberExpiry.Imminent(); + db.Execute("HPEXPIREAT", "user:user1", deadline, "FIELDS", "1", "Title"); + await MemberExpiry.WaitUntilPastAsync(deadline).ConfigureAwait(false); result = db.HashValues("user:user1"); ClassicAssert.AreEqual(2, result.Length); @@ -445,11 +483,16 @@ public async Task CanDoHIncrByWithExpire() using var redis = ConnectionMultiplexer.Connect(TestUtils.GetConfig()); var db = redis.GetDatabase(0); db.HashSet("user:user1", [new HashEntry("Field1", "StringValue"), new HashEntry("Field2", "1")]); - db.HashFieldExpire("user:user1", ["Field2"], TimeSpan.FromMilliseconds(100)); + + // Give the field a pending expiration that is not reached while the test runs + db.Execute("HPEXPIREAT", "user:user1", MemberExpiry.Pending(), "FIELDS", "1", "Field2"); var result = db.HashIncrement(new RedisKey("user:user1"), new RedisValue("Field2"), -4); ClassicAssert.AreEqual(-3, result); - await Task.Delay(200).ConfigureAwait(false); + // Expire the field + var deadline = MemberExpiry.Imminent(); + db.Execute("HPEXPIREAT", "user:user1", deadline, "FIELDS", "1", "Field2"); + await MemberExpiry.WaitUntilPastAsync(deadline).ConfigureAwait(false); result = db.HashIncrement(new RedisKey("user:user1"), new RedisValue("Field2"), -4); ClassicAssert.AreEqual(-4, result); @@ -503,11 +546,16 @@ public async Task CheckHashIncrementDoublePrecisionWithExpire() using var redis = ConnectionMultiplexer.Connect(TestUtils.GetConfig()); var db = redis.GetDatabase(0); db.HashSet("user:user1", [new HashEntry("Field1", "1.1111111111")]); - db.HashFieldExpire("user:user1", ["Field1"], TimeSpan.FromMilliseconds(100)); + + // Give the field a pending expiration that is not reached while the test runs + db.Execute("HPEXPIREAT", "user:user1", MemberExpiry.Pending(), "FIELDS", "1", "Field1"); var result = db.HashIncrement(new RedisKey("user:user1"), new RedisValue("Field1"), 2.2222222222); ClassicAssert.AreEqual(3.3333333333, result, 1e-15); - await Task.Delay(200).ConfigureAwait(false); + // Expire the field + var deadline = MemberExpiry.Imminent(); + db.Execute("HPEXPIREAT", "user:user1", deadline, "FIELDS", "1", "Field1"); + await MemberExpiry.WaitUntilPastAsync(deadline).ConfigureAwait(false); result = db.HashIncrement(new RedisKey("user:user1"), new RedisValue("Field1"), 2.2222222222); ClassicAssert.AreEqual(2.2222222222, result, 1e-15); @@ -532,10 +580,20 @@ public async Task CanDoHSETNXCommandWithExpire() using var redis = ConnectionMultiplexer.Connect(TestUtils.GetConfig()); var db = redis.GetDatabase(0); db.HashSet(new RedisKey("user:user1"), new RedisValue("Field"), new RedisValue("Hello")); - db.HashFieldExpire("user:user1", ["Field"], TimeSpan.FromMilliseconds(100)); + + // Give the field a pending expiration that is not reached while the test runs + var pending = MemberExpiry.Pending(); + db.Execute("HPEXPIREAT", "user:user1", pending, "FIELDS", "1", "Field"); db.HashSet(new RedisKey("user:user1"), new RedisValue("Field"), new RedisValue("Hello"), When.NotExists); - await Task.Delay(200).ConfigureAwait(false); + // SetNX on a field that is present is a no-op and leaves the expiration untouched + var expireTime = (RedisResult[])db.Execute("HPEXPIRETIME", "user:user1", "FIELDS", "1", "Field"); + ClassicAssert.AreEqual(pending, (long)expireTime[0]); + + // Expire the field + var deadline = MemberExpiry.Imminent(); + db.Execute("HPEXPIREAT", "user:user1", deadline, "FIELDS", "1", "Field"); + await MemberExpiry.WaitUntilPastAsync(deadline).ConfigureAwait(false); string result = db.HashGet("user:user1", "Field"); ClassicAssert.IsNull(result); // SetNX should not reset the expiration @@ -618,11 +676,16 @@ public async Task CanDoRandomFieldWithExpire() var hashKey = new RedisKey("user:user1"); HashEntry[] hashEntries = [new HashEntry("Title", "Tsavorite")]; db.HashSet(hashKey, hashEntries); - db.HashFieldExpire("user:user1", ["Title"], TimeSpan.FromMilliseconds(100)); + + // Give the field a pending expiration that is not reached while the test runs + db.Execute("HPEXPIREAT", "user:user1", MemberExpiry.Pending(), "FIELDS", "1", "Title"); string field = db.HashRandomField(hashKey); ClassicAssert.AreEqual(field, "Title"); - await Task.Delay(200).ConfigureAwait(false); + // Expire the field + var deadline = MemberExpiry.Imminent(); + db.Execute("HPEXPIREAT", "user:user1", deadline, "FIELDS", "1", "Title"); + await MemberExpiry.WaitUntilPastAsync(deadline).ConfigureAwait(false); field = db.HashRandomField(hashKey); ClassicAssert.IsNull(field); @@ -637,12 +700,17 @@ public async Task CanDoRandomFieldsWithExpire() var hashKey = new RedisKey("user:user1"); HashEntry[] hashEntries = [new HashEntry("Title", "Tsavorite")]; db.HashSet(hashKey, hashEntries); - db.HashFieldExpire("user:user1", ["Title"], TimeSpan.FromMilliseconds(100)); + + // Give the field a pending expiration that is not reached while the test runs + db.Execute("HPEXPIREAT", "user:user1", MemberExpiry.Pending(), "FIELDS", "1", "Title"); var field = db.HashRandomFields(hashKey, 10).Select(x => (string)x).ToArray(); ClassicAssert.AreEqual(field.Length, 1); ClassicAssert.AreEqual("Title", field[0]); - await Task.Delay(200).ConfigureAwait(false); + // Expire the field + var deadline = MemberExpiry.Imminent(); + db.Execute("HPEXPIREAT", "user:user1", deadline, "FIELDS", "1", "Title"); + await MemberExpiry.WaitUntilPastAsync(deadline).ConfigureAwait(false); field = db.HashRandomFields(hashKey, 10).Select(x => (string)x).ToArray(); ClassicAssert.AreEqual(field.Length, 0); @@ -721,13 +789,18 @@ public async Task CanDoHashScanWithExpire() var db = redis.GetDatabase(0); db.HashSet("user:user789", [new HashEntry("email", "email@example.com"), new HashEntry("email1", "email1@example.com"), new HashEntry("email2", "email2@example.com"), new HashEntry("email3", "email3@example.com"), new HashEntry("age", "25")]); - db.HashFieldExpire("user:user789", ["email"], TimeSpan.FromMilliseconds(100)); + + // Give the field a pending expiration that is not reached while the test runs + db.Execute("HPEXPIREAT", "user:user789", MemberExpiry.Pending(), "FIELDS", "1", "email"); var members = db.HashScan("user:user789", "email*"); ClassicAssert.IsTrue(((IScanningCursor)members).Cursor == 0); ClassicAssert.IsTrue(members.Count() == 4, "HSCAN with MATCH failed."); - await Task.Delay(200).ConfigureAwait(false); + // Expire the field + var deadline = MemberExpiry.Imminent(); + db.Execute("HPEXPIREAT", "user:user789", deadline, "FIELDS", "1", "email"); + await MemberExpiry.WaitUntilPastAsync(deadline).ConfigureAwait(false); // HSCAN with match members = db.HashScan("user:user789", "email*"); @@ -824,13 +897,18 @@ public async Task CanDoHMGETWithExpire() var db = redis.GetDatabase(0); db.HashSet("user:user789", [new HashEntry("email", "email@example.com"), new HashEntry("email1", "email1@example.com"), new HashEntry("email2", "email2@example.com"), new HashEntry("email3", "email3@example.com"), new HashEntry("age", "25")]); - db.HashFieldExpire("user:user789", ["email"], TimeSpan.FromMilliseconds(100)); + + // Give the field a pending expiration that is not reached while the test runs + db.Execute("HPEXPIREAT", "user:user789", MemberExpiry.Pending(), "FIELDS", "1", "email"); var members = (string[])db.Execute("HMGET", "user:user789", "email", "email1"); ClassicAssert.AreEqual("email@example.com", members[0]); ClassicAssert.AreEqual("email1@example.com", members[1]); - await Task.Delay(200).ConfigureAwait(false); + // Expire the field + var deadline = MemberExpiry.Imminent(); + db.Execute("HPEXPIREAT", "user:user789", deadline, "FIELDS", "1", "email"); + await MemberExpiry.WaitUntilPastAsync(deadline).ConfigureAwait(false); members = (string[])db.Execute("HMGET", "user:user789", "email", "email1"); ClassicAssert.IsNull(members[0]); @@ -1026,26 +1104,31 @@ public async Task CanDoHashExpire() var db = redis.GetDatabase(0); db.HashSet("myhash", [new HashEntry("field1", "hello"), new HashEntry("field2", "world"), new HashEntry("field3", "value3"), new HashEntry("field4", "value4"), new HashEntry("field5", "value5"), new HashEntry("field6", "value6")]); - var result = db.Execute("HEXPIRE", "myhash", "4", "FIELDS", "3", "field1", "field5", "nonexistfield"); + var pendingSeconds = MemberExpiry.PendingTtlMs / 1000; + var pendingDeadlineMs = MemberExpiry.Pending(); + var pendingDeadlineSeconds = pendingDeadlineMs / 1000; + + // Pending expirations that are not reached while the test runs + var result = db.Execute("HEXPIRE", "myhash", pendingSeconds, "FIELDS", "3", "field1", "field5", "nonexistfield"); var results = (RedisResult[])result; ClassicAssert.AreEqual(3, results.Length); ClassicAssert.AreEqual(1, (long)results[0]); ClassicAssert.AreEqual(1, (long)results[1]); ClassicAssert.AreEqual(-2, (long)results[2]); - result = db.Execute("HPEXPIRE", "myhash", "4000", "FIELDS", "2", "field2", "nonexistfield"); + result = db.Execute("HPEXPIRE", "myhash", MemberExpiry.PendingTtlMs, "FIELDS", "2", "field2", "nonexistfield"); results = (RedisResult[])result; ClassicAssert.AreEqual(2, results.Length); ClassicAssert.AreEqual(1, (long)results[0]); ClassicAssert.AreEqual(-2, (long)results[1]); - result = db.Execute("HEXPIREAT", "myhash", DateTimeOffset.UtcNow.AddSeconds(4).ToUnixTimeSeconds().ToString(), "FIELDS", "2", "field3", "nonexistfield"); + result = db.Execute("HEXPIREAT", "myhash", pendingDeadlineSeconds, "FIELDS", "2", "field3", "nonexistfield"); results = (RedisResult[])result; ClassicAssert.AreEqual(2, results.Length); ClassicAssert.AreEqual(1, (long)results[0]); ClassicAssert.AreEqual(-2, (long)results[1]); - result = db.Execute("HPEXPIREAT", "myhash", DateTimeOffset.UtcNow.AddSeconds(4).ToUnixTimeMilliseconds().ToString(), "FIELDS", "2", "field4", "nonexistfield"); + result = db.Execute("HPEXPIREAT", "myhash", pendingDeadlineMs, "FIELDS", "2", "field4", "nonexistfield"); results = (RedisResult[])result; ClassicAssert.AreEqual(2, results.Length); ClassicAssert.AreEqual(1, (long)results[0]); @@ -1053,26 +1136,25 @@ public async Task CanDoHashExpire() var ttl = (RedisResult[])db.Execute("HTTL", "myhash", "FIELDS", "2", "field1", "nonexistfield"); ClassicAssert.AreEqual(2, ttl.Length); - ClassicAssert.LessOrEqual((long)ttl[0], 4); - ClassicAssert.Greater((long)ttl[0], 1); + ClassicAssert.LessOrEqual((long)ttl[0], pendingSeconds); + ClassicAssert.Greater((long)ttl[0], 0); ClassicAssert.AreEqual(-2, (long)results[1]); ttl = (RedisResult[])db.Execute("HPTTL", "myhash", "FIELDS", "2", "field1", "nonexistfield"); ClassicAssert.AreEqual(2, ttl.Length); - ClassicAssert.LessOrEqual((long)ttl[0], 4000); - ClassicAssert.Greater((long)ttl[0], 1000); + ClassicAssert.LessOrEqual((long)ttl[0], MemberExpiry.PendingTtlMs); + ClassicAssert.Greater((long)ttl[0], 0); ClassicAssert.AreEqual(-2, (long)results[1]); - ttl = (RedisResult[])db.Execute("HEXPIRETIME", "myhash", "FIELDS", "2", "field1", "nonexistfield"); + // An absolute deadline is reported back exactly as it was set + ttl = (RedisResult[])db.Execute("HEXPIRETIME", "myhash", "FIELDS", "2", "field3", "nonexistfield"); ClassicAssert.AreEqual(2, ttl.Length); - ClassicAssert.LessOrEqual((long)ttl[0], DateTimeOffset.UtcNow.AddSeconds(4).ToUnixTimeSeconds()); - ClassicAssert.Greater((long)ttl[0], DateTimeOffset.UtcNow.AddSeconds(1).ToUnixTimeSeconds()); + ClassicAssert.AreEqual(pendingDeadlineSeconds, (long)ttl[0]); ClassicAssert.AreEqual(-2, (long)results[1]); - ttl = (RedisResult[])db.Execute("HPEXPIRETIME", "myhash", "FIELDS", "2", "field1", "nonexistfield"); + ttl = (RedisResult[])db.Execute("HPEXPIRETIME", "myhash", "FIELDS", "2", "field4", "nonexistfield"); ClassicAssert.AreEqual(2, ttl.Length); - ClassicAssert.LessOrEqual((long)ttl[0], DateTimeOffset.UtcNow.AddSeconds(4).ToUnixTimeMilliseconds()); - ClassicAssert.Greater((long)ttl[0], DateTimeOffset.UtcNow.AddSeconds(1).ToUnixTimeMilliseconds()); + ClassicAssert.AreEqual(pendingDeadlineMs, (long)ttl[0]); ClassicAssert.AreEqual(-2, (long)results[1]); results = (RedisResult[])db.Execute("HPERSIST", "myhash", "FIELDS", "3", "field5", "field6", "nonexistfield"); @@ -1081,7 +1163,10 @@ public async Task CanDoHashExpire() ClassicAssert.AreEqual(-1, (long)results[1]); // -1 if the field exists but has no associated expiration set. ClassicAssert.AreEqual(-2, (long)results[2]); - await Task.Delay(4500).ConfigureAwait(false); + // Expire the fields that still carry an expiration + var deadline = MemberExpiry.Imminent(); + db.Execute("HPEXPIREAT", "myhash", deadline, "FIELDS", "4", "field1", "field2", "field3", "field4"); + await MemberExpiry.WaitUntilPastAsync(deadline).ConfigureAwait(false); var items = db.HashGetAll("myhash"); ClassicAssert.AreEqual(2, items.Length); @@ -1114,16 +1199,20 @@ public async Task CanDoHashExpireLTM() string[] smallExpireKeys = ["user:user0", "user:user1"]; string[] largeExpireKeys = ["user:user2", "user:user3"]; + // Expirations are absolute deadlines, so each phase is reached by waiting for a deadline to + // pass rather than by assuming a bound on the time that elapses between commands. + var deadline = MemberExpiry.Imminent(); foreach (var key in smallExpireKeys) { db.HashSet(key, [new HashEntry("Field1", "StringValue"), new HashEntry("Field2", "1")]); - db.Execute("HEXPIRE", key, "2", "FIELDS", "1", "Field1"); + db.Execute("HPEXPIREAT", key, deadline, "FIELDS", "1", "Field1"); } + // Pending expirations that are not reached while the test runs foreach (var key in largeExpireKeys) { db.HashSet(key, [new HashEntry("Field1", "StringValue"), new HashEntry("Field2", "1")]); - db.Execute("HEXPIRE", key, "4", "FIELDS", "1", "Field1"); + db.Execute("HPEXPIREAT", key, MemberExpiry.Pending(), "FIELDS", "1", "Field1"); } // Create LTM (larger than memory) DB by inserting 1000 keys @@ -1137,7 +1226,7 @@ public async Task CanDoHashExpireLTM() // Ensure data has spilled to disk ClassicAssert.Greater(info.HeadAddress, info.BeginAddress); - await Task.Delay(2000).ConfigureAwait(false); + await MemberExpiry.WaitUntilPastAsync(deadline).ConfigureAwait(false); var result = db.HashExists(smallExpireKeys[0], "Field1"); ClassicAssert.IsFalse(result); @@ -1150,13 +1239,18 @@ public async Task CanDoHashExpireLTM() var ttl = db.HashFieldGetTimeToLive(largeExpireKeys[0], ["Field1"]); ClassicAssert.AreEqual(ttl.Length, 1); ClassicAssert.Greater(ttl[0], 0); - ClassicAssert.LessOrEqual(ttl[0], 2000); + ClassicAssert.LessOrEqual(ttl[0], MemberExpiry.PendingTtlMs); ttl = db.HashFieldGetTimeToLive(largeExpireKeys[1], ["Field1"]); ClassicAssert.AreEqual(ttl.Length, 1); ClassicAssert.Greater(ttl[0], 0); - ClassicAssert.LessOrEqual(ttl[0], 2000); + ClassicAssert.LessOrEqual(ttl[0], MemberExpiry.PendingTtlMs); - await Task.Delay(2000).ConfigureAwait(false); + // Expire the remaining fields. These hashes carry field expirations and have been evicted, so + // the expiration is applied by copy-updating the record read back from disk. + deadline = MemberExpiry.Imminent(); + foreach (var key in largeExpireKeys) + db.Execute("HPEXPIREAT", key, deadline, "FIELDS", "1", "Field1"); + await MemberExpiry.WaitUntilPastAsync(deadline).ConfigureAwait(false); result = db.HashExists(largeExpireKeys[0], "Field1"); ClassicAssert.IsFalse(result); @@ -1183,7 +1277,7 @@ public async Task CanDoHashExpireWithAofRecoveryAsync() HashEntry[] values2_1 = [new HashEntry("key2_1", "val2_1"), new HashEntry("key2_2", "val2_2")]; HashEntry[] values2_2 = [new HashEntry("key2_3", "val2_3"), new HashEntry("key2_4", "val2_4")]; - var expireTime = DateTimeOffset.UtcNow + TimeSpan.FromMinutes(1); + var expireTime = DateTimeOffset.FromUnixTimeMilliseconds(MemberExpiry.Pending()); using (var redis = ConnectionMultiplexer.Connect(TestUtils.GetConfig())) { @@ -1195,18 +1289,18 @@ public async Task CanDoHashExpireWithAofRecoveryAsync() // Set 1st hash set, add short expiry to 1 entry db.HashSet(key2, values2_1); - db.Execute("HEXPIRE", key2, 1, "FIELDS", 1, "key2_1"); + var deadline = MemberExpiry.Imminent(); + db.Execute("HPEXPIREAT", key2, deadline, "FIELDS", 1, "key2_1"); // Wait for short expiry to pass - Thread.Sleep(2000); + await MemberExpiry.WaitUntilPastAsync(deadline).ConfigureAwait(false); // Add more entries to 1st and 2nd hash sets db.HashSet(key1, values1_2); db.HashSet(key2, values2_2); // Add longer expiry to entry in 2nd hash set - db.Execute("HPEXPIRE", key2, 15000, "FIELDS", 1, "key2_2"); - Thread.Sleep(2000); + db.Execute("HPEXPIREAT", key2, MemberExpiry.Pending(), "FIELDS", 1, "key2_2"); // Verify 1st hash set contains all added entries var recoveredValues = db.HashGetAll(key1); @@ -1228,7 +1322,7 @@ public async Task CanDoHashExpireWithAofRecoveryAsync() ClassicAssert.IsNotNull(recoveredValuesTtl); ClassicAssert.AreEqual(4, recoveredValuesTtl!.Length); ClassicAssert.AreEqual(-2, (long)recoveredValuesTtl[0]); - ClassicAssert.LessOrEqual((long)recoveredValuesTtl[1], 13); + ClassicAssert.LessOrEqual((long)recoveredValuesTtl[1], MemberExpiry.PendingTtlMs / 1000); ClassicAssert.Greater((long)recoveredValuesTtl[1], 0); ClassicAssert.AreEqual(-1, (long)recoveredValuesTtl[2]); ClassicAssert.AreEqual(-1, (long)recoveredValuesTtl[3]); @@ -1264,7 +1358,7 @@ public async Task CanDoHashExpireWithAofRecoveryAsync() ClassicAssert.IsNotNull(recoveredValuesTtl); ClassicAssert.AreEqual(4, recoveredValuesTtl!.Length); ClassicAssert.AreEqual(-2, (long)recoveredValuesTtl[0]); - ClassicAssert.Less((long)recoveredValuesTtl[1], 13000); + ClassicAssert.LessOrEqual((long)recoveredValuesTtl[1], MemberExpiry.PendingTtlMs); ClassicAssert.Greater((long)recoveredValuesTtl[1], 0); ClassicAssert.AreEqual(-1, (long)recoveredValuesTtl[2]); ClassicAssert.AreEqual(-1, (long)recoveredValuesTtl[3]); @@ -1291,13 +1385,18 @@ public async Task CanDoHashCollect() var server = redis.GetServers().First(); db.HashSet("myhash", [new HashEntry("field1", "hello"), new HashEntry("field2", "world"), new HashEntry("field3", "value3"), new HashEntry("field4", "value4"), new HashEntry("field5", "value5"), new HashEntry("field6", "value6")]); - var result = db.Execute("HPEXPIRE", "myhash", "500", "FIELDS", "2", "field1", "field2"); + // Set the pending expirations first: HEXPIRE-family commands purge already-expired fields, so no + // mutating command may run between arming field1/field2 and the HCOLLECT that must reclaim them. + var result = db.Execute("HPEXPIREAT", "myhash", MemberExpiry.Pending(), "FIELDS", "2", "field3", "field4"); var results = (RedisResult[])result; ClassicAssert.AreEqual(2, results.Length); ClassicAssert.AreEqual(1, (long)results[0]); ClassicAssert.AreEqual(1, (long)results[1]); - result = db.Execute("HPEXPIRE", "myhash", "1500", "FIELDS", "2", "field3", "field4"); + // Expire field1 and field2, leaving field3 and field4 with a pending expiration that is + // not reached while the test runs + var deadline = MemberExpiry.Imminent(); + result = db.Execute("HPEXPIREAT", "myhash", deadline, "FIELDS", "2", "field1", "field2"); results = (RedisResult[])result; ClassicAssert.AreEqual(2, results.Length); ClassicAssert.AreEqual(1, (long)results[0]); @@ -1305,7 +1404,7 @@ public async Task CanDoHashCollect() var orginalMemory = (long)db.Execute("MEMORY", "USAGE", "myhash"); - await Task.Delay(600).ConfigureAwait(false); + await MemberExpiry.WaitUntilPastAsync(deadline).ConfigureAwait(false); var newMemory = (long)db.Execute("MEMORY", "USAGE", "myhash"); ClassicAssert.AreEqual(newMemory, orginalMemory); @@ -1315,9 +1414,13 @@ public async Task CanDoHashCollect() newMemory = (long)db.Execute("MEMORY", "USAGE", "myhash"); ClassicAssert.Less(newMemory, orginalMemory); - orginalMemory = newMemory; - await Task.Delay(1100).ConfigureAwait(false); + // Expire field3 and field4. Re-setting a deadline enqueues another expiration entry, so the + // baseline for the lazy-expiration comparison is taken after the command. + deadline = MemberExpiry.Imminent(); + db.Execute("HPEXPIREAT", "myhash", deadline, "FIELDS", "2", "field3", "field4"); + orginalMemory = (long)db.Execute("MEMORY", "USAGE", "myhash"); + await MemberExpiry.WaitUntilPastAsync(deadline).ConfigureAwait(false); newMemory = (long)db.Execute("MEMORY", "USAGE", "myhash"); ClassicAssert.AreEqual(newMemory, orginalMemory); @@ -1355,10 +1458,10 @@ public void CanDoHashExpireWithOptions(string command, string option) (var expireTimeField1, var expireTimeField3, var newExpireTimeField) = command switch { - "HEXPIRE" => ("2", "6", "4"), - "HPEXPIRE" => ("2000", "6000", "4000"), - "HEXPIREAT" => (DateTimeOffset.UtcNow.AddSeconds(2).ToUnixTimeSeconds().ToString(), DateTimeOffset.UtcNow.AddSeconds(6).ToUnixTimeSeconds().ToString(), DateTimeOffset.UtcNow.AddSeconds(4).ToUnixTimeSeconds().ToString()), - "HPEXPIREAT" => (DateTimeOffset.UtcNow.AddSeconds(2).ToUnixTimeMilliseconds().ToString(), DateTimeOffset.UtcNow.AddSeconds(6).ToUnixTimeMilliseconds().ToString(), DateTimeOffset.UtcNow.AddSeconds(4).ToUnixTimeMilliseconds().ToString()), + "HEXPIRE" => ("3600", "10800", "7200"), + "HPEXPIRE" => ("3600000", "10800000", "7200000"), + "HEXPIREAT" => (DateTimeOffset.UtcNow.AddHours(1).ToUnixTimeSeconds().ToString(), DateTimeOffset.UtcNow.AddHours(3).ToUnixTimeSeconds().ToString(), DateTimeOffset.UtcNow.AddHours(2).ToUnixTimeSeconds().ToString()), + "HPEXPIREAT" => (DateTimeOffset.UtcNow.AddHours(1).ToUnixTimeMilliseconds().ToString(), DateTimeOffset.UtcNow.AddHours(3).ToUnixTimeMilliseconds().ToString(), DateTimeOffset.UtcNow.AddHours(2).ToUnixTimeMilliseconds().ToString()), _ => throw new ArgumentException("Invalid command") }; diff --git a/test/standalone/Garnet.test.collections/RespSortedSetTests.cs b/test/standalone/Garnet.test.collections/RespSortedSetTests.cs index 44cc545e422..a836c072ec0 100644 --- a/test/standalone/Garnet.test.collections/RespSortedSetTests.cs +++ b/test/standalone/Garnet.test.collections/RespSortedSetTests.cs @@ -548,10 +548,10 @@ public void ZCountAndZCardWithExpiredAndExpiringItems() db.SortedSetAdd("key1", "d", 4); db.SortedSetAdd("key1", "e", 5); - // Set expiration for the minimum, maximum and middle items - db.Execute("ZPEXPIRE", "key1", "200", "MEMBERS", "3", "a", "e", "c"); - - Thread.Sleep(300); + // Expire the minimum, maximum and middle items + var deadline = MemberExpiry.Imminent(); + db.Execute("ZPEXPIREAT", "key1", deadline, "MEMBERS", "3", "a", "e", "c"); + MemberExpiry.WaitUntilPast(deadline); // ZCARD var count = db.SortedSetLength("key1"); @@ -2487,14 +2487,14 @@ public void ZDiffWithExpiredAndExpiringItems() db.SortedSetAdd("key2", "b", 2); db.SortedSetAdd("key2", "c", 3); - // Set expiration for some items in key1 - db.Execute("ZPEXPIRE", "key1", "200", "MEMBERS", "2", "a", "c"); - db.Execute("ZPEXPIRE", "key1", "500", "MEMBERS", "1", "d"); - - // Set expiration for matching items in key2 - db.Execute("ZPEXPIRE", "key2", "200", "MEMBERS", "1", "a"); + // Give "d" a pending expiration that is not reached while the test runs + db.Execute("ZPEXPIREAT", "key1", MemberExpiry.Pending(), "MEMBERS", "1", "d"); - Thread.Sleep(300); + // Expire some items in key1 and the matching item in key2 + var deadline = MemberExpiry.Imminent(); + db.Execute("ZPEXPIREAT", "key1", deadline, "MEMBERS", "2", "a", "c"); + db.Execute("ZPEXPIREAT", "key2", deadline, "MEMBERS", "1", "a"); + MemberExpiry.WaitUntilPast(deadline); // Perform ZDIFF var diff = db.SortedSetCombine(SetOperation.Difference, ["key1", "key2"]); @@ -2508,7 +2508,10 @@ public void ZDiffWithExpiredAndExpiringItems() ClassicAssert.AreEqual("e", diffWithScores[1].Element.ToString()); ClassicAssert.AreEqual(5, diffWithScores[1].Score); - Thread.Sleep(300); + // Expire "d" + deadline = MemberExpiry.Imminent(); + db.Execute("ZPEXPIREAT", "key1", deadline, "MEMBERS", "1", "d"); + MemberExpiry.WaitUntilPast(deadline); // Perform ZDIFF again after more items have expired diff = db.SortedSetCombine(SetOperation.Difference, ["key1", "key2"]); @@ -2538,14 +2541,14 @@ public void ZDiffStoreWithExpiredAndExpiringItems() db.SortedSetAdd("key2", "b", 2); db.SortedSetAdd("key2", "c", 3); - // Set expiration for some items in key1 - db.Execute("ZPEXPIRE", "key1", "200", "MEMBERS", "2", "a", "c"); - db.Execute("ZPEXPIRE", "key1", "500", "MEMBERS", "1", "d"); + // Give "d" a pending expiration that is not reached while the test runs + db.Execute("ZPEXPIREAT", "key1", MemberExpiry.Pending(), "MEMBERS", "1", "d"); - // Set expiration for matching items in key2 - db.Execute("ZPEXPIRE", "key2", "200", "MEMBERS", "1", "a"); - - Thread.Sleep(300); + // Expire some items in key1 and the matching item in key2 + var deadline = MemberExpiry.Imminent(); + db.Execute("ZPEXPIREAT", "key1", deadline, "MEMBERS", "2", "a", "c"); + db.Execute("ZPEXPIREAT", "key2", deadline, "MEMBERS", "1", "a"); + MemberExpiry.WaitUntilPast(deadline); // Perform ZDIFFSTORE var diffStoreCount = db.SortedSetCombineAndStore(SetOperation.Difference, "key3", ["key1", "key2"]); @@ -2559,7 +2562,10 @@ public void ZDiffStoreWithExpiredAndExpiringItems() ClassicAssert.AreEqual("e", diffStoreResult[1].Element.ToString()); ClassicAssert.AreEqual(5, diffStoreResult[1].Score); - Thread.Sleep(300); + // Expire "d" + deadline = MemberExpiry.Imminent(); + db.Execute("ZPEXPIREAT", "key1", deadline, "MEMBERS", "1", "d"); + MemberExpiry.WaitUntilPast(deadline); // Perform ZDIFFSTORE again after more items have expired diffStoreCount = db.SortedSetCombineAndStore(SetOperation.Difference, "key3", ["key1", "key2"]); @@ -2583,10 +2589,9 @@ public void ZIncrByWithExpiredAndExpiringItems() db.SortedSetAdd("key1", "b", 2); db.SortedSetAdd("key1", "c", 3); - // Set expiration for some items in key1 - db.Execute("ZPEXPIRE", "key1", "200", "MEMBERS", "1", "a"); - - Thread.Sleep(10); + // Give "a" a pending expiration that is not reached while the test runs + var pending = MemberExpiry.Pending(); + db.Execute("ZPEXPIREAT", "key1", pending, "MEMBERS", "1", "a"); // Try to increment the score of an expiring item var newScore = db.SortedSetIncrement("key1", "a", 5); @@ -2594,10 +2599,17 @@ public void ZIncrByWithExpiredAndExpiringItems() // Check the TTL of the expiring item var ttl = db.Execute("ZPTTL", "key1", "MEMBERS", "1", "a"); - ClassicAssert.LessOrEqual((long)ttl, 200); + ClassicAssert.LessOrEqual((long)ttl, MemberExpiry.PendingTtlMs); ClassicAssert.Greater((long)ttl, 0); - Thread.Sleep(200); + // The increment leaves the deadline of the item untouched + var expireTime = db.Execute("ZPEXPIRETIME", "key1", "MEMBERS", "1", "a"); + ClassicAssert.AreEqual(pending, (long)expireTime); + + // Expire "a" + var deadline = MemberExpiry.Imminent(); + db.Execute("ZPEXPIREAT", "key1", deadline, "MEMBERS", "1", "a"); + MemberExpiry.WaitUntilPast(deadline); // Check the item has expired ttl = db.Execute("ZPTTL", "key1", "MEMBERS", "1", "a"); @@ -2629,21 +2641,30 @@ public void ZInterWithExpiredAndExpiringItems() db.SortedSetAdd("key2", "b", 2); db.SortedSetAdd("key2", "c", 3); - db.Execute("ZPEXPIRE", "key1", "200", "MEMBERS", "1", "c"); - db.Execute("ZPEXPIRE", "key1", "500", "MEMBERS", "1", "b"); - db.Execute("ZPEXPIRE", "key2", "200", "MEMBERS", "1", "a"); + // Give the items that must survive the first two phases a pending expiration that is not + // reached while the test runs + var pending = MemberExpiry.Pending(); + db.Execute("ZPEXPIREAT", "key1", pending, "MEMBERS", "2", "c", "b"); + db.Execute("ZPEXPIREAT", "key2", pending, "MEMBERS", "1", "a"); var inter = db.SortedSetCombine(SetOperation.Intersect, ["key1", "key2"]); ClassicAssert.AreEqual(3, inter.Length); - Thread.Sleep(300); + // Expire "c" in key1 and "a" in key2 + var deadline = MemberExpiry.Imminent(); + db.Execute("ZPEXPIREAT", "key1", deadline, "MEMBERS", "1", "c"); + db.Execute("ZPEXPIREAT", "key2", deadline, "MEMBERS", "1", "a"); + MemberExpiry.WaitUntilPast(deadline); var interWithScores = db.SortedSetCombineWithScores(SetOperation.Intersect, ["key1", "key2"]); ClassicAssert.AreEqual(1, interWithScores.Length); // Only "b" should remain ClassicAssert.AreEqual("b", interWithScores[0].Element.ToString()); ClassicAssert.AreEqual(4, interWithScores[0].Score); // Sum of scores - Thread.Sleep(300); + // Expire "b" in key1 + deadline = MemberExpiry.Imminent(); + db.Execute("ZPEXPIREAT", "key1", deadline, "MEMBERS", "1", "b"); + MemberExpiry.WaitUntilPast(deadline); inter = db.SortedSetCombine(SetOperation.Intersect, ["key1", "key2"]); ClassicAssert.AreEqual(0, inter.Length); @@ -2665,16 +2686,22 @@ public void ZInterCardWithExpiredAndExpiringItems() db.SortedSetAdd("key2", "b", 2); db.SortedSetAdd("key2", "c", 3); - db.Execute("ZPEXPIRE", "key1", "200", "MEMBERS", "1", "c"); - db.Execute("ZPEXPIRE", "key1", "500", "MEMBERS", "1", "b"); - db.Execute("ZPEXPIRE", "key2", "200", "MEMBERS", "1", "a"); + // Give "b" a pending expiration that is not reached while the test runs + db.Execute("ZPEXPIREAT", "key1", MemberExpiry.Pending(), "MEMBERS", "1", "b"); - Thread.Sleep(300); + // Expire "c" in key1 and "a" in key2 + var deadline = MemberExpiry.Imminent(); + db.Execute("ZPEXPIREAT", "key1", deadline, "MEMBERS", "1", "c"); + db.Execute("ZPEXPIREAT", "key2", deadline, "MEMBERS", "1", "a"); + MemberExpiry.WaitUntilPast(deadline); var interCardCount = (long)db.Execute("ZINTERCARD", "2", "key1", "key2"); ClassicAssert.AreEqual(1, interCardCount); // Only "b" should remain - Thread.Sleep(300); + // Expire "b" in key1 + deadline = MemberExpiry.Imminent(); + db.Execute("ZPEXPIREAT", "key1", deadline, "MEMBERS", "1", "b"); + MemberExpiry.WaitUntilPast(deadline); interCardCount = (long)db.Execute("ZINTERCARD", "2", "key1", "key2"); ClassicAssert.AreEqual(0, interCardCount); // No items should remain @@ -2696,11 +2723,14 @@ public void ZInterStoreWithExpiredAndExpiringItems() db.SortedSetAdd("key2", "b", 2); db.SortedSetAdd("key2", "c", 3); - db.Execute("ZPEXPIRE", "key1", "200", "MEMBERS", "1", "c"); - db.Execute("ZPEXPIRE", "key1", "500", "MEMBERS", "1", "b"); - db.Execute("ZPEXPIRE", "key2", "200", "MEMBERS", "1", "a"); + // Give "b" a pending expiration that is not reached while the test runs + db.Execute("ZPEXPIREAT", "key1", MemberExpiry.Pending(), "MEMBERS", "1", "b"); - Thread.Sleep(300); + // Expire "c" in key1 and "a" in key2 + var deadline = MemberExpiry.Imminent(); + db.Execute("ZPEXPIREAT", "key1", deadline, "MEMBERS", "1", "c"); + db.Execute("ZPEXPIREAT", "key2", deadline, "MEMBERS", "1", "a"); + MemberExpiry.WaitUntilPast(deadline); var interStoreCount = db.SortedSetCombineAndStore(SetOperation.Intersect, "key3", ["key1", "key2"]); ClassicAssert.AreEqual(1, interStoreCount); // Only "b" should remain @@ -2710,7 +2740,10 @@ public void ZInterStoreWithExpiredAndExpiringItems() ClassicAssert.AreEqual("b", interStoreResult[0].Element.ToString()); ClassicAssert.AreEqual(4, interStoreResult[0].Score); // Sum of scores - Thread.Sleep(300); + // Expire "b" in key1 + deadline = MemberExpiry.Imminent(); + db.Execute("ZPEXPIREAT", "key1", deadline, "MEMBERS", "1", "b"); + MemberExpiry.WaitUntilPast(deadline); interStoreCount = db.SortedSetCombineAndStore(SetOperation.Intersect, "key3", ["key1", "key2"]); ClassicAssert.AreEqual(0, interStoreCount); // No items should remain @@ -2731,10 +2764,13 @@ public void ZLexCountWithExpiredAndExpiringItems() db.SortedSetAdd("key1", "d", 4); db.SortedSetAdd("key1", "e", 5); - db.Execute("ZPEXPIRE", "key1", "500", "MEMBERS", "3", "a", "e", "c"); - db.Execute("ZPEXPIRE", "key1", "2000", "MEMBERS", "1", "b"); + // Give "b" a pending expiration that is not reached while the test runs + db.Execute("ZPEXPIREAT", "key1", MemberExpiry.Pending(), "MEMBERS", "1", "b"); - Thread.Sleep(1000); + // Expire "a", "e" and "c" + var deadline = MemberExpiry.Imminent(); + db.Execute("ZPEXPIREAT", "key1", deadline, "MEMBERS", "3", "a", "e", "c"); + MemberExpiry.WaitUntilPast(deadline); var lexCount = (int)db.Execute("ZLEXCOUNT", "key1", "-", "+"); // SortedSetLengthByValue will check - and + to [- and [+ ClassicAssert.AreEqual(2, lexCount); // Only "b" and "d" should remain @@ -2742,7 +2778,10 @@ public void ZLexCountWithExpiredAndExpiringItems() var lexCountRange = db.SortedSetLengthByValue("key1", "b", "d", Exclude.Stop); ClassicAssert.AreEqual(1, lexCountRange); // Only "b" should remain within the range - Thread.Sleep(1500); + // Expire "b" + deadline = MemberExpiry.Imminent(); + db.Execute("ZPEXPIREAT", "key1", deadline, "MEMBERS", "1", "b"); + MemberExpiry.WaitUntilPast(deadline); lexCount = (int)db.Execute("ZLEXCOUNT", "key1", "-", "+"); ClassicAssert.AreEqual(1, lexCount); // Only "d" should remain @@ -3379,10 +3418,9 @@ public void ZScoreWithExpiredAndExpiringItems() db.SortedSetAdd("key1", "b", 2); db.SortedSetAdd("key1", "c", 3); - // Set expiration for some items in key1 - db.Execute("ZPEXPIRE", "key1", "200", "MEMBERS", "1", "a"); - - Thread.Sleep(10); + // Give "a" a pending expiration that is not reached while the test runs + var pending = MemberExpiry.Pending(); + db.Execute("ZPEXPIREAT", "key1", pending, "MEMBERS", "1", "a"); // Check the score of an expiring item var score = db.SortedSetScore("key1", "a"); @@ -3390,10 +3428,13 @@ public void ZScoreWithExpiredAndExpiringItems() // Check the TTL of the expiring item var ttl = db.Execute("ZPTTL", "key1", "MEMBERS", "1", "a"); - ClassicAssert.LessOrEqual((long)ttl, 200); + ClassicAssert.LessOrEqual((long)ttl, MemberExpiry.PendingTtlMs); ClassicAssert.Greater((long)ttl, 0); - Thread.Sleep(200); + // Expire "a" + var deadline = MemberExpiry.Imminent(); + db.Execute("ZPEXPIREAT", "key1", deadline, "MEMBERS", "1", "a"); + MemberExpiry.WaitUntilPast(deadline); // Check the item has expired ttl = db.Execute("ZPTTL", "key1", "MEMBERS", "1", "a"); diff --git a/test/standalone/Garnet.test/RespTests.cs b/test/standalone/Garnet.test/RespTests.cs index d7cac3cb456..8fd815aafab 100644 --- a/test/standalone/Garnet.test/RespTests.cs +++ b/test/standalone/Garnet.test/RespTests.cs @@ -5564,18 +5564,23 @@ public async Task MultipleClientsUnblockAndAddTest(int numberOfItems) // Wait for client to enter blocking state await Task.Delay(1000).ConfigureAwait(false); - // Start parallel unblock and add tasks - var unblockTasks = new List>(); + // Start parallel unblock and add tasks. + // These use the asynchronous StackExchange.Redis APIs deliberately: the synchronous overloads block a + // thread pool thread for the duration of the round trip, and issuing numberOfItems + 3 of them at once + // exhausts the pool on a 2-core machine. The pool then injects replacement threads at roughly one per + // second, so the commands reach the server several seconds late - after the blocking command has already + // timed out - and the concurrency this test exists to exercise never actually happens. + var unblockTasks = new List>(); var addTasks = new List(); for (int i = 0; i < numberOfItems; i++) { var _i = i; - addTasks.Add(Task.Run(() => redis.GetDatabase(0).ListLeftPush(key, $"{value}{_i}"))); + addTasks.Add(db.ListLeftPushAsync(key, $"{value}{_i}")); } for (int i = 0; i < 3; i++) { - unblockTasks.Add(Task.Run(() => (int)redis.GetDatabase(0).Execute("CLIENT", "UNBLOCK", clientId, "ERROR"))); + unblockTasks.Add(db.ExecuteAsync("CLIENT", "UNBLOCK", clientId, "ERROR")); } await Task.WhenAll(unblockTasks).ConfigureAwait(false); @@ -5590,11 +5595,11 @@ public async Task MultipleClientsUnblockAndAddTest(int numberOfItems) if (numberOfItemsReturned == 0) { ClassicAssert.IsTrue(blockingResult.StartsWith("-UNBLOCKED")); - ClassicAssert.IsTrue(unblockTasks.Any(x => x.Result == 1)); + ClassicAssert.IsTrue(unblockTasks.Any(x => (int)x.Result == 1)); } else { - ClassicAssert.IsTrue(unblockTasks.All(x => x.Result == 0)); + ClassicAssert.IsTrue(unblockTasks.All(x => (int)x.Result == 0)); } } From 58e3764dc2980551500f989f2eb459fa78710f25 Mon Sep 17 00:00:00 2001 From: Badrish Chandramouli Date: Fri, 11 Sep 2026 18:17:47 -0700 Subject: [PATCH 18/27] Load the test client certificate once instead of per connection TLS-enabled tests build their SslClientAuthenticationOptions from a factory delegate that the client invokes once per physical connection, and that factory imported testcert.pfx from disk every time. Importing a PKCS#12 file is expensive - on Windows it materializes an on-disk key container - so the import landed inside the client's 2s connect timeout, repeated on every connection and every retry. The imported X509Certificate2 was also never disposed, so each import leaked a key handle for the lifetime of the run. Import the certificate once and share it. Sharing a single X509Certificate2 across SslStream connections is the normal pattern. Measured on a TLS cluster test pinned to 2 cores, PKCS#12 imports per test drop from 7 to 3, with the 4 removed imports all on the client connect path. This targets ClusterManagementTests.ReplicasRestartAsReplicasAsync and PrimaryUnavailableRecoveryAsync, the only two TLS tests in that fixture and the only two that flake there. Those failures reproduce only on the Windows runner, so this removes measured redundant work from the path that times out rather than being verified against the failure itself. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 30f31d5b-9e53-4111-91a3-e16c94510a77 --- .../Garnet.test.cluster/ClusterTestUtils.cs | 5 ++-- test/standalone/Garnet.test/TestUtils.cs | 26 +++++++++++++++---- 2 files changed, 23 insertions(+), 8 deletions(-) diff --git a/test/cluster/Garnet.test.cluster/ClusterTestUtils.cs b/test/cluster/Garnet.test.cluster/ClusterTestUtils.cs index 3be0afc4fc5..e66ae8785cd 100644 --- a/test/cluster/Garnet.test.cluster/ClusterTestUtils.cs +++ b/test/cluster/Garnet.test.cluster/ClusterTestUtils.cs @@ -16,7 +16,6 @@ using Garnet.client; using Garnet.common; using Garnet.server; -using Garnet.server.TLS; using GarnetClusterManagement; using Microsoft.Extensions.Logging; using NUnit.Framework; @@ -930,7 +929,7 @@ public GarnetClientSession GetGarnetClientSession(int nodeIndex, bool useTLS = f { sslOptions = new SslClientAuthenticationOptions { - ClientCertificates = [CertificateUtils.GetMachineCertificateByFile(certFile, certPassword)], + ClientCertificates = [TestUtils.GetClientCertificate()], TargetHost = "GarnetTest", AllowRenegotiation = false, RemoteCertificateValidationCallback = TestUtils.ValidateServerCertificate, @@ -952,7 +951,7 @@ public GarnetClientSession CreateGarnetClientSession(int nodeIndex, bool useTLS { sslOptions = new SslClientAuthenticationOptions { - ClientCertificates = [CertificateUtils.GetMachineCertificateByFile(certFile, certPassword)], + ClientCertificates = [TestUtils.GetClientCertificate()], TargetHost = "GarnetTest", AllowRenegotiation = false, RemoteCertificateValidationCallback = TestUtils.ValidateServerCertificate, diff --git a/test/standalone/Garnet.test/TestUtils.cs b/test/standalone/Garnet.test/TestUtils.cs index 886e732a36f..d1c2f649aba 100644 --- a/test/standalone/Garnet.test/TestUtils.cs +++ b/test/standalone/Garnet.test/TestUtils.cs @@ -146,6 +146,22 @@ internal static string AzureTestContainer public const string certFile = "testcert.pfx"; public const string certPassword = "placeholder"; + static X509Certificate2 clientCertificate; + static readonly object clientCertificateLock = new(); + + /// + /// Returns the client certificate used by TLS-enabled tests, importing it from disk on first use. + /// Importing a PKCS#12 file is expensive - on Windows it materializes a key container - and the + /// options factory a client calls runs once per connection attempt, so importing per connection + /// spends that cost inside the client's connect timeout on every attempt and retry. + /// + public static X509Certificate2 GetClientCertificate() + { + if (clientCertificate is not null) return clientCertificate; + lock (clientCertificateLock) + return clientCertificate ??= CertificateUtils.GetMachineCertificateByFile(certFile, certPassword); + } + public const string pemCertFile = "testcert.pem"; public const string pemCertKeyFile = "testcert.key.pem"; @@ -858,7 +874,7 @@ public static GarnetServerOptions GetGarnetServerOptions( tlsServerOptionsOverride: null, clusterTlsClientOptionsOverride: new SslClientAuthenticationOptions { - ClientCertificates = certificates ?? [CertificateUtils.GetMachineCertificateByFile(certFile, certPassword)], + ClientCertificates = certificates ?? [GetClientCertificate()], TargetHost = "GarnetTest", AllowRenegotiation = false, RemoteCertificateValidationCallback = ValidateServerCertificate, @@ -995,7 +1011,7 @@ public static ConfigurationOptions GetConfig( ( new SslClientAuthenticationOptions { - ClientCertificates = certificates ?? [CertificateUtils.GetMachineCertificateByFile(certFile, certPassword)], + ClientCertificates = certificates ?? [GetClientCertificate()], TargetHost = "GarnetTest", AllowRenegotiation = false, RemoteCertificateValidationCallback = ValidateServerCertificate, @@ -1012,7 +1028,7 @@ public static GarnetClient GetGarnetClient(EndPoint endpoint = null, bool useTLS { sslOptions = new SslClientAuthenticationOptions { - ClientCertificates = [CertificateUtils.GetMachineCertificateByFile(certFile, certPassword)], + ClientCertificates = [GetClientCertificate()], TargetHost = "GarnetTest", AllowRenegotiation = false, RemoteCertificateValidationCallback = ValidateServerCertificate, @@ -1028,7 +1044,7 @@ public static GarnetClientSession GetGarnetClientSession(bool useTLS = false, bo { sslOptions = new SslClientAuthenticationOptions { - ClientCertificates = [CertificateUtils.GetMachineCertificateByFile(certFile, certPassword)], + ClientCertificates = [GetClientCertificate()], TargetHost = "GarnetTest", AllowRenegotiation = false, RemoteCertificateValidationCallback = ValidateServerCertificate, @@ -1044,7 +1060,7 @@ public static LightClientRequest CreateRequest(LightClient.OnResponseDelegateUns { sslOptions = new SslClientAuthenticationOptions { - ClientCertificates = [CertificateUtils.GetMachineCertificateByFile(certFile, certPassword)], + ClientCertificates = [GetClientCertificate()], TargetHost = "GarnetTest", AllowRenegotiation = false, RemoteCertificateValidationCallback = ValidateServerCertificate, From 2e252038348c4a1006954cadc2d6283faccb40d8 Mon Sep 17 00:00:00 2001 From: Badrish Chandramouli Date: Fri, 11 Sep 2026 19:16:11 -0700 Subject: [PATCH 19/27] Settle background context cleanup before sampling Vector Set slot state RespVectorSetTests.RenamesAsync reads VectorManager context state straight after a rename. A context released by the preceding delete or overwrite is only marked for cleanup by the background cleanup task, so until that task runs the released context still reads as in-use and the slot reports two live contexts rather than one. Wait for that work to quiesce before sampling, using the primitive VectorSetOverwriteTests already uses for the same reason. The assertion is unchanged and no less strict. Captured at the moment of failure, the slot reported 16 namespaces; calling WaitForQuiescence at that same instant reported 8, and it stayed 8 three seconds later. The context is released correctly and only the sampling was early, so there is nothing to fix in VectorManager itself. Pinned to two cores under CPU load the test failed on iterations 11 and 14 of two separate 60-run loops. With this change 60 runs are clean. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 30f31d5b-9e53-4111-91a3-e16c94510a77 --- test/standalone/Garnet.test.vectorset/RespVectorSetTests.cs | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/test/standalone/Garnet.test.vectorset/RespVectorSetTests.cs b/test/standalone/Garnet.test.vectorset/RespVectorSetTests.cs index 57b2e690b09..8529e777613 100644 --- a/test/standalone/Garnet.test.vectorset/RespVectorSetTests.cs +++ b/test/standalone/Garnet.test.vectorset/RespVectorSetTests.cs @@ -3402,6 +3402,11 @@ static void AssertExpectedHashSlot(GarnetServer server, string indexKey, ushort var store = server.Provider.StoreWrapper; var vectorManager = store.DefaultDatabase.VectorManager; + // A context released by a delete or an overwrite is only marked for cleanup on a background + // task, so it still reads as live until that task runs. Settle that work before sampling the + // slot, otherwise the released context is counted alongside the live one. + vectorManager.WaitForQuiescence(); + unsafe { fixed (byte* indexKeyPtr = Encoding.ASCII.GetBytes(indexKey)) From 44da67ee7328b5aaaaed8f38a7903ba5d92c3387 Mon Sep 17 00:00:00 2001 From: Badrish Chandramouli Date: Sat, 12 Sep 2026 09:38:05 -0700 Subject: [PATCH 20/27] Address review feedback on comments Correct the description of the concurrent flush in ObjectLogCheckpointTransferTests: the test shifts ReadOnlyAddress, so what it models is the ReadOnly flush rather than eviction driven by LogSizeTracker. Document recoveredIndexes and recoveredMetadata, and the closeRequested flag. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 30f31d5b-9e53-4111-91a3-e16c94510a77 --- libs/client/ClientSession/GarnetClientSession.cs | 5 +++++ libs/server/Resp/Vector/VectorManager.cs | 12 ++++++++++++ .../ObjectLogCheckpointTransferTests.cs | 7 ++++--- 3 files changed, 21 insertions(+), 3 deletions(-) diff --git a/libs/client/ClientSession/GarnetClientSession.cs b/libs/client/ClientSession/GarnetClientSession.cs index 17e70b709e9..b7e1d96b8e6 100644 --- a/libs/client/ClientSession/GarnetClientSession.cs +++ b/libs/client/ClientSession/GarnetClientSession.cs @@ -43,6 +43,11 @@ public sealed partial class GarnetClientSession : IServerHook, IMessageConsumer Socket socket; + /// + /// Set by and never cleared, so a close is sticky: it also lets + /// observe a close that lands while the connection is still + /// being established, which would otherwise leave that thread awaiting a reply that never comes. + /// int closeRequested; int disposed; diff --git a/libs/server/Resp/Vector/VectorManager.cs b/libs/server/Resp/Vector/VectorManager.cs index 043f07bdd8b..5a948c2965a 100644 --- a/libs/server/Resp/Vector/VectorManager.cs +++ b/libs/server/Resp/Vector/VectorManager.cs @@ -171,7 +171,19 @@ private static void EnsureFilterBitmapSize(ref SpanByteAndMemory buffer, int res private readonly int dbId; + /// + /// Context of each Vector Set index record found during recovery, mapped to the hash slot of the key + /// that owns it. restores the reservation for each of these + /// contexts, and needs the hash slot recorded here because the record itself is not retained. + /// private ConcurrentDictionary recoveredIndexes; + + /// + /// Newest context metadata recovered for each index into contextMetadatas. A snapshot holds + /// every version of a record written in the range it covers, so the same index is presented more than + /// once; entries are kept by ContextMetadata.Version and trimmed by + /// . + /// private ConcurrentDictionary recoveredMetadata; public VectorManager(int dbId, GarnetServerOptions serverOptions, Func getTempSession, ILoggerFactory loggerFactory) diff --git a/libs/storage/Tsavorite/cs/test/test.recovery/ObjectLogCheckpointTransferTests.cs b/libs/storage/Tsavorite/cs/test/test.recovery/ObjectLogCheckpointTransferTests.cs index 6841f9699aa..26e3c302f2c 100644 --- a/libs/storage/Tsavorite/cs/test/test.recovery/ObjectLogCheckpointTransferTests.cs +++ b/libs/storage/Tsavorite/cs/test/test.recovery/ObjectLogCheckpointTransferTests.cs @@ -48,9 +48,10 @@ public void Setup() [TearDown] public void TearDown() => TestUtils.OnTearDown(); - // flushMainLogDuringCheckpoint models a concurrent main-log flush (as issued by LogSizeTracker eviction) that - // starts after the checkpoint has captured its snapshot-start object-log position: it advances both - // FlushedUntilAddress and the main object-log tail before the checkpoint records them at PERSISTENCE_CALLBACK. + // flushMainLogDuringCheckpoint models a concurrent main-log ReadOnly flush - the shift of + // ReadOnlyAddress that flushes the pages it makes immutable - starting after the checkpoint has + // captured its snapshot-start object-log position: it advances both FlushedUntilAddress and the + // main object-log tail before the checkpoint records them at PERSISTENCE_CALLBACK. [Test] [Category("TsavoriteKV"), Category("CheckpointRestore")] public async Task TransferredSnapshotCheckpointRecoversAllObjects([Values] bool flushMainLogDuringCheckpoint) From 9d80ee9e24ba8276d9144cee6065776acf746077 Mon Sep 17 00:00:00 2001 From: Badrish Chandramouli Date: Sat, 12 Sep 2026 15:26:24 -0700 Subject: [PATCH 21/27] Fix two migration and recovery defects found in review, and drop two timeout increases Two contexts could end up sharing one namespace block. A context chosen by the PRIMARY can be occupied on the node receiving the migration, so that migration is steered onto a context that is free locally, and the context it lands on is marked migrating. That mark is also how an interrupted and resumed migration looks, so a later incoming context whose own value happened to equal that one read it as its own reservation and adopted it, leaving both migrations writing to one context. Remembering which local contexts have already been claimed keeps the mapping injective. Reconciling recovered state rebuilt the context metadata array from the recovered metadata alone and defaulted every index that was not represented. A cluster node with AOF enabled reconciles twice while starting up, once from RecoverCheckpointAndAOFAsync and again from StoreWrapper, and the first pass consumes the recovered metadata, so the second had only the index records to go on. A context that is in use without an index record - a Vector Set that was deleted but whose data has not finished being cleaned up - was dropped instead of being marked for cleanup, freeing it for the next Vector Set while its old data was still present, and the persisted version was reset so later metadata writes were cancelled as stale. Entries with nothing recovered now keep what is already in memory. The multi-slot Vector Set migration test polled without sleeping on the path where the migration had not finished, so it spun a core flat out for the whole window and starved the cluster nodes it was waiting on. Sleeping on every path is the fix, and the window it was given goes back to the five seconds the single-slot variant uses: on two cores, main fails five runs in ten at five seconds, and sleeping on every path fails none. The testhost connection timeout increase goes with it. Ten failed runs, thirty failed jobs, contain no instance of that timeout elapsing, so there was nothing to justify raising it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 30f31d5b-9e53-4111-91a3-e16c94510a77 --- .github/workflows/ci.yml | 5 -- .../Resp/Vector/VectorManager.Replication.cs | 28 +++++++- libs/server/Resp/Vector/VectorManager.cs | 12 +++- .../VectorSets/ClusterVectorSetTests.cs | 12 ++-- .../VectorSetMigratedContextRemapTests.cs | 65 +++++++++++++++++ ...ctorSetRecoveredContextReservationTests.cs | 69 +++++++++++++++++++ 6 files changed, 178 insertions(+), 13 deletions(-) create mode 100644 test/standalone/Garnet.test.vectorset/VectorSetMigratedContextRemapTests.cs diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index cb90be7f159..aaeb906f4ae 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -13,11 +13,6 @@ on: env: DOTNET_SKIP_FIRST_TIME_EXPERIENCE: 1 DOTNET_NOLOGO: true - # vstest.console waits this many seconds for the testhost process it just launched to connect back - # to it. The default of 90 is occasionally not enough on a loaded two-core runner, and when it - # elapses the job fails without having run a single test. This governs process startup only; a test - # that hangs is still caught by --blame-hang-timeout. - VSTEST_CONNECTION_TIMEOUT: 180 permissions: contents: read diff --git a/libs/server/Resp/Vector/VectorManager.Replication.cs b/libs/server/Resp/Vector/VectorManager.Replication.cs index 7ceb513e83e..ff2d5cef843 100644 --- a/libs/server/Resp/Vector/VectorManager.Replication.cs +++ b/libs/server/Resp/Vector/VectorManager.Replication.cs @@ -174,6 +174,17 @@ internal void ReplicateVectorSetSetAttribute(ReadOnlySpan key, ReadOnlySpa /// private readonly Dictionary migratedContextRemap = new(); + /// + /// The local contexts already claimed as the target of a remapping in . + /// + /// A claimed context is marked migrating, and a later incoming context whose own value happens to equal + /// that target would otherwise read that mark as its own interrupted migration and adopt the context + /// directly, leaving two migrated Vector Sets sharing one context. + /// + /// Guarded by lock (this), like the rest of the context metadata. + /// + private readonly HashSet migratedContextTargets = []; + /// /// Discard every in-flight migration remapping. /// @@ -184,6 +195,7 @@ private void ClearMigratedContextRemap() Debug.Assert(Monitor.IsEntered(this), "Migration remap is guarded by lock (this)"); migratedContextRemap.Clear(); + migratedContextTargets.Clear(); } /// @@ -214,10 +226,13 @@ private ulong ResolveMigratedContext(StorageSession currentSession, ulong migrat // so it cannot be adopted as-is var contextKnownLocally = contextIndex < contextMetadatas.Length; - if (contextKnownLocally && contextMetadatas[contextIndex].IsMigrating(contextIndex != 0, contextValue)) + if (contextKnownLocally + && !migratedContextTargets.Contains(migratedContext) + && contextMetadatas[contextIndex].IsMigrating(contextIndex != 0, contextValue)) { // Already reserved for a migration, which is how a migration that was interrupted and resumed looks migratedContextRemap[migratedContext] = migratedContext; + _ = migratedContextTargets.Add(migratedContext); return migratedContext; } @@ -238,6 +253,7 @@ private ulong ResolveMigratedContext(StorageSession currentSession, ulong migrat contextMetadatas[contextIndex].MarkMigrating(contextIndex != 0, contextValue); migratedContextRemap[migratedContext] = localContext; + _ = migratedContextTargets.Add(localContext); _ = dirtyContextMetadatas.Add(contextIndex); } @@ -247,6 +263,16 @@ private ulong ResolveMigratedContext(StorageSession currentSession, ulong migrat return localContext; } + /// + /// For testing purposes, resolve a context the way an incoming migrated record would. + /// + public ulong ResolveMigratedContextForTest(ulong migratedContext) + { + using var session = (RespServerSession)getTempSession(); + + return ResolveMigratedContext(session.storageSession, migratedContext); + } + /// /// Vector Set adds are phrased as reads (once the index is created), so they require special handling. /// diff --git a/libs/server/Resp/Vector/VectorManager.cs b/libs/server/Resp/Vector/VectorManager.cs index 5a948c2965a..d3139805952 100644 --- a/libs/server/Resp/Vector/VectorManager.cs +++ b/libs/server/Resp/Vector/VectorManager.cs @@ -335,13 +335,23 @@ public void ReconcileRecoveredState(bool requireNoReservedContexts = false) if (maxIndex >= 0) { + var priorMetadatas = contextMetadatas; + contextMetadatas = new ContextMetadata[maxIndex + 1]; for (var i = 0; i < contextMetadatas.Length; i++) { if (!recoveredMetadata.TryGetValue(i, out contextMetadatas[i]) || contextMetadatas[i].IsEmpty) { - contextMetadatas[i] = new(); + // Nothing was recovered for this index, so keep whatever is already in memory rather + // than defaulting it. A cluster node with AOF enabled reconciles twice while starting + // up - once from RecoverCheckpointAndAOFAsync and again from StoreWrapper - and the + // first pass consumes recoveredMetadata, so the second has only the index records to + // go on. Defaulting here would drop a reservation that has no index record instead of + // marking it for cleanup below, leaving that context free to hand to the next Vector + // Set while the data behind it is still present, and would reset the persisted + // version so that later metadata writes are cancelled as stale. + contextMetadatas[i] = i < priorMetadatas.Length ? priorMetadatas[i] : new(); } } } diff --git a/test/cluster/Garnet.test.cluster.vectorsets/VectorSets/ClusterVectorSetTests.cs b/test/cluster/Garnet.test.cluster.vectorsets/VectorSets/ClusterVectorSetTests.cs index 624933e58cc..5634d080390 100644 --- a/test/cluster/Garnet.test.cluster.vectorsets/VectorSets/ClusterVectorSetTests.cs +++ b/test/cluster/Garnet.test.cluster.vectorsets/VectorSets/ClusterVectorSetTests.cs @@ -1031,7 +1031,7 @@ public async Task VectorSetMigrateSingleBySlotAsync() var success = false; string lastResponse = null; - while (Stopwatch.GetElapsedTime(start) < TimeSpan.FromSeconds(15)) + while (Stopwatch.GetElapsedTime(start) < TimeSpan.FromSeconds(5)) { try { @@ -1337,14 +1337,14 @@ public void VectorSetMigrateManyBySlot() ClassicAssert.IsTrue(exc0.StartsWith("Key has MOVED to ")); } - // Same convergence window the single-slot variant of this test uses: the source replica keeps - // serving the migrated keys for a while, then passes through a window where the slot is not yet - // known-stable from its point of view, before it settles on redirecting to the new primary. + // Every path through this loop has to sleep. Polling without one spins a core flat out for the + // whole window, and on a two-core runner that starves the cluster nodes this is waiting on, so + // the convergence it is polling for cannot happen until the window has already elapsed. var start = Stopwatch.GetTimestamp(); var success = false; string lastResponse = null; - while (Stopwatch.GetElapsedTime(start) < TimeSpan.FromSeconds(15)) + while (Stopwatch.GetElapsedTime(start) < TimeSpan.FromSeconds(5)) { try { @@ -1394,7 +1394,7 @@ public void VectorSetMigrateManyBySlot() // result even for a poll that the very next one resolves. List<(string Key, byte[] Data, byte[][] Reply)> replicated = []; - while (Stopwatch.GetElapsedTime(start) < TimeSpan.FromSeconds(15)) + while (Stopwatch.GetElapsedTime(start) < TimeSpan.FromSeconds(5)) { success = true; replicated.Clear(); diff --git a/test/standalone/Garnet.test.vectorset/VectorSetMigratedContextRemapTests.cs b/test/standalone/Garnet.test.vectorset/VectorSetMigratedContextRemapTests.cs new file mode 100644 index 00000000000..6b146dc2064 --- /dev/null +++ b/test/standalone/Garnet.test.vectorset/VectorSetMigratedContextRemapTests.cs @@ -0,0 +1,65 @@ +// Copyright (c) Microsoft Corporation. +// Licensed under the MIT license. + +using Garnet.server; +using NUnit.Framework; +using NUnit.Framework.Legacy; + +namespace Garnet.test +{ + /// + /// Contexts are chosen independently by each node, so a context that was free on the PRIMARY which + /// picked it can be occupied on the node receiving the migration, and that migration is steered onto a + /// context that is free locally. The context it is steered onto gets marked migrating, which is also how + /// a migration that was interrupted and resumed looks, so a later incoming context whose own value + /// happens to equal that one must not read that mark as its own reservation and adopt it. + /// + [TestFixture] + public class VectorSetMigratedContextRemapTests : TestBase + { + private global::Garnet.GarnetServer server; + + [SetUp] + public void Setup() + { + TestUtils.DeleteDirectory(TestUtils.MethodTestDir, wait: true); + + server = TestUtils.CreateGarnetServer(TestUtils.MethodTestDir, enableVectorSetPreview: true); + server.Start(); + } + + [TearDown] + public void TearDown() + { + server?.Dispose(); + server = null; + + TestUtils.DeleteDirectory(TestUtils.MethodTestDir); + TestUtils.OnTearDown(); + } + + [Test] + public void RemappedMigrationTargetIsNotAdoptedByALaterMigration() + { + var vectorManager = server.Provider.StoreWrapper.DefaultDatabase.VectorManager; + + // Occupy the low contexts, so the first incoming context cannot be adopted as-is + vectorManager.AllocateTestContexts(4); + + var occupiedLocally = VectorManager.ContextStep; + + var firstLocal = vectorManager.ResolveMigratedContextForTest(occupiedLocally); + ClassicAssert.AreNotEqual(occupiedLocally, firstLocal, "A context that is in use locally must be steered elsewhere"); + + // The PRIMARY assigns contexts independently, so the value the first migration was steered onto + // can equal the value a second, unrelated migration arrives with + var secondLocal = vectorManager.ResolveMigratedContextForTest(firstLocal); + + ClassicAssert.AreNotEqual(firstLocal, secondLocal, "Two migrated Vector Sets were mapped onto a single context"); + + // Resolution is stable, so replaying either migration's records keeps landing in the same place + ClassicAssert.AreEqual(firstLocal, vectorManager.ResolveMigratedContextForTest(occupiedLocally)); + ClassicAssert.AreEqual(secondLocal, vectorManager.ResolveMigratedContextForTest(firstLocal)); + } + } +} \ No newline at end of file diff --git a/test/standalone/Garnet.test.vectorset/VectorSetRecoveredContextReservationTests.cs b/test/standalone/Garnet.test.vectorset/VectorSetRecoveredContextReservationTests.cs index b12056bdfbb..2a19b19b45c 100644 --- a/test/standalone/Garnet.test.vectorset/VectorSetRecoveredContextReservationTests.cs +++ b/test/standalone/Garnet.test.vectorset/VectorSetRecoveredContextReservationTests.cs @@ -4,6 +4,7 @@ using System.Collections.Generic; using System.Linq; using System.Threading.Tasks; +using Garnet.server; using NUnit.Framework; using NUnit.Framework.Legacy; using StackExchange.Redis; @@ -122,6 +123,74 @@ public async Task RecoveredVectorSetDoesNotShareContextWithNewVectorSetAsync() } } + /// + /// A cluster node with AOF enabled reconciles twice while starting up, once from + /// RecoverCheckpointAndAOFAsync and again from StoreWrapper, and the first pass consumes the + /// recovered metadata. The second pass therefore rebuilds the reservations from the index records + /// alone, and a context that is in use without an index record - a Vector Set that was deleted but + /// whose data has not finished being cleaned up - must still be marked for cleanup rather than + /// silently dropped, or it is handed to the next Vector Set while its old data is still present. + /// + [Test] + public async Task ReconcileWithoutRecoveredMetadataStillMarksUnbackedContextsAsync() + { + const string Live = "live-vs"; + const int Elements = 50; + const int Dim = 32; + const int UnbackedContexts = 4; + + StartServer(tryRecover: false); + + var vectorManager = server.Provider.StoreWrapper.DefaultDatabase.VectorManager; + + // Reservations with no index record of their own, which is what a Vector Set whose data is still + // being cleaned up looks like + vectorManager.AllocateTestContexts(UnbackedContexts); + + using (var redis = ConnectionMultiplexer.Connect(TestUtils.GetConfig(allowAdmin: true))) + { + var db = redis.GetDatabase(0); + + for (var i = 0; i < Elements; i++) + { + ClassicAssert.AreEqual(1, (int)db.Execute("VADD", BuildVaddArgs(Live, Dim, i, $"element_{i}"))); + } + } + + var unbacked = new List(); + for (var context = VectorManager.ContextStep; context <= UnbackedContexts * VectorManager.ContextStep; context += VectorManager.ContextStep) + { + vectorManager.GetContextState(context, out var isInUse, out _, out _); + if (isInUse) + { + unbacked.Add(context); + } + } + + ClassicAssert.AreEqual(UnbackedContexts, unbacked.Count, "expected every allocated context to be reserved"); + + // Hold cleanup so the reconcile's decision stays observable instead of being processed away + await vectorManager.PauseCleanupAsync(); + try + { + // recoveredMetadata is empty here, exactly as it is on the second reconcile of a startup + vectorManager.ReconcileRecoveredState(); + + foreach (var context in unbacked) + { + vectorManager.GetContextState(context, out var isInUse, out var isCleaningUp, out _); + + ClassicAssert.IsTrue( + isInUse && isCleaningUp, + $"context {context} was dropped rather than marked for cleanup, so it can be reused while its data is still present"); + } + } + finally + { + vectorManager.ResumeCleanup(); + } + } + private static object[] BuildVaddArgs(string key, int dim, int seed, string element) { var args = new List { key, "VALUES", $"{dim}" }; From 004436c1f47c13d0f5f3e62678f88be78485603e Mon Sep 17 00:00:00 2001 From: Badrish Chandramouli Date: Tue, 15 Sep 2026 19:39:52 -0700 Subject: [PATCH 22/27] Take StackExchange.Redis 3.2.1 3.2.0 assumed IBufferWriter.GetSpan(sizeHint) returns at least sizeHint bytes. It guarantees one byte; the hint is advisory. When RESPite's CycleBuffer handed back a recycled segment shorter than the hint, MessageWriter.WriteUnifiedSpan sliced past the end of it and the connection was torn down with an ArgumentOutOfRangeException, failing the in-flight command. That showed up here as intermittent RedisConnectionException on concurrent VADD. 3.2.1 is StackExchange/StackExchange.Redis#3220, which checks the returned length and falls back to a piecewise write. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 30f31d5b-9e53-4111-91a3-e16c94510a77 --- Directory.Packages.props | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Directory.Packages.props b/Directory.Packages.props index c917e3623e0..72f013b4e0a 100644 --- a/Directory.Packages.props +++ b/Directory.Packages.props @@ -25,7 +25,7 @@ - + From 45c274af48605a6a26a0af93e40ca437714f89d9 Mon Sep 17 00:00:00 2001 From: Badrish Chandramouli Date: Wed, 16 Sep 2026 11:35:12 -0700 Subject: [PATCH 23/27] Tighten two comments to describe current behaviour Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 30f31d5b-9e53-4111-91a3-e16c94510a77 --- test/cluster/Garnet.test.cluster/ClusterTestUtils.cs | 2 +- .../Garnet.test.vectorset/ExceptionInjectionShutdownTests.cs | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/test/cluster/Garnet.test.cluster/ClusterTestUtils.cs b/test/cluster/Garnet.test.cluster/ClusterTestUtils.cs index e66ae8785cd..b8b05dff8eb 100644 --- a/test/cluster/Garnet.test.cluster/ClusterTestUtils.cs +++ b/test/cluster/Garnet.test.cluster/ClusterTestUtils.cs @@ -420,7 +420,7 @@ private async Task WaitForSyncAsync(ClientClusterConfig clusterConfig) /// /// Describes which of the expected nodes a server has not yet agreed on, so a cluster that never - /// converges reports what it was waiting for instead of only that it ran out of time. + /// converges reports what it was waiting for. /// /// Server whose view was inspected. /// Expected cluster configuration, keyed by node id. diff --git a/test/standalone/Garnet.test.vectorset/ExceptionInjectionShutdownTests.cs b/test/standalone/Garnet.test.vectorset/ExceptionInjectionShutdownTests.cs index 6598430876e..98f847c3ce4 100644 --- a/test/standalone/Garnet.test.vectorset/ExceptionInjectionShutdownTests.cs +++ b/test/standalone/Garnet.test.vectorset/ExceptionInjectionShutdownTests.cs @@ -84,7 +84,7 @@ public void ServerDisposeCompletesWhileAnInjectionPointIsParked() /// Releasing only the waiters that are already parked leaves the same hang one moment later: /// disposal closes listeners before it drains handlers, so a request that is already in flight can /// reach a still-armed injection point after shutdown has begun and park there instead. Nothing - /// re-enables it, so it strands its pooled buffer exactly as before. + /// re-enables it, so it strands its pooled buffer the same way. /// [Test] public void ArrivingAtAnInjectionPointDuringShutdownDoesNotPark() From b53ecdd9136186cc2753511806edc4b020dd5220 Mon Sep 17 00:00:00 2001 From: Badrish Chandramouli Date: Wed, 16 Sep 2026 12:19:29 -0700 Subject: [PATCH 24/27] Wait for the client to observe a promotion before writing to a new primary StackExchange.Redis caches a per-endpoint replica flag and rejects a primary-only command before it leaves the client, refreshing that flag only on its periodic configuration check. A test that promotes a node and immediately writes to it therefore races an incidental multiplexer reconfiguration, and fails with "Command cannot be issued to a replica" even though the node reports the primary role. WaitForPrimaryRole now also waits until the multiplexer no longer regards the node as a replica, forcing it to re-read the topology. The failover tests use it in place of their own role polls. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 30f31d5b-9e53-4111-91a3-e16c94510a77 --- .../ClusterReplicationDisklessSyncTests.cs | 8 +------- .../ClusterReplicationBaseTests.cs | 6 ++++++ .../Garnet.test.cluster/ClusterTestContext.cs | 13 +------------ .../Garnet.test.cluster/ClusterTestUtils.cs | 16 +++++++++++++--- 4 files changed, 21 insertions(+), 22 deletions(-) diff --git a/test/cluster/Garnet.test.cluster.replication.disklesssync/ReplicationTests/ClusterReplicationDisklessSyncTests.cs b/test/cluster/Garnet.test.cluster.replication.disklesssync/ReplicationTests/ClusterReplicationDisklessSyncTests.cs index 4e8467b0cc3..9dfd72c7ed7 100644 --- a/test/cluster/Garnet.test.cluster.replication.disklesssync/ReplicationTests/ClusterReplicationDisklessSyncTests.cs +++ b/test/cluster/Garnet.test.cluster.replication.disklesssync/ReplicationTests/ClusterReplicationDisklessSyncTests.cs @@ -86,13 +86,7 @@ void ResetAndReAttach(int replicaIndex, int primaryIndex, bool soft) void Failover(int replicaIndex, string option = null) { _ = context.clusterTestUtils.ClusterFailover(replicaIndex, option, logger: context.logger); - var role = context.clusterTestUtils.RoleCommand(replicaIndex, logger: context.logger); - while (!role.Value.Equals("master")) - { - ClusterTestUtils.BackOff(cancellationToken: context.cts.Token); - role = context.clusterTestUtils.RoleCommand(replicaIndex, logger: context.logger); - } - + context.clusterTestUtils.WaitForPrimaryRole(replicaIndex, context.logger); } // Deterministic field value so a huge hash can be verified without holding every value in memory. diff --git a/test/cluster/Garnet.test.cluster.replication/ReplicationTests/ClusterReplicationBaseTests.cs b/test/cluster/Garnet.test.cluster.replication/ReplicationTests/ClusterReplicationBaseTests.cs index 33a41674a1d..56a74f583de 100644 --- a/test/cluster/Garnet.test.cluster.replication/ReplicationTests/ClusterReplicationBaseTests.cs +++ b/test/cluster/Garnet.test.cluster.replication/ReplicationTests/ClusterReplicationBaseTests.cs @@ -672,6 +672,9 @@ public void ClusterReplicationSimpleFailover([Values] bool performRMW, [Values] // Enable when old primary becomes replica context.clusterTestUtils.WaitForReplicaRecovery(primaryIndex, logger: context.logger); + // The promoted node has to be usable as a primary by this client before writes are sent to it + context.clusterTestUtils.WaitForPrimaryRole(replicaIndex, context.logger); + // Check if allowed to write to new Primary if (!performRMW) context.PopulatePrimary(ref context.kvPairs, keyLength, kvpairCount, replicaIndex, slotMap: slotMap); @@ -744,6 +747,9 @@ public void ClusterFailoverAttachReplicas([Values] bool performRMW, [Values] boo context.clusterTestUtils.WaitForNoFailover(1, context.logger); context.clusterTestUtils.WaitForNoFailover(2, context.logger); + // Node 1 has to be usable as a primary by this client before writes are sent to it + context.clusterTestUtils.WaitForPrimaryRole(1, context.logger); + // Wait for replica to recover context.clusterTestUtils.WaitForReplicaRecovery(2, context.logger); diff --git a/test/cluster/Garnet.test.cluster/ClusterTestContext.cs b/test/cluster/Garnet.test.cluster/ClusterTestContext.cs index d5e3e90da60..b7437ed2c1e 100644 --- a/test/cluster/Garnet.test.cluster/ClusterTestContext.cs +++ b/test/cluster/Garnet.test.cluster/ClusterTestContext.cs @@ -909,18 +909,7 @@ public void ClusterFailoverSpinWait(int replicaNodeIndex, ILogger logger) _ = clusterTestUtils.ClusterFailover(replicaNodeIndex, "ABORT", logger); _ = clusterTestUtils.ClusterFailover(replicaNodeIndex, logger: logger); - var retryCount = 0; - while (true) - { - var role = clusterTestUtils.GetReplicationRole(replicaNodeIndex, logger: logger); - if (role.Equals("master")) break; - if (retryCount++ > 10000) - { - logger?.LogError("CLUSTER FAILOVER retry count reached"); - Assert.Fail(); - } - Thread.Sleep(1000); - } + clusterTestUtils.WaitForPrimaryRole(replicaNodeIndex, logger); } /// Opens a connection, assigns all slots to primaryIndex, and introduces every other created node. diff --git a/test/cluster/Garnet.test.cluster/ClusterTestUtils.cs b/test/cluster/Garnet.test.cluster/ClusterTestUtils.cs index b8b05dff8eb..ed48e209847 100644 --- a/test/cluster/Garnet.test.cluster/ClusterTestUtils.cs +++ b/test/cluster/Garnet.test.cluster/ClusterTestUtils.cs @@ -3456,13 +3456,23 @@ public void WaitForFailoverCompleted(IPEndPoint endPoint, ILogger logger = null) } } + /// + /// Waits until the node reports the primary role and the multiplexer has observed it. + /// The multiplexer caches a per-endpoint replica flag and rejects a primary-only command + /// before it leaves the client, refreshing that flag only on its periodic configuration + /// check, so a promotion is not usable until the topology has been re-read. + /// public void WaitForPrimaryRole(int nodeIndex, ILogger logger = null) { + var endPoint = GetEndPoint(nodeIndex); while (true) { - var role = RoleCommand(nodeIndex, logger); - if (role.Value.Equals("master")) break; - BackOff(cancellationToken: context.cts.Token); + if (RoleCommand(endPoint, logger).Value.Equals("master")) + { + if (!redis.GetServer(endPoint).IsReplica) break; + _ = redis.Configure(); + } + BackOff(cancellationToken: context.cts.Token, msg: nameof(WaitForPrimaryRole)); } } From d547109c246ee7be6b60b67387fd9197050a37f5 Mon Sep 17 00:00:00 2001 From: Badrish Chandramouli Date: Wed, 16 Sep 2026 15:41:16 -0700 Subject: [PATCH 25/27] Gate the AOF tail witness pulse on a sharded log The pulse was gated on MultiLogEnabled, which is true when either the log is sharded across physical sublogs or a single sublog is replayed by several tasks. The pulse value comes from the sequence number generator, which is constructed only for a sharded log, so a single sublog with multiple replay tasks dereferenced a null generator and killed the AOF sync task, stranding every subsequent write on the primary. A single physical sublog orders records by log address rather than by generator sequence number, and every replay task rendezvouses on the same batch and publishes the same address, so no virtual sublog can lag behind another for a pulse to relieve. Gate the pulse on AofPhysicalSublogCount instead, matching how GarnetLog and GarnetAppendOnlyFile already select between the two ordering schemes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 30f31d5b-9e53-4111-91a3-e16c94510a77 --- .../PrimaryOps/AofOperations/AofSyncTask.cs | 20 +++++-------------- 1 file changed, 5 insertions(+), 15 deletions(-) diff --git a/libs/cluster/Server/Replication/PrimaryOps/AofOperations/AofSyncTask.cs b/libs/cluster/Server/Replication/PrimaryOps/AofOperations/AofSyncTask.cs index 78ef037d893..5f46fa1cdc6 100644 --- a/libs/cluster/Server/Replication/PrimaryOps/AofOperations/AofSyncTask.cs +++ b/libs/cluster/Server/Replication/PrimaryOps/AofOperations/AofSyncTask.cs @@ -29,7 +29,6 @@ public class AofSyncTask : IBulkLogEntryConsumer, IDisposable long previousAddress; readonly bool timePulseEnabled; - readonly bool pulseUsesSequenceNumbers; readonly GarnetAppendOnlyFile appendOnlyFile; readonly TsavoriteLog physicalSublog; readonly long[] pulseTailSnapshot; @@ -115,13 +114,12 @@ public AofSyncTask( this.remoteNodeId = remoteNodeId; this.cts = cts; appendOnlyFile = clusterProvider.storeWrapper.appendOnlyFile; - timePulseEnabled = clusterProvider.serverOptions.MultiLogEnabled; + // Only a log sharded across several physical sublogs needs the pulse. On a single + // physical sublog every replay task rendezvouses on the same batch and publishes the + // same log address, so no virtual sublog can lag behind another for a pulse to relieve. + timePulseEnabled = clusterProvider.serverOptions.AofPhysicalSublogCount > 1; if (timePulseEnabled) { - // Records carry a generator sequence number only when the log is sharded across - // several physical sublogs; a single physical sublog times its virtual sublogs by - // log address instead, and has no generator to read from. - pulseUsesSequenceNumbers = clusterProvider.serverOptions.AofPhysicalSublogCount > 1; physicalSublog = appendOnlyFile.Log.GetSubLog(physicalSublogIdx); pulseTailSnapshot = new long[clusterProvider.serverOptions.AofPhysicalSublogCount]; pulseTailScratch = new long[clusterProvider.serverOptions.AofPhysicalSublogCount]; @@ -300,15 +298,7 @@ void SendAdvanceTimePulse() } } - // The pulse has to carry a value in whatever domain the replay side measures virtual - // sublog time in. Sharded across several physical sublogs that is a generator sequence - // number, because that is what each record carries. On a single physical sublog records - // carry no sequence number and replay publishes the log address it has reached, so the - // pulse is the observed tail: the address replay itself would publish once it has - // consumed everything at or below it. - var sequenceNumber = pulseUsesSequenceNumbers - ? appendOnlyFile.GetLargerThanMaximumSequenceNumber() - : pulseTailScratch[physicalSublogIdx]; + var sequenceNumber = appendOnlyFile.GetLargerThanMaximumSequenceNumber(); if (iter.NextAddress < physicalSublog.TailAddress) { lastAdvanceTimePulse = now; From f4ca40c3e4d1cf0f179848e01a58a6b298272fbd Mon Sep 17 00:00:00 2001 From: Badrish Chandramouli Date: Wed, 16 Sep 2026 16:21:56 -0700 Subject: [PATCH 26/27] Use the shared harness configuration in ClusterReplicateFails ClusterReplicateFails was the only cluster test that hand-built a connection string, so its client configuration did not match the servers it creates: the default command map advertises SUBSCRIBE and PUBLISH against nodes started with pub/sub disabled, and exception detail and client names were off, leaving a connect failure with no attributable information. Route both connections through TestUtils.GetConfig, which trims the command map to the server's command set, applies the suite's connect retry policy, and enables exception detail and per-test client names. The protocol is left unpinned, matching the previous behaviour. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 30f31d5b-9e53-4111-91a3-e16c94510a77 --- .../ClusterNegativeTests.cs | 22 +++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-) diff --git a/test/cluster/Garnet.test.cluster/ClusterNegativeTests.cs b/test/cluster/Garnet.test.cluster/ClusterNegativeTests.cs index b58f56b9fa1..b966212938f 100644 --- a/test/cluster/Garnet.test.cluster/ClusterNegativeTests.cs +++ b/test/cluster/Garnet.test.cluster/ClusterNegativeTests.cs @@ -641,6 +641,20 @@ public void ClusterDontKnowReplicaFailTest([Values] bool useReplicaOf) } } + /// + /// Build a single-endpoint configuration matching the servers these tests create. + /// + /// Endpoint to connect to. + /// ACL user to authenticate as. + /// Password for . + static ConfigurationOptions GetNegativeTestConfig(IPEndPoint endPoint, string user, string password) + => TestUtils.GetConfig( + new EndPointCollection { endPoint }, + allowAdmin: true, + disablePubSub: true, + authUsername: user, + authPassword: password); + [Test, Order(13)] [Category("REPLICATION")] public void ClusterReplicateFails() @@ -662,13 +676,17 @@ public void ClusterReplicateFails() ClassicAssert.AreNotEqual(primaryEndpoint, replicaEndpoint, "Should have different endpoints for nodes"); - using var primaryConnection = ConnectionMultiplexer.Connect($"{primaryEndpoint.Address}:{primaryEndpoint.Port},user={UserName},password={Password},allowAdmin=true"); + // Connect through the shared harness configuration rather than a hand-built connection + // string: it trims the command map to match the pub/sub-disabled servers created above, + // applies the suite's connect retry policy, and enables exception detail and client names + // so a connect failure identifies the test and endpoint. + using var primaryConnection = ConnectionMultiplexer.Connect(GetNegativeTestConfig(primaryEndpoint, UserName, Password)); var primaryServer = primaryConnection.GetServer(primaryEndpoint); ClassicAssert.AreEqual("OK", (string)primaryServer.Execute("CLUSTER", ["ADDSLOTSRANGE", "0", "16383"], flags: CommandFlags.NoRedirect)); ClassicAssert.AreEqual("OK", (string)primaryServer.Execute("CLUSTER", ["MEET", replicaEndpoint.Address.ToString(), replicaEndpoint.Port.ToString()], flags: CommandFlags.NoRedirect)); - using var replicaConnection = ConnectionMultiplexer.Connect($"{replicaEndpoint.Address}:{replicaEndpoint.Port},user={UserName},password={Password},allowAdmin=true"); + using var replicaConnection = ConnectionMultiplexer.Connect(GetNegativeTestConfig(replicaEndpoint, UserName, Password)); var replicaServer = replicaConnection.GetServer(replicaEndpoint); // Try to replicate from a server that doesn't exist From 54cead22d076ea5372a17cf15a38320c3cb82e15 Mon Sep 17 00:00:00 2001 From: Badrish Chandramouli Date: Wed, 16 Sep 2026 16:48:03 -0700 Subject: [PATCH 27/27] Wait for the primary to be known before CLUSTER REPLICATE in tests CLUSTER REPLICATE resolves the primary's node id against the replica's own cluster configuration. CLUSTER MEET only makes the node it is sent to aware of its target; the reverse direction arrives through the gossip handshake that follows. Tests issue MEET at the primary and REPLICATE at the replica, so the replica may not yet know the primary and the command fails with "I don't know about node". Measured immediately after Meet returns, the replica does not know the primary in any run; the tests pass only because unrelated setup work between the two calls usually gives gossip enough time. Issuing REPLICATE at that point reproduces the failure in every run. The index-based ClusterReplicate helper now waits for the replica to know the primary's node id. Tests that assert on the unknown-node error use the node-id overload and are unaffected. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 30f31d5b-9e53-4111-91a3-e16c94510a77 --- test/cluster/Garnet.test.cluster/ClusterTestUtils.cs | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/test/cluster/Garnet.test.cluster/ClusterTestUtils.cs b/test/cluster/Garnet.test.cluster/ClusterTestUtils.cs index ed48e209847..bebe108af4d 100644 --- a/test/cluster/Garnet.test.cluster/ClusterTestUtils.cs +++ b/test/cluster/Garnet.test.cluster/ClusterTestUtils.cs @@ -2305,6 +2305,10 @@ public void ResetReplica(int replicaIndex, int primaryIndex, ILogger logger = nu public string ClusterReplicate(int replicaNodeIndex, int primaryNodeIndex, bool async = false, bool failEx = true, ILogger logger = null) { var primaryId = ClusterMyId(primaryNodeIndex, logger: logger); + // CLUSTER REPLICATE resolves the primary's node id against the replica's own configuration. + // MEET only makes the node it is sent to aware of its target; the reverse direction arrives + // through the gossip handshake that follows, so wait for it before issuing the command. + WaitUntilNodeIdIsKnown(replicaNodeIndex, primaryId, logger: logger); return ClusterReplicate(replicaNodeIndex, primaryId, async: async, failEx: failEx, logger: logger); }