fix(redundancy): make promote() CAS-based to close a concurrent-failover race

[SoundMatt]

Bug (HIGH severity, cpp-RCP v3.0.0 deep audit finding, batch 2)

include/rcp/redundancy.hpp's RedundantRequestFn::send() reads active_ under a short lock, calls the RequestFn pointer outside the lock, and on ErrClosed/ErrTimeout calls promote(). promote() did a blind, unconditional toggle:

active_ = (active_ == &primary_) ? &standby_ : &primary_;

The toggle direction is computed from whatever active_ happens to be at the moment promote() runs, not from what the calling thread actually observed failing. Two send() calls that both start on the primary, both fail concurrently, and both call promote() apply the toggle twice (serialized by the internal mutex): thread A flips primary→standby, thread B's promote() — blind to what B actually saw fail — flips it right back standby→primary.

Net effect: zero net toggles, active_ left pointing at the confirmed-bad primary for every subsequent caller — silently defeating the entire hot-standby failover mechanism, and violating the project's own accepted requirement REQ-RED-006 in .fusa-reqs.json:

"Once RedundantRequestFn has promoted the standby, subsequent send() calls shall continue to be served by the standby without reverting to the primary on their own."

Fix

Added a private, CAS-style promote_from(RequestFn* observed_active):

void promote_from(RequestFn* observed_active) {
    std::lock_guard<std::mutex> lk(mu_);
    if (active_ == observed_active) {
        active_ = (active_ == &primary_) ? &standby_ : &primary_;
    }
}

send() now captures the pointer it actually read under its lock (already done, just wasn't being used) and passes it to promote_from(active) on the auto-promote path, instead of calling the no-arg promote(). If active_ has already moved on by the time a racing caller's promote_from() acquires the lock, it's a no-op instead of a re-toggle.

The public promote() is untouched — it's still an unconditional manual toggle, preserving REQ-RED-004 ("promote() shall switch the active RequestFn between primary and standby") and all existing manual-promote() test behavior exactly as-is. This is a targeted fix to the auto-promote path inside send(), not a redesign of the public API.

No other files or unrelated code touched.

Test

tests/test_redundancy.cpp gains one new [thread]-tagged case (tagged REQ-RED-006 per the existing // fusa:test REQ-ID convention):

• Drives 8 threads concurrently through send() against a primary RequestFn that always fails, using a two-phase handshake (entered_cv/release_cv mutex+condition_variable gate) — the same deterministic-concurrency pattern already established in this codebase (see tests/test_shmem.cpp's "admits up to queue_capacity concurrent callers" case).
• The gate guarantees every thread has already read active_ == &primary_ under send()'s lock before any of them can reach promote_from() — making the race deterministic rather than relying on OS scheduling luck.
• All 8 are released together, forcing a genuine concurrent race on promote_from().
• Asserts active_ ends up on the standby, never reverted to primary.
• The standby is a trivial stateless always-succeeds closure (not a mock::Server), so the test doesn't introduce an unrelated race of its own from concurrent mock::Server::dispatch() calls, which isn't documented as thread-safe.

Verification

1. Full rebuild from scratch: rm -rf build && cmake -S . -B build -DRCP_BUILD_TESTS=ON -DCMAKE_BUILD_TYPE=Debug && cmake --build build -j — 0 errors, 0 warnings under -Wall -Wextra -Wpedantic.
2. Full test suite: ctest --output-on-failure — 100% pass, 58/58 test binaries, including the new case.
3. Flakiness check (fixed code): ran the new test standalone with --rng-seed time, 30/30 passed (20 before the mutation check + 30 after reapplying the fix).
4. Mutation-testing sanity check: temporarily reverted only the promote_from(active) call back to the old blind promote() toggle, rebuilt clean (still 0 warnings), and ran the new test standalone 20x — 20/20 reliably FAILED with:

   REQUIRE_FALSE( rr.is_primary_active() )
   with expansion:
     !true
   

   confirming the test deterministically catches the exact bug. Reapplied the fix, rebuilt clean, reran 30x — 30/30 passed.

Closes a finding from the cpp-RCP v3.0.0 deep audit.