Fix ConfigNode client failover with stale candidates

[d-wang-commit]

Description

Recover from stale ConfigNode candidates

A pooled ConfigNode client created during startup can retain a seed-only address list after its DataNode has learned additional ConfigNode addresses. If the former leader remains reachable over TCP but returns REDIRECTION_RECOMMEND without a redirect address, reconnecting to it can succeed repeatedly while the client continues using stale candidates.

This change updates fallback selection in AbstractConfigNodeClient:

1. Refresh candidates from the DataNode's local ConfigNodeInfo before scanning fallback endpoints. This also covers RPC failures and connection failures to a cached leader.
2. Locate the last successfully connected endpoint in the refreshed list and continue from the following endpoint. This preserves retry progress when the list is reordered. If that endpoint has been removed, selection starts from the first candidate.
3. Replace the candidate list and reset the cursor only when the latest list is non-empty. An empty refresh preserves both the existing candidates and cursor.

Re-enable all nine cases in IoTDBNodeStatusFailoverIT by reverting the temporary skip from 4541192. The original scenarios and timeouts are preserved.

Tests

• Added AbstractConfigNodeClientTest: 12 unit tests covering stale and reordered candidates, repeated redirects without a target, explicit redirects, removed and empty candidates, RPC and connection failures, bounded connection attempts, successful RPCs, and leader warm-up.
• Added AbstractConfigNodeClientConnectionTest: two unit tests covering connection pool reuse and closure, and concurrent reconnection with 100 clients using real loopback TCP connections. These verify client connection lifecycle rather than a 100-DataNode deployment.
• Restored the existing IoTDBNodeStatusFailoverIT; no new IT class was added.
• Ran both new UT classes and the existing ConfigNodeInfoTest: 15 tests passed.
• Ran all nine IoTDBNodeStatusFailoverIT cases and both existing IoTDBRemoveConfigNodeSchemaPartitionIT cases: 11 tests passed. These include the live former-leader partition scenario and retrying after a ConfigNode returns REDIRECTION_RECOMMEND without a redirect address.
• Validation completed in one successful Maven reactor build on Windows with JDK 17, with zero failures, errors, or skipped tests. Spotless and git diff --check passed.

Side effects and risks

Entering fallback selection adds a local read-lock acquisition, candidate-list copy, and endpoint lookup, with work proportional to the number of ConfigNodes. Shared-list updates can contend with these reads. No discovery RPC, additional connection pool, protocol change, or configuration change is introduced.

Successful RPCs, explicit redirect priority, and known-leader warm-up behavior are preserved. Empty refreshes retain previously known addresses, which may themselves be stale.

The existing retry counter and backoff condition are unchanged; there is no additional refresh before the backoff check. A stale singleton list can therefore still trigger a two-second backoff before fallback selection refreshes the candidates. The counter does not track distinct endpoints, so this change does not guarantee that every candidate is tried before sleeping or that recovery completes within a fixed timeout.

---

This PR has:

• been self-reviewed.
• added comments explaining the intent of the code.
• added unit tests to cover changed retry behavior.
• been tested in a test IoTDB cluster.

---

Key changed/added classes (or packages if there are too many classes) in this PR

• AbstractConfigNodeClient: refresh candidates before fallback selection, resume after the last connected endpoint, and preserve candidates and cursor on empty refreshes.
• AbstractConfigNodeClientTest: verify candidate selection and retry behavior.
• AbstractConfigNodeClientConnectionTest: verify connection reuse and release during concurrent reconnects.
• IoTDBNodeStatusFailoverIT: restore all nine temporarily skipped integration tests.