Fix stale IoTConsensus peers after restart during removal

[d-wang-commit]

Description

Restart during IoTConsensus peer removal

A surviving replica can restart after the removed peer's sync channel has been deleted but before ConfigNode commits the updated replica set. The restart response still includes the removed peer, so IoTConsensus recreates its dispatcher. Previously, completing the metadata update did not correct that recovered peer list. Replication to the removed peer kept failing and pinned the minimum sync index, leaving syncLag above zero.

Reconcile before completing removal

For IoTConsensus V1 DataRegions, commit the replica-set removal first, then use the existing resetPeerList RPC to send the final membership to every retained replica. The existing DataNode handler waits for consensus startup before applying the list. Finish the procedure only after every retained replica acknowledges success.

Metadata failures, missing or unsuccessful responses, and exceptions keep the existing final state pending and retry after a five-second timeout. Offline replicas are included. ProcedureExecutor registers the timeout after persisting the suspended state, allowing retries to resume automatically and after recovery. No procedure state IDs, serialized fields, DataNode production code, or RPC definitions are changed.

Tests

• UT added: RestartDuringRemovePeerTest covers metadata ordering, failed or missing responses, exceptions, recovery, idempotent replay, and protocol gating. StateMachineTimeoutTest covers persistence-before-wakeup and executor recovery. RemovePeerRestartConsensusTest uses real IoTConsensus instances and loopback Thrift RPCs to reproduce the stale dispatcher, verify its removal and syncLag convergence, and verify subsequent writes and another restart.
• UT run: All 17 tests passed across the three new classes, RemoveRegionPeerProcedureTest, and STMProcedureTest. The same restart regression fails with the original removal procedure because no final reset is sent; with the fix, syncLag falls from 8 to 0 and remains 0 after further writes and restart.
• IT: No full cluster IT was added or run. The consensus regression restarts instances in one JVM with a test internal-service facade and an in-memory state machine; it does not launch full DataNode processes.
• Build checks: ConfigNode Spotless and the full reactor Chinese-locale package build passed. The selected UTs ran with the default English locale.

Side effects and risks

• An unavailable retained replica now delays completion and causes repeated metadata/RPC attempts. The procedure remains pending instead of silently leaving stale membership behind.
• The serialized procedure layout is unchanged, but ConfigNodes taking over a suspended retry must contain this fix; older leaders retain the previous generic timeout behavior.
• This reuses existing DataNode dispatcher cleanup, which can wait for its stop timeout. The tests verify removal from membership and syncLag accounting, not complete dispatcher-thread termination.
• Already completed removal procedures are not replayed automatically, so previously affected Regions need separate reconciliation. The reconciliation is limited to IoTConsensus V1 DataRegions.

---

This PR has:

• been self-reviewed.
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.

---

Key changed/added classes (or packages if there are too many classes) in this PR

• RemoveRegionPeerProcedure: retain and retry finalization until membership reconciliation succeeds.
• RegionMaintainHandler: verify metadata removal and reset every retained replica's peer list.
• ProcedureExecutor: schedule suspended procedures after persistence.
• ProcedureMessages (en/zh): add reconciliation and retry diagnostics.
• RestartDuringRemovePeerTest, RemovePeerRestartConsensusTest, StateMachineTimeoutTest: cover removal recovery and timeout execution.