Skip to content

[To dev/1.3] Fence IoTConsensus writes before storage shutdown (#18841, #18893) - #18896

Open
Caideyipi wants to merge 2 commits into
apache:dev/1.3from
Caideyipi:cp/iotconsensus-shutdown-dev-1.3
Open

Caideyipi wants to merge 2 commits into
apache:dev/1.3from
Caideyipi:cp/iotconsensus-shutdown-dev-1.3

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Oct 15, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Queued IoTConsensus requests can pass the RPC read-only check before shutdown and apply after storage or WAL has stopped, reopening processors and WAL nodes. This backport closes write admission and drains admitted writes, including their WAL separators, before the final flush and storage shutdown.

Cherry-picks:

Adaptation for dev/1.3

  • Reuse DataRegion.insertLock for admission and WAL separators, keeping memory backpressure and retry waits outside the lock. Apply the same fencing to tree deletions, which already hold the outer visitor lock in 1.3.
  • Adapt the 1.3 multi-tablet implementation so the complete admitted batch finishes under the write lock with one WAL separator. Add a regression test for shutdown between tablets.
  • Use the 1.3 request, state-machine, and status APIs. Preserve its independent per-request SyncLog status handling.
  • Backport the tree-model replication check to pipe.it.autocreate.IoTDBPipeClusterIT. Table-model code and tests do not apply to 1.3.

Validation

  • Spotless formatting passed for consensus, datanode, and integration-test.
  • Reactor build and targeted tests: mvn -B -nsu test -pl iotdb-core/datanode -am -Dtest=IoTConsensusServerImplTest,IoTConsensusShutdownTest,DataRegionStateMachineTest,DataRegionTest -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false -DforkCount=1.
  • Result: BUILD SUCCESS; 49 tests passed and 1 existing DataRegion test was skipped. The configured Maven test executions both passed.
  • Coverage includes queued writes, the pre-apply race, draining admitted writes and separators, retry waits, memory backpressure, all 1.3 write paths after stop, other consensus protocols, and WAL checkpoint recovery.
  • Cluster integration tests were not run locally.

This PR has:

  • been self-reviewed, including concurrent writes and shutdown.
  • added and adapted regression tests.

* Fix queued IoTConsensus writes during shutdown

* Test queued IoTConsensus writes across storage shutdown

* Reproduce WAL checkpoint loss from writes after shutdown

* Fence IoTConsensus applies before storage shutdown

(cherry picked from commit 532d6c1)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant