Skip to content

Fix schema region NPE during DataNode shutdown #18821

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jt2594838 merged 5 commits into from
Oct 12, 2026
Merged

Fix schema region NPE during DataNode shutdown #18821

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
126b14e
Fix schema region NPE during DataNode shutdown
jt2594838 Oct 10, 2026
df08230
Handle missing schema regions in schema writes
jt2594838 Oct 10, 2026
f90ab64
Handle cleared MTree roots during lookup
jt2594838 Oct 12, 2026
705d388
Return retryable status while SchemaEngine clears regions
jt2594838 Oct 12, 2026
ef05aa2
Return retryable status for alter during schema teardown
jt2594838 Oct 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions iotdb-core/datanode/src/main/i18n/en/org/apache/iotdb/db/i18n/DataNodeQueryMessages.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@
package org.apache.iotdb.db.i18n;

public final class DataNodeQueryMessages {
public static final String MESSAGE_SCHEMA_REGION_IS_UNAVAILABLE_PLEASE_RETRY_LATER_D642510A =
"Schema region is unavailable. Please retry later.";

@jt2594838 jt2594838 Oct 10, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adds the retryable missing-region message to the English message catalog so the new validation result participates in the existing compile-time localization mechanism.


public static final String EXCEPTION_FRAGMENT_INSTANCE_ARG_IS_ALREADY_ARG_B44984B4 =
"Fragment instance %s is already %s";
Expand Down
2 changes: 2 additions & 0 deletions iotdb-core/datanode/src/main/i18n/zh/org/apache/iotdb/db/i18n/DataNodeQueryMessages.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,8 @@
package org.apache.iotdb.db.i18n;

public final class DataNodeQueryMessages {
public static final String MESSAGE_SCHEMA_REGION_IS_UNAVAILABLE_PLEASE_RETRY_LATER_D642510A =
"SchemaRegion 不可用，请稍后重试。";

@jt2594838 jt2594838 Oct 10, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adds the same message key in Chinese to preserve locale parity. DataNode and ConfigNode compiled successfully with the Chinese profile; the full reactor stopped later at distribution because SNAPSHOT ZIP dependencies were unavailable.


public static final String EXCEPTION_FRAGMENT_INSTANCE_ARG_IS_ALREADY_ARG_B44984B4 =
"Fragment instance %s 已处于 %s 状态";
Expand Down
36 changes: 34 additions & 2 deletions ...src/main/java/org/apache/iotdb/db/queryengine/execution/executor/RegionWriteExecutor.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -176,6 +176,12 @@ public RegionExecutionResult execute(final ConsensusGroupId groupId, final PlanN
return RegionExecutionResult.create(
false, errorMsg, RpcUtils.getStatus(TSStatusCode.NO_AVAILABLE_REGION_GROUP));
}
// SchemaEngine clears regions before DataNodeRegionManager removes their validation locks.
// Failures from an in-flight schema write during that window must remain retryable.
if (groupId instanceof SchemaRegionId && schemaEngine.isClearing()) {

@jt2594838 jt2594838 Oct 12, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maps exceptions from in-flight schema writes to retryable status while SchemaEngine teardown is active. The validation lock may still be registered after clear begins, so the general outer catch cannot infer region availability from that lock.

return RegionExecutionResult.create(
false, e.getMessage(), RpcUtils.getStatus(TSStatusCode.NO_AVAILABLE_REGION_GROUP));
}
LOGGER.warn(e.getMessage(), e);
return RegionExecutionResult.create(
false,
Expand Down Expand Up @@ -842,16 +848,21 @@ private RegionExecutionResult executeInternalCreateMultiTimeSeries(
}

/**
* Check the quota before creating time series.
* Check region availability, data types and quota before creating time series.
*
* @return null if the quota is not exceeded, otherwise return the execution result.
* @return null if all checks pass, otherwise return the execution result.
*/
private RegionExecutionResult checkQuotaAndTypeBeforeCreatingTimeSeries(
final ISchemaRegion schemaRegion,
final PartialPath path,
final int size,
final List<String> measurements,
final List<TSDataType> dataTypes) {
// Shutdown may clear SchemaEngine while the region's validation lock still exists.
// Reject the request as retryable before dereferencing the missing region.
if (schemaRegion == null) {
return createSchemaRegionUnavailableResult();

@jt2594838 jt2594838 Oct 10, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This reuses the centralized retryable result for internal timeseries creation after the original guard. Keeping one result builder ensures shutdown races consistently return NO_AVAILABLE_REGION_GROUP (906) with the localized message across all schema-write prechecks; live-region quota and type validation remain unchanged.

}
for (int i = 0; i < measurements.size(); ++i) {
if (dataTypes.get(i) == TSDataType.OBJECT) {
final String errorStr =
Expand Down Expand Up @@ -933,6 +944,9 @@ private RegionExecutionResult executeAlterTimeSeries(
AlterTimeSeriesNode node, WritePlanNodeExecutionContext context, boolean receivedFromPipe) {
ISchemaRegion schemaRegion =
schemaEngine.getSchemaRegion((SchemaRegionId) context.getRegionId());
if (schemaRegion == null) {

@jt2594838 jt2594838 Oct 10, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AlterTimeSeries fetches its measurement before consensus, so a cleared SchemaEngine entry could otherwise be dereferenced while the validation lock still exists. Returning the same retryable 906 result here preserves the shutdown behavior established for timeseries creation; the available-region path remains unchanged.

return createSchemaRegionUnavailableResult();
}
Comment on lines +947 to +949

@Caideyipi Caideyipi Oct 12, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Handle a region cleared after the availability check

This guard handles a region that is already absent from the map. An in-flight request can still obtain a non-null region and call fetchMeasurementPath() after shutdown clears that same object. In Memory mode, SchemaRegionMemoryImpl.clear() calls MTreeBelowSGMemoryImpl.clear(), which sets databaseMNode to null; getNodeByPath() subsequently dereferences it. That NullPointerException reaches the outer execute() catch, which still returns INTERNAL_SERVER_ERROR (305) if the validation lock remains registered.

Please coordinate this local validation with region teardown, or convert the failure to NO_AVAILABLE_REGION_GROUP (906) when the region is confirmed to have been cleared. The existing validation lock alone is insufficient because SchemaEngine.clear() does not acquire it.

The unavailable-region cases in the new tests start with getSchemaRegion() returning null. Please add a case that pauses after region lookup, clears the region, and resumes the request to cover this interleaving.

@jt2594838 jt2594838 Oct 12, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 705d388. SchemaEngine now publishes its teardown state before clearing regions, RegionWriteExecutor returns retryable NO_AVAILABLE_REGION_GROUP for in-flight schema-write failures during teardown, and RegionWriteExecutorTest covers the after-lookup/clear interleaving.

try {
MeasurementPath measurementPath = schemaRegion.fetchMeasurementPath(node.getPath());
if (node.isAlterView() && !measurementPath.getMeasurementSchema().isLogicalView()) {
Expand All @@ -957,6 +971,9 @@ private RegionExecutionResult executeAlterTimeSeries(
new PipeEnrichedWritePlanNode(node), context)
: PlanVisitor.super.visitAlterTimeSeries(node, context);
} catch (MetadataException e) {
if (schemaEngine.isClearing()) {
return createSchemaRegionUnavailableResult();

@jt2594838 jt2594838 Oct 12, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The MTree cleared-root guard reports PathNotExistException, which this validation method catches locally. Mapping MetadataException to the shared retryable result while SchemaEngine is clearing prevents this catch from returning a normal path-not-exist status during shutdown; ordinary metadata errors retain their existing behavior.

}
return RegionExecutionResult.create(
true, e.getMessage(), RpcUtils.getStatus(e.getErrorCode(), e.getMessage()));
}
Expand Down Expand Up @@ -1123,6 +1140,9 @@ private RegionExecutionResult executeCreateLogicalView(
final ISchemaRegion schemaRegion =
schemaEngine.getSchemaRegion((SchemaRegionId) context.getRegionId());
if (CONFIG.getSchemaRegionConsensusProtocolClass().equals(ConsensusFactory.RATIS_CONSENSUS)) {
if (schemaRegion == null) {

@jt2594838 jt2594838 Oct 10, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ratis validates logical-view targets locally before writing consensus state. Guarding the missing region prevents a shutdown-time null dereference and makes the request retryable; SimpleConsensus continues to use its existing direct write path.

return createSchemaRegionUnavailableResult();
}
context.getRegionWriteValidationRWLock().writeLock().lock();
try {
// step 1. make sure all target paths do NOT exist.
Expand Down Expand Up @@ -1180,6 +1200,9 @@ private RegionExecutionResult executeCreateOrUpdateTableDevice(
final boolean receivedFromPipe) {
final ISchemaRegion schemaRegion =
schemaEngine.getSchemaRegion((SchemaRegionId) context.getRegionId());
if (schemaRegion == null) {

@jt2594838 jt2594838 Oct 10, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Table-device quota validation also reads the schema region before consensus. This guard covers the same shutdown window as the timeseries path and returns 906 before checkSchemaQuota is called, while preserving normal validation for an available region.

return createSchemaRegionUnavailableResult();
}
try {
schemaRegion.checkSchemaQuota(node.getTableName(), node.getDeviceIdList());
} catch (final SchemaQuotaExceededException e) {
Expand All @@ -1192,6 +1215,15 @@ private RegionExecutionResult executeCreateOrUpdateTableDevice(
: PlanVisitor.super.visitCreateOrUpdateTableDevice(node, context);
}

private RegionExecutionResult createSchemaRegionUnavailableResult() {

@jt2594838 jt2594838 Oct 10, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Centralizing the unavailable-region result keeps all guarded schema-write paths aligned on the retryable status and localized message. The helper also documents why a still-registered validation lock must not be treated as proof that the region is usable.

// A validation lock can outlive its region during shutdown, so all local schema checks
// must report a missing region as retryable even when the lock is still registered.
final String message =
DataNodeQueryMessages.MESSAGE_SCHEMA_REGION_IS_UNAVAILABLE_PLEASE_RETRY_LATER_D642510A;
return RegionExecutionResult.create(
false, message, RpcUtils.getStatus(TSStatusCode.NO_AVAILABLE_REGION_GROUP, message));
}

@Override
public RegionExecutionResult visitPipeEnrichedWritePlanNode(
final PipeEnrichedWritePlanNode node, final WritePlanNodeExecutionContext context) {
Expand Down
9 changes: 9 additions & 0 deletions iotdb-core/datanode/src/main/java/org/apache/iotdb/db/schemaengine/SchemaEngine.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,9 @@ public class SchemaEngine {

private SchemaMetricManager schemaMetricManager;

// Published so write handlers can identify teardown failures before region locks are removed.
private volatile boolean clearing;

@jt2594838 jt2594838 Oct 12, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Publishes teardown state across threads because write execution may observe clear() from another thread. init() resets it for a subsequent engine lifecycle, while clear() marks it before schema resources are torn down.


private final DataNodeSchemaQuotaManager schemaQuotaManager =
DataNodeSchemaQuotaManager.getInstance();

Expand All @@ -110,6 +113,7 @@ public static SchemaEngine getInstance() {
}

public void init() {
clearing = false;
logger.info(
DataNodeSchemaMessages.USED_SCHEMA_ENGINE_MODE,
CommonDescriptor.getInstance().getConfig().getSchemaEngineMode());
Expand Down Expand Up @@ -244,6 +248,7 @@ public void forceMlog() {
}

public synchronized void clear() {
clearing = true;
schemaRegionLoader.clear();

// clearSchemaResource will shut down release and flush task in PBTree mode, which must be
Expand All @@ -268,6 +273,10 @@ public synchronized void clear() {
ClusterTemplateManager.getInstance().clear();
}

public boolean isClearing() {
return clearing;
}

public ISchemaRegion getSchemaRegion(final SchemaRegionId regionId) {
return schemaRegionMap.get(regionId);
}
Expand Down
6 changes: 6 additions & 0 deletions .../org/apache/iotdb/db/schemaengine/schemaregion/mtree/impl/mem/MTreeBelowSGMemoryImpl.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1041,6 +1041,9 @@ protected Void collectEntity(final IDeviceMNode<IMemMNode> node) {
public IMemMNode getNodeByPath(final PartialPath path) throws PathNotExistException {
final String[] nodes = path.getNodes();
IMemMNode cur = databaseMNode;
if (cur == null) {

@jt2594838 jt2594838 Oct 12, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SchemaRegion.clear() nulls the MTree database root. Converting that cleared state to the existing PathNotExistException lets checkDeviceNodeExists handle shutdown without an NPE in quota validation; live-tree path lookup behavior is unchanged.

throw new PathNotExistException(path.getFullPath(), true);
}
IMemMNode next;
for (int i = levelOfDB + 1; i < nodes.length; i++) {
next = cur.getChild(nodes[i]);
Expand Down Expand Up @@ -1924,6 +1927,9 @@ protected Void collectMeasurement(final IMeasurementMNode<IMemMNode> node) {
// region table device management

public int getTableDeviceNotExistNum(final String tableName, final List<Object[]> deviceIdList) {
if (databaseMNode == null) {

@jt2594838 jt2594838 Oct 12, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Table-device quota validation calls this count after the schema MTree may have been cleared. Treating every requested device as absent avoids dereferencing the missing root and keeps the count consistent with an empty tree.

return deviceIdList.size();
}
final IMemMNode tableNode = databaseMNode.getChild(tableName);
int notExistNum = deviceIdList.size();
if (tableNode == null) {
Expand Down
7 changes: 6 additions & 1 deletion ...g/apache/iotdb/db/schemaengine/schemaregion/mtree/impl/pbtree/MTreeBelowSGCachedImpl.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1074,6 +1074,9 @@ public ICachedMNode getNodeByPath(PartialPath path) throws MetadataException {
ICachedMNode cur = databaseMNode;
ICachedMNode next;
try {
if (cur == null) {

@jt2594838 jt2594838 Oct 12, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After PBTree clear, the database root is null. This maps that state to PathNotExistException, and the guarded finally block avoids trying to unpin a null node. checkDeviceNodeExists can then handle the shutdown state without an NPE.

throw new PathNotExistException(path.getFullPath(), true);
}
for (int i = levelOfDB + 1; i < nodes.length; i++) {
next = store.getChild(cur, nodes[i]);
if (next == null) {
Expand All @@ -1090,7 +1093,9 @@ public ICachedMNode getNodeByPath(PartialPath path) throws MetadataException {
pinMNode(cur);
return cur;
} finally {
unPinPath(cur);
if (cur != null) {
unPinPath(cur);
}
}
}

Expand Down
Loading
Loading