Skip to content

[To dev/1.3] [Pipe] Wake parser workers when admission is available - #18865

Merged
jt2594838 merged 1 commit into
apache:dev/1.3from
Caideyipi:cp-aec1ad-dev-1.3
Oct 14, 2026
Merged

jt2594838 merged 1 commit into
apache:dev/1.3from
Caideyipi:cp-aec1ad-dev-1.3

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Description

When TsFile parser admission becomes available, processor workers can remain asleep until their next polling interval, delaying pipes behind the fair admission queue. Wake the relevant worker immediately and retain notifications that arrive before the worker begins its idle wait.

Backport of #18843, cherry-picked from aec1ad6139cb4dd1450725f53418cf40c26efb4e to dev/1.3.

  • Connect pending parser reservations to their processor worker's wake-up callback and signal the next eligible reservation while preserving fair queue ordering.
  • Reuse one memory-budget snapshot during admission checks and expose admission reject/yield counters, a wait timer, and in-flight/waiting gauges.
  • Preserve the 1.3 memory-accounting implementation when resolving PipeMemoryManager conflicts, and retain the existing disk-I/O counter when resolving PipeResourceMetrics conflicts.
  • Backport regression coverage for notification ordering, notifications before idle wait, 100 waiting pipes, and stop/start behavior.

Validation

  • mvn -B -nsu spotless:apply -pl iotdb-core/datanode,iotdb-core/node-commons — passed; no formatting changes.
  • mvn -B -nsu test-compile -DskipTests — full reactor passed.
  • mvn -B -nsu test-compile -P with-zh-locale -DskipTests — full reactor passed with MAVEN_OPTS="-Xmx2048m -XX:MaxMetaspaceSize=512m"; the initial uncapped attempt exhausted local native memory after DataNode compiled successfully.
  • mvn -B -nsu test -pl iotdb-core/datanode -am -Dtest=PipeProcessorSubtaskExecutionGuardTest,PipeProcessorSubtaskWorkerTest,PipeMemoryManagerTest -Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false — all 18 tests passed, with no failures, errors, or skipped tests.
  • git diff --check — passed.

Checklist

  • Self-reviewed, including notification ordering and concurrent admission/release.
  • Backported unit tests for the changed behavior.

* Fix pipe parser admission wake-ups

* Avoid reflective test dependency

(cherry picked from commit aec1ad6)
@jt2594838
jt2594838 merged commit 6527fb0 into apache:dev/1.3 Oct 14, 2026
15 of 16 checks passed
@jt2594838
jt2594838 deleted the cp-aec1ad-dev-1.3 branch October 14, 2026 08:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants