Skip to content

[To dev/1.3] [Performance] Pre-size collections with known cardinality (#18851) - #18854

Merged
jt2594838 merged 1 commit into
dev/1.3from
cp-18851-dev-1.3
Oct 14, 2026
Merged

jt2594838 merged 1 commit into
dev/1.3from
cp-18851-dev-1.3

Conversation

@Caideyipi

Copy link
Copy Markdown
Collaborator

Description

Backport #18851 (a78f5a736c7e34a951fa1dda34a314fc8a47b7c9) to dev/1.3 by re-scanning this branch's production Java code and adapting the collection allocation optimization to its implementations.

Collections with a known number of entries currently grow their backing arrays or hash tables while being filled. Pre-size 152 collections in 80 production Java files across six modules:

  • Session batch-write buffers and schema lists, and both JDBC and Session result column metadata.
  • Schema/path, query and write plan, logical-view, template, and ConfigNode plan/procedure deserialization.
  • Query projections, expression transformers, CASE evaluation, and spill readers.
  • Memtable row/tablet schema lists and recovery, TsFile device sets, Pipe payloads and progress metadata, and UDF point/tangent materialization.

Use serialized counts or the number of outputs produced from the input. Guava expected-size factories account for the hash-table load factor. Optional empty lists use capacity zero; memtable schema-list sizing handles nullable measurement arrays.

The dev/1.3 scan includes its aggregation/device-view/templated plan deserialization and logical-view maps, and uses the branch's existing DataNode transformation, AuthorPlan, DeviceTimeIndex, and PipeConsensus implementations.

Validation

  • Spotless applied to all six affected modules; Checkstyle, Spotless, and forbidden API checks passed in the reactor build.
  • English reactor test-compile passed for the affected modules and their dependencies (28 modules).
  • 48 existing test classes passed: 378 tests, zero failures/errors/skips. Coverage includes Session, schema/path and plan/procedure serialization, memtable partial inserts and recovery, TsFile metadata, Pipe payloads/progress metadata, expression/operator behavior, and UDF pattern processing.
  • Chinese locale reactor test-compile passed for the same 28 modules with -P with-zh-locale.
  • git diff --cached --check passed.

This PR has:

  • been self-reviewed.

Backport #18851 by re-scanning dev/1.3 for equivalent collection
materialization paths and adapting capacity hints to this branch's
schema, plan, Session, Pipe, memtable, and UDF implementations.

(cherry picked from commit a78f5a7)
@codecov

codecov Bot commented Oct 13, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 69.23077% with 52 lines in your changes missing coverage. Please review.
✅ Project coverage is 45.41%. Comparing base (c056ce6) to head (65a6768).

Files with missing lines Patch % Lines
...n/expression/visitor/ColumnTransformerVisitor.java 0.00% 4 Missing ⚠️
...quest/PipeConsensusTransferFileSealWithModReq.java 0.00% 4 Missing ⚠️
...ion/operator/process/FilterAndProjectOperator.java 0.00% 3 Missing ⚠️
...tion/dag/column/CaseWhenThenColumnTransformer.java 0.00% 3 Missing ⚠️
...in/java/org/apache/iotdb/rpc/IoTDBJDBCDataSet.java 50.00% 2 Missing ⚠️
...ain/java/org/apache/iotdb/rpc/IoTDBRpcDataSet.java 50.00% 2 Missing ⚠️
...tdb/db/queryengine/plan/analyze/TemplatedInfo.java 0.00% 2 Missing ⚠️
...plan/parameter/model/ModelInferenceDescriptor.java 0.00% 2 Missing ⚠️
.../logfile/visitor/SchemaRegionPlanDeserializer.java 81.81% 2 Missing ⚠️
...geengine/dataregion/memtable/AbstractMemTable.java 83.33% 2 Missing ⚠️
... and 23 more
Additional details and impacted files
@@            Coverage Diff             @@
##             dev/1.3   #18854   +/-   ##
==========================================
  Coverage      45.41%   45.41%           
  Complexity       198      198           
==========================================
  Files           3667     3667           
  Lines         251096   251115   +19     
  Branches       31089    31109   +20     
==========================================
+ Hits          114038   114052   +14     
- Misses        137058   137063    +5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@jt2594838
jt2594838 merged commit dfcd489 into dev/1.3 Oct 14, 2026
15 of 17 checks passed
@jt2594838
jt2594838 deleted the cp-18851-dev-1.3 branch October 14, 2026 04:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants