[Feature] Support TAG value filtering for table subscriptions by Caideyipi · Pull Request #18639 · apache/iotdb

Caideyipi · 2026-09-15T07:55:41Z

Description

Table subscriptions can select databases, tables, and output columns, but they cannot filter individual rows by TAG value. This PR adds the tag-filter topic attribute for table-model subscriptions. Its default value is true, so existing topics keep their current behavior.

Filter semantics

Parse a restricted SQL predicate over TAG columns. The supported forms are AND, OR, NOT, =, !=, IN, LIKE, REGEXP, and IS NULL, evaluated with SQL three-valued logic.
CREATE/ALTER validates the topic attributes and filter expression syntax. Each DataNode binds referenced identifiers against its current table schema when it refreshes the matcher. Missing schema, stale bindings, unknown fields, and non-TAG fields fail closed instead of leaking unfiltered rows.
Filter Tablet rows while preserving row order, data types, timestamps, and null bitmaps. TAG filtering runs before column-filter projection.

Delivery paths and configuration changes

Apply the filter to Record/Tablet and TsFile delivery, including normal subscription queues, consensus subscription queues, and Tree View projection.
Snapshot filter configuration per event batch so ALTER TOPIC takes effect on subsequent batches without changing the meaning of an in-flight batch.
Rebind matchers when table schema versions change, and clean up temporary TsFile rewrite resources on empty output and failure paths.
Append TagFilterStatus and TagFilterMessage to table-model SHOW TOPICS. The status reports MATCH_ALL, MATCH_NONE, ACTIVE, or ERROR for the DataNode executing the statement; the message contains the local matcher failure when the status is ERROR. Tree-model output remains unchanged.

Validation

Spotless and Checkstyle: passed for the changed modules.
Focused DataNode tests: 14 tests passed (ShowTopicsTaskTest, TabletTagFilterTest, and SubscriptionTopicAgentTest).
TopicConfigTest: 10 tests passed.
Chinese-locale DataNode test compilation: passed (mvn test-compile -pl iotdb-core/datanode -P with-zh-locale -DskipTests).
git diff --check: passed.

Performance comparison

Benchmarked commit 5e9f4c07cd7 with the subscription feature gate enabled in the test build; the TAG filtering implementation was unchanged. iot-benchmark generated a fixed table with 1,000,000 rows, 200 devices, 10 random DOUBLE fields, and one uniformly distributed TAG (tag_0, 10 values). All columns were retained.

Snapshot subscriptions; one warm-up and three measured runs per case, with shuffled order. The table reports medians. Consumption time runs from the start of subscribe() until all expected field cells are delivered and the last data batch is acknowledged. Topic creation, the later automatic snapshot-termination notification, and subsequent TsFile content verification are excluded. Input throughput uses all 1,000,000 source rows for every case.

Delivery	TAG predicate	Delivered rows	Consumption median [min–max] (s)	Time vs. no filter	Input throughput (M rows/s)	DataNode CPU (s)
Tablet/Record	Unset (default `true`)	1,000,000	4.056 [3.633–4.511]	baseline	0.247	5.63
Tablet/Record	`tag_0 IS NOT NULL` (100%)	1,000,000	5.073 [4.489–5.125]	+25.1%	0.197	6.47
Tablet/Record	`tag_0 IN ("value_0", "value_1", "value_2", "value_3", "value_4")` (50%)	500,000	3.244 [3.125–3.400]	-20.0%	0.308	3.98
Tablet/Record	`tag_0 = "value_0"` (10%)	100,000	2.378 [2.058–2.465]	-41.4%	0.420	2.57
TsFile	Unset (default `true`)	1,000,000	0.887 [0.489–0.988]	baseline	1.127	0.09
TsFile	`tag_0 IS NOT NULL` (100%)	1,000,000	92.360 [92.098–92.664]	104.1×	0.011	79.69
TsFile	`tag_0 IN ("value_0", "value_1", "value_2", "value_3", "value_4")` (50%)	500,000	47.003 [46.898–47.107]	53.0×	0.021	39.65
TsFile	`tag_0 = "value_0"` (10%)	100,000	10.480 [10.200–10.775]	11.8×	0.095	8.55

TsFile TAG predicate	Delivered files	Delivered bytes (MiB)
Unset (default `true`)	1	1.775
`tag_0 IS NOT NULL` (100%)	900	2.995
`tag_0 IN ("value_0", "value_1", "value_2", "value_3", "value_4")` (50%)	452	1.486
`tag_0 = "value_0"` (10%)	94	0.310

All 32 runs (including warm-ups) passed TAG-value, per-field row completeness, and duplicate-cell verification. Expected outputs were 1,000,000, 1,000,000, 500,000, and 100,000 logical rows with all 10 fields. Column-split Tablet/TsFile payloads were verified by device, timestamp, and field. TsFile contents were read after the measured delivery interval.

Tablet/Record: the 100%-matching predicate increased consumption time by 25.1%. Selecting 50% and 10% of rows reduced total time by 20.0% and 41.4%, respectively.
TsFile: nontrivial TAG predicates require parsing/filtering/rebuilding instead of forwarding the original file. The 100%-matching predicate produced 900 files and took 104.1× the baseline time. Even the 10%-matching predicate produced 94 files and took 11.8× the baseline. The resulting file fragmentation and delivery overhead are a substantial optimization target.
These results describe this flushed snapshot workload and do not measure incremental/consensus subscription throughput.

This PR has:

been self-reviewed.
- concurrent read
- concurrent write
- concurrent read and write
added comments explaining the intent where it would not be obvious to an unfamiliar reader.
added unit tests or modified existing tests to cover new code paths.
added integration tests.

Key changed/added classes (or packages if there are too many classes) in this PR

org.apache.iotdb.db.subscription.tagfilter
SubscriptionFilterSnapshot
SubscriptionTreeViewProjector
SubscriptionBrokerAgent
ConsensusLogToTabletConverter
PipeTabletEventTsFileBatch
ShowTopicsTask
TopicConfig and TopicMeta

jt2594838 · 2026-10-08T10:19:46Z

+  @Test
+  public void testFilteredTabletBatchIsAckedWithoutReturningEmptyMessage()
+      throws SubscriptionException {
+    final TestPullConsumer consumer = new TestPullConsumer(1, 0, true);
+    try {
+      consumer.open();
+      consumer.subscribeTopic();
+
+      final List<SubscriptionMessage> messages = consumer.pollForTest(1_000L);
+
+      Assert.assertTrue(messages.isEmpty());
+      Assert.assertEquals(1, consumer.getAcknowledgedCommitContexts().size());
+      Assert.assertEquals(
+          new SubscriptionCommitContext(1, 0, TOPIC, CONSUMER_GROUP_ID, 0L),
+          consumer.getAcknowledgedCommitContexts().get(0));


Why do you call the test WithoutReturningEmptyMessage but the messages are empty?

Renamed the test to testFullyFilteredTabletBatchIsAcknowledgedAndPollReturnsNoMessages. The expected result is an empty poll result, rather than a SubscriptionMessage carrying an empty Tablet payload; the filtered batch is still acknowledged.

Commit: 7caec10.

jt2594838 · 2026-10-08T10:20:42Z

+  public static final String
+      EXCEPTION_FAILED_TO_CREATE_OR_ALTER_TOPIC_DUPLICATE_ARG_ATTRIBUTES_ARE_NOT_ALLOWED_27315578 =
+          "\u521b\u5efa\u6216\u4fee\u6539 Topic \u5931\u8d25\uff0c\u4e0d\u5141\u8bb8\u91cd\u590d\u7684 %s \u5c5e\u6027";
+  public static final String
+      EXCEPTION_FAILED_TO_CREATE_OR_ALTER_TOPIC_ARG_IS_ONLY_SUPPORTED_FOR_TABLE_TOPICS_A5126607 =
+          "\u521b\u5efa\u6216\u4fee\u6539 Topic \u5931\u8d25\uff0c%s \u4ec5\u652f\u6301\u8868\u6a21\u578b Topic";
+  public static final String
+      EXCEPTION_FAILED_TO_CREATE_OR_ALTER_TOPIC_ARG_SHOULD_NOT_BE_EMPTY_767B1148 =
+          "\u521b\u5efa\u6216\u4fee\u6539 Topic \u5931\u8d25\uff0c%s \u4e0d\u5e94\u4e3a\u7a7a";


Replaced the Unicode escapes with readable UTF-8 Chinese text for all three messages, preserving the format placeholders.

Commit: 7caec10.

jt2594838 · 2026-10-08T10:34:46Z

-          retainedAlignedFlags.add(insertNodeTabletInsertionEvent.isAligned(i));
+        final TabletTransformResult transformedTablet =
+            transformTablet(
+                isTableModel ? insertNodeTabletInsertionEvent.getTableModelDatabaseName() : null,


Is it possible to let getTableModelDatabaseName return null when it is not a table model event?

Applied: PipeInsertionEvent.getTableModelDatabaseName() now returns null for tree-model events. Both transformation call sites use the getter directly. Added coverage for the database name passed for both event models.

Commit: 7caec10.

jt2594838 · 2026-10-08T10:36:10Z

+          calculateTabletsSizeInBytes(
+              retainedTablets.stream()
+                  .map(TabletTransformResult::getTablet)
+                  .collect(java.util.stream.Collectors.toList())));


mind the import

Added the Collectors import and removed the fully qualified reference.

Commit: 7caec10.

jt2594838 · 2026-10-08T10:38:05Z

+                isTableModel ? insertNodeTabletInsertionEvent.getTableModelDatabaseName() : null,
+                tablet,
+                isTableModel,
+                !isTableModel && insertNodeTabletInsertionEvent.isAligned(i));


Why is ' isAligned ' false when isTableModel is true?

Table-model data is aligned. Changed both transformation inputs and projected Tree View table results to report true; tree-model inputs preserve their original alignment. Added coverage for both models and alignment values.

Commit: 7caec10.

jt2594838 · 2026-10-12T06:23:43Z

+      throw new TagFilterEvaluationException(
+          DataNodeMiscMessages.EXCEPTION_TABLET_TIMESTAMPS_ARE_INCOMPLETE_24F8CE6F);
+    }
+    final org.apache.tsfile.utils.BitMap[] bitMaps = tablet.getBitMaps();


Mind the import

Added the BitMap import and removed the fully qualified references. The array-type validation helpers, including the qualified LocalDate reference, were removed with the redundant structural validation.

Commit: 7caec10.

jt2594838 · 2026-10-12T06:26:06Z

+    for (int i = 0; i < schemas.size(); i++) {
+      final IMeasurementSchema schema = schemas.get(i);
+      if (schema == null || schema.getMeasurementName() == null || schema.getType() == null) {
+        throw TagFilterEvaluationException.schemaBinding(
+            DataNodeMiscMessages.EXCEPTION_TABLET_MEASUREMENT_SCHEMA_IS_INCOMPLETE_6A813472);
+      }
+      if (values[i] == null) {
+        // A column whose every row is NULL may legitimately omit its value array and represent
+        // the values solely through a bitmap.  It still has a complete schema and must remain
+        // distinguishable from a missing or malformed column.
+        if (!isEntirelyNull(bitMaps, i, tablet.getRowSize())) {
+          throw new TagFilterEvaluationException(
+              DataNodeMiscMessages.EXCEPTION_TABLET_VALUE_COLUMN_IS_INCOMPLETE_845721FE);
+        }
+        continue;
+      }
+      if (!hasArrayLength(values[i], tablet.getRowSize())) {
+        throw new TagFilterEvaluationException(
+            DataNodeMiscMessages.EXCEPTION_TABLET_VALUE_COLUMN_IS_INCOMPLETE_845721FE);
+      }
+      if (categories.get(i) == null) {
+        throw TagFilterEvaluationException.schemaBinding(
+            DataNodeMiscMessages.EXCEPTION_TABLET_COLUMN_CATEGORY_IS_MISSING_A812B500);
+      }
+    }


Too many validations in this class.
The tablet coming from upper streams should have already been validated.
Check the necessity of the checks here and do not let unnecessary checks slow down and complicate the process.

Removed the full Tablet structural scan, value-array length/type checks, per-cell null validation, and repeated schema/category validation. Upstream conversion supplies the structural guarantees. The filter retains predicate-specific checks for missing bindings, missing referenced columns, and non-TAG columns. It also uses primitive row indexes and copies typed arrays directly during compaction.

Commit: 7caec10.

jt2594838 · 2026-10-12T06:32:47Z

+        }
+        hasUnknown |= result == TruthValue.UNKNOWN;
+      }


Break on first unknown?

We need to continue after UNKNOWN to preserve SQL three-valued logic: UNKNOWN AND FALSE is FALSE, and UNKNOWN OR TRUE is TRUE. In particular, NOT (UNKNOWN AND FALSE) must match. Kept short-circuiting on FALSE for AND and TRUE for OR, documented why UNKNOWN is not decisive, and added regression assertions for both operand orders and NOT. These assertions also caught and fixed the shared parser's false rejection of valid NOT (...) expressions.

Commit: 7caec10.

jt2594838 · 2026-10-14T09:44:07Z

+
+  private static TableBinding bindTable(
+      final List<Identifier> referencedFields, final TsTable table) {
+    final java.util.Set<String> tagNames = new java.util.HashSet<>();


Mind the import

Added normal imports for Set, HashSet, and Locale, and removed their fully qualified references throughout the class.

Commit: 7caec10.

jt2594838 · 2026-10-14T09:51:48Z

+  private static String canonicalizeTopicAttributeKey(final String key) {
+    if (Objects.isNull(key)) {
+      return null;
+    }
+    final String normalizedKey = key.trim().toLowerCase(Locale.ROOT);
+    switch (normalizedKey) {
+      case TopicConstant.PATH_KEY:
+        return TopicConstant.PATH_KEY;
+      case TopicConstant.PATTERN_KEY:
+        return TopicConstant.PATTERN_KEY;
+      case TopicConstant.DATABASE_KEY:
+        return TopicConstant.DATABASE_KEY;
+      case TopicConstant.TABLE_KEY:
+        return TopicConstant.TABLE_KEY;
+      case TopicConstant.COLUMN_FILTER_KEY:
+        return TopicConstant.COLUMN_FILTER_KEY;
+      case TopicConstant.TAG_FILTER_KEY:
+        return TopicConstant.TAG_FILTER_KEY;
+      case TopicConstant.RETENTION_BYTES_KEY:
+        return TopicConstant.RETENTION_BYTES_KEY;
+      case TopicConstant.RETENTION_MS_KEY:
+        return TopicConstant.RETENTION_MS_KEY;
+      case TopicConstant.START_TIME_KEY:
+        return TopicConstant.START_TIME_KEY;
+      case TopicConstant.END_TIME_KEY:
+        return TopicConstant.END_TIME_KEY;


What is this for?

This canonicalizes case-insensitive built-in topic keys before creation, ALTER merging, and deserialization. For example, ALTER with TAG-FILTER must replace an existing tag-filter rather than leave two values that different getters could read inconsistently. Replaced the switch with the shared TopicProperty registry, documented the purpose, and added tests for mixed-case ALTER and both deserialization paths. Extension keys and predicate values retain their original case.

Commit: 7caec10.

Preserve subscription configuration tests and bilingual message additions. Reuse the existing table-cache version API and adapt subscription tests to topic metadata and model-specific lookups.

Caideyipi added 2 commits September 15, 2026 15:53

Subscription: support table-model tag filtering

32d7822

Subscription: expose tag filter diagnostics in SHOW TOPICS

5e9f4c0

jt2594838 reviewed Oct 14, 2026

View reviewed changes

Caideyipi added 3 commits October 15, 2026 16:18

Merge origin/master into feat/subscription-tag-filter

eafa679

Preserve subscription configuration tests and bilingual message additions. Reuse the existing table-cache version API and adapt subscription tests to topic metadata and model-specific lookups.

Merge latest origin/master into feat/subscription-tag-filter

1c3e476

Address review feedback for table subscription TAG filters

7caec10

Conversation

Caideyipi commented Sep 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Filter semantics

Delivery paths and configuration changes

Validation

Performance comparison

Key changed/added classes (or packages if there are too many classes) in this PR

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Caideyipi commented Sep 15, 2026 •

edited

Loading