[yujun777]

What problem does this PR solve?

Issue Number: N/A

Related PR: #68646

Trace issue: #65418

Problem Summary:

A refresh of one MV partition reads each base table through the key range of that MV partition, while the snapshot it records next to the read names the base partitions the MV partition's mapping keeps. The two sets are not the same one: the mapping is what MTMV.calculatePartitionMappings answers, and partition_sync_limit filters it, so a refresh reads base partitions the snapshot does not describe -- and the rows they put in the MV partition stay there unaccounted for.

That difference is what a silent base partition change hides behind. DROP PARTITION, TRUNCATE PARTITION, REPLACE PARTITION and RECOVER PARTITION remove visible rows through metadata rather than through row binlog entries, so a refresh has to be told about them. An IVM MV is: its per-partition baseline is invalidated, which reaches every MV partition that reads the changed one. A plain MV is judged by its snapshot, and when the changed partition is one the snapshot never named, the base partition set is back to what the snapshot does name once the change is done: the MV partition is judged synchronized, the transparent rewrite serves the rows of a partition the base table no longer has, and no refresh plans that partition again.

This PR makes the read and the record the same set. MTMVTask#mappedBasePartitions decides which base partitions a batch may read, from the same mapping the snapshots taken next to that read are generated from, and UpdateMvByPartitionCommand builds each base table's read predicate from exactly those partitions:

• A base table the mapping gives no partition of is read as nothing rather than in full, which is the reading the IVM rewrite already gives such a scope.
• A table the caller does not scope keeps the MV partition's own key range, which is what an external base table and the full refresh of an IVM MV use.
• The predicate is built from the base partition's key at the position the MV's partition column has in that base table: a partition of a list partitioned base table holds one key per partition column, and the MV's column is not necessarily the first of them.
• A scoped range table keeps the MV partition's own key range alongside the mapped partitions' bounds, and both sides are written at the base partition column's type: an MV partition's key carries only its primitive type, so the bound of one whose bounds come from the base partitions would be written as the bare second and the rows of the last fraction of the partition -- which this refresh is to write -- would be left out of it, silently, with the snapshot recording the partition all the same. A base range partition is not always inside the MV partition it is rolled up into -- the roll-up keeps one whose upper bound the last second of belongs to the next bucket -- so without that range the read reaches a row no partition of the batch can take: the sink filters it and, with enable_insert_strict on by default, the refresh fails where it used to succeed. A scoped list table keeps the whole-key pin alone, since a list partition's key is a value the MV partition's key set was written from.
• The partitions are named by BaseTableInfo, so a table is identified by the table it names rather than by which of the MV's partition info and the mapping's own two objects it was read from. Keying by the object would have made "the lookup missed" read as "this table may not be read at all", which would empty the MV quietly.
• A table whose partitions include a list partition's default one is neither scoped nor windowed. The default partition takes the rows no other partition of the table claims, and ADD PARTITION claims a key without moving the rows the default partition already holds under it, so which side of a window a row is on is not something a predicate on the partition columns can say. The MV partition keeps the whole of what its key range covers and is recorded with every partition of it, plus the default one. Leaving the window is also what makes such a table's partitions meet at the MV's column -- one partition holding several keys of it, one of them shared with another partition -- so MTMVRelatedPartitionDescTransferGenerator merges descs whose keys meet into the one partition whose keys are their union, which is the partition the windowed pipeline already produced for the shape; an MV cannot have two partitions repeating a key.

Only the olap base tables of a non-IVM MV are scoped: the changes this answers are olap DDL, and an IVM MV already invalidates its baseline for them and scopes the reads of its delta in the IVM rewrite.

With the read equal to the record, a base partition that was read is one the snapshot names, so a later silent change to it is what the sync check compares against, and the partition is planned for a rebuild instead of being served stale.

The read is pinned where the predicates are written. The union compensation writes its whole-key predicate into a plan that is already bound and that nothing binds again, so it builds it from the slots that plan's relation reads; a partition column the relation does not read has no column to write the key against, and that case refuses the rewrite rather than narrowing it to a guess. The same path asks for a NULL list key as IS NULL, through the conversion the scoped read uses, instead of k IN (NULL), which is UNKNOWN even for a NULL row.

EXPLAIN REFRESH ... COMPLETE builds its scope from the same mapping, through the helper the refresh also uses (MTMVPartitionUtil#mappedBasePartitions), so it describes the read the refresh makes rather than the MV partitions' own key ranges.

Two places decide whether an MV may answer a query at all, and both are now closed against an MV that stands on less than the query reads:

• A partition inside its grace period is answered usable without being compared, but that says nothing about the base partitions the query reads. The coverage is read before the grace period answers, so a base partition the query reads that no MV partition is mapped from takes the MV out of the running -- the grace period used to answer first and the query was rewritten to the MV, dropping that partition's rows. That is for the MV-only rewrite: with the union rewrite on, such a partition is taken out of the MV plan and read from the base table, so it is passed over rather than taking the whole MV out of the running.
• With enable_materialized_view_union_rewrite off there is no compensation for the partitions that are not valid, so a non-empty subset of them was enough for the MV to answer. It now answers only when every MV partition the query's base partitions are mapped from is valid.

Two readings that were taken twice are now taken once. The sync check compares exactly the MV partitions grace does not answer for, so the snapshots prepared for it are those of that same set -- the one grace is decided on, once and after the version read, rather than a second reading of the grace window. The read is not free to reorder it: on cloud it writes each partition's cached mtime back (0 where the meta service has none), so a partition whose timestamp was inside the window when it was selected can be outside it once the read is done and grace does not answer for it after all -- the second reading had already left it out of the preload, and the comparison then asked for a snapshot nobody had prepared. A partition inside its window that no refresh has written is in that same position, for the same reason. The check reports the unprepared snapshot as "not synchronized", which drops the partition and, with every partition in that state, the MV. And the check that a batch's MV partitions are described by some base partition is taken for every batch: an external table does not make the olap side of a mixed batch any less scoped, so its presence is no reason to skip the check.

Behaviour changed: Yes

• With partition_sync_limit, an MV partition no longer keeps the rows of base partitions the window no longer keeps: the refresh reads the partitions the window keeps rather than the whole key range of the MV partition, which can be wider. Queries that also read a base partition the window has expired are answered from the base table by the union rewrite, which is what that path is for and is on by default (enable_materialized_view_rewrite).
• A base table that has a list partition's default partition is not windowed: its MV partition keeps every base partition its key range covers and is recorded with them, so a row whose key a later ADD PARTITION claimed stays in the MV partition it was read into, and a change to any partition of that range reaches it.
• A query against a recently refreshed MV partition now reads the partition mapping, where the grace period used to answer without it.
• With enable_materialized_view_union_rewrite off, a query whose MV partitions are only partly valid is answered from the base table instead of reading the partitions that are not valid as they are.
• A materialized view partitioned on a base table whose list partitions include a default partition is no longer used by transparent rewrite: a key only that partition holds has no MV partition to be read into, so the view cannot be a complete answer for a query that may read it. The view is otherwise unaffected.
• An aggregate query that the union compensation would answer is now judged by its grouping. The compensation unions two finished aggregates -- what the view answers for the partitions it keeps and what the base table answers for the rest -- so it is right while no group is answerable by both, and only a grouping that takes every column the view is partitioned on by itself keeps them apart: the view's partitions are disjoint in that tuple. A grouping that does not is answered from the base table instead, which gives the same rows: by an expression over a partition column (date_trunc(d, 'year'), to_date(d), which put several partitions' values in one group), by a ROLLUP/CUBE/GROUPING SETS set that leaves a partition column out (the () total puts every partition value in one group), or over a view partitioned on several columns whose grouping holds fewer than all of them (two of its partitions can share one value of one of them).
• For the same reason, an aggregate query whose base branch would read a base partition the window left out beside an MV partition recorded with keys meeting it -- a merged LIST component holds several keys of the view's partition column -- is answered from the base table rather than by a union that would answer that group twice. A base partition whose keys meet no retained MV partition's, which is every RANGE case, keeps the union rewrite.

Release note

With partition_sync_limit, a materialized view partition no longer keeps the rows of base partitions the sync window no longer keeps. A refresh reads exactly the base partitions the view's partition is recorded with, so what the partition holds and what it is reported as synchronized with agree; a query that also reads expired base partitions is answered from the base table by the union rewrite, as before. A base table that has a list partition's default partition is not windowed: the MV partition keeps every base partition its key range covers, so nothing read into it is left unrecorded. With enable_materialized_view_union_rewrite disabled, a view no longer answers a query when part of the MV partitions the query's base partitions are mapped from are not valid. A materialized view over a base table whose list partitions include a default partition is no longer used to answer queries by transparent rewrite (those queries read the base table); the view itself is created, refreshed and queried directly as before. An aggregate query grouped by an expression over the view's partition column, or by a grouping set that leaves it out, is read from the base table instead of by the union compensation; its rows are unchanged.

Check List (For Author)

• Test: Unit Test / Regression test
  • FE unit tests: UpdateMvByPartitionCommandTest (a scoped table is read from the partitions named, a table named with none is read as nothing, a partition predicate follows the key at the position the MV's partition column has, the read of a scoped range table ends at the MV partition's own upper bound, on the case of a DATETIME(3) partition ending at 2025-01-01 00:00:00.123 under a year MV, whose row at 2025-01-01 00:00:00.100 no partition of the batch can take, and a compensation filter is written with the relation's own slots), MTMVRewriteUtilTest (a partition in grace does not answer for a base partition the query reads that no MV partition is mapped from -- and the partition grace would have admitted is the one the coverage has to reject, so the case now gives it a written version and reaches grace at all; without the union rewrite every mapped partition has to be valid; an MV over a table with a default list partition offers no partition at all; a partition inside its window that no refresh has written has the snapshots prepared for the comparison the sync check then makes -- the case the second reading of the grace window had left out; and a partition the version read itself moves out of the window is compared with the snapshots of the same set), MTMVRelatedPartitionDescGeneratorTest (descs whose keys meet are one MV partition holding all of their keys -- a chain of three, and two tables of a multi-table MV -- naming each table's partitions; a key set is the same desc whether it was merged or not -- including two keys that hash alike (Aa, BB) written down in either order -- so ADD PARTITION over a covered key cannot re-derive the MV partition; a pruned query keeps the merged desc; disjoint ones keep their own), MTMVRelatedPartitionDescSyncLimitGeneratorTest (a table with a default list partition is kept out of the window, a range table is not), MTMVTaskTest, MTMVTest, MTMVPartitionUtilTest, ExplainRefreshMTMVCommandTest -- all pass. Each case was checked to fail with the change it covers reverted.
  • Regression: test_mtmv_base_partition_read_scope (daily base partitions with a year MV partition and a partition_sync_limit window; a list partitioned base table whose partitions hold several keys of the MV's partition column; a default-partition table under a window, where a row the default partition holds under a key a later ADD PARTITION claimed stays in the MV partition, which is recorded with that partition; a default partition whose rows are read for the MV partition they belong to; a two-table join whose second table has a default partition; a key written with milliseconds; a key only a default partition holds, whose MV holds it not at all and whose query is answered whole from the base table; and the aggregate compensation's own declinations -- a group split by an uncovered partition whose keys meet a retained MV partition's, a grouping by an expression over the partition column, a grouping set that leaves it out, a stale partition re-read through the base branch beside an expression grouping, and a key that met only a partition the staleness pass had already scheduled for removal, which stays union-rewritten), the mtmv_p0 suites around list partitions, roll-up, partition limits, recreate and column changes, and mtmv_p0/ivm's two partition_sync_limit suites -- all pass, and the suites' other cases keep their goldens. test_mtmv_base_partition_read_scope carries the read's own boundary as well: a DATETIME(3) range partition of [2024-01-01 00:00:00.123, 2025-01-01 00:00:00.123) under a year-granularity MV with a row at 2025-01-01 00:00:00.100 -- measured both ways, the refresh task fails with Insert has filtered data in strict mode / no partition for this tuple for that row when the MV partition's own range is left out of the read. A second case, range_ms_direct, puts a direct range MV over that partition: it reads the same partition with the MV's own bound written at the base column's scale, and measured against a build that writes that bound from the partition key the refresh succeeds and the row at 2024-02-01 00:00:00.100 is not in the MV at all -- the golden names it. nereids_rules_p0/mv/union_rewrite_grace_big failed on a cloud cluster before the snapshots fix (Partition snapshot was not prepared out of the sync check, which took the MV out of the rewrite) and passes after it; nereids_rules_p0/mv/partition_union_rewrite/partition_mv_rewrite and mtmv_p0/test_iceberg_mtmv -- the refresh of a view over an external table, the shape the undescribed-partition check is about -- pass as well. In nereids_rules_p0/mv/union_all_compensate, query5_0 groups a daily-range view by to_date(data_date), the shape the compensation now declines, so its expectation moved from mv_rewrite_success to mv_rewrite_fail: the query is answered from the base table and its golden rows are unchanged.
• Behavior changed: Yes (see above)
• Does this need documentation: Yes, in a follow-up (no doc PR yet): the pages for partition_sync_limit and for the incremental materialized view's refresh modes are tracked in the design document-change list, which records the exact locations and wording.

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[yujun777]

run buildall

[yujun777]

/review

[github-actions]

Static review complete for PR #68648 at head 908d784ad5076034117789eb966f653ae7b7f09a after two review rounds, an independent risk scan, and a final sweep of all seven changed files. I request changes for one correctness finding and include one test-coverage finding. There are no existing P0/P1 inline findings on this head.

• Goal and data correctness: The new non-IVM path derives base read predicates from the same mapping used for refresh snapshots. It works for the covered single-column range and empty-scope cases, but a valid multi-column LIST partition layout still reads an unmapped base partition because the predicate uses only the projected MV partition column (P1). A later partition drop can leave a stale row in an MV partition that is judged synchronized.
• Scope and parallel paths: The change is localized to refresh task planning, command predicate construction, EXPLAIN wiring, and tests. BaseTableInfo preserves table identity across resolved objects. IVM refresh deliberately passes a null scope; EXPLAIN retains the MV key-range approximation; external PCT tables are unscoped. I found no separate defect in those paths.
• Concurrency, lifecycle, and errors: A batch's map and snapshots come from one refresh context and the map is retained through retries. No new thread, lock order, or static lifecycle was introduced. If a mapped base partition disappears before live item lookup, the refresh fails explicitly under stale metadata; I did not treat that as a correctness defect.
• Configuration, compatibility, and writes: This adds no configuration key, persisted format, FE/BE field, or rolling-upgrade interface. It uses the existing partition_sync_limit. Insert overwrite still targets the selected MV partitions and publishes task snapshots after a successful batch; the P1 case is a mismatch between what the scan reads and those recorded snapshots. No separate EditLog or crash-path regression was substantiated.
• Tests, observability, and performance: The FE tests exercise range scope, empty scope, and multi-key LIST key position. The new regression checks base rows and MV counts, but on January 2-5 it passes even with the old predicate (P2), and it does not cover the P1 overlap. Existing logs and failure propagation remain in place; no new metric or independently substantiated performance defect was found. This was static inspection only: the review contract prohibited builds, tests, and source edits.

The user supplied no additional focus points. All candidates were accepted, dismissed with code evidence, or deduplicated; the second round returned NO_NEW_VALUABLE_FINDINGS from every required reviewer.

[github-actions]

[P1] Restrict the scan to the mapped base partitions, not just their projected partition-column values. For a valid LIST(d, region) table, let expired p_old contain (old, 'US') and retained p_kept contain (old, 'EU') and (new, 'EU'). With partition_sync_limit=2 and an MV partitioned by d, the mapping and snapshot name only p_kept, but this predicate becomes d IN (old, new) and still reads the p_old row. Dropping p_old later leaves that row in the MV while the snapshot comparison sees only unchanged p_kept, so rewrite can serve stale data. The old predicate had this hole too; the new scope does not close it. Use exact base partition IDs or full LIST tuples for the scoped read, and cover this overlap in a regression test.

[yujun777]

Fixed in 43be1b3. The scope is now built from the partitions themselves rather than from the value they take at one column: a list partition is pinned to the whole of each key it holds (each key a conjunction, IS NULL for a value a row does not have), and a range partition to its bounds, which is the same thing since a range partitioned base table has a single partition column.

The LIST(d, region) shape you describe is a regression case now. With the change the MV holds only the kept partition's rows; with the predicate reverted to the projected one it reads the expired partition's row, measured as RealRow: [2020-01-01, US, 1] against a golden that expects only the EU rows.

One shape is deliberately left as it was: a list partitioned table's default partition, which holds no key at all. What it holds cannot be said with a predicate on the partition columns, so it is read in full as before -- a read that can be seen to be too wide, rather than one that loses its rows quietly.

[github-actions]

[P2] Keep the expired and retained days in one MV year for every test run. On January 2-5, expiredDay falls in the previous year while both retained days are in the current year. The sync limit therefore creates only a current-year MV partition, whose old key-range predicate already excludes expiredDay; both assertions pass even if this PR's scoped-read change is reverted. Please use a fixture or controlled clock that always leaves an expired base partition inside a retained MV partition's year, so this regression consistently fails without the fix.

[yujun777]

Fixed in 43be1b3. You are right, and the fixture turned out to rest on an assumption of mine rather than on a measurement: while fixing it I measured the property, and partition_sync_limit=1 DAY expires nothing at all (six days of partitions all kept), while limit=2 DAY puts its edge at today-2. So both the number of retained days and the identity of the expired one were guesses.

The two sides are now asserted separately through order_qt -- the expired day must be missing from the MV, the days the window kept must be there -- so a total cannot let them cancel out, and the case is skipped with assumeTrue where the calendar does not leave the expired day inside a retained MV partition. Skipped rather than asserted, so that a run which exercises the case is a run that can fail without the change.

[hello-stephen]

TPC-H: Total hot run time: 27828 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 908d784ad5076034117789eb966f653ae7b7f09a, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17741	3955	3991	3955
q2	2192	364	297	297
q3	10105	1410	800	800
q4	4701	482	346	346
q5	7468	831	552	552
q6	178	167	134	134
q7	740	782	604	604
q8	9293	1326	1443	1326
q9	5441	4222	4190	4190
q10	6832	1327	1036	1036
q11	436	269	239	239
q12	634	411	294	294
q13	18097	2625	2000	2000
q14	261	265	234	234
q15	q16	731	722	660	660
q17	1620	1131	1000	1000
q18	6556	5585	5531	5531
q19	1307	1259	1099	1099
q20	488	396	257	257
q21	5703	3261	2959	2959
q22	450	371	315	315
Total cold run time: 100974 ms
Total hot run time: 27828 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4691	4939	4550	4550
q2	762	571	559	559
q3	4852	5155	4530	4530
q4	2185	2372	1429	1429
q5	4489	4362	4636	4362
q6	220	173	126	126
q7	1797	1723	1544	1544
q8	2366	2031	1956	1956
q9	7279	7215	7190	7190
q10	3662	3559	3081	3081
q11	517	368	344	344
q12	697	693	510	510
q13	2299	2576	1997	1997
q14	283	285	243	243
q15	q16	664	689	591	591
q17	7307	6709	6621	6621
q18	11867	10982	11795	10982
q19	1086	1006	1021	1006
q20	2194	2161	1878	1878
q21	5001	4073	4273	4073
q22	532	442	386	386
Total cold run time: 64750 ms
Total hot run time: 57958 ms

[hello-stephen]

TPC-DS: Total hot run time: 152080 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 908d784ad5076034117789eb966f653ae7b7f09a, data reload: false

query5	4329	630	491	491
query6	443	214	212	212
query7	4898	552	302	302
query8	324	183	166	166
query9	8815	4008	3958	3958
query10	453	325	254	254
query11	5846	3565	3234	3234
query12	147	97	88	88
query13	1282	611	413	413
query14	6521	4570	4253	4253
query14_1	3986	4117	4046	4046
query15	205	204	181	181
query16	978	441	425	425
query17	913	674	538	538
query18	2425	459	330	330
query19	199	177	142	142
query20	84	82	79	79
query21	217	134	117	117
query22	13006	13022	12767	12767
query23	13978	12996	12469	12469
query23_1	12387	12450	12580	12450
query24	7349	1137	645	645
query24_1	709	641	763	641
query25	563	443	377	377
query26	1278	308	172	172
query27	2700	588	338	338
query28	4596	1989	1973	1973
query29	1643	721	528	528
query30	298	222	189	189
query31	928	760	662	662
query32	153	98	98	98
query33	519	314	260	260
query34	1195	1100	610	610
query35	727	777	651	651
query36	805	759	735	735
query37	141	116	95	95
query38	1838	1769	1678	1678
query39	708	709	642	642
query39_1	660	676	644	644
query40	222	125	118	118
query41	77	69	69	69
query42	104	97	98	97
query43	349	344	305	305
query44	1383	695	726	695
query45	190	180	163	163
query46	1094	1228	725	725
query47	1461	1513	1431	1431
query48	391	379	301	301
query49	573	390	305	305
query50	950	349	249	249
query51	10547	10488	10387	10387
query52	90	89	76	76
query53	238	259	183	183
query54	252	196	203	196
query55	81	72	67	67
query56	239	221	205	205
query57	1469	1470	1150	1150
query58	296	243	250	243
query59	1990	2073	1872	1872
query60	270	238	223	223
query61	150	150	145	145
query62	404	321	262	262
query63	218	173	182	173
query64	2789	993	802	802
query65	3474	3432	3410	3410
query66	1782	429	301	301
query67	19753	20182	19984	19984
query68	3169	1565	900	900
query69	416	303	262	262
query70	902	817	789	789
query71	292	231	215	215
query72	2637	2536	1821	1821
query73	802	761	411	411
query74	4626	4495	4287	4287
query75	2294	2271	1930	1930
query76	2321	1094	711	711
query77	362	391	301	301
query78	9027	9016	8519	8519
query79	1330	1214	769	769
query80	585	449	374	374
query81	538	322	279	279
query82	641	161	122	122
query83	301	223	207	207
query84	325	144	114	114
query85	902	456	394	394
query86	323	251	236	236
query87	2006	1991	1835	1835
query88	3683	2743	2697	2697
query89	375	287	249	249
query90	1911	192	188	188
query91	168	154	126	126
query92	103	91	88	88
query93	1475	1445	932	932
query94	536	355	304	304
query95	682	472	336	336
query96	1044	781	339	339
query97	2426	2424	2329	2329
query98	156	152	147	147
query99	719	726	622	622
Total cold run time: 235837 ms
Total hot run time: 152080 ms

[hello-stephen]

ClickBench: Total hot run time: 23.92 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 908d784ad5076034117789eb966f653ae7b7f09a, data reload: false

query1	0.01	0.01	0.01
query2	0.09	0.05	0.05
query3	0.26	0.13	0.12
query4	1.61	0.14	0.14
query5	0.24	0.22	0.23
query6	1.15	0.93	0.93
query7	0.04	0.00	0.00
query8	0.06	0.04	0.04
query9	0.39	0.34	0.35
query10	0.54	0.58	0.58
query11	0.20	0.15	0.14
query12	0.18	0.14	0.14
query13	0.47	0.46	0.46
query14	0.95	0.95	0.95
query15	0.63	0.59	0.58
query16	0.32	0.32	0.32
query17	1.08	1.09	1.09
query18	0.21	0.19	0.20
query19	2.00	1.90	1.97
query20	0.01	0.01	0.01
query21	15.48	0.21	0.15
query22	4.84	0.05	0.05
query23	16.16	0.32	0.12
query24	2.95	0.41	0.32
query25	0.11	0.05	0.04
query26	0.73	0.21	0.15
query27	0.05	0.03	0.04
query28	3.59	0.77	0.32
query29	12.48	4.06	3.21
query30	0.28	0.15	0.15
query31	2.76	0.55	0.32
query32	3.24	0.59	0.48
query33	3.24	3.24	3.15
query34	15.47	3.90	3.27
query35	3.22	3.20	3.22
query36	0.54	0.44	0.41
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.04	0.04
query40	0.18	0.15	0.14
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.04
Total cold run time: 96.07 s
Total hot run time: 23.92 s

[hello-stephen]

FE Regression Coverage Report

Increment line coverage 100.00% (47/47) 🎉
Increment coverage report
Complete coverage report

[yujun777]

run buildall

[yujun777]

/review

[github-actions]

Static review complete for PR #68648 at head 43be1b3. Two full-review rounds, separate risk scans, and a final sweep covered all seven changed files. I request changes for two P1 correctness findings and include one P2 regression finding. This is static inspection only; the review contract prohibited builds, tests, and source edits.

Critical checkpoints:

• Goal and tests: The change aims to read the same OLAP base partitions that a non-IVM MV refresh records in its snapshots. Ordinary range and full LIST tuple cases follow that mapping, and the new tests cover selected RANGE partitions, an empty scope, and one ordinary LIST overlap. The goal is not met for fractional DATETIMEV2 keys or default LIST partitions (P1 comments); no added test covers either.

• Scope and clarity: The implementation is localized to refresh planning, predicate construction, EXPLAIN wiring, and tests. BaseTableInfo avoids an object-identity mismatch. The default-partition comment assumes an empty key list that Doris does not produce for normal defaults.

• Concurrency: Refresh jobs and base DDL can overlap. The batch read set and its snapshots use one MTMVRefreshContext mapping, the same set is retained across retries, and missing mapped names fail rather than silently widening a scan. No new thread, lock-order, or independently substantiated race defect was found.

• Lifecycle: COMPLETE and PARTITIONS non-IVM attempts reach the per-batch scope; overwrite completion precedes snapshot publication. I found no new static-initialization, ownership, or termination issue.

• Configuration: No configuration item was added. The change uses the existing partition_sync_limit; no new dynamic-update behavior is required.

• Compatibility: No persisted format, function symbol, or FE/BE protocol field changed. Existing valid fractional partition keys and default LIST layouts do regress under the new predicates (P1 comments). No separate rolling-upgrade defect was found.

• Parallel paths: IVM deliberately keeps its stream and partition-ID selection, EXPLAIN COMPLETE keeps an approximate MV-range plan without an execution context, and external PCT tables remain unscoped. No additional parallel-path omission was substantiated.

• Conditions and errors: An explicitly empty readable set becomes FALSE, which matches its snapshot set. NULL tuple members use IS NULL. The default LIST fallback checks for an empty key list, but normal defaults contain a synthetic MIN key (P1). Missing partition lookups fail explicitly.

• Test coverage and results: The golden output matches the fixture's intended ordered results, but the new date guard throws a JUnit assumption that this Groovy runner counts as a failed suite on January 1-5 (P2), and it prevents the independent LIST case from running then. Fractional and default LIST/RANGE refresh cases need coverage. No tests were run in this review; author and CI statements were not treated as independent validation.

• Observability: Existing task logs and failure propagation remain. The two missing-row cases can be recorded as synchronized, so ordinary logs do not expose their incorrect contents; this is covered by the P1 findings. No separate metric or logging defect was substantiated.

• Transaction, persistence, and writes: Insert overwrite still targets the selected MV partitions and publishes snapshots after successful batches. No EditLog, commit, or crash-path change was added. The P1 defects instead make a successful overwrite persist an incomplete MV image beside a matching snapshot.

• FE/BE variable passing: No new cross-process variable or thrift field was introduced.

• Performance: Base-partition predicates can grow with the mapped partition count, but this review found no concrete performance failure or further actionable optimization defect.

• Other issues: No additional substantiated issue remains after candidate-by-candidate verification and the final changed-file sweep.

Existing review context: Inline P1 #4134420545's ordinary LIST(d,region) overlap is addressed by the full-tuple predicate and regression fixture on this head; its original scenario is not a still-applicable blocking comment. The new default-LIST underread is a distinct failure despite the author's reply expecting a broad default read. Inline P2 #4134420556's former false-positive assertion was replaced, while the newly added assumption produces the separate suite failure reported here. There are no still-applicable existing P0/P1 inline IDs.

User focus: No additional focus points were supplied. All candidate findings were accepted, dismissed with code evidence, or deduplicated; every required reviewer returned NO_NEW_VALUABLE_FINDINGS in round two.

[github-actions]

[P1] Keep default LIST rows reachable when scoping by full tuples. With LIST(k1,k2), an explicit partition for (1,2), and a default partition containing (1,3), an MV partitioned by k1 previously read both rows for key 1. This predicate now reads only (1,2). Doris represents the default partition with a synthetic MIN key, so keys.isEmpty() is never true; its separate MIN MV partition also cannot supply (1,3). The MV loses a committed row while its mapped snapshots can report synchronized. Handle default partition coverage in the mapping/read together, or reject this shape until it can be represented; add a regression with a default row sharing an explicit MV key.

[yujun777]

Fixed in 5bf185b, and you were right about the representation: a default partition's item is not empty, it holds the sentinel key the rows no other partition claims are placed by, so the keys.isEmpty() I had written to read it in full was dead code and the partition was read as none of its rows. Measured on the shape in your comment -- LIST(k1, k2) with an explicit (1, 2) and a default partition holding (1, 3), an MV partitioned by k1 -- the MV held only (1, 2) before the change and holds both after it.

The two cannot be handled apart, as you say: which rows of a default partition belong to an MV partition is the MV partition's own question, since it takes the rows whose key falls in its range wherever the base table placed them, and that partition is not one the MV partition is recorded with. So a base table that has one is read the way an unscoped one is -- the MV partition's own key range -- which is a reading that can be seen to be too wide rather than one that drops rows belonging to the partition being refreshed. A regression case pins it (list_default in the read scope suite), and it fails when that fallback is removed.

[github-actions]

[P1] Preserve DATETIMEV2 scale in scoped base-partition predicates. Type.fromPrimitiveType(DATETIMEV2) supplies scale 0, so this conversion rounds fractional partition keys before building both LIST tuple equalities and RANGE bounds. For example, a RANGE(ts DATETIMEV2(3)) partition ending at 2024-02-01 00:00:00.123 in a year-rollup MV gets a new ts < ...00.000 filter, omitting rows that belong to the mapped partition; a secondary LIST key at .123 is similarly compared to .000. The snapshot still records that partition, so the MV can appear synchronized while missing rows. Convert with the base partition column's full Type and cover fractional keys in refresh tests.

[yujun777]

Fixed in 5bf185b. The key is converted with its partition column's full Type now, so a value written with milliseconds is compared at that scale; the callers that have the column pass its type and the ones that do not keep the primitive one they were given before.

Measured on a fractional key rather than assumed: LIST(ts) with a partition of ('2024-02-01 00:00:00.123') and an MV partitioned by ts left that MV partition empty before the change -- the predicate had been rounded to .000 -- and holds its row after it. A regression case pins it (fractional_key), failing when the type is taken from the primitive again.

One thing your example led me to check: a RANGE(ts DATETIME(3)) partition ending at a fractional value cannot back a year rollup MV at all -- MTMVPartitionExprDateTrunc refuses it at CREATE with "partition values not equal" -- so the range bound is not a way to reach this; the list key is.

[github-actions]

[P2] Avoid a JUnit assumption as this suite's calendar guard. From January 1 through 5, expiredDay is in the prior year and assumeTrue throws, but the Groovy regression runner catches every Throwable in SuiteContext.start and records it through onSuiteFailed; this makes the suite fail rather than skip, and the independent LIST case never runs. Use a calendar-independent fixture or a supported conditional that leaves the LIST case runnable.

[yujun777]

Fixed in 5bf185b. You are right about the runner: SuiteContext#start catches every Throwable from the suite body and records it through onSuiteFailed, so the assumption was a failure of the whole suite -- and being at the top of it, it took the calendar independent cases down with it for five days a year.

The calendar dependent part is now guarded by a condition, so it is the part that is left out rather than the suite, and the rest runs either way (verified by forcing the guard off: the same suite passes with the other cases alone). The days themselves stay relative because what the case is about -- a base partition the sync window left out, inside a retained MV partition's range -- cannot be written with fixed dates; the fixed-date shapes that can be are the list partition cases, which are calendar independent and always run.

[hello-stephen]

TPC-H: Total hot run time: 27608 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 43be1b3ba23db1bf0aee4885827187c06d6454d7, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17894	3837	3837	3837
q2	2126	362	286	286
q3	10125	1324	769	769
q4	4678	473	354	354
q5	7460	805	539	539
q6	174	168	134	134
q7	754	774	606	606
q8	9368	1526	1600	1526
q9	5324	4160	4129	4129
q10	6817	1314	1012	1012
q11	431	273	243	243
q12	634	423	294	294
q13	18057	2600	1971	1971
q14	262	257	235	235
q15	q16	737	717	661	661
q17	1756	1114	992	992
q18	6494	5588	5516	5516
q19	1308	1256	1045	1045
q20	455	390	253	253
q21	5659	3066	2897	2897
q22	449	358	309	309
Total cold run time: 100962 ms
Total hot run time: 27608 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4610	4488	4418	4418
q2	715	545	534	534
q3	4782	5342	4566	4566
q4	2186	2337	1523	1523
q5	4648	4418	4588	4418
q6	231	176	128	128
q7	1781	1678	1482	1482
q8	2284	2074	1994	1994
q9	7377	6833	6784	6784
q10	3620	3531	3059	3059
q11	522	367	335	335
q12	697	700	488	488
q13	2276	2578	1976	1976
q14	263	270	249	249
q15	q16	652	678	597	597
q17	7255	6654	6613	6613
q18	11855	11066	11736	11066
q19	1058	967	982	967
q20	2192	2159	1901	1901
q21	4934	4061	4258	4061
q22	494	445	394	394
Total cold run time: 64432 ms
Total hot run time: 57553 ms

[hello-stephen]

TPC-DS: Total hot run time: 152644 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 43be1b3ba23db1bf0aee4885827187c06d6454d7, data reload: false

query5	4312	585	468	468
query6	425	232	191	191
query7	4817	512	314	314
query8	324	174	163	163
query9	8793	3998	4022	3998
query10	462	311	259	259
query11	5844	3540	3224	3224
query12	152	91	87	87
query13	1282	589	447	447
query14	6531	4539	4241	4241
query14_1	3965	3973	3956	3956
query15	213	200	180	180
query16	1015	464	433	433
query17	896	685	538	538
query18	2432	460	341	341
query19	198	186	140	140
query20	82	81	81	81
query21	219	135	113	113
query22	13083	12914	12857	12857
query23	13877	12871	12465	12465
query23_1	12445	12418	12527	12418
query24	7227	1114	691	691
query24_1	730	677	734	677
query25	525	414	346	346
query26	1245	301	162	162
query27	2726	557	347	347
query28	5121	1953	2006	1953
query29	1670	718	530	530
query30	293	223	192	192
query31	900	759	643	643
query32	144	90	95	90
query33	522	304	249	249
query34	1219	1165	629	629
query35	705	730	626	626
query36	810	823	712	712
query37	141	101	97	97
query38	1813	1758	1682	1682
query39	708	707	639	639
query39_1	664	673	645	645
query40	218	131	106	106
query41	70	63	62	62
query42	98	92	95	92
query43	335	343	299	299
query44	1374	694	699	694
query45	184	175	157	157
query46	1076	1185	709	709
query47	1471	1501	1365	1365
query48	396	386	290	290
query49	578	409	284	284
query50	967	344	247	247
query51	10569	10731	10716	10716
query52	90	88	82	82
query53	239	255	179	179
query54	257	197	199	197
query55	81	73	68	68
query56	226	208	213	208
query57	1440	1305	1365	1305
query58	290	267	259	259
query59	1979	2048	1868	1868
query60	276	244	236	236
query61	170	163	169	163
query62	404	325	269	269
query63	216	177	180	177
query64	2916	1107	891	891
query65	3464	3419	3406	3406
query66	1804	426	317	317
query67	20164	19934	19947	19934
query68	3286	1490	836	836
query69	426	313	271	271
query70	895	868	791	791
query71	290	231	200	200
query72	2635	2507	2247	2247
query73	832	713	438	438
query74	4608	4504	4286	4286
query75	2318	2256	1935	1935
query76	2310	1112	795	795
query77	353	400	313	313
query78	9068	9076	8375	8375
query79	1337	1168	742	742
query80	590	461	361	361
query81	549	325	283	283
query82	609	170	126	126
query83	315	224	195	195
query84	319	140	113	113
query85	857	455	375	375
query86	326	243	227	227
query87	2007	1969	1840	1840
query88	3636	2700	2710	2700
query89	368	283	247	247
query90	1916	179	177	177
query91	170	149	123	123
query92	101	89	82	82
query93	1538	1433	883	883
query94	548	338	298	298
query95	670	453	347	347
query96	1086	824	335	335
query97	2412	2414	2309	2309
query98	163	146	145	145
query99	729	734	606	606
Total cold run time: 236843 ms
Total hot run time: 152644 ms

[hello-stephen]

ClickBench: Total hot run time: 24.02 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 43be1b3ba23db1bf0aee4885827187c06d6454d7, data reload: false

query1	0.01	0.01	0.01
query2	0.09	0.05	0.05
query3	0.26	0.14	0.13
query4	1.61	0.14	0.14
query5	0.24	0.22	0.22
query6	1.15	0.94	0.93
query7	0.04	0.01	0.01
query8	0.06	0.04	0.04
query9	0.40	0.33	0.34
query10	0.54	0.55	0.59
query11	0.19	0.14	0.14
query12	0.18	0.14	0.15
query13	0.46	0.47	0.47
query14	0.95	0.95	0.95
query15	0.60	0.58	0.58
query16	0.30	0.32	0.34
query17	1.07	1.06	1.05
query18	0.21	0.20	0.20
query19	1.97	1.94	1.93
query20	0.02	0.02	0.01
query21	15.49	0.22	0.14
query22	4.80	0.05	0.05
query23	16.13	0.31	0.13
query24	2.96	0.43	0.32
query25	0.10	0.05	0.04
query26	0.74	0.22	0.14
query27	0.03	0.04	0.03
query28	3.58	0.85	0.35
query29	12.48	4.04	3.22
query30	0.28	0.15	0.16
query31	2.77	0.57	0.31
query32	3.22	0.60	0.49
query33	3.23	3.15	3.21
query34	15.56	3.97	3.30
query35	3.26	3.25	3.23
query36	0.57	0.43	0.41
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.18	0.15	0.14
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.03
Total cold run time: 96.08 s
Total hot run time: 24.02 s

[yujun777]

run buildall

[yujun777]

/review

[hello-stephen]

TPC-H: Total hot run time: 28179 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5bf185b273d40d859ff1ed0c4dd89bf15eae8c0f, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17904	4120	4117	4117
q2	2209	359	310	310
q3	10077	1650	807	807
q4	4686	478	342	342
q5	7486	831	565	565
q6	180	172	139	139
q7	758	801	604	604
q8	9317	1560	1499	1499
q9	5452	4203	4147	4147
q10	6831	1327	1030	1030
q11	440	270	237	237
q12	630	411	291	291
q13	18063	2611	1985	1985
q14	258	260	239	239
q15	q16	736	722	660	660
q17	1713	1113	1011	1011
q18	6554	5631	5589	5589
q19	1389	1229	1064	1064
q20	494	404	268	268
q21	5679	3437	2970	2970
q22	445	375	305	305
Total cold run time: 101301 ms
Total hot run time: 28179 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4875	4967	4832	4832
q2	788	605	567	567
q3	5279	5121	4676	4676
q4	2209	2343	1451	1451
q5	4649	4727	4502	4502
q6	226	172	134	134
q7	1880	1740	1538	1538
q8	2504	2147	2055	2055
q9	7294	7301	7310	7301
q10	3617	3587	3082	3082
q11	556	409	375	375
q12	694	702	498	498
q13	2276	2573	2003	2003
q14	275	287	244	244
q15	q16	661	688	622	622
q17	7525	6855	6735	6735
q18	11898	11104	11849	11104
q19	1075	978	1044	978
q20	2223	2217	1944	1944
q21	5099	4323	4376	4323
q22	513	461	413	413
Total cold run time: 66116 ms
Total hot run time: 59377 ms

[hello-stephen]

TPC-DS: Total hot run time: 152265 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5bf185b273d40d859ff1ed0c4dd89bf15eae8c0f, data reload: false

query5	4326	597	473	473
query6	443	201	194	194
query7	4825	531	292	292
query8	330	176	160	160
query9	8831	3983	4005	3983
query10	449	305	258	258
query11	5840	3534	3211	3211
query12	150	97	86	86
query13	1257	559	417	417
query14	6574	4566	4244	4244
query14_1	3934	3936	3959	3936
query15	204	193	184	184
query16	983	473	414	414
query17	911	671	532	532
query18	2438	467	339	339
query19	220	191	160	160
query20	86	83	80	80
query21	220	139	114	114
query22	13019	13032	12880	12880
query23	13952	12986	12469	12469
query23_1	12434	12485	12555	12485
query24	7268	1121	666	666
query24_1	730	728	676	676
query25	558	428	370	370
query26	1258	316	169	169
query27	2689	526	360	360
query28	4596	1990	1974	1974
query29	1683	698	526	526
query30	300	222	188	188
query31	890	754	643	643
query32	180	116	95	95
query33	518	321	237	237
query34	1225	1089	634	634
query35	714	745	648	648
query36	805	839	737	737
query37	143	108	94	94
query38	1860	1765	1716	1716
query39	704	686	663	663
query39_1	666	646	642	642
query40	226	125	100	100
query41	71	67	71	67
query42	106	99	91	91
query43	334	338	297	297
query44	1359	715	711	711
query45	191	187	163	163
query46	1086	1165	698	698
query47	1502	1516	1363	1363
query48	416	401	288	288
query49	586	411	295	295
query50	941	346	253	253
query51	10727	10464	10506	10464
query52	89	88	74	74
query53	239	250	177	177
query54	254	223	186	186
query55	76	74	68	68
query56	222	227	201	201
query57	1420	1409	1297	1297
query58	286	255	240	240
query59	1956	2059	1847	1847
query60	280	233	216	216
query61	145	146	149	146
query62	394	324	274	274
query63	215	172	177	172
query64	2786	991	842	842
query65	3471	3410	3394	3394
query66	1833	408	304	304
query67	19980	20134	20048	20048
query68	3269	1451	864	864
query69	398	310	253	253
query70	936	801	781	781
query71	306	236	212	212
query72	2619	2489	2218	2218
query73	824	746	426	426
query74	4602	4480	4252	4252
query75	2281	2284	1936	1936
query76	2328	1074	733	733
query77	350	387	292	292
query78	9145	8957	8372	8372
query79	1404	1100	750	750
query80	1242	448	367	367
query81	582	320	278	278
query82	1162	157	118	118
query83	293	224	193	193
query84	317	148	116	116
query85	899	463	375	375
query86	456	227	224	224
query87	1993	1966	1830	1830
query88	3630	2678	2667	2667
query89	366	285	244	244
query90	1935	178	175	175
query91	177	154	127	127
query92	102	84	80	80
query93	1518	1451	837	837
query94	728	323	294	294
query95	668	354	428	354
query96	1068	786	304	304
query97	2439	2459	2323	2323
query98	160	153	143	143
query99	725	733	632	632
Total cold run time: 237882 ms
Total hot run time: 152265 ms

[hello-stephen]

ClickBench: Total hot run time: 24.1 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5bf185b273d40d859ff1ed0c4dd89bf15eae8c0f, data reload: false

query1	0.01	0.01	0.01
query2	0.10	0.05	0.05
query3	0.27	0.14	0.13
query4	1.61	0.14	0.14
query5	0.25	0.23	0.23
query6	1.16	0.94	0.94
query7	0.05	0.01	0.01
query8	0.06	0.04	0.04
query9	0.40	0.35	0.34
query10	0.55	0.56	0.54
query11	0.20	0.15	0.15
query12	0.19	0.15	0.14
query13	0.45	0.47	0.47
query14	0.96	0.94	0.95
query15	0.63	0.59	0.60
query16	0.31	0.32	0.33
query17	1.11	1.05	1.15
query18	0.23	0.22	0.22
query19	2.01	1.96	1.93
query20	0.02	0.01	0.02
query21	15.50	0.21	0.15
query22	4.82	0.06	0.05
query23	16.12	0.30	0.13
query24	2.95	0.42	0.34
query25	0.11	0.07	0.04
query26	0.72	0.21	0.15
query27	0.05	0.03	0.04
query28	3.53	0.76	0.36
query29	12.54	3.98	3.20
query30	0.28	0.15	0.15
query31	2.78	0.57	0.31
query32	3.22	0.59	0.49
query33	3.13	3.20	3.23
query34	15.57	3.90	3.27
query35	3.24	3.22	3.24
query36	0.55	0.43	0.41
query37	0.10	0.06	0.07
query38	0.04	0.04	0.04
query39	0.03	0.03	0.03
query40	0.17	0.14	0.15
query41	0.08	0.03	0.03
query42	0.03	0.03	0.02
query43	0.05	0.04	0.03
Total cold run time: 96.18 s
Total hot run time: 24.1 s

[github-actions]

Review complete — changes requested. I reviewed all seven changed files at head 5bf185b in three static rounds. The final round's two full reviews and separate risk scan found no new valuable findings. This review adds three P1 and one P2 inline findings. Earlier P1 comment 4134420545 remains applicable only to the default-LIST fallback variant: its original no-default fixture is fixed, but an expired explicit partition can still be read without appearing in the recorded mapping when a default partition triggers the fallback. I did not repost that issue.

Critical checkpoints

• Goal and proof: The change aims to make non-IVM MV refresh reads match mapped base partitions and their snapshots. The RANGE, explicit LIST-tuple, and fractional-key cases have unit or regression coverage, but default LIST and multi-PCT cases still violate that goal (inline findings). The golden output is consistent with the checked positive cases; it does not prove later default-partition DML or the combined edge cases.
• Scope and clarity: The new mapping is passed only through the intended refresh path; IVM and EXPLAIN retain their existing unscoped behavior. The empty-set and default fallbacks introduce the correctness gaps identified inline. No unrelated source change was found.
• Concurrency and lifecycle: The refresh context builds a mapping, batches derive readable names, the overwrite reads base data, and partition snapshots are published afterward. Default-partition data can enter an MV partition without a matching dependency in its snapshot. Concurrent partition DDL can also mutate the newly traversed metadata map after its read lock has been released. No separate lock-order deadlock, static-initialization, or one-replacement false-synchronization case was substantiated.
• Configuration and compatibility: No new configuration, FE-BE field, function symbol, storage format, or edit-log schema is added. Existing partition_sync_limit and one-partition batch settings expose the residual over-read and repeated metadata-scan cost; no separate dynamic-config or rolling-upgrade issue was established.
• Parallel paths and conditions: Scoped RANGE and full LIST tuples, NULL keys, views/CTEs, retries, IVM, SELF_MANAGE, and EXPLAIN were traced. The default LIST fallback and the empty-mapping branch need the fixes described inline.
• Tests and results: The added FE assertions and ordered regression golden cover the positive RANGE, LIST, default-row, and fractional-key examples. They omit a later default-partition write, a multi-PCT default row, and a fractional key on a table with a default partition. This invocation did not run builds or tests, as the review prompt prohibits them; author replies and the golden are not independent runtime validation.
• Persistence, writes, visibility, and observability: The overwrite and journal formats are unchanged. The findings concern which committed rows the refresh reads and which base-partition versions its snapshot records, so they can affect MV contents and transparent rewrite despite ordinary visible-version handling. Existing task logging remains available; no separate metrics or logging defect was substantiated.
• Performance and other invariants: The new default-partition lookup is repeated for each batch and adds quadratic partition-map visits under the default batch size. No BE memory-safety or additional error-handling issue was established.

User focus: No additional focus points were supplied. Conclusions are static-only.

Existing P0/P1 findings confirmed for this head: #68648 (comment)

[github-actions]

[P1] Check default LIST coverage before treating an empty mapping as FALSE. In a supported two-PCT MV, A has an explicit LIST partition for key 1, B has an explicit partition for key 2 plus a default partition holding key 1, and the MV joins A and B on that key. The key-1 MV partition is created from A; B has no mapped name for it because its default projects the MIN sentinel. This branch injects FALSE into B's scan, making the join empty and dropping the committed key-1 row on the first refresh. The earlier default-partition thread covers the nonempty-map branch. Apply the MV-key fallback here too, with matching snapshot dependencies, and test the two-PCT case.

[yujun777]

Fixed in 800af51, in the mapping rather than in that branch, because that is where the read and the record have to agree.

A base table's default list partition is now named in the mapping of every MV partition that reads the table, not only in the mapping of the one its sentinel key maps to. Your shape is measured: with a_t holding key 1 explicitly and b_t holding key 2 explicitly plus a default partition with key 1, the MV partition for key 1 is recorded with b_default, so it is not read as nothing and the join holds the row (1 10 100), where before the change the MV was empty.

That also settles where the empty mapping comes from: with the mapping naming the default partition in every MV partition, "this table feeds nothing to these partitions" can no longer be true of a table that has one. The check for it is still made before the empty branch answers FALSE, and it is now read from the mapped partitions themselves, so a table's partitions are not scanned per batch.

[github-actions]

[P1] Track default LIST partitions in every MV partition that reads them. In the new list_default fixture, this fallback puts the p_default row with d=2020-01-01 into the explicit 2020 MV partition, but its snapshot records only p_explicit because the default sentinel maps elsewhere. A later insert into p_default changes none of the versions checked for 2020, so automatic refresh skips it and transparent rewrite can serve stale rows. The existing default-partition thread covers the initial read; this is the missing dependency after that read. Add a post-refresh default insert test.

[yujun777]

Fixed in 800af51. The default list partition of a base table is now named in the mapping of every MV partition that reads it, which is the same set the read uses and the set the refresh records, so isSyncWithPartitions compares it: an insert into it changes a version the MV partition checks and the partition stops calling itself synchronized.

Pinned as a case in the read scope suite (list_default_tracked): the MV is refreshed, a row is inserted into the default partition afterwards, and SyncWithBaseTables is false for it, where before the change it stayed true.

[github-actions]

[P1] Preserve DATETIMEV2 scale when the new default-LIST fallback reaches this conversion. A LIST(ts DATETIME(3)) base table with an explicit key 2024-02-01 00:00:00.123 and a default partition sends its PARTITION BY ts MV through constructPredicates(mvItems, colName). This Optional.empty() uses Type.fromPrimitiveType(DATETIMEV2) (scale 0), so the generated ts IN predicate compares against .000 and omits the committed .123 row. The earlier scale thread covered the scoped base-partition path; the fallback bypasses that fix. Add a combined fractional/default regression.

[yujun777]

Fixed in 800af51. The fallback that reads a table through the MV partition's own key range now converts keys with the partition column's full Type, like the scoped path does, so the scale is not lost on the way: a LIST(ts DATETIME(3)) table with an explicit 2024-02-01 00:00:00.123 and a default partition keeps that row in its PARTITION BY ts MV, which was empty before.

The regression case is the combined shape you asked for (fractional_default), and it fails when the type is taken from the primitive again.

[github-actions]

[P2] Resolve default-LIST presence once from stable partition metadata. getIdToItem(false) returns a mutable HashMap and requires the table read lock, but this call runs after buildRefreshContext released its locks; concurrent ADD/DROP PARTITION can make anyMatch throw an unretried ConcurrentModificationException. With no default, it also walks all N partitions for every refresh batch; the default one-partition batch size adds O(N²) metadata visits for N one-to-one MV partitions. Capture this fact under the table lock and reuse it across batches.

[yujun777]

Fixed in 800af51, in the direction you point at: the fact is no longer read from getIdToItem at all.

Whether a table is read this way is now taken from the mapped partitions a refresh already has in hand -- one of them being a default partition is the condition -- so there is no lock-free walk of the table's partition metadata, no per-batch O(N) scan, and nothing new to serialise against a concurrent ADD or DROP PARTITION. The underlying fact, which table has such a partition, is computed once per mapping in MTMV#calculatePartitionMappings and merged into every MV partition's entry there, which is also what makes it available across batches instead of per batch.

[yujun777]

run buildall

[yujun777]

/review

[github-actions]

Static review of PR #68648 at 800af51. Review complete after three rounds and a final sweep of all eight changed files. Request changes: one new P1 rewrite correctness finding and two new P2 test findings are inline below. No builds or tests were run, as required by the review task; the conclusions are code-path analysis, not runtime validation.

Critical checkpoints:

• Goal and proof: The PR aims to make MV refresh read exactly the recorded base partitions, including default LIST dependencies. Exact LIST/RANGE predicates improve that path, but the default mapping still has wrong-result cases, so the goal is not fully met. The new regression suite does not reliably prove its date-window or per-partition dependency claims (two P2 comments).
• Scope and clarity: The changes are concentrated in mapping, refresh predicate plumbing, and matching tests. The new fan-out also changes rewrite eligibility; that shared consumer needs to be handled explicitly.
• Concurrency: Refresh context, invalidation, and rewrite planning hold the relevant table locks during mapping. The unlocked metadata status probe catches analysis/runtime failures and can transiently report unsynchronized; no separate correctness or lock-order finding was substantiated. No new thread or global mutable state is introduced.
• Lifecycle: Manual named-partition refresh remains exact, and batch snapshots publish only refreshed partitions. That lifecycle is central to the new P1: another default-dependent MV partition can remain stale while rewrite checks a different, refreshed partition.
• Configuration: No new setting is introduced. The existing SET-able enable_materialized_view_union_rewrite=false path skips invalid-partition removal and exposes the P1 wrong result.
• Compatibility: No changed storage, serialization, FE/BE protocol, or function symbol was found in this PR; no rolling-upgrade finding was substantiated.
• Parallel paths: Scoped LIST/RANGE reads, default fallback, empty mapping, EXPLAIN, batch refresh, invalidation, and rewrite were traced. The new mapping must be represented faithfully in the rewrite inverse; it currently is not.
• Conditions: The default LIST and empty-map branches were checked against their upstream mappings and related cases. The still-applicable too-wide default read and missing real-key coverage are covered by existing P1 threads below.
• Tests: FE unit and regression changes cover several intended shapes, including LIST, RANGE, default, and fractional keys, but the date fixture can vary between the runner and FE calendars, and the whole-MV flag can pass without proving the 2020 partition dependency. The partial-refresh/default-only rewrite case needs coverage.
• Expected results: The affected golden entries are deterministic only under an unverified common calendar, and SyncWithBaseTables=false is too coarse to prove the claimed dependency. No other distinct golden mismatch was substantiated.
• Observability: Existing debug/status paths provide context; this PR adds no metric or logging requirement supported by a distinct issue.
• Transactions and persistence: No EditLog format or new persistent field is changed. The reviewed snapshot and partition-version relationships can nevertheless mark an incomplete or stale MV as rewrite-eligible in the cases reported.
• Writes and crashes: No new base-table write or crash-recovery mechanism is introduced. Refresh publication and exact partial scope were checked; no separate atomicity or crash finding was substantiated.
• FE/BE variables: No new variable or FE-to-BE transfer is added. The relevant union-rewrite session setting is handled in FE planner control flow.
• Performance and remaining issues: Default detection runs once per mapping rather than once per refresh batch. No further concrete performance defect or unresolved candidate remained after the final sweep.

Existing blocking findings still applicable on this head, without duplicate inline comments: r4134420545 (default LIST plus sync limit can read an expired explicit partition absent from the snapshot) and r4140476259 (a default row whose real MV key has no created partition remains unreachable while rewrite can regard the base default as covered). The earlier threads already describe these failure modes. User focus: none was provided.

Existing P0/P1 findings confirmed for this head: #68648 (comment), #68648 (comment)

[github-actions]

[P1] Check every MV partition mapped from a default LIST base partition before allowing rewrite. This fan-out puts p_default in multiple MV mappings, but MTMVRewriteUtil.getPctToMv inverts them into one String and keeps only an arbitrary destination. With explicit LIST tuples (2020,2) and (2021,2) plus default rows for (2020,3) and (2021,3), an insert into p_default dirties both MV partitions. Refresh only the inverse-selected partition, then query the other default-only tuple: Filter(d=other AND k=3) -> Scan(base[p_default]) can rewrite to Filter(d=other AND k=3) -> Scan(MV[other stale partition]) because eligibility checks only the refreshed destination. Setting enable_materialized_view_union_rewrite=false skips the later invalid-partition removal, so this can return stale data. Preserve the full inverse set or reject ambiguous default mappings for rewrite.

[yujun777]

Fixed in 5ba21b7, by preserving the whole inverse set. MTMVRewriteUtil.getPctToMv answers each base partition with the set of MV partitions that read it now, and the rewrite's partition selection adds all of them, so a query that reads a default partition is held against every MV partition its rows can be in rather than against whichever one the map happened to keep. The UT covers the fan-out (t1_default read by mv_a and mv_b, answered with both).

That is the other half of the fan-out the previous commit introduced, and it is what the mapping's own contract says: the mapping is what the read and the record agree on, so anything inverting it has to keep all of it.

[yujun777]

Fixed in 0dbc408, on the side that cannot drop rows. Preserving the whole inverse set made every MV partition that reads a base partition a candidate, but the eligibility still answered a query when only some of those partitions were valid. With enable_materialized_view_union_rewrite off there is nothing to compensate the others: the MV alone answers the query and reads the partitions that are not valid as they are.

getMTMVCanRewritePartitions now returns no partition for a query unless every MV partition its base partitions are mapped from is valid, when the union rewrite is off. Read as a subset rather than as a count, since a partition inside its grace period is answered usable without having been compared and is not necessarily one the query reads. With the union rewrite on (the default) a partially valid answer is still a complete one, and the valid partitions still answer.

[github-actions]

[P2] Use the FE refresh calendar for this date-window fixture. LocalDate.now() uses the regression runner's default time zone, while partition_sync_limit computes its cutoff from DateTimeAcquire.now() in the FE session time zone. If the runner is on September 30 and FE is on October 1 (or setup crosses midnight), p_kept has an upper bound equal to FE's cutoff and is filtered out, so scope_kept_days returns 1 against the fixed golden value 2. Derive and hold the fixture date in the same calendar as the refresh, or recheck the boundary before asserting it.

[yujun777]

Fixed in 5ba21b7. The fixture's days now come from select curdate(), the FE session's own date, which is the calendar partition_sync_limit cuts its window from, instead of the runner's local date.

[github-actions]

[P2] Assert the default dependency on the 2020 MV partition itself. Before the fan-out change, p_default still mapped to the separate MIN MV partition. This insert makes MIN unsynchronized, so mv_infos.SyncWithBaseTables becomes false even while the 2020 partition still omits p_default from its snapshot. The new golden therefore passes with the tracking fix reverted. Check the 2020 partition's sync state before and after the insert, or refresh again and assert that its new row appears.

[yujun777]

You are right, and the case asserts the row now instead of the flag. list_default_tracked inserts into the default partition, refreshes the MV, and compares the rows: the new row has to be there, which is exactly what a refresh that judged the 2020 partition synchronized would leave out. The MV-level SyncWithBaseTables was not a check -- as you say, the sentinel partition moves it on its own.

The mapping fix cannot be reverted on its own to control this one, since reverting it fails the suite earlier (list_default, the read half); the row assertion is the part that pins the tracking.

[yujun777]

run buildall

[yujun777]

/review

[github-actions]

Static review of #68648 at dc1d04c. One new P2 finding: the aggregate-group fallback also suppresses safe union rewrites over disjoint RANGE partitions.

Two existing P1 findings remain applicable and are carried as existing_blocking_comment_ids rather than reposted: external PCT read/snapshot mismatch and absent IVM PCT input read outside recorded scope. Both were independently traced through current sources, including the relocated Paimon connector and IVM full/reset/delta paths. The existing aggregate-group finding 4356885832 is addressed by the new fallback for its stated LIST trigger.

Required checkpoint conclusions:

1. Goal and evidence: the change aligns ordinary OLAP refresh reads with recorded base partitions, coalesces overlapping LIST components, and tightens rewrite eligibility. Added unit/regression fixtures cover these mechanisms; the two known P1 exemptions mean read/record agreement is incomplete.
2. Scope and reuse: changes stay within partition mapping and its refresh, EXPLAIN, and rewrite consumers; grouping and mapped-scope helpers are reused. The aggregate fallback is broader than necessary (new P2).
3. Concurrency and locks: task alignment/context construction use sorted base-table read locks; default metadata walks hold the table lock. Mapping and later predicate lookup are separate phases. The existing DDL/retry threads already cover that transient window; the new descriptor guard fails loudly but does not itself retry alignment. No distinct new concurrency defect was substantiated.
4. Lifecycle: new maps and the undescribed-partition set are task/statement scoped. No new thread, ownership cycle, C++ static initializer, or release obligation is introduced.
5. Configuration: no new setting. Existing sync-window, grace, and union settings control the paths; default LIST tables deliberately bypass the sync window.
6. Compatibility: no new wire/storage format or journal member. Stored and generated direct LIST descriptors both use canonical ordering; legacy physical key order alone does not require replacement.
7. Thrift/protobuf: no gensrc changes.
8. Parallel paths: ordinary/default LIST, RANGE, fractional keys, SELF_MANAGE, EXPLAIN, aggregate-on-nonaggregate, self/multi-table joins, external PCT, and IVM paths were inspected. The last two retain the known P1 scope mismatches.
9. Special conditions: NULL-aware full-key predicates, explicit empty scope, unmatched-descriptor guard, and post-version-read grace decisions are justified. The aggregate rejection needs narrowing to overlapping groups.
10. Test coverage: added unit and regression cases cover transitive grouping, key-order collisions, pruned components, read scope, default dependencies, millisecond bounds, grace/preload, compensation slots, and aggregate-group fallback. The known P1 scenarios still need focused end-to-end coverage. No builds or tests were run, as required by the static-only contract.
11. Changed results: inspected new expected rows against the fixtures. The three changed grace-big no-cache blocks match their base-query blocks. Actual execution and golden-generation provenance were not verified.
12. Observability: existing task failure/snapshot logs and optimizer fail reasons carry useful MV/partition context; no additional critical observability defect found.
13. Persistence and failover: existing snapshot publication and journal machinery remain in use; no new persisted state or commit point is added. Read/record correctness remains limited by the two known P1s.
14. Writes and atomicity: refresh continues through the existing transactional INSERT OVERWRITE path and publishes snapshots after successful batches. An atomic write can still record the wrong dependency set in the known external/IVM cases.
15. FE/BE propagation: no new transmitted variable or protocol field.
16. Performance: key-indexed grouping, factored RANGE predicates, early RANGE pruning, and bulk cloud-version reads avoid the historical repeated-work paths. The new P2 unnecessarily removes MV acceleration for disjoint RANGE aggregate compensation.
17. Further-issue recheck: reviewed wrapper/subtree aggregates, stale-only versus uncovered compensation, self aliases, bound slots, NULL/scale, descriptor identity, and DDL/default fanout. The apparent LIST roll-up ordering issue is unreachable through supported CREATE: the only partition-expression service rejects LIST inputs.

User focus: none supplied. Coverage includes all 30 changed paths and required upstream/downstream mechanisms. Limits are static inspection only: no runtime interleaving, connector integration, build, or test execution is claimed; no unresolved concrete candidate remains.

Review converged after 2 round(s).

Existing P0/P1 findings confirmed for this head: #68648 (comment), #68648 (comment)

[github-actions]

[P2] Preserve safe RANGE aggregate union rewrites

This also rejects ordinary windowed RANGE MVs whose aggregate groups are disjoint. For example, let t have populated daily RANGE(d) partitions for 2020-01-01 and 2038-01-01, plus an unrelated 2040 partition. Refresh PARTITION BY(d) AS SELECT d,SUM(v) FROM t GROUP BY d with a 2 YEAR sync limit, then query those first two dates:

Aggregate(d, SUM(v))
  Filter(d IN (2020-01-01, 2038-01-01))
    Scan t[old, kept]

calcInvalidPartitions reports old in uncoveredNarrowedBasePartitions, so this guard rejects the candidate before canUnionRewrite. But UnionAll(Scan MV[kept], Aggregate(d,SUM(v)) -> Scan t[old]) is correct: its groups have different d values. The query now scans and aggregates the retained base data too, losing the normal TTL/union optimization (also expected by partition_union_rewrite/partition_mv_rewrite.groovy). Limit the fallback to cases where uncovered partitions can overlap retained aggregate groups, and preserve this disjoint RANGE case.

[yujun777]

Fixed in a3cdee32fd1.

The fallback you pointed at is now only the key-overlap question, and that set is filled from a ListPartitionInfo: overlappingUncoveredListPartitions returns empty for any other partitioning, because a RANGE partition is either covered by a retained MV partition -- then it is not uncovered -- or lies outside its range, where it answers groups of its own. Your windowed RANGE case keeps its UnionAll(Scan MV[kept], Aggregate(d,SUM(v)) -> Scan t[old]), and so does every ordinary TTL-shaped query.

What I took out of the base rule is the group-expression test, which is what over-declined here; it now lives in AbstractMaterializedViewAggregateRule#canUnionRewrite, which asks the grouping itself: the rewrite stands when the grouping holds every column the MV is partitioned on, each of them by itself, and is declined otherwise. A bare GROUP BY d therefore keeps the rewrite -- which is what nereids_rules_p0/mv/partition_union_rewrite and nereids_rules_p0/mv/union_rewrite_grace_big need. Those two suites were failing on the branch before this commit (the roll-up query of partition_mv_rewrite, which groups by date_trunc(l_shipdate,'day') and bare l_shipdate, was declined by the expression test); both pass now, along with union_rewrite, grouping_sets, agg_with_roll_up, agg_without_roll_up and date_trunc -- 14 suites, none failed.

One thing to flag rather than bury: the grouping test declines an expression over the partition column even when the expression happens not to coarsen. union_all_compensate's query5 groups by to_date(data_date) over daily RANGE partitions, where the branches do come out disjoint; telling that apart needs the expression evaluated per partition value, which is not something the grouping can answer, so that candidate is now declined and the suite's expectation moved from mv_rewrite_success to mv_rewrite_fail. Its rows are unchanged -- the .out compares the base-table answer against the same values -- only the plan differs.

[yujun777]

run buildall

[yujun777]

/review

[github-actions]

Static review of #68648 at 1de7a0f. No additional inline findings; request changes for three previously reported P1 concerns independently confirmed against this head. Existing threads were treated as duplicate fences.

• 4271997436: external PCT reads remain wider than the recorded partition set for newly merged LIST components. The OLAP-only scope leaves the external scan on the MV key domain, while Paimon freshness compares only mapped per-partition timestamps; removing an unrecorded partition can leave its rows eligible through the MV.
• 4278938895: an IVM COMPLETE batch can read an entirely absent PCT input through an ordinary scan. No reset IDs or freshness dependency record that expired partition, and a subsequent DELETE can be skipped by the all-synchronized incremental exit.
• 4356885832: the exact GROUP BY d reproduction is fixed, but aggregate-group splitting remains for a many-to-one grouping expression. For LIST(d,k), use kept={(2020-01-02,1),(2038-01-01,1)}, bridge={(2038-01-01,2)}, old={(2020-01-01,2)}, and a 2 YEAR sync window. An MV partitioned/grouped by d records kept+bridge. With v=2 at kept's January 2 row and v=1 at old's January 1 row, query d IN ('2020-01-01','2020-01-02') grouped by date_trunc(d,'year'). The raw-key overlap helper sees distinct dates, but canUnionRewrite accepts the grouping expression containing d. The reduced static result is UnionAll(Aggregate(year(d),SUM(s))->MV[M], Aggregate(year(d),SUM(v))->base[old]): two year-2020 rows of 2 and 1 instead of one row of 3. The existing issue remains open for this variant; no duplicate inline is posted.

The safe disjoint RANGE case in 4359578727 is addressed: the helper exempts RANGE and preserves ordinary GROUP BY d union compensation.

Required checkpoint conclusions:

1. Goal/correctness: non-IVM OLAP refresh reads and snapshots now share the mapping, with full LIST tuples and precise RANGE bounds; the three concerns above prevent a general correctness sign-off.
2. Scope/clarity/reuse: changes are focused on MV mapping, refresh, rewrite and tests; EXPLAIN and refresh share the scope helper, and predicate construction is reused.
3. Concurrency/locks: refresh workers, planners and DDL can overlap. Sorted context table locks and locked default walks are retained. The known post-context live partition lookup remains covered by existing thread 4220192710; no distinct new lock-order issue was established. The descriptor-miss check fails before overwrite, rather than automatically realigning in that call.
4. Lifecycle: new contexts/scopes are statement/task/batch local; no new static initialization or resource lifecycle issue found.
5. Configuration: no new settings. Existing sync-window behavior changes for default LIST tables; session union/grace decisions remain runtime reads.
6. Compatibility: no new persisted format or wire contract; snapshot compatibility handling remains intact, and canonical LIST ordering is shared by current descriptors and stored items.
7. Thrift/Protobuf: no changes; branch field-ID comparison is inapplicable.
8. Parallel paths: checked OLAP/external/IVM, SELF_MANAGE, COMPLETE/partition refresh/EXPLAIN, LIST/RANGE/default/NULL/precision, aggregate-on-nonaggregate, multi-PCT and self-join compensation. Exceptions are identified above.
9. Conditions/errors: missing full-key slots reject rewrite; unmatched descriptors and invalid key/type assumptions fail explicitly. Original exact-key overlap and cloud-grace preload cases are covered by current logic.
10. Test coverage: inspected all eight changed Java test files and four regression scripts. Fixtures cover tuple scope, defaults, fractional bounds, component pruning, grace changes and exact-key aggregate fallback. The known default-insert fixture weakness (4142955465) remains: COMPLETE does not prove dependency-driven partition selection. The residual grouping-expression case lacks a regression.
11. Expected results: inspected all three changed goldens against their fixtures; restored outer-join rows and fractional/default/group totals are consistent. Generation history and runtime results were not independently verified.
12. Observability: existing task/query/snapshot logging and optimizer failure reasons cover the changed decisions; no additional essential metric identified.
13. Persistence/failover: new data structures are transient; existing committed-batch snapshot publication and replay structures are reused. No new EditLog schema is introduced.
14. Writes/atomicity: overwrite transaction/swap and partial-progress publication remain unchanged; freshness-record correctness is limited by the concerns above. Crash and concurrent schedules were not executed.
15. FE/BE propagation: no new transmitted variable or protocol field.
16. Performance: reviewed indexed component grouping, canonical ordering, early RANGE filtering, factored RANGE predicates and bulk cloud version reads; no additional verified performance defect found.

No additional user focus was supplied. This was a complete static review of the supplied changed-file list and authoritative diff, including related control flow and tests. No builds, tests, source changes or runtime/performance experiments were performed, as required by the runner contract. Runtime reproduction, concurrent schedules and golden provenance remain validation limits, not unclassified review candidates.

Review converged after 2 round(s).

Existing P0/P1 findings confirmed for this head: #68648 (comment), #68648 (comment), #68648 (comment)

[yujun777]

run buildall

[yujun777]

/review

[github-actions]

Request changes at e57d08a. Three new verified findings are inline: a mixed-stale computed-group bypass and a grouping-sets bypass can return duplicate aggregate groups; overlap detection also rejects a safe LIST compensation by counting an MV partition already scheduled for removal.

Two existing P1s remain applicable and are retained without duplicate inline comments: external PCT read/record mismatch and absent IVM PCT input scope. The external read still uses the MV domain while recording only mapped partitions; an absent IVM PCT input still gets an ordinary scan and an empty partition record. The exact earlier raw LIST aggregate split and safe RANGE fallback examples are fixed on this head; the new inline cases exercise distinct remaining mechanisms.

Critical checkpoint conclusions:

1. Goal/data correctness: ordinary non-IVM OLAP reads now follow their recorded mapping, including full LIST tuples, NULLs, defaults and fractional RANGE bounds. The new aggregate failures and two existing exceptions prevent an overall correctness endorsement.
2. Scope/reuse: refresh, mapping, rewrite and canonical LIST descriptors support the same invariant; shared scope/literal/grouping helpers are reused. No unrelated production changes identified.
3. Concurrency/locks: no new threads. Context construction uses existing sorted table locks; new default metadata walks use read locks. The post-context partition lookup window is existing thread context (#4220192710); no additional lock-order defect verified.
4. Lifecycle/static initialization: new maps/sets and snapshot preparation are invocation-local. Cloud version reads precede the single grace decision; that same set drives comparison preloading. No new static initialization dependency.
5. Configuration: none added. Existing sync-window, grace, batch-size and union settings are read per invocation; default-LIST window/rewrite behavior changes are explicit.
6. Compatibility: no persisted format change. Canonical descriptors are computed consistently on both alignment sides; stored LIST keys and SQL serialization stay unchanged. Internal command/rule signatures are propagated.
7. Thrift/protobuf: unchanged; no field-ID or enum review required.
8. Parallel paths: COMPLETE/PARTITIONS/EXPLAIN, LIST/RANGE/default, OLAP/external/IVM, self joins, grace/force-consistent, union enabled/disabled and aggregate-on-nonaggregate paths were traced. External/IVM exceptions are the existing P1s above.
9. Conditions/error handling: empty scoped input emits FALSE, absent scope keeps the MV domain, descriptor misses fail before overwrite, missing bound key slots refuse compensation. Aggregate safety conditions still have the inline defects.
10. Test coverage: changed JUnit and regression artifacts cover grouping/canonical collisions, pruning, scope/precision/defaults, grace/preload and union behavior. The three new mixed-stale/grouping-sets/removed-overlap cases are missing. The default dependency regression still uses unconditional COMPLETE, so it cannot isolate freshness selection; this is already covered by the existing test thread. Deterministic metadata interleavings and merged external/IVM mutation cases also lack direct coverage.
11. Expected results: changed scope/group totals and partition names agree with fixture data; repaired grace-big rows match base-only before blocks. Golden generation provenance was not independently established.
12. Observability: descriptor errors name the MV/partition; rewrite fallbacks record plan/partition reasons and version/preload failures log warnings. No additional metric requirement identified.
13. Persistence/transactions: existing overwrite and snapshot publication ordering remains; no new journal record or replay format.
14. Writes/atomicity: refresh retains batch INSERT OVERWRITE and completed-group accounting; descriptor misses fail before erasing data. No new crash leak or deadlock verified statically.
15. FE/BE propagation: predicates use existing expression analysis/translation; no new cross-process variable.
16. Performance: bulk grace version reads, one descriptor per component, early RANGE filtering and factored RANGE predicates bound redundant work. The new removed-partition overlap finding loses a safe optimization; conservative full-key LIST grouping fallback is also existing safe-union-thread context.
17. Omission sweep: all 30 changed paths and related invocation chains were reviewed. No additional unclassified candidate remains after the bounded reviewer passes.

No additional user focus was supplied. This was a static review only: no builds, tests, cluster runs or source modifications were performed. Runtime behavior, race reproduction and reported author test runs were not independently validated.

Review converged after 2 round(s).

Existing P0/P1 findings confirmed for this head: #68648 (comment), #68648 (comment)

[github-actions]

[P1] Keep computed-group protection when only some MV partitions are removed

Removing a stale partition does not imply that every MV partition is removed. Extend the new group_split_expr fixture with a populated p_stale={(2037-01-01,3)}, refresh the MV, then write p_stale. Query d IN ('2020-01-01','2020-01-02','2037-01-01') GROUP BY date_trunc(d,'year'). The fresh merged component still holds Jan 2 with v=2; uncovered p_old holds Jan 1 with v=1, while the separate 2037 MV partition is stale.

calcInvalidPartitions removes only the 2037 partition and compensates {p_old,p_stale}. Raw dates do not overlap, so uncoveredNarrowedBasePartitions is empty; needRemoveMvPartitions=true then disables this expression check. canUnionRewrite accepts the PCT slot inside date_trunc, producing this reduced plan:

UnionAll
  Aggregate(year(d),sum(s)) -> Scan MV[fresh merged component]
  Aggregate(year(d),sum(v)) -> Filter(query AND keys(old,stale)) -> Scan t

The two branches emit separate year-2020 rows with 2 and 1, instead of one row with 3. Unlike #4356885832's fixed same-date case, this bypass needs computed groups and a simultaneous stale removal. Check overlap against the MV partitions that actually remain, even when another partition is removed, or merge aggregate states above the union.

[yujun777]

Fixed in a3cdee32fd1.

The bypass was the !needRemoveMvPartitions term, and the whole expression test has left the base rule: what the union of two finished aggregates needs is a question about the grouping, not about which partitions were dropped, and the aggregate rule is where the grouping is known. AbstractMaterializedViewAggregateRule#canUnionRewrite now requires every column the MV is partitioned on to appear in the grouping by itself, and AbstractMaterializedViewRule consults it for any compensation union -- needBaseTableUnion alone, with removal beside the point -- so a stale partition removed beside the compensation no longer switches the question off.

Your shape is the new group_split_stale_expr fixture of mtmv_p0/test_mtmv_base_partition_read_scope: p_kept={(2020-01-01,1),(2038-01-01,1)}, p_bridge={(2038-01-01,2)}, p_stale={(2037-01-01,3)}, p_old={(2020-01-01,2)}, 2 YEAR window, MV PARTITION BY (d) AS SELECT d, SUM(v) FROM t GROUP BY d; after the refresh, 100 more go into p_stale. The query over the three dates grouped by date_trunc(d,'year') removes only the 2037 MV partition and compensates {p_old, p_stale}, so raw dates do not overlap and the key-overlap set stays empty. Before the change the plan carried your UnionAll and year 2020 came out twice (2 and 1); the explain now asserts no UNION and the golden is one row per year, 2020-01-01 -> 3 and 2037-01-01 -> 104.

[github-actions]

[P1] Check each grouping set before treating slot grouping keys as disjoint

The normalized group-by list hides grouping sets that omit the partition column. With the new group_split_expr fixture unchanged and fresh, query SELECT d,SUM(v) FROM group_split_expr_base WHERE d IN ('2020-01-01','2020-01-02') GROUP BY GROUPING SETS ((d),()). The MV holds Jan 2 (v=2), and uncovered p_old holds Jan 1 (v=1). Their raw dates do not overlap.

NormalizeRepeat builds aggregate keys [d, grouping_id] and preserves sourceRepeat=[[d],[]]. Both keys remain SlotReferences through lineage, so this loop skips them and returns false. canUnionRewrite accepts d, while aggregateRewriteByView explicitly rewrites the repeat over the MV. The reduced result is:

UnionAll
  Aggregate[sets(d),()] -> Repeat -> Scan MV // (Jan2,2), (NULL,2)
  Aggregate[sets(d),()] -> Repeat -> Filter(keys(old)) -> Scan t
                                             // (Jan1,1), (NULL,1)

There is no aggregate above this union, so it returns two grand totals instead of (NULL,3). This is separate from the stale-partition bypass: no partition removal or computed group expression is needed. Inspect sourceRepeat/each grouping set and reject compensation when a set can span both branches, or combine its aggregate states correctly.

[yujun777]

Fixed in a3cdee32fd1.

Done as you describe, and in the place that can ask it: the method at line 547 you pointed at is gone, and AbstractMaterializedViewAggregateRule#canUnionRewrite now takes the grouping apart itself. When the aggregate has a source Repeat it walks getGroupingSets() and requires, of every set, that each column the MV is partitioned on appears by itself; the normalized key list is no longer trusted to say which set a slot came from. Your [[d],[]] case fails on the empty set, and a set that groups only by an expression over the column fails on the same rule as the plain case.

group_split_sets in mtmv_p0/test_mtmv_base_partition_read_scope is the case: the group_split_expr_base fixture unchanged, SELECT d, SUM(v) ... WHERE d IN ('2020-01-01','2020-01-02') GROUP BY GROUPING SETS ((d),()). Before, the plan was the UnionAll of two Aggregate -> Repeat -> ... branches and the grand total came out as two rows; the explain now asserts no UNION, and the golden is the three rows with one total: \N 3, 2020-01-01 1, 2020-01-02 2.

nereids_rules_p0/mv/grouping_sets and mtmv_p0/test_multi_pct_rollup_mtmv / test_rollup_partition_mtmv / test_create_rollup_mtmv pass with it, so the tightening is limited to the shapes that can actually span the two branches.

[github-actions]

[P2] Exclude removed MV partitions from the retained-key overlap check

This passes every selected MV partition, including those already scheduled for removal, as the helper's retainedMvPartitions. For LIST(d,k), use p_stale={(2020-01-01,1),(2037-01-01,1)}, p_bridge={(2037-01-01,2)}, p_old={(2020-01-01,2)}, and p_fresh={(2038-01-01,1)} with a 2 YEAR sync limit. Populate both retained components, refresh PARTITION BY(d) AS SELECT d,SUM(v) FROM t GROUP BY d, then write p_stale. Query dates 2020-01-01 and 2038-01-01, grouped by d.

The invalid merged {2020,2037} MV partition is removed; only the fresh 2038 partition remains. The base branch reads old plus the stale bucket (the query filter excludes bridge's 2037 rows), so the valid target is:

UnionAll
  Scan MV[fresh 2038 aggregate]
  Aggregate(d,sum(v)) -> Filter(d=2020 AND compensation keys) -> Scan t

Its groups are disjoint. But the helper sees 2020 in the stale MV partition, flags p_old, and the aggregate guard rejects the rewrite before removal. Unlike #4359578727's fixed RANGE case, this false overlap comes specifically from including a removed LIST component. Subtract the MV removal set before computing retained keys and add this partial-staleness case.

[yujun777]

Fixed in a3cdee32fd1.

calcInvalidPartitions now computes the set the overlap is projected against as the plan's MV partitions minus everything the removal pass has named:

Set<String> removedMvPartitions = Sets.newHashSet();
mvPartitionNeedRemoveNameMap.values().forEach(removedMvPartitions::addAll);
Set<String> retainedMvPartitions = Sets.difference(rewrittenPlanUsePartitionNameSet, removedMvPartitions);

A partition scheduled for removal is read back through the compensation branch, so its keys are the compensation's own and cannot be the retained group an uncovered partition would meet.

partial_stale in mtmv_p0/test_mtmv_base_partition_read_scope is your case: p_stale={(2020-01-01,1),(2037-01-01,1)}, p_bridge={(2037-01-01,2)}, p_old={(2020-01-01,2)}, p_fresh={(2038-01-01,1)}, 2 YEAR window, both retained components populated before the refresh and p_stale written after it. The merged {2020,2037} MV partition is removed and only the fresh 2038 partition remains, so p_old is no longer flagged for a 2020 it shares with a partition that is going away; SELECT d, SUM(v) ... WHERE d IN ('2020-01-01','2038-01-01') GROUP BY d keeps the union, and its explain asserts both UNION and partial_stale_mv (rows 2020-01-01 103, 2038-01-01 0). Note it needs the bare-column grouping too: this is the case where the partition keys are the only thing separating the two branches' groups.

[hello-stephen]

TPC-H: Total hot run time: 28720 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e57d08ac3bb243922169b1e5b9c33f3975a5252d, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17599	5192	5064	5064
q2	2115	344	257	257
q3	10223	1405	798	798
q4	4679	468	336	336
q5	7449	788	531	531
q6	178	181	152	152
q7	747	807	599	599
q8	9266	1396	1086	1086
q9	5602	4173	4189	4173
q10	6821	1277	1039	1039
q11	433	262	235	235
q12	646	400	278	278
q13	18105	3709	3039	3039
q14	278	277	250	250
q15	q16	729	715	679	679
q17	1205	750	576	576
q18	7064	6192	6122	6122
q19	1242	1089	604	604
q20	399	340	238	238
q21	5490	2388	2483	2388
q22	377	344	276	276
Total cold run time: 100647 ms
Total hot run time: 28720 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5958	5809	5999	5809
q2	722	609	586	586
q3	5633	5806	5221	5221
q4	2013	2138	1581	1581
q5	5623	5551	5692	5551
q6	243	188	141	141
q7	2301	2031	1819	1819
q8	3029	2687	2691	2687
q9	7876	7906	7793	7793
q10	3734	3602	3140	3140
q11	579	402	375	375
q12	665	649	504	504
q13	3353	3629	3049	3049
q14	283	288	254	254
q15	q16	656	692	612	612
q17	7736	7007	6808	6808
q18	12911	12176	12860	12176
q19	891	760	772	760
q20	2142	2133	1942	1942
q21	5753	4599	4681	4599
q22	490	467	411	411
Total cold run time: 72591 ms
Total hot run time: 65818 ms

[hello-stephen]

TPC-DS: Total hot run time: 150332 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e57d08ac3bb243922169b1e5b9c33f3975a5252d, data reload: false

query5	4321	610	449	449
query6	428	207	196	196
query7	4805	436	242	242
query8	323	161	146	146
query9	8754	3060	3061	3060
query10	470	300	264	264
query11	5934	3857	3545	3545
query12	145	90	87	87
query13	1258	440	323	323
query14	6152	3910	3643	3643
query14_1	3421	3411	3353	3353
query15	211	201	182	182
query16	946	475	464	464
query17	840	662	521	521
query18	2412	410	298	298
query19	193	155	125	125
query20	80	79	91	79
query21	209	122	103	103
query22	12887	12840	12801	12801
query23	13510	12885	12406	12406
query23_1	12559	12471	12575	12471
query24	7395	795	463	463
query24_1	470	460	469	460
query25	515	395	334	334
query26	1252	261	137	137
query27	2794	435	270	270
query28	4605	1921	1916	1916
query29	1529	584	424	424
query30	298	217	182	182
query31	1018	920	814	814
query32	152	91	85	85
query33	509	293	232	232
query34	881	843	490	490
query35	746	740	634	634
query36	818	817	739	739
query37	120	97	85	85
query38	1780	1701	1670	1670
query39	789	769	746	746
query39_1	733	733	736	733
query40	236	118	97	97
query41	74	69	66	66
query42	89	84	94	84
query43	348	358	315	315
query44	1331	682	681	681
query45	182	179	165	165
query46	821	953	560	560
query47	2969	3059	2830	2830
query48	298	316	227	227
query49	581	405	296	296
query50	672	274	213	213
query51	10314	10537	10373	10373
query52	79	79	71	71
query53	198	211	149	149
query54	251	215	204	204
query55	73	68	64	64
query56	239	238	218	218
query57	1689	1602	1556	1556
query58	298	277	258	258
query59	2142	2224	1969	1969
query60	302	259	234	234
query61	174	170	170	170
query62	399	338	285	285
query63	191	156	161	156
query64	2882	1073	831	831
query65	3320	3250	3237	3237
query66	1769	416	300	300
query67	19973	20035	19958	19958
query68	3183	990	623	623
query69	378	291	246	246
query70	844	778	755	755
query71	302	230	202	202
query72	2835	2555	2279	2279
query73	509	513	293	293
query74	4603	4506	4320	4320
query75	2347	2283	1927	1927
query76	2393	1005	588	588
query77	333	376	260	260
query78	9806	9694	9133	9133
query79	1897	840	519	519
query80	1624	432	360	360
query81	601	313	279	279
query82	751	129	101	101
query83	299	194	185	185
query84	316	125	100	100
query85	886	444	384	384
query86	398	238	239	238
query87	2006	1913	1759	1759
query88	3680	2643	2632	2632
query89	330	281	238	238
query90	1932	163	152	152
query91	164	145	122	122
query92	98	93	92	92
query93	1132	988	572	572
query94	618	313	257	257
query95	567	325	291	291
query96	649	480	229	229
query97	2423	2422	2312	2312
query98	160	149	149	149
query99	774	751	652	652
Total cold run time: 237065 ms
Total hot run time: 150332 ms

[hello-stephen]

ClickBench: Total hot run time: 24.37 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e57d08ac3bb243922169b1e5b9c33f3975a5252d, data reload: false

query1	0.01	0.01	0.01
query2	0.09	0.05	0.05
query3	0.26	0.09	0.09
query4	1.61	0.10	0.10
query5	0.22	0.24	0.24
query6	1.17	0.68	0.67
query7	0.04	0.01	0.01
query8	0.06	0.04	0.04
query9	0.39	0.34	0.34
query10	0.56	0.56	0.61
query11	0.20	0.14	0.15
query12	0.18	0.15	0.14
query13	0.49	0.52	0.49
query14	1.04	1.01	1.02
query15	0.63	0.61	0.62
query16	0.35	0.34	0.34
query17	1.09	1.12	1.15
query18	0.20	0.19	0.19
query19	2.06	1.94	2.02
query20	0.02	0.02	0.02
query21	15.52	0.20	0.14
query22	4.86	0.05	0.05
query23	16.28	0.33	0.13
query24	2.98	0.62	0.41
query25	0.11	0.05	0.04
query26	1.40	0.22	0.16
query27	0.06	0.03	0.04
query28	3.08	0.81	0.36
query29	12.54	4.17	3.27
query30	0.28	0.13	0.10
query31	2.77	0.55	0.32
query32	3.22	0.59	0.51
query33	3.10	3.25	3.28
query34	15.34	3.98	3.38
query35	3.33	3.35	3.34
query36	0.55	0.49	0.42
query37	0.09	0.06	0.07
query38	0.05	0.04	0.03
query39	0.04	0.03	0.03
query40	0.17	0.16	0.14
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.03
Total cold run time: 96.6 s
Total hot run time: 24.37 s

[hello-stephen]

FE UT Coverage Report

Increment line coverage 64.16% (324/505) 🎉
Increment coverage report
Complete coverage report

[hello-stephen]

FE Regression Coverage Report

Increment line coverage 83.56% (422/505) 🎉
Increment coverage report
Complete coverage report

[hello-stephen]

TPC-H: Total hot run time: 28535 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e57d08ac3bb243922169b1e5b9c33f3975a5252d, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17586	5098	5088	5088
q2	2162	331	252	252
q3	10247	1362	826	826
q4	4687	461	332	332
q5	7434	798	542	542
q6	178	181	153	153
q7	751	801	597	597
q8	9260	1336	1036	1036
q9	5447	4340	4298	4298
q10	6824	1274	1005	1005
q11	417	247	230	230
q12	636	402	286	286
q13	18059	3512	3014	3014
q14	271	269	250	250
q15	q16	708	715	683	683
q17	1183	768	583	583
q18	6964	6145	6101	6101
q19	1103	949	584	584
q20	379	345	225	225
q21	4638	2195	2313	2195
q22	351	297	255	255
Total cold run time: 99285 ms
Total hot run time: 28535 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5611	5617	5620	5617
q2	675	585	533	533
q3	5174	5345	4894	4894
q4	2000	2160	1560	1560
q5	5280	5143	5384	5143
q6	263	200	163	163
q7	2455	2065	1907	1907
q8	3061	2712	2720	2712
q9	7999	7889	7876	7876
q10	3751	3659	3268	3268
q11	614	452	414	414
q12	734	811	553	553
q13	3543	3897	3357	3357
q14	322	319	282	282
q15	q16	683	738	687	687
q17	8182	7580	7070	7070
q18	13038	12125	12858	12125
q19	920	824	807	807
q20	2137	2134	1906	1906
q21	5729	4803	4799	4799
q22	501	446	412	412
Total cold run time: 72672 ms
Total hot run time: 66085 ms

[hello-stephen]

TPC-DS: Total hot run time: 150682 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e57d08ac3bb243922169b1e5b9c33f3975a5252d, data reload: false

query5	4318	602	460	460
query6	426	222	197	197
query7	4807	445	226	226
query8	324	169	153	153
query9	8771	3093	3116	3093
query10	485	304	256	256
query11	5894	3845	3553	3553
query12	162	93	91	91
query13	1262	452	340	340
query14	6113	3957	3615	3615
query14_1	3423	3411	3388	3388
query15	207	200	180	180
query16	944	428	401	401
query17	861	662	538	538
query18	2414	431	311	311
query19	200	173	146	146
query20	86	85	83	83
query21	212	132	110	110
query22	12974	12975	12864	12864
query23	13449	12694	12447	12447
query23_1	12539	12490	12436	12436
query24	7418	815	455	455
query24_1	472	485	480	480
query25	546	424	358	358
query26	1263	262	139	139
query27	2776	436	276	276
query28	4667	1955	1995	1955
query29	1899	586	430	430
query30	289	215	182	182
query31	1020	920	808	808
query32	138	88	94	88
query33	533	299	244	244
query34	1037	863	504	504
query35	733	741	636	636
query36	804	836	723	723
query37	128	93	85	85
query38	1772	1777	1674	1674
query39	785	799	743	743
query39_1	730	717	736	717
query40	235	116	92	92
query41	67	63	65	63
query42	85	84	80	80
query43	349	369	310	310
query44	1339	679	682	679
query45	174	174	170	170
query46	818	962	567	567
query47	3016	2987	2864	2864
query48	290	308	221	221
query49	568	391	286	286
query50	683	285	218	218
query51	10489	10303	10546	10303
query52	82	78	73	73
query53	185	209	158	158
query54	243	186	179	179
query55	77	70	64	64
query56	222	211	215	211
query57	1656	1621	1585	1585
query58	282	254	258	254
query59	2103	2119	1933	1933
query60	279	246	225	225
query61	142	139	141	139
query62	387	336	302	302
query63	195	158	170	158
query64	2718	925	790	790
query65	3317	3369	3277	3277
query66	1771	414	299	299
query67	20005	20085	19909	19909
query68	3019	1001	617	617
query69	389	277	253	253
query70	882	814	782	782
query71	303	241	202	202
query72	2765	2610	2284	2284
query73	525	538	300	300
query74	4636	4515	4350	4350
query75	2336	2305	1899	1899
query76	2318	1019	614	614
query77	334	395	275	275
query78	9683	9664	9180	9180
query79	872	910	523	523
query80	536	414	329	329
query81	543	318	269	269
query82	367	130	109	109
query83	315	217	195	195
query84	315	122	97	97
query85	841	422	363	363
query86	322	236	242	236
query87	1991	1918	1819	1819
query88	3669	2680	2654	2654
query89	327	278	241	241
query90	1786	174	172	172
query91	164	150	125	125
query92	97	90	87	87
query93	1012	1011	579	579
query94	446	312	278	278
query95	581	392	314	314
query96	664	495	235	235
query97	2430	2450	2323	2323
query98	163	150	147	147
query99	774	778	639	639
Total cold run time: 233849 ms
Total hot run time: 150682 ms

[hello-stephen]

ClickBench: Total hot run time: 24.37 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e57d08ac3bb243922169b1e5b9c33f3975a5252d, data reload: false

query1	0.01	0.01	0.00
query2	0.09	0.05	0.04
query3	0.26	0.09	0.09
query4	1.62	0.09	0.10
query5	0.26	0.24	0.25
query6	1.18	0.68	0.68
query7	0.04	0.01	0.00
query8	0.06	0.04	0.04
query9	0.41	0.36	0.36
query10	0.57	0.62	0.59
query11	0.21	0.14	0.14
query12	0.19	0.15	0.15
query13	0.49	0.48	0.49
query14	1.04	1.00	1.01
query15	0.64	0.62	0.61
query16	0.35	0.36	0.35
query17	1.10	1.13	1.08
query18	0.23	0.21	0.20
query19	2.14	2.01	2.00
query20	0.02	0.02	0.01
query21	15.47	0.18	0.14
query22	4.96	0.06	0.05
query23	16.31	0.33	0.12
query24	2.88	0.53	0.53
query25	0.08	0.07	0.06
query26	1.39	0.21	0.14
query27	0.06	0.04	0.05
query28	3.56	0.80	0.36
query29	12.48	4.20	3.28
query30	0.28	0.13	0.10
query31	2.77	0.58	0.32
query32	3.22	0.60	0.51
query33	3.27	3.14	3.29
query34	15.27	4.08	3.34
query35	3.36	3.33	3.38
query36	0.55	0.43	0.41
query37	0.09	0.07	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.18	0.15	0.14
query41	0.09	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 97.34 s
Total hot run time: 24.37 s

[hello-stephen]

FE UT Coverage Report

Increment line coverage 64.16% (324/505) 🎉
Increment coverage report
Complete coverage report

[hello-stephen]

FE Regression Coverage Report

Increment line coverage 83.56% (422/505) 🎉
Increment coverage report
Complete coverage report

[hello-stephen]

FE Regression Coverage Report

Increment line coverage 83.56% (422/505) 🎉
Increment coverage report
Complete coverage report

[hello-stephen]

FE Regression Coverage Report

Increment line coverage 83.56% (422/505) 🎉
Increment coverage report
Complete coverage report

[hello-stephen]

FE Regression Coverage Report

Increment line coverage 83.56% (422/505) 🎉
Increment coverage report
Complete coverage report

[yujun777]

run buildall

[yujun777]

/review

[github-actions]

Reviewed exact head a3cdee3 against the supplied aggregate diff. Changes are requested because two existing P1 correctness findings still apply; neither is reposted. One new P2 performance regression is reported inline.

Existing blockers independently revalidated:

• 4271997436: external PCT inputs still read the MV key domain while recording only the sync-window mapping. The newly admitted mixed LIST component can materialize an unrecorded Paimon partition; removing it can leave the MV falsely synchronized.
• 4278938895: an IVM PCT table absent from a batch mapping still lacks an explicit empty reset scope. COMPLETE can read its excluded partition, while snapshot comparison and subsequent incremental scope omit it, preserving a row after DELETE.

The earlier full-tuple, default LIST, fractional precision, descriptor-miss, partition identity, grace eligibility, uncovered-input, aggregate-overlap, computed-group/stale-removal, and grouping-set findings were rechecked. Their reported failure paths are now fixed or refused by upstream rewrite eligibility. The new inline finding is distinct: equivalent PCT join columns are treated as independent grouping dimensions.

Applicable code-review skill checkpoints:

Checkpoint	Conclusion
Goal and evidence	Non-IVM OLAP reads now share the recorded mapping and overlapping LIST descriptors form connected components. External/IVM exceptions remain the two correctness blockers. Added FE/regression tests supply static coverage; no runtime verification was performed.
Scope and clarity	Changes are confined to FE MTMV mapping, refresh, rewrite, and related tests. Shared descriptor and typed whole-key helpers avoid duplicated implementations.
Concurrency, threads, protected state	Scheduler refresh and partition DDL share metadata; alignment/context creation retain ID-sorted base-table read locks. New default-partition walks use table read locks with finally unlock. New maps are local; no thread or shared cache is added.
Work under locks and lock order	Added locked work is metadata traversal; cloud grace uses bulk version reads. No new lock-order inversion was substantiated. Existing external metadata planning under caller locks remains.
Lifecycle and static initialization	Contexts and batch scopes retain existing task/call lifetimes and cleanup. No C++ static initialization or ownership change.
Configuration and dynamic changes	No configuration added. Existing sync-window, grace, and union settings are read on refresh/query; narrowed-window behavior is covered.
Compatibility and rolling upgrades	No persisted field, storage format, wire schema, or exported function change. Single-PCT snapshot compatibility handling remains.
Thrift/protobuf IDs	No gensrc changes; inapplicable.
Parallel paths	OLAP/external, IVM/non-IVM, refresh/EXPLAIN, full/incremental, default LIST, and cloud/shared-nothing paths were traced. The two existing blockers are verified implementation defects, not unreviewed scope.
Special conditions and failure handling	Empty mapped OLAP reads become FALSE; undescribed MV partitions fail before overwrite, including mixed PCT batches. Default-bearing tables bypass the window consistently. Grace requires a written partition version. Missing bound compensation slots refuse rewrite.
Tests, negative cases, modularity	Unit cases cover component chains/order/pruning, typed/empty reads, grace/preload/coverage and compensation inputs. Regressions cover default dependencies, fractional bounds and aggregate/grouping-set cases. The inline safe-equijoin case and existing external/IVM failures need coverage.
Changed expected results	Fixture arithmetic supports the new ordered goldens; changed grace results restore rows present in base-result blocks. Static review cannot prove golden-generation provenance or test execution.
Observability	Existing task/query identifiers, partition-aware errors, and rewrite failure reasons cover changed decisions. No new service or critical metric surface was added.
Transactions, persistence, failover	Snapshot capture remains before reads and recording after successful overwrite; existing publication/journaling is retained. No new EditLog format. Dependency completeness still fails in the two existing P1s.
Writes, atomicity, crashes	Descriptor misses fail before writes; committed batch progress uses existing overwrite/snapshot lifecycle. No new transaction or crash/leak/deadlock path was substantiated.
FE-BE variables	No new variable or sending path.
Performance	Bulk cloud versions, single merged descriptor per component, early RANGE pruning, and factored RANGE predicates avoid known repeated work. The new allMatch grouping gate unnecessarily loses safe multi-PCT union rewrites, as reported inline.
Other invariants	Whole-key predicates retain NULL and full column types; version and snapshot identity checks were inspected. BE allocation accounting and MoW delete bitmap code are unchanged.

No additional user focus was supplied. All 32 changed files and related invocation paths were covered by the main scan and independent reviewers. Stale-only multi-input compensation was rechecked against partition-key equality/whole-component guarantees; the speculative mixed-pair issue was dismissed. LIST rollup ordering is unreachable through currently supported expression DDL, which requires RANGE.

Limits: this was static-only review: no builds, tests, clusters, or source edits. Some result fixtures can also pass on base-only plans, so those results alone do not prove rewrite selection. The descriptor-miss guard safely fails the current task; despite its comment, it does not itself retry alignment inside that task. No unverified candidate is included inline.

Review converged after 2 round(s).

Existing P0/P1 findings confirmed for this head: #68648 (comment), #68648 (comment)

[github-actions]

[P2] Accept equivalent PCT join keys when checking group disjointness

pctInfos lists the base columns that derive the same MV partition key: PartitionIncrementMaintainer follows A.d = B.d, so a two-PCT inner join records both A.d and B.d. Requiring every one as a grouping slot rejects the ordinary GROUP BY A.d; lineage substitution does not add the equivalent B.d.

For aligned daily RANGE partitions old, kept, and an unrelated third partition, refresh PARTITION BY(a_d) AS SELECT a.d AS a_d, SUM(a.v) AS s FROM A a JOIN B b ON a.d=b.d GROUP BY a.d, then write to old. Query only old and kept, grouping by a.d. The safe compensation is:

UnionAll
  Scan MV[kept]
  Aggregate(a.d, SUM(a.v))
    Join(a.d=b.d)
      Scan A[old]
      Scan B[old]

Its groups are disjoint by a.d, but this allMatch fails on B.d, discarding the MV and scanning/aggregating the retained base data too. Unlike the earlier RANGE overlap guard, this rejection remains in the new aggregate helper. Check the actual partition dimension using join equivalence, retain the per-grouping-set check, and add a two-PCT partial-union regression grouped by one join side.

[hello-stephen]

TPC-H: Total hot run time: 28271 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a3cdee32fd1d9c355d6ef7f4e59e7f43d9193420, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17609	5134	5075	5075
q2	2114	291	239	239
q3	10298	1355	819	819
q4	4679	463	330	330
q5	7420	792	528	528
q6	180	181	154	154
q7	737	788	585	585
q8	9259	1308	1065	1065
q9	5454	4173	4155	4155
q10	6829	1284	1029	1029
q11	435	242	236	236
q12	631	393	279	279
q13	18109	3514	3021	3021
q14	265	269	242	242
q15	q16	728	714	654	654
q17	1291	752	573	573
q18	6921	6149	6098	6098
q19	1106	955	594	594
q20	381	334	229	229
q21	4719	2115	2279	2115
q22	351	288	251	251
Total cold run time: 99516 ms
Total hot run time: 28271 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5610	5585	5602	5585
q2	661	562	533	533
q3	5165	5343	4925	4925
q4	1988	2157	1554	1554
q5	5250	5141	5442	5141
q6	258	198	155	155
q7	2408	2057	1875	1875
q8	3008	2647	2685	2647
q9	7930	7975	7941	7941
q10	3720	3636	3244	3244
q11	608	419	557	419
q12	703	709	522	522
q13	3528	3909	3278	3278
q14	322	317	289	289
q15	q16	688	732	675	675
q17	8172	7528	6852	6852
q18	13021	12181	12809	12181
q19	915	769	791	769
q20	2135	2153	1933	1933
q21	5582	4550	4659	4550
q22	482	479	401	401
Total cold run time: 72154 ms
Total hot run time: 65469 ms

[hello-stephen]

TPC-DS: Total hot run time: 150603 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a3cdee32fd1d9c355d6ef7f4e59e7f43d9193420, data reload: false

query5	4311	594	430	430
query6	428	216	190	190
query7	4800	445	232	232
query8	309	162	154	154
query9	8761	3090	3076	3076
query10	471	317	258	258
query11	6005	3883	3607	3607
query12	147	94	87	87
query13	1248	421	311	311
query14	6159	3932	3636	3636
query14_1	3456	3428	3440	3428
query15	213	193	183	183
query16	954	427	410	410
query17	863	665	534	534
query18	2426	435	335	335
query19	194	161	127	127
query20	80	81	80	80
query21	207	123	102	102
query22	12970	12926	12701	12701
query23	13379	13032	12380	12380
query23_1	12562	12517	12507	12507
query24	7441	855	461	461
query24_1	485	462	457	457
query25	520	408	348	348
query26	1251	252	138	138
query27	2758	445	271	271
query28	4613	1907	1887	1887
query29	1538	580	429	429
query30	303	213	190	190
query31	1037	934	813	813
query32	142	86	88	86
query33	513	289	232	232
query34	891	849	486	486
query35	726	737	638	638
query36	827	835	715	715
query37	119	98	80	80
query38	1799	1733	1649	1649
query39	799	794	746	746
query39_1	740	734	741	734
query40	213	109	91	91
query41	72	63	62	62
query42	83	79	81	79
query43	337	367	312	312
query44	1281	667	668	667
query45	176	194	169	169
query46	830	956	563	563
query47	2966	2867	2925	2867
query48	286	299	231	231
query49	576	417	288	288
query50	659	266	208	208
query51	10300	10625	10556	10556
query52	77	79	73	73
query53	183	204	154	154
query54	242	193	181	181
query55	68	67	67	67
query56	229	227	203	203
query57	1690	1620	1581	1581
query58	283	261	254	254
query59	2113	2146	1917	1917
query60	289	234	221	221
query61	149	134	136	134
query62	388	331	278	278
query63	186	158	160	158
query64	2741	915	775	775
query65	3331	3277	3250	3250
query66	1775	437	330	330
query67	20127	20276	20036	20036
query68	2974	994	604	604
query69	395	290	249	249
query70	850	771	786	771
query71	301	238	218	218
query72	2804	2639	2394	2394
query73	509	533	285	285
query74	4647	4480	4344	4344
query75	2321	2278	1927	1927
query76	2317	1009	593	593
query77	349	357	268	268
query78	9653	9751	9160	9160
query79	1009	914	510	510
query80	540	436	343	343
query81	538	323	269	269
query82	576	144	103	103
query83	312	194	197	194
query84	320	127	101	101
query85	815	429	360	360
query86	309	235	231	231
query87	1985	1967	1802	1802
query88	3632	2673	2624	2624
query89	363	283	240	240
query90	1940	169	164	164
query91	161	145	125	125
query92	95	89	89	89
query93	1014	977	563	563
query94	438	314	235	235
query95	578	421	301	301
query96	670	484	226	226
query97	2431	2440	2308	2308
query98	157	154	153	153
query99	759	778	636	636
Total cold run time: 233455 ms
Total hot run time: 150603 ms

[hello-stephen]

ClickBench: Total hot run time: 24.4 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a3cdee32fd1d9c355d6ef7f4e59e7f43d9193420, data reload: false

query1	0.01	0.01	0.00
query2	0.09	0.05	0.05
query3	0.25	0.09	0.09
query4	1.61	0.10	0.09
query5	0.24	0.24	0.24
query6	1.17	0.69	0.68
query7	0.04	0.01	0.01
query8	0.05	0.03	0.03
query9	0.40	0.34	0.35
query10	0.56	0.60	0.60
query11	0.20	0.15	0.14
query12	0.17	0.15	0.14
query13	0.50	0.48	0.50
query14	1.05	1.01	1.01
query15	0.65	0.60	0.62
query16	0.35	0.35	0.35
query17	1.12	1.16	1.12
query18	0.21	0.19	0.18
query19	2.04	1.89	1.92
query20	0.02	0.01	0.01
query21	15.49	0.21	0.13
query22	4.86	0.06	0.05
query23	16.25	0.33	0.12
query24	2.75	0.64	0.62
query25	0.11	0.05	0.06
query26	1.67	0.21	0.16
query27	0.06	0.04	0.03
query28	3.48	0.72	0.38
query29	12.55	4.12	3.23
query30	0.29	0.12	0.10
query31	2.76	0.54	0.32
query32	3.22	0.61	0.50
query33	3.27	3.18	3.20
query34	15.27	4.02	3.37
query35	3.37	3.35	3.36
query36	0.55	0.49	0.41
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.17	0.14	0.13
query41	0.08	0.04	0.03
query42	0.04	0.03	0.03
query43	0.05	0.04	0.03
Total cold run time: 97.2 s
Total hot run time: 24.4 s