Skip to content

[fix](tvf) Secure LOCAL TVF access - #69526

Open
OIiveirra wants to merge 3 commits into
apache:masterfrom
OIiveirra:fix/jira-sdbsec-112
Open

OIiveirra wants to merge 3 commits into
apache:masterfrom
OIiveirra:fix/jira-sdbsec-112

Conversation

@OIiveirra

@OIiveirra OIiveirra commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

What problem does this PR solve?

Issue Number: N/A

Related PR: N/A

Problem Summary:

Glob expansion could return paths outside user_files_secure_path because
the expanded targets were not checked against a nonempty canonical
secure root.

  • Disable LOCAL TVF and FILE TVF with local storage by default using the
    startup-only enable_local_tvf setting.
  • Run the startup gate before the existing ADMIN/OPERATOR checks,
    property parsing, backend selection and file discovery.
  • Canonicalize the secure root and each expanded glob target before
    checking directory containment. Reject empty secure roots, remote
    schemes and invalid local paths. Return generic errors for failed or
    out-of-root target resolution, including dangling symlinks.
  • Explicitly enable LOCAL TVF in existing regression deployments to
    retain their LOCAL test coverage.

Release note

LOCAL TVF and FILE TVF with local storage are disabled by default. To
use them, set enable_local_tvf=true in fe.conf on every query-serving FE
and restart those FEs. The setting cannot be changed at runtime. Enabled
reads still require ADMIN/OPERATOR and must remain within the configured
BE user_files_secure_path. Errors no longer disclose expanded or
resolved target names.

Check List (For Author)

  • Test
    • Regression test (added; not run locally)
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been
        changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

Local validation:

  • 11 focused FE unit tests and 5 focused BE unit tests passed with zero
    failures or skips. These earlier results still match the current
    source; the BE link used a private legacy-glibc adapter.

  • 34 real SQL checks passed on a private FE built from this PR. They
    cover default-disable and restart opt-in, startup-only configuration,
    ADMIN/ordinary users, LOCAL/FILE and SELECT/EXPLAIN. No BE was
    registered, so these validate FE policy and rejection paths.

  • The original Groovy suites have not completed: their JDBC
    initialization requires a BE. Full Groovy and end-to-end file reads
    remain unverified locally.

  • Relevant Java Checkstyle, changed-line clang-format 16 and git diff
    --check passed.

  • Behavior changed:

    • No.
    • Yes. LOCAL and FILE-local require explicit startup opt-in;
      enabled reads enforce authorization and secure-root containment.
  • Does this need documentation?

    • No.
    • Yes. Document enable_local_tvf and the LOCAL/FILE-local
      upgrade behavior. Documentation PR pending; the release note
      above does not replace the user documentation.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

### What problem does this PR solve?

Issue Number: N/A

Problem Summary: Add the startup-only enable_local_tvf setting, defaulting to false. Reject LOCAL TVF and FILE TVF with local storage before property parsing or backend access. Explicit opt-in restores each branch's existing behavior. Dedicated regression deployments opt in to retain LOCAL coverage.

### Release note

LOCAL TVF and FILE TVF with local storage are disabled by default. Set enable_local_tvf=true in fe.conf on every query-serving FE and restart those FEs to restore the existing behavior.

### Check List (For Author)

- Test: 11 focused FE unit tests passed with zero failures or skips; focused FE Checkstyle passed. SQL regression coverage added; runtime validation recorded separately.
- Behavior changed: Yes. Local TVF access requires explicit startup opt-in.
- Does this need documentation: Yes. Release note documents the default and opt-in.
Copilot AI balanced review requested due to automatic review settings October 23, 2026 09:44
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

OIiveirra added a commit to OIiveirra/doris that referenced this pull request Oct 23, 2026
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69526

Problem Summary: Canonical glob results were not checked against the secure root, and LOCAL Lance bypassed local-path validation. Check the canonical root and resolved paths, enforce ADMIN/OPERATOR before file access, validate Lance schema and scan paths, and preserve LOCAL/S3 range types. Reject local storage selected through S3 Lance before dataset opening.

### Release note

Enabled LOCAL reads remain restricted to user_files_secure_path and require ADMIN/OPERATOR. LOCAL Lance accepts only local paths inside that directory. The existing default-disabled startup gate is retained.

### Check List (For Author)

- Test: 26 focused FE tests, Checkstyle and changed-line clang-format 16 passed. BE tests could not build because the local dependency package lacks libgoogle_cloud_cpp_oauth2.a.
- Behavior changed: Yes, enforce secure local path containment and authorization after explicit opt-in.
- Does this need documentation: Covered by the PR release note.
### What problem does this PR solve?

Issue Number: N/A

Related PR: N/A

Problem Summary: Glob expansion and symlinks can produce canonical paths outside the configured secure directory. Resolve a nonempty secure root and each local result canonically before checking containment. The shared resolver rejects remote schemes and embedded NULs. Keep the default-disabled startup gate and existing local authorization. Bind the related glob test fixture to its own secure root across build profiles.

### Release note

Enabled LOCAL reads remain restricted to user_files_secure_path after glob expansion and symlink resolution.

### Check List (For Author)

- Test: 5 focused BE tests passed with zero failures/skips; changed-line clang-format 16 and build hygiene passed. BE tests used the documented private legacy-glibc link adapter with branch/toolchain inputs verified. FE source is unchanged by this supplement; its existing focused tests passed previously.
- Behavior changed: Yes, enforce canonical secure-directory containment after explicit startup opt-in.
- Does this need documentation: Covered by the PR release note.
@OIiveirra OIiveirra changed the title [fix](fe) Disable LOCAL TVF by default [fix](tvf) Restrict local paths and require explicit startup opt-in Oct 23, 2026
OIiveirra added a commit to OIiveirra/doris that referenced this pull request Oct 23, 2026
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69526

Problem Summary: FILE_LOCAL also identifies existing local Lance catalogs. Apply the new secure-root check only to the documented LOCAL TVF version-zero ranges, retaining fixed-version catalog warehouse paths. Add catalog compatibility coverage, qualify the nested DatasetKey type in the new tests, and bind the existing glob fixture to its own root across build profiles.

### Release note

Fixed-version local Lance catalog reads keep their existing warehouse paths.

### Check List (For Author)

- Test: changed-line clang-format 16 passed. Local BE execution remains blocked by missing GCS dependencies; targeted compilation diagnostics also identify an existing Lance-C SDK API mismatch. Existing final FE changes have 26 passing tests.
- Behavior changed: Scope the new path check to LOCAL TVF.
- Does this need documentation: Covered by the PR release note.
OIiveirra added a commit to OIiveirra/doris that referenced this pull request Oct 23, 2026
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69526

Problem Summary: Canonical glob results were not checked against the secure root, and LOCAL Lance bypassed local-path validation. Validate canonical roots and resolved paths, check ADMIN/OPERATOR before file access, validate LOCAL Lance schema/scan paths, and preserve LOCAL/S3 range types. Limit Lance validation to the existing LOCAL TVF version-zero contract so fixed-version local catalogs retain their warehouse paths. Reject LocalProperties selected through S3 Lance before dataset opening.

### Release note

Enabled LOCAL reads remain restricted to user_files_secure_path and require ADMIN/OPERATOR. LOCAL Lance accepts only local paths inside that directory. The default-disabled startup gate is retained.

### Check List (For Author)

- Test: 26 focused FE tests, Java Checkstyle and changed-line clang-format 16 passed. BE execution is blocked by the installed Lance-C SDK missing the existing lance_scanner_set_distance_range API; no BE runtime pass is claimed.
- Behavior changed: Yes, enforce secure local paths and authorization after explicit opt-in.
- Does this need documentation: Covered by the PR release note.
OIiveirra added a commit to OIiveirra/doris that referenced this pull request Oct 23, 2026
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69526

Problem Summary: Canonical glob results were not checked against the secure root. Validate canonical local paths against a nonempty canonical secure root, reject remote schemes in the shared local resolver, and enforce ADMIN/OPERATOR before local file access and during privilege checking. Keep the existing default-disabled startup gate. Give the existing glob fixture an explicit secure root so profile-specific test directories do not change its assumptions.

### Release note

Enabled LOCAL reads remain restricted to user_files_secure_path and require ADMIN/OPERATOR privileges.

### Check List (For Author)

- Test: 10 focused FE tests and 4 focused BE tests passed with zero failures/skips; Java Checkstyle and changed-line clang-format 16 passed. BE tests used the supported Clang16 ASAN profile.
- Behavior changed: Yes, canonical secure-root containment and local authorization after explicit opt-in.
- Does this need documentation: Covered by the PR release note.
OIiveirra added a commit to OIiveirra/doris that referenced this pull request Oct 23, 2026
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69526

Problem Summary: Canonical glob results were not checked against the secure root. Validate canonical local paths against a nonempty canonical secure root, reject remote schemes in the shared local resolver, and enforce ADMIN/OPERATOR before local file access and during privilege checking. Keep the existing default-disabled startup gate. Give the existing glob fixture an explicit secure root so profile-specific test directories do not change its assumptions.

### Release note

Enabled LOCAL reads remain restricted to user_files_secure_path and require ADMIN/OPERATOR privileges.

### Check List (For Author)

- Test: 6 focused FE tests and 4 focused BE tests passed with zero failures/skips; Java Checkstyle and changed-line clang-format 16 passed. BE tests used the supported Clang16 ASAN profile.
- Behavior changed: Yes, canonical secure-root containment and local authorization after explicit opt-in.
- Does this need documentation: Covered by the PR release note.
### What problem does this PR solve?

Issue Number: N/A

Related PR: N/A

Problem Summary: A rejected glob result could disclose its expanded filename through a canonical-path error or a failed resolution of a dangling symlink. Return one generic target-validation error for both cases before the status reaches SQL clients. Extend temporary-directory tests to assert that neither real target names nor paths appear in errors.

### Release note

LOCAL path validation errors no longer reveal expanded or resolved target filenames.

### Check List (For Author)

- Test: 5 focused BE tests passed with zero failures/skips, including external wildcard, dangling symlink and error redaction assertions. Modified lines pass clang-format 16; independent Standards and Spec review found no remaining issue in this supplement.
- Behavior changed: Yes, target-validation errors use a generic message.
- Does this need documentation: Covered by the PR release note.
OIiveirra added a commit to OIiveirra/doris that referenced this pull request Oct 23, 2026
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69526

Problem Summary: A rejected glob result could disclose its expanded filename through a canonical-path error or a failed resolution of a dangling symlink. Return one generic target-validation error for both cases before the status reaches SQL clients. Extend temporary-directory tests to assert that neither real target names nor paths appear in errors.

### Release note

LOCAL path validation errors no longer reveal expanded or resolved target filenames.

### Check List (For Author)

- Test: Targeted compilation diagnostics passed for the two modified C++ files. Full BE execution remains blocked by the previously recorded Lance-C SDK/GCS dependency gaps. Modified lines pass clang-format 16; independent Standards and Spec review found no remaining issue in this supplement.
- Behavior changed: Yes, target-validation errors use a generic message.
- Does this need documentation: Covered by the PR release note.
OIiveirra added a commit to OIiveirra/doris that referenced this pull request Oct 23, 2026
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69526

Problem Summary: A rejected glob result could disclose its expanded filename through a canonical-path error or a failed resolution of a dangling symlink. Return one generic target-validation error for both cases before the status reaches SQL clients. Extend temporary-directory tests to assert that neither real target names nor paths appear in errors.

### Release note

LOCAL path validation errors no longer reveal expanded or resolved target filenames.

### Check List (For Author)

- Test: Targeted compilation diagnostics passed for the two modified C++ files. Full BE execution remains blocked by the previously recorded Lance-C SDK/GCS dependency gaps. Modified lines pass clang-format 16; independent Standards and Spec review found no remaining issue in this supplement.
- Behavior changed: Yes, target-validation errors use a generic message.
- Does this need documentation: Covered by the PR release note.
OIiveirra added a commit to OIiveirra/doris that referenced this pull request Oct 23, 2026
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69526

Problem Summary: A rejected glob result could disclose its expanded filename through a canonical-path error or a failed resolution of a dangling symlink. Return one generic target-validation error for both cases before the status reaches SQL clients. Extend temporary-directory tests to assert that neither real target names nor paths appear in errors.

### Release note

LOCAL path validation errors no longer reveal expanded or resolved target filenames.

### Check List (For Author)

- Test: 4 focused BE tests passed with zero failures/skips, including external wildcard, dangling symlink and error redaction assertions. Modified lines pass clang-format 16; independent Standards and Spec review found no remaining issue in this supplement.
- Behavior changed: Yes, target-validation errors use a generic message.
- Does this need documentation: Covered by the PR release note.
OIiveirra added a commit to OIiveirra/doris that referenced this pull request Oct 23, 2026
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69526

Problem Summary: A rejected glob result could disclose its expanded filename through a canonical-path error or a failed resolution of a dangling symlink. Return one generic target-validation error for both cases before the status reaches SQL clients. Extend temporary-directory tests to assert that neither real target names nor paths appear in errors.

### Release note

LOCAL path validation errors no longer reveal expanded or resolved target filenames.

### Check List (For Author)

- Test: 4 focused BE tests passed with zero failures/skips, including external wildcard, dangling symlink and error redaction assertions. Modified lines pass clang-format 16; independent Standards and Spec review found no remaining issue in this supplement.
- Behavior changed: Yes, target-validation errors use a generic message.
- Does this need documentation: Covered by the PR release note.
@OIiveirra OIiveirra changed the title [fix](tvf) Restrict local paths and require explicit startup opt-in [fix](tvf) Secure LOCAL TVF access Oct 23, 2026
OIiveirra added a commit to OIiveirra/doris that referenced this pull request Oct 23, 2026
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69526

Problem Summary:
S3 Lance could open a local dataset through BrokerProperties because the
entry point rejected only LocalProperties. Validate the final dataset URI
before loading metadata. Allow the pinned SDK's remote providers and reject
local aliases and malformed authorities that could fall back to local paths.
Keep valid ports, underscore buckets, hostless GooseFS and normalized S3/OSS
HTTP locations working without changing catalog or non-Lance paths.

### Release note

S3 Lance rejects local dataset URIs and malformed remote URI authorities
before opening a dataset, regardless of storage-property selection.

### Check List (For Author)

- Test: 31 focused FE unit tests passed on this branch; changed Java
  files pass Checkstyle and git diff --check passes.
- Behavior changed: Yes. S3 Lance requires a valid supported remote URI.
- Does this need documentation: No additional documentation for this
  security validation; the parent PR's configuration documentation is pending.
OIiveirra added a commit to OIiveirra/doris that referenced this pull request Oct 23, 2026
### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#69526

Problem Summary:
S3 Lance could open a local dataset through BrokerProperties because the
entry point rejected only LocalProperties. Validate the final dataset URI
before loading metadata. Allow the pinned SDK's remote providers and reject
local aliases and malformed authorities that could fall back to local paths.
Keep valid ports, underscore buckets, hostless GooseFS and normalized S3/OSS
HTTP locations working without changing catalog or non-Lance paths.

### Release note

S3 Lance rejects local dataset URIs and malformed remote URI authorities
before opening a dataset, regardless of storage-property selection.

### Check List (For Author)

- Test: 31 focused FE unit tests passed on this branch; changed Java
  files pass Checkstyle and git diff --check passes.
- Behavior changed: Yes. S3 Lance requires a valid supported remote URI.
- Does this need documentation: No additional documentation for this
  security validation; the parent PR's configuration documentation is pending.
@OIiveirra

Copy link
Copy Markdown
Collaborator Author

run buildall

@OIiveirra

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review of PR #69526 at d353b6d, covering all 16 changed files and related SQL-to-BE discovery/read and LOCAL write-sink flows. Two P2 correctness issues need attention: canonical-root glob syntax breaks some valid enabled reads, and the new LOCAL disable switch does not cover INSERT INTO LOCAL. No existing inline threads were present, and no additional user focus was supplied.

Critical checkpoint conclusions:

  1. Goal and proof: read-side startup rejection precedes authorization, property parsing, backend selection and discovery. Canonical target containment and generic target-resolution errors address the stated discovery problem. The two inline findings identify remaining enabled-read and disable-switch gaps.
  2. Scope and reuse: changes are focused and reuse existing config, URI conversion and containment helpers; a shared LOCAL startup check would cover the separate sink path.
  3. Concurrency and locks: no new production threads, mutable shared state or lock ordering. Glob work runs on the existing heavy-work pthread pool; no new blocking under metadata locks was found.
  4. Lifecycle and initialization: no new resource ownership, caches, cycles or cross-TU initialization dependencies. Test fixtures restore config; production config is loaded before serving requests.
  5. Configuration: enable_local_tvf defaults false and is startup-only because ConfField.mutable defaults false and ConfigBase rejects runtime changes. Opt-in is per FE; fe_custom.conf can override fe.conf. All seven changed regression deployment configs explicitly enable it.
  6. Compatibility: default-off reads are an intentional operator-visible change documented in the PR release note; every query-serving FE needs configuration and restart. The canonical-root glob regression is unintended. No wire/storage-format or function-symbol changes were found.
  7. Thrift/protobuf: no definitions or field/enum IDs changed.
  8. Parallel paths: LOCAL/FILE-local SELECT, schema discovery, DESCRIBE and ordinary EXPLAIN share the read constructor; remote FILE storage stays independent. Parquet metadata wildcard expansion also delegates to LOCAL, while its exact-path behavior is unchanged. INSERT LOCAL uses the separate unguarded sink chain described inline.
  9. Conditions and errors: new BE checks have concrete failure comments and checked Status propagation; failed/outside matched targets return generic errors. Glob RPC suppresses partial file results on failure. FE rejection preserves the existing exception family and gives restart instructions.
  10. Test coverage: BE cases cover traversal glob classes, sibling-prefix escapes, symlinks, broken targets, relative/trailing-slash/symlink roots, invalid roots and helper URI/NUL inputs. FE/Groovy cases cover read gate ordering, ADMIN/OPERATOR authorization, FILE-local delegation, remote FILE dispatch and SELECT/EXPLAIN. Missing cases are metacharacters introduced by canonical roots and disabled INSERT LOCAL. Standard pipeline configs run enabled, so disabled SQL integration additionally needs a default-off deployment.
  11. Test results: no expected-result files changed or were handwritten in this diff; new Groovy error assertions are appropriate. Tests were inspected, not executed.
  12. Observability: the disabled error, generic target rejection and existing discovery/RPC context suffice for these guards; no new metrics or high-volume logging are needed.
  13. Transactions and persistence: no edit-log, replay, failover, visible-version or delete-bitmap changes.
  14. Writes and atomicity: the production diff changes read admission/discovery, with no new write protocol. The supported existing LOCAL sink remains reachable despite the new off setting, as the second finding explains.
  15. FE/BE transmission: no new transmitted state or protocol field; gate stays on FE and checked absolute paths continue through existing file statuses/ranges to BE readers.
  16. Performance and memory: the FE guard is constant time. BE retains linear per-match work and adds root canonicalization/stat calls; no new significant persistent allocation, ownership leak or asymptotic regression was identified. No performance measurement was run.
  17. Other concerns: user documentation for the new startup opt-in/upgrade behavior remains pending in the PR description. No additional substantiated issue or unresolved static candidate remains beyond the two inline findings.

Validation and limits: the pure-text build-support/check-build-hygiene.sh passed header layering/budgets, extern-template pairing and unity-skip coverage. No source modifications, builds, unit/regression execution, cluster reads or binary ABI audit were performed; author-reported execution results were not independently reproduced. The conclusions above are based on static control-flow and diff evidence.

Review converged after 2 round(s).

}
std::string secure_path;
RETURN_IF_ERROR(canonicalize(config::user_files_secure_path, &secure_path));
std::string full_path = secure_path + "/" + path;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Escape the canonical root before glob expansion

A configured root such as /srv/user-files can be a symlink to /mnt/user_files[prod]. With enable_local_tvf=true and file_path=*.csv, this change sends /mnt/user_files[prod]/*.csv to glob(), which interprets [prod] as a character class and returns no matches even when inside.csv exists. Previously glob used the literal /srv/user-files prefix and the read succeeded. Keep the canonical root for containment, but escape its glob metacharacters before prefixing the user pattern, and add a symlink-root regression case.

private boolean sharedStorage = false;

public LocalTableValuedFunction(Map<String, String> properties) throws AnalysisException {
if (!Config.enable_local_tvf) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Apply the LOCAL startup gate to the write sink too

With enable_local_tvf=false, an ADMIN can still run INSERT INTO local("file_path"="/tmp/local_optin_", "backend_id"="1", "format"="csv") SELECT 1 against a live backend and create a local CSV file. That path goes through InsertIntoTVFCommand -> BindSink.bindTVFTableSink -> TVFTableSink -> VTVFTableWriter and never constructs this class, so the new constructor check is bypassed. The config description and release note promise that LOCAL TVF is disabled by default, but currently only reads are disabled. Apply the startup check to local sink binding as well and cover INSERT/EXPLAIN INSERT with the flag off.

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28584 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d353b6d45bac9dd12a1b388d13d4f19e1e694994, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17594	5175	5053	5053
q2	2144	344	271	271
q3	10184	1377	828	828
q4	4676	453	332	332
q5	7421	790	529	529
q6	177	177	147	147
q7	749	785	597	597
q8	9240	1348	1082	1082
q9	5573	4185	4159	4159
q10	6812	1290	1030	1030
q11	426	245	230	230
q12	637	393	275	275
q13	18101	3533	2999	2999
q14	268	273	237	237
q15	q16	718	715	647	647
q17	1188	739	564	564
q18	6941	6166	6129	6129
q19	1113	955	595	595
q20	372	325	222	222
q21	5162	2375	2433	2375
q22	392	332	283	283
Total cold run time: 99888 ms
Total hot run time: 28584 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	5893	5879	5909	5879
q2	735	574	550	550
q3	5542	5741	5308	5308
q4	1981	2147	1559	1559
q5	5839	5756	5760	5756
q6	249	189	140	140
q7	2266	2005	1809	1809
q8	3003	2692	2644	2644
q9	7889	7522	7503	7503
q10	3664	3589	3187	3187
q11	585	399	371	371
q12	647	664	502	502
q13	3197	3577	3009	3009
q14	291	288	268	268
q15	q16	650	677	625	625
q17	7783	7114	6997	6997
q18	13236	12295	12964	12295
q19	898	773	781	773
q20	2119	2144	1910	1910
q21	5636	4571	4729	4571
q22	501	454	433	433
Total cold run time: 72604 ms
Total hot run time: 66089 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 149989 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d353b6d45bac9dd12a1b388d13d4f19e1e694994, data reload: false

query5	4312	579	439	439
query6	424	206	190	190
query7	4808	444	227	227
query8	326	163	150	150
query9	8759	3069	3046	3046
query10	487	297	258	258
query11	5863	3864	3570	3570
query12	147	97	89	89
query13	1237	435	316	316
query14	6022	3920	3651	3651
query14_1	3475	3382	3390	3382
query15	207	192	182	182
query16	936	404	348	348
query17	843	629	520	520
query18	2408	420	291	291
query19	188	157	129	129
query20	83	80	80	80
query21	207	122	104	104
query22	13047	12928	12791	12791
query23	13367	12800	12321	12321
query23_1	12599	12452	12548	12452
query24	7375	798	460	460
query24_1	464	456	458	456
query25	508	392	344	344
query26	1265	247	134	134
query27	2779	432	261	261
query28	4669	1894	1920	1894
query29	1544	603	463	463
query30	300	210	180	180
query31	1009	921	812	812
query32	136	89	89	89
query33	518	294	233	233
query34	901	829	492	492
query35	718	747	626	626
query36	820	796	707	707
query37	127	94	82	82
query38	1769	1743	1663	1663
query39	785	782	746	746
query39_1	728	730	736	730
query40	220	113	92	92
query41	65	62	62	62
query42	82	82	81	81
query43	341	368	311	311
query44	1295	672	676	672
query45	174	179	160	160
query46	831	953	553	553
query47	2952	2752	2723	2723
query48	285	298	211	211
query49	579	397	290	290
query50	682	283	206	206
query51	10763	10326	10391	10326
query52	78	80	73	73
query53	182	200	154	154
query54	241	199	175	175
query55	67	62	60	60
query56	215	208	218	208
query57	1631	1637	1559	1559
query58	284	260	252	252
query59	2157	2185	1883	1883
query60	302	245	225	225
query61	143	148	145	145
query62	383	348	284	284
query63	190	168	162	162
query64	2766	933	747	747
query65	3312	3276	3301	3276
query66	1770	415	299	299
query67	20198	20272	19955	19955
query68	2982	988	613	613
query69	368	295	255	255
query70	844	730	737	730
query71	299	242	210	210
query72	2739	2669	2401	2401
query73	505	524	294	294
query74	4643	4542	4346	4346
query75	2328	2315	1924	1924
query76	2318	1015	587	587
query77	342	373	277	277
query78	9826	9624	9157	9157
query79	885	842	516	516
query80	499	440	358	358
query81	536	328	278	278
query82	206	136	107	107
query83	199	208	189	189
query84	290	130	105	105
query85	838	493	440	440
query86	264	235	229	229
query87	1979	1952	1850	1850
query88	3553	2661	2645	2645
query89	375	283	242	242
query90	2081	167	158	158
query91	156	153	125	125
query92	94	84	79	79
query93	975	1010	563	563
query94	434	307	263	263
query95	572	323	308	308
query96	660	482	232	232
query97	2437	2420	2299	2299
query98	154	154	143	143
query99	757	758	653	653
Total cold run time: 233117 ms
Total hot run time: 149989 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.43 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d353b6d45bac9dd12a1b388d13d4f19e1e694994, data reload: false

query1	0.00	0.00	0.00
query2	0.10	0.05	0.04
query3	0.26	0.09	0.09
query4	1.61	0.10	0.10
query5	0.24	0.24	0.23
query6	1.18	0.69	0.68
query7	0.04	0.01	0.01
query8	0.06	0.04	0.04
query9	0.41	0.35	0.36
query10	0.60	0.62	0.60
query11	0.19	0.15	0.13
query12	0.17	0.14	0.15
query13	0.48	0.50	0.49
query14	1.04	1.01	1.01
query15	0.63	0.60	0.60
query16	0.35	0.36	0.35
query17	1.15	1.15	1.11
query18	0.20	0.19	0.18
query19	2.10	2.02	1.95
query20	0.02	0.02	0.01
query21	15.48	0.21	0.14
query22	4.92	0.06	0.06
query23	16.23	0.32	0.12
query24	2.67	0.57	0.58
query25	0.09	0.06	0.06
query26	1.66	0.22	0.16
query27	0.06	0.05	0.05
query28	4.51	0.84	0.37
query29	12.49	4.18	3.21
query30	0.29	0.13	0.10
query31	2.77	0.55	0.32
query32	3.22	0.59	0.51
query33	3.15	3.32	3.20
query34	15.45	4.01	3.37
query35	3.33	3.34	3.36
query36	0.56	0.45	0.41
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.03	0.03
query40	0.18	0.15	0.14
query41	0.09	0.02	0.02
query42	0.04	0.02	0.03
query43	0.04	0.04	0.03
Total cold run time: 98.24 s
Total hot run time: 24.43 s

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 77.13% (35906/46552)
Line Coverage 62.44% (405687/649772)
Region Coverage 58.85% (342566/582103)
Branch Coverage 59.83% (158459/264853)

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 0.00% (0/2) 🎉
Increment coverage report
Complete coverage report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants