Repository navigation
Conversation
### What problem does this PR solve? Issue Number: N/A Problem Summary: Add the startup-only enable_local_tvf setting, defaulting to false. Reject LOCAL TVF and FILE TVF with local storage before property parsing or backend access. Explicit opt-in restores each branch's existing behavior. Dedicated regression deployments opt in to retain LOCAL coverage. ### Release note LOCAL TVF and FILE TVF with local storage are disabled by default. Set enable_local_tvf=true in fe.conf on every query-serving FE and restart those FEs to restore the existing behavior. ### Check List (For Author) - Test: 11 focused FE unit tests passed with zero failures or skips; focused FE Checkstyle passed. SQL regression coverage added; runtime validation recorded separately. - Behavior changed: Yes. Local TVF access requires explicit startup opt-in. - Does this need documentation: Yes. Release note documents the default and opt-in.
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
### What problem does this PR solve? Issue Number: N/A Related PR: apache#69526 Problem Summary: Canonical glob results were not checked against the secure root, and LOCAL Lance bypassed local-path validation. Check the canonical root and resolved paths, enforce ADMIN/OPERATOR before file access, validate Lance schema and scan paths, and preserve LOCAL/S3 range types. Reject local storage selected through S3 Lance before dataset opening. ### Release note Enabled LOCAL reads remain restricted to user_files_secure_path and require ADMIN/OPERATOR. LOCAL Lance accepts only local paths inside that directory. The existing default-disabled startup gate is retained. ### Check List (For Author) - Test: 26 focused FE tests, Checkstyle and changed-line clang-format 16 passed. BE tests could not build because the local dependency package lacks libgoogle_cloud_cpp_oauth2.a. - Behavior changed: Yes, enforce secure local path containment and authorization after explicit opt-in. - Does this need documentation: Covered by the PR release note.
### What problem does this PR solve? Issue Number: N/A Related PR: N/A Problem Summary: Glob expansion and symlinks can produce canonical paths outside the configured secure directory. Resolve a nonempty secure root and each local result canonically before checking containment. The shared resolver rejects remote schemes and embedded NULs. Keep the default-disabled startup gate and existing local authorization. Bind the related glob test fixture to its own secure root across build profiles. ### Release note Enabled LOCAL reads remain restricted to user_files_secure_path after glob expansion and symlink resolution. ### Check List (For Author) - Test: 5 focused BE tests passed with zero failures/skips; changed-line clang-format 16 and build hygiene passed. BE tests used the documented private legacy-glibc link adapter with branch/toolchain inputs verified. FE source is unchanged by this supplement; its existing focused tests passed previously. - Behavior changed: Yes, enforce canonical secure-directory containment after explicit startup opt-in. - Does this need documentation: Covered by the PR release note.
### What problem does this PR solve? Issue Number: N/A Related PR: apache#69526 Problem Summary: FILE_LOCAL also identifies existing local Lance catalogs. Apply the new secure-root check only to the documented LOCAL TVF version-zero ranges, retaining fixed-version catalog warehouse paths. Add catalog compatibility coverage, qualify the nested DatasetKey type in the new tests, and bind the existing glob fixture to its own root across build profiles. ### Release note Fixed-version local Lance catalog reads keep their existing warehouse paths. ### Check List (For Author) - Test: changed-line clang-format 16 passed. Local BE execution remains blocked by missing GCS dependencies; targeted compilation diagnostics also identify an existing Lance-C SDK API mismatch. Existing final FE changes have 26 passing tests. - Behavior changed: Scope the new path check to LOCAL TVF. - Does this need documentation: Covered by the PR release note.
### What problem does this PR solve? Issue Number: N/A Related PR: apache#69526 Problem Summary: Canonical glob results were not checked against the secure root, and LOCAL Lance bypassed local-path validation. Validate canonical roots and resolved paths, check ADMIN/OPERATOR before file access, validate LOCAL Lance schema/scan paths, and preserve LOCAL/S3 range types. Limit Lance validation to the existing LOCAL TVF version-zero contract so fixed-version local catalogs retain their warehouse paths. Reject LocalProperties selected through S3 Lance before dataset opening. ### Release note Enabled LOCAL reads remain restricted to user_files_secure_path and require ADMIN/OPERATOR. LOCAL Lance accepts only local paths inside that directory. The default-disabled startup gate is retained. ### Check List (For Author) - Test: 26 focused FE tests, Java Checkstyle and changed-line clang-format 16 passed. BE execution is blocked by the installed Lance-C SDK missing the existing lance_scanner_set_distance_range API; no BE runtime pass is claimed. - Behavior changed: Yes, enforce secure local paths and authorization after explicit opt-in. - Does this need documentation: Covered by the PR release note.
### What problem does this PR solve? Issue Number: N/A Related PR: apache#69526 Problem Summary: Canonical glob results were not checked against the secure root. Validate canonical local paths against a nonempty canonical secure root, reject remote schemes in the shared local resolver, and enforce ADMIN/OPERATOR before local file access and during privilege checking. Keep the existing default-disabled startup gate. Give the existing glob fixture an explicit secure root so profile-specific test directories do not change its assumptions. ### Release note Enabled LOCAL reads remain restricted to user_files_secure_path and require ADMIN/OPERATOR privileges. ### Check List (For Author) - Test: 10 focused FE tests and 4 focused BE tests passed with zero failures/skips; Java Checkstyle and changed-line clang-format 16 passed. BE tests used the supported Clang16 ASAN profile. - Behavior changed: Yes, canonical secure-root containment and local authorization after explicit opt-in. - Does this need documentation: Covered by the PR release note.
### What problem does this PR solve? Issue Number: N/A Related PR: apache#69526 Problem Summary: Canonical glob results were not checked against the secure root. Validate canonical local paths against a nonempty canonical secure root, reject remote schemes in the shared local resolver, and enforce ADMIN/OPERATOR before local file access and during privilege checking. Keep the existing default-disabled startup gate. Give the existing glob fixture an explicit secure root so profile-specific test directories do not change its assumptions. ### Release note Enabled LOCAL reads remain restricted to user_files_secure_path and require ADMIN/OPERATOR privileges. ### Check List (For Author) - Test: 6 focused FE tests and 4 focused BE tests passed with zero failures/skips; Java Checkstyle and changed-line clang-format 16 passed. BE tests used the supported Clang16 ASAN profile. - Behavior changed: Yes, canonical secure-root containment and local authorization after explicit opt-in. - Does this need documentation: Covered by the PR release note.
### What problem does this PR solve? Issue Number: N/A Related PR: N/A Problem Summary: A rejected glob result could disclose its expanded filename through a canonical-path error or a failed resolution of a dangling symlink. Return one generic target-validation error for both cases before the status reaches SQL clients. Extend temporary-directory tests to assert that neither real target names nor paths appear in errors. ### Release note LOCAL path validation errors no longer reveal expanded or resolved target filenames. ### Check List (For Author) - Test: 5 focused BE tests passed with zero failures/skips, including external wildcard, dangling symlink and error redaction assertions. Modified lines pass clang-format 16; independent Standards and Spec review found no remaining issue in this supplement. - Behavior changed: Yes, target-validation errors use a generic message. - Does this need documentation: Covered by the PR release note.
### What problem does this PR solve? Issue Number: N/A Related PR: apache#69526 Problem Summary: A rejected glob result could disclose its expanded filename through a canonical-path error or a failed resolution of a dangling symlink. Return one generic target-validation error for both cases before the status reaches SQL clients. Extend temporary-directory tests to assert that neither real target names nor paths appear in errors. ### Release note LOCAL path validation errors no longer reveal expanded or resolved target filenames. ### Check List (For Author) - Test: Targeted compilation diagnostics passed for the two modified C++ files. Full BE execution remains blocked by the previously recorded Lance-C SDK/GCS dependency gaps. Modified lines pass clang-format 16; independent Standards and Spec review found no remaining issue in this supplement. - Behavior changed: Yes, target-validation errors use a generic message. - Does this need documentation: Covered by the PR release note.
### What problem does this PR solve? Issue Number: N/A Related PR: apache#69526 Problem Summary: A rejected glob result could disclose its expanded filename through a canonical-path error or a failed resolution of a dangling symlink. Return one generic target-validation error for both cases before the status reaches SQL clients. Extend temporary-directory tests to assert that neither real target names nor paths appear in errors. ### Release note LOCAL path validation errors no longer reveal expanded or resolved target filenames. ### Check List (For Author) - Test: Targeted compilation diagnostics passed for the two modified C++ files. Full BE execution remains blocked by the previously recorded Lance-C SDK/GCS dependency gaps. Modified lines pass clang-format 16; independent Standards and Spec review found no remaining issue in this supplement. - Behavior changed: Yes, target-validation errors use a generic message. - Does this need documentation: Covered by the PR release note.
### What problem does this PR solve? Issue Number: N/A Related PR: apache#69526 Problem Summary: A rejected glob result could disclose its expanded filename through a canonical-path error or a failed resolution of a dangling symlink. Return one generic target-validation error for both cases before the status reaches SQL clients. Extend temporary-directory tests to assert that neither real target names nor paths appear in errors. ### Release note LOCAL path validation errors no longer reveal expanded or resolved target filenames. ### Check List (For Author) - Test: 4 focused BE tests passed with zero failures/skips, including external wildcard, dangling symlink and error redaction assertions. Modified lines pass clang-format 16; independent Standards and Spec review found no remaining issue in this supplement. - Behavior changed: Yes, target-validation errors use a generic message. - Does this need documentation: Covered by the PR release note.
### What problem does this PR solve? Issue Number: N/A Related PR: apache#69526 Problem Summary: A rejected glob result could disclose its expanded filename through a canonical-path error or a failed resolution of a dangling symlink. Return one generic target-validation error for both cases before the status reaches SQL clients. Extend temporary-directory tests to assert that neither real target names nor paths appear in errors. ### Release note LOCAL path validation errors no longer reveal expanded or resolved target filenames. ### Check List (For Author) - Test: 4 focused BE tests passed with zero failures/skips, including external wildcard, dangling symlink and error redaction assertions. Modified lines pass clang-format 16; independent Standards and Spec review found no remaining issue in this supplement. - Behavior changed: Yes, target-validation errors use a generic message. - Does this need documentation: Covered by the PR release note.
### What problem does this PR solve? Issue Number: N/A Related PR: apache#69526 Problem Summary: S3 Lance could open a local dataset through BrokerProperties because the entry point rejected only LocalProperties. Validate the final dataset URI before loading metadata. Allow the pinned SDK's remote providers and reject local aliases and malformed authorities that could fall back to local paths. Keep valid ports, underscore buckets, hostless GooseFS and normalized S3/OSS HTTP locations working without changing catalog or non-Lance paths. ### Release note S3 Lance rejects local dataset URIs and malformed remote URI authorities before opening a dataset, regardless of storage-property selection. ### Check List (For Author) - Test: 31 focused FE unit tests passed on this branch; changed Java files pass Checkstyle and git diff --check passes. - Behavior changed: Yes. S3 Lance requires a valid supported remote URI. - Does this need documentation: No additional documentation for this security validation; the parent PR's configuration documentation is pending.
### What problem does this PR solve? Issue Number: N/A Related PR: apache#69526 Problem Summary: S3 Lance could open a local dataset through BrokerProperties because the entry point rejected only LocalProperties. Validate the final dataset URI before loading metadata. Allow the pinned SDK's remote providers and reject local aliases and malformed authorities that could fall back to local paths. Keep valid ports, underscore buckets, hostless GooseFS and normalized S3/OSS HTTP locations working without changing catalog or non-Lance paths. ### Release note S3 Lance rejects local dataset URIs and malformed remote URI authorities before opening a dataset, regardless of storage-property selection. ### Check List (For Author) - Test: 31 focused FE unit tests passed on this branch; changed Java files pass Checkstyle and git diff --check passes. - Behavior changed: Yes. S3 Lance requires a valid supported remote URI. - Does this need documentation: No additional documentation for this security validation; the parent PR's configuration documentation is pending.
|
run buildall |
|
/review |
There was a problem hiding this comment.
Static review of PR #69526 at d353b6d, covering all 16 changed files and related SQL-to-BE discovery/read and LOCAL write-sink flows. Two P2 correctness issues need attention: canonical-root glob syntax breaks some valid enabled reads, and the new LOCAL disable switch does not cover INSERT INTO LOCAL. No existing inline threads were present, and no additional user focus was supplied.
Critical checkpoint conclusions:
- Goal and proof: read-side startup rejection precedes authorization, property parsing, backend selection and discovery. Canonical target containment and generic target-resolution errors address the stated discovery problem. The two inline findings identify remaining enabled-read and disable-switch gaps.
- Scope and reuse: changes are focused and reuse existing config, URI conversion and containment helpers; a shared LOCAL startup check would cover the separate sink path.
- Concurrency and locks: no new production threads, mutable shared state or lock ordering. Glob work runs on the existing heavy-work pthread pool; no new blocking under metadata locks was found.
- Lifecycle and initialization: no new resource ownership, caches, cycles or cross-TU initialization dependencies. Test fixtures restore config; production config is loaded before serving requests.
- Configuration: enable_local_tvf defaults false and is startup-only because ConfField.mutable defaults false and ConfigBase rejects runtime changes. Opt-in is per FE; fe_custom.conf can override fe.conf. All seven changed regression deployment configs explicitly enable it.
- Compatibility: default-off reads are an intentional operator-visible change documented in the PR release note; every query-serving FE needs configuration and restart. The canonical-root glob regression is unintended. No wire/storage-format or function-symbol changes were found.
- Thrift/protobuf: no definitions or field/enum IDs changed.
- Parallel paths: LOCAL/FILE-local SELECT, schema discovery, DESCRIBE and ordinary EXPLAIN share the read constructor; remote FILE storage stays independent. Parquet metadata wildcard expansion also delegates to LOCAL, while its exact-path behavior is unchanged. INSERT LOCAL uses the separate unguarded sink chain described inline.
- Conditions and errors: new BE checks have concrete failure comments and checked Status propagation; failed/outside matched targets return generic errors. Glob RPC suppresses partial file results on failure. FE rejection preserves the existing exception family and gives restart instructions.
- Test coverage: BE cases cover traversal glob classes, sibling-prefix escapes, symlinks, broken targets, relative/trailing-slash/symlink roots, invalid roots and helper URI/NUL inputs. FE/Groovy cases cover read gate ordering, ADMIN/OPERATOR authorization, FILE-local delegation, remote FILE dispatch and SELECT/EXPLAIN. Missing cases are metacharacters introduced by canonical roots and disabled INSERT LOCAL. Standard pipeline configs run enabled, so disabled SQL integration additionally needs a default-off deployment.
- Test results: no expected-result files changed or were handwritten in this diff; new Groovy error assertions are appropriate. Tests were inspected, not executed.
- Observability: the disabled error, generic target rejection and existing discovery/RPC context suffice for these guards; no new metrics or high-volume logging are needed.
- Transactions and persistence: no edit-log, replay, failover, visible-version or delete-bitmap changes.
- Writes and atomicity: the production diff changes read admission/discovery, with no new write protocol. The supported existing LOCAL sink remains reachable despite the new off setting, as the second finding explains.
- FE/BE transmission: no new transmitted state or protocol field; gate stays on FE and checked absolute paths continue through existing file statuses/ranges to BE readers.
- Performance and memory: the FE guard is constant time. BE retains linear per-match work and adds root canonicalization/stat calls; no new significant persistent allocation, ownership leak or asymptotic regression was identified. No performance measurement was run.
- Other concerns: user documentation for the new startup opt-in/upgrade behavior remains pending in the PR description. No additional substantiated issue or unresolved static candidate remains beyond the two inline findings.
Validation and limits: the pure-text build-support/check-build-hygiene.sh passed header layering/budgets, extern-template pairing and unity-skip coverage. No source modifications, builds, unit/regression execution, cluster reads or binary ABI audit were performed; author-reported execution results were not independently reproduced. The conclusions above are based on static control-flow and diff evidence.
Review converged after 2 round(s).
| } | ||
| std::string secure_path; | ||
| RETURN_IF_ERROR(canonicalize(config::user_files_secure_path, &secure_path)); | ||
| std::string full_path = secure_path + "/" + path; |
There was a problem hiding this comment.
[P2] Escape the canonical root before glob expansion
A configured root such as /srv/user-files can be a symlink to /mnt/user_files[prod]. With enable_local_tvf=true and file_path=*.csv, this change sends /mnt/user_files[prod]/*.csv to glob(), which interprets [prod] as a character class and returns no matches even when inside.csv exists. Previously glob used the literal /srv/user-files prefix and the read succeeded. Keep the canonical root for containment, but escape its glob metacharacters before prefixing the user pattern, and add a symlink-root regression case.
| private boolean sharedStorage = false; | ||
|
|
||
| public LocalTableValuedFunction(Map<String, String> properties) throws AnalysisException { | ||
| if (!Config.enable_local_tvf) { |
There was a problem hiding this comment.
[P2] Apply the LOCAL startup gate to the write sink too
With enable_local_tvf=false, an ADMIN can still run INSERT INTO local("file_path"="/tmp/local_optin_", "backend_id"="1", "format"="csv") SELECT 1 against a live backend and create a local CSV file. That path goes through InsertIntoTVFCommand -> BindSink.bindTVFTableSink -> TVFTableSink -> VTVFTableWriter and never constructs this class, so the new constructor check is bypassed. The config description and release note promise that LOCAL TVF is disabled by default, but currently only reads are disabled. Apply the startup check to local sink binding as well and cover INSERT/EXPLAIN INSERT with the flag off.
TPC-H: Total hot run time: 28584 ms |
TPC-DS: Total hot run time: 149989 ms |
ClickBench: Total hot run time: 24.43 s |
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
FE Regression Coverage ReportIncrement line coverage |
What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary:
Glob expansion could return paths outside user_files_secure_path because
the expanded targets were not checked against a nonempty canonical
secure root.
startup-only enable_local_tvf setting.
property parsing, backend selection and file discovery.
checking directory containment. Reject empty secure roots, remote
schemes and invalid local paths. Return generic errors for failed or
out-of-root target resolution, including dangling symlinks.
retain their LOCAL test coverage.
Release note
LOCAL TVF and FILE TVF with local storage are disabled by default. To
use them, set enable_local_tvf=true in fe.conf on every query-serving FE
and restart those FEs. The setting cannot be changed at runtime. Enabled
reads still require ADMIN/OPERATOR and must remain within the configured
BE user_files_secure_path. Errors no longer disclose expanded or
resolved target names.
Check List (For Author)
changed.
Local validation:
11 focused FE unit tests and 5 focused BE unit tests passed with zero
failures or skips. These earlier results still match the current
source; the BE link used a private legacy-glibc adapter.
34 real SQL checks passed on a private FE built from this PR. They
cover default-disable and restart opt-in, startup-only configuration,
ADMIN/ordinary users, LOCAL/FILE and SELECT/EXPLAIN. No BE was
registered, so these validate FE policy and rejection paths.
The original Groovy suites have not completed: their JDBC
initialization requires a BE. Full Groovy and end-to-end file reads
remain unverified locally.
Relevant Java Checkstyle, changed-line clang-format 16 and git diff
--check passed.
Behavior changed:
enabled reads enforce authorization and secure-root containment.
Does this need documentation?
upgrade behavior. Documentation PR pending; the release note
above does not replace the user documentation.
Check List (For Reviewer who merge this PR)