Repository navigation
Conversation
### What problem does this PR solve?
Problem Summary: At verbose log levels, FragmentMgr formats complete pipeline parameters and external query plans. File scan properties, table descriptors and transaction parameters can contain storage credentials, passwords or tokens. Replace these dumps with a shared summary of query and fragment identifiers and instance counts, and keep only identifiers in the external open log. Execution parameters remain unchanged.
### Release note
Verbose fragment execution logs no longer include full execution parameters, query options or external query plans that may contain credentials.
### Check List (For Author)
- Test: clang-format 16 and whitespace checks passed; focused BE unit tests and ASAN build are in progress
- Unit Test: add synthetic credential coverage and capture both pipeline and external request logs
- Behavior changed: Yes; verbose diagnostics contain identifiers and counts instead of full parameter dumps
- Does this need documentation: No
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
|
/review |
There was a problem hiding this comment.
Static review found no new actionable issues at head 3fa38be. All five changed files, both FragmentMgr entry paths, the shared formatter, test registration, and relevant upstream/downstream paths were reviewed. The credential-bearing request, query-options, and external-plan dumps are replaced with fixed numeric identifiers and instance counts without changing execution inputs. No existing inline findings require retention; no additional user focus was supplied.
Critical checkpoint conclusions:
- Goal and evidence: the numeric allowlist addresses the targeted credential exposure. Five new tests cover identifiers/counts, known and unknown property keys, JDBC passwords, transaction tokens, optional defaults, immutability, and actual entry-path log capture.
- Scope and clarity: three logging replacements and one reusable const formatter keep the change focused.
- Concurrency: no production shared state or locking is added. The test sink protects captured text with a mutex and does not log while holding it; shutdown joins background work before sink removal.
- Lifecycle and static initialization: no new retained production state or cross-translation-unit initializer dependency. The fixture stops/destroys FragmentMgr and restores its logging and debug-point settings.
- Configuration: no production configuration added or changed.
- Compatibility: diagnostic text changes intentionally; execution interfaces, wire/storage formats, and rolling-upgrade behavior are unchanged.
- Thrift/protobuf definitions: none modified, so cross-release field-ID comparison is inapplicable.
- Parallel paths: internal exec/prepare, transaction callbacks, and external execution reach the updated common logging path. Recursive rebuild directly reuses saved inputs without a full-request dump; both external log sites were checked separately.
- Conditional logic: no new production condition; generated defaults support absent optional fields and empty instance lists.
- Test coverage: static control-flow inspection confirms both request tests reach the intended logs before the existing injected error. CMake includes both test files; successful query execution is deliberately outside these focused tests.
- Test results: no expected-output files changed. Expected strings agree with the local ID formatter; the author's reported runtime results were not independently rerun.
- Observability: query/fragment identifiers, external instance ID, and local instance counts remain available at the existing verbosity levels; new metrics are unnecessary for this diagnostic replacement.
- Transactions and persistence: no transaction state, EditLog, visibility, publish, or failover behavior is modified.
- Writes and crash safety: the formatter performs const reads only; no new write, atomicity, recovery, or ownership behavior.
- FE-to-BE variables: no transmitted variable added or changed.
- Performance and memory: bounded formatting replaces request-sized serialization, preserves lazy VLOG evaluation, and introduces no significant allocation or retained buffer.
- Other invariants: no change to version/delete-bitmap consistency, nullable-column handling, or error propagation was identified.
Validation was static only, as required: no builds, test runs, source modifications, or submodule initialization. Runtime/sanitizer behavior remains unverified by this review.
Scope boundary: the separate HTTP stream-load handler still contains a pre-existing full-parameter VLOG_NOTICE at be/src/service/http/action/stream_load.cpp:832-833, unchanged from the supplied base. This FragmentMgr-focused fix does not establish credential-safe logging across all BE services; that unchanged site is not reported as a new inline defect.
Review converged after 2 round(s).
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
What problem does this PR solve?
Related PR: #69540
Problem Summary: FragmentMgr prints complete pipeline execution parameters at VLOG(10), including file-scan properties, descriptor passwords and transaction tokens. The external open path also prints a complete query plan at VLOG(1). Replace these dumps with query/fragment identifiers and instance counts without modifying execution parameters.
Release note
Verbose fragment execution logs no longer include full execution parameters, query options or external query plans that may contain credentials.
Evidence
FragmentParamsLogTest.*:FragmentMgrLogTest.*: 5 executed, 5 passed, 0 failed, 0 skipped (Clang21/ASAN). Fresh GTest XML and unchanged source inputs were checked.Check List (For Author)
Merge Danger
Door: two-way
Blast Radius: diagnostics
Execution and protocol parameters are unchanged; only verbose log contents change.
Check List (For Reviewer who merge this PR)