Skip to content

[fix](be) Keep credentials out of fragment execution logs (branch-4.1) - #69542

Open
OIiveirra wants to merge 1 commit into
apache:branch-4.1from
OIiveirra:fix/jira-sdbsec-276-4.1
Open

OIiveirra wants to merge 1 commit into
apache:branch-4.1from
OIiveirra:fix/jira-sdbsec-276-4.1

Conversation

@OIiveirra

@OIiveirra OIiveirra commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

What problem does this PR solve?

Related PR: #69540

Problem Summary: FragmentMgr prints complete pipeline execution parameters at VLOG(10), including file-scan properties, descriptor passwords and transaction tokens. The external open path also prints a complete query plan at VLOG(1). Replace these dumps with query/fragment identifiers and instance counts without modifying execution parameters.

- Log complete execution parameters and external query plans
+ Log fixed identifiers and instance counts

Release note

Verbose fragment execution logs no longer include full execution parameters, query options or external query plans that may contain credentials.

Evidence

  • FragmentParamsLogTest.*:FragmentMgrLogTest.*: 5 executed, 5 passed, 0 failed, 0 skipped (Clang21/ASAN). Fresh GTest XML and unchanged source inputs were checked.
  • Tests cover synthetic S3/token/password values, unknown property keys, unchanged execution parameters, and real pipeline/external request log capture with VLOG(10) enabled.
  • clang-format 16 and whitespace checks passed.
  • Task-private build selection retained the two requested test files, native initialization/link support, and complete branch BE libraries. This validates the selected tests; full native UT and regression groups were not run.

Check List (For Author)

  • Test:
    • Unit Test: the five focused fragment log tests above.
  • Behavior changed:
    • Yes. Verbose diagnostics contain fixed identifiers and counts instead of full parameter dumps.
  • Does this need documentation?
    • No.

Merge Danger

Door: two-way

Blast Radius: diagnostics

Execution and protocol parameters are unchanged; only verbose log contents change.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

### What problem does this PR solve?

Problem Summary: At verbose log levels, FragmentMgr formats complete pipeline parameters and external query plans. File scan properties, table descriptors and transaction parameters can contain storage credentials, passwords or tokens. Replace these dumps with a shared summary of query and fragment identifiers and instance counts, and keep only identifiers in the external open log. Execution parameters remain unchanged.

### Release note

Verbose fragment execution logs no longer include full execution parameters, query options or external query plans that may contain credentials.

### Check List (For Author)

- Test: clang-format 16 and whitespace checks passed; focused BE unit tests and ASAN build are in progress
    - Unit Test: add synthetic credential coverage and capture both pipeline and external request logs
- Behavior changed: Yes; verbose diagnostics contain identifiers and counts instead of full parameter dumps
- Does this need documentation: No
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@OIiveirra
OIiveirra marked this pull request as ready for review October 23, 2026 14:03
@OIiveirra
OIiveirra requested a review from yiguolei as a code owner October 23, 2026 14:03
Copilot AI balanced review requested due to automatic review settings October 23, 2026 14:03
@OIiveirra

Copy link
Copy Markdown
Collaborator Author

run buildall

@OIiveirra

Copy link
Copy Markdown
Collaborator Author

/review

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review found no new actionable issues at head 3fa38be. All five changed files, both FragmentMgr entry paths, the shared formatter, test registration, and relevant upstream/downstream paths were reviewed. The credential-bearing request, query-options, and external-plan dumps are replaced with fixed numeric identifiers and instance counts without changing execution inputs. No existing inline findings require retention; no additional user focus was supplied.

Critical checkpoint conclusions:

  • Goal and evidence: the numeric allowlist addresses the targeted credential exposure. Five new tests cover identifiers/counts, known and unknown property keys, JDBC passwords, transaction tokens, optional defaults, immutability, and actual entry-path log capture.
  • Scope and clarity: three logging replacements and one reusable const formatter keep the change focused.
  • Concurrency: no production shared state or locking is added. The test sink protects captured text with a mutex and does not log while holding it; shutdown joins background work before sink removal.
  • Lifecycle and static initialization: no new retained production state or cross-translation-unit initializer dependency. The fixture stops/destroys FragmentMgr and restores its logging and debug-point settings.
  • Configuration: no production configuration added or changed.
  • Compatibility: diagnostic text changes intentionally; execution interfaces, wire/storage formats, and rolling-upgrade behavior are unchanged.
  • Thrift/protobuf definitions: none modified, so cross-release field-ID comparison is inapplicable.
  • Parallel paths: internal exec/prepare, transaction callbacks, and external execution reach the updated common logging path. Recursive rebuild directly reuses saved inputs without a full-request dump; both external log sites were checked separately.
  • Conditional logic: no new production condition; generated defaults support absent optional fields and empty instance lists.
  • Test coverage: static control-flow inspection confirms both request tests reach the intended logs before the existing injected error. CMake includes both test files; successful query execution is deliberately outside these focused tests.
  • Test results: no expected-output files changed. Expected strings agree with the local ID formatter; the author's reported runtime results were not independently rerun.
  • Observability: query/fragment identifiers, external instance ID, and local instance counts remain available at the existing verbosity levels; new metrics are unnecessary for this diagnostic replacement.
  • Transactions and persistence: no transaction state, EditLog, visibility, publish, or failover behavior is modified.
  • Writes and crash safety: the formatter performs const reads only; no new write, atomicity, recovery, or ownership behavior.
  • FE-to-BE variables: no transmitted variable added or changed.
  • Performance and memory: bounded formatting replaces request-sized serialization, preserves lazy VLOG evaluation, and introduces no significant allocation or retained buffer.
  • Other invariants: no change to version/delete-bitmap consistency, nullable-column handling, or error propagation was identified.

Validation was static only, as required: no builds, test runs, source modifications, or submodule initialization. Runtime/sanitizer behavior remains unverified by this review.

Scope boundary: the separate HTTP stream-load handler still contains a pre-existing full-parameter VLOG_NOTICE at be/src/service/http/action/stream_load.cpp:832-833, unchanged from the supplied base. This FragmentMgr-focused fix does not establish credential-safe logging across all BE services; that unchanged site is not reported as a new inline defect.

Review converged after 2 round(s).

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 100.00% (9/9) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 61.50% (26994/43894)
Line Coverage 46.20% (281567/609448)
Region Coverage 41.81% (222704/532620)
Branch Coverage 43.35% (103496/238750)

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100.00% (9/9) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 75.12% (32066/42684)
Line Coverage 59.52% (359821/604566)
Region Coverage 56.21% (300005/533677)
Branch Coverage 57.09% (136121/238451)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants