Skip to content

[fix](be) Keep credentials out of fragment execution logs (branch-4.2) - #69543

Open
OIiveirra wants to merge 1 commit into
apache:branch-4.2from
OIiveirra:fix/jira-sdbsec-254-4.2
Open

OIiveirra wants to merge 1 commit into
apache:branch-4.2from
OIiveirra:fix/jira-sdbsec-254-4.2

Conversation

@OIiveirra

@OIiveirra OIiveirra commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

What problem does this PR solve?

Related PR: #69540

Problem Summary: FragmentMgr prints complete pipeline execution parameters at VLOG(10), including file-scan properties, descriptor passwords and transaction tokens. The external open path also prints a complete query plan at VLOG(1). Replace these dumps with query/fragment identifiers and instance counts without modifying execution parameters.

- Log complete execution parameters and external query plans
+ Log fixed identifiers and instance counts

Release note

Verbose fragment execution logs no longer include full execution parameters, query options or external query plans that may contain credentials.

Evidence

  • FragmentParamsLogTest.*:FragmentMgrLogTest.*: 5 executed, 5 passed, 0 failed, 0 skipped (Clang21/ASAN). Fresh GTest XML and unchanged source inputs were checked.
  • Tests cover synthetic S3/token/password values, unknown property keys, unchanged execution parameters, and real pipeline/external request log capture with VLOG(10) enabled.
  • clang-format 16 and whitespace checks passed.
  • Task-private build selection retained the two requested test files, native initialization/link support, and complete branch BE libraries. This validates the selected tests; full native UT and regression groups were not run.

Check List (For Author)

  • Test:
    • Unit Test: the five focused fragment log tests above.
  • Behavior changed:
    • Yes. Verbose diagnostics contain fixed identifiers and counts instead of full parameter dumps.
  • Does this need documentation?
    • No.

Merge Danger

Door: two-way

Blast Radius: diagnostics

Execution and protocol parameters are unchanged; only verbose log contents change.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

### What problem does this PR solve?

Problem Summary: At verbose log levels, FragmentMgr formats complete pipeline parameters and external query plans. File scan properties, table descriptors and transaction parameters can contain storage credentials, passwords or tokens. Replace these dumps with a shared summary of query and fragment identifiers and instance counts, and keep only identifiers in the external open log. Execution parameters remain unchanged.

### Release note

Verbose fragment execution logs no longer include full execution parameters, query options or external query plans that may contain credentials.

### Check List (For Author)

- Test: clang-format 16 and whitespace checks passed; focused BE unit tests and ASAN build are in progress
    - Unit Test: add synthetic credential coverage and capture both pipeline and external request logs
- Behavior changed: Yes; verbose diagnostics contain identifiers and counts instead of full parameter dumps
- Does this need documentation: No
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@OIiveirra
OIiveirra marked this pull request as ready for review October 23, 2026 14:03
@OIiveirra
OIiveirra requested a review from yiguolei as a code owner October 23, 2026 14:03
Copilot AI balanced review requested due to automatic review settings October 23, 2026 14:03
@OIiveirra

Copy link
Copy Markdown
Collaborator Author

run buildall

@OIiveirra

Copy link
Copy Markdown
Collaborator Author

/review

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No actionable new findings in the five changed files at head 01ffe8c. No existing inline findings require carry-forward. No additional user focus was supplied.

Critical checkpoint conclusions:

  1. Goal and evidence: all three FragmentMgr full request/options/external-plan logging sites now emit only numeric identifiers and instance counts. The formatter does not read credential-bearing strings or mutate execution parameters. Five added tests cover summaries, unknown credential properties, defaults, immutability and actual internal/external logging sites; inspected statically only.
  2. Scope: the shared const formatter and logging replacements are small and focused.
  3. Concurrency: request-local reads add no shared state or locks; the test sink protects its buffer with a mutex. Existing dispatch/map locking remains intact.
  4. Lifecycle/static initialization: no production ownership or static initialization changes. Tests stop/join the manager before removing the sink and restore logging/debug-point state; native harness initialization and ExecEnv teardown were traced.
  5. Configuration: no production configuration added or changed.
  6. Compatibility: no wire/storage format or existing interface changes; helper declaration, includes and source collection are consistent with this branch.
  7. Thrift/protobuf: no definitions modified; existing field types and consumers were checked.
  8. Parallel paths: normal/prepare RPC, callback, transaction stream-load and external-open paths reach the sanitized common overload; the external pre-construction log is also replaced.
  9. Conditions/error handling: no production branches or Status propagation changed; default/absent fields do not require indexing or credential-specific filtering.
  10. Test coverage: both files are included by the native test glob. Site-level tests assert the intended early debug-point failure and log contents; they do not exercise later preparation/execution.
  11. Expected results: no golden files changed; exact ID/count assertions agree with the formatter.
  12. Observability: query/fragment correlation and instance count remain available; external instance correlation is retained. Removing full diagnostic payloads is intentional; no new metrics are needed.
  13. Transactions/persistence: no commit, EditLog, version, delete-bitmap or failover logic changes.
  14. Writes/atomicity/crashes: no execution-data mutation or write-order change; committed-data visibility and memory ownership invariants are unaffected.
  15. FE/BE variables: no transmitted variable or sending path changed.
  16. Performance/memory: bounded fixed-field formatting replaces recursive full-object serialization; disabled VLOG evaluation remains lazy, with no significant retained allocation.
  17. Other risks: upstream/downstream dumps, shared parent parameters, external descriptor creation and fixture global-state interactions were checked and dispositioned; no unresolved in-scope candidate remains.

Coverage limits: this was a static review; no build or test was run, and generated-header compilation was not verified. A separate pre-existing HTTP stream-load parameter dump remains outside these edited FragmentMgr logging sites, so this review does not establish a repository-wide absence of sensitive logs. All changed-file coverage and the two identified risk questions are complete.

Review converged after 2 round(s).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants