Skip to content

[fix](lakesoul) Avoid logging credentials on branch-4.1 - #69473

Merged
yiguolei merged 1 commit into
apache:branch-4.1from
OIiveirra:fix/jira-sdbsec-273
Oct 23, 2026
Merged

yiguolei merged 1 commit into
apache:branch-4.1from
OIiveirra:fix/jira-sdbsec-273

Conversation

@OIiveirra

Copy link
Copy Markdown
Collaborator

What problem does this PR solve?

Issue Number: N/A (no public GitHub issue)

Problem Summary:

The retained LakeSoul scanner logs complete scanner parameters at INFO, including object storage credentials carried in options. The FE scan node also logs complete catalog properties at DEBUG.

Log only the query ID and batch size when opening the scanner, and remove the catalog-property dump. Authentication options continue to reach the reader unchanged.

Add focused regression tests and an explicit lakesoul-scanner-tests Maven profile so the deprecated scanner can be tested while remaining excluded from default builds.

Release note

Prevent sensitive LakeSoul scanner parameters and catalog properties from being written to logs.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:

    Validation on JDK 17:

    • LakeSoulJniScannerTest: 3 tests passed, covering INFO, DEBUG, and a missing query ID; credentials still reach the native reader.
    • LakeSoulScanNodeTest: 1 test passed, verifying credentials stay in scan options and out of DEBUG logs.
    • Restoring the original production logging makes all 4 tests fail on credential-leak assertions; restoring the fix makes all 4 pass.
    • Targeted Checkstyle: 0 violations. git diff --check passed.

    Native I/O is isolated with Mockito. No live LakeSoul SQL scan was run.

  • Behavior changed:

    • No.
    • Yes. Scanner startup logs contain the query ID and batch size instead of the full parameters; FE no longer logs complete catalog properties.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

Replace the complete scanner parameter dump with query ID and batch size,
and remove the FE catalog-property dump. Preserve authentication options.

Add focused log-capture regressions and an opt-in test profile for the
retained LakeSoul scanner without enabling it in default builds.
Copilot AI balanced review requested due to automatic review settings October 22, 2026 14:03
@OIiveirra
OIiveirra requested a review from yiguolei as a code owner October 22, 2026 14:03
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@OIiveirra

Copy link
Copy Markdown
Collaborator Author

run buildall

@linrrzqqq

Copy link
Copy Markdown
Collaborator

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all seven changed files and the related FE -> Thrift options -> BE/JNI -> native-reader flow at b75a4de. No actionable new finding was identified. The scanner startup log now selects query ID and batch size, the FE catalog-property dump is removed, and authentication option forwarding is preserved.

The review converged after one round: both complete-scope reviewers and the separate test/runtime risk reviewer returned NO_NEW_VALUABLE_FINDINGS on the unchanged empty finding set. All candidate and risk questions were dismissed with evidence; no unresolved candidate or uncovered changed path remains. There were no existing inline findings to carry forward and no additional user-provided focus.

Critical-checkpoint conclusions:

  1. Goal and tests: the two credential-bearing map dumps are addressed. The three new scanner cases and one FE case check log suppression and preserved options at source level.
  2. Scope: the two production logging changes and their test/profile support are focused.
  3. Concurrency and locking: no new production thread, shared mutation, lock or lock-order dependency. Tests restore logger state in finally.
  4. Lifecycle and initialization: production native/Arrow ownership and cleanup are unchanged. Mock/appender cleanup and the mocked native-construction boundary were checked statically, including exact-version dependency source.
  5. Configuration: no runtime setting is added. The lakesoul-scanner-tests Maven profile is explicitly opt-in and preserves the normal module exclusion.
  6. Compatibility: no production signature, storage format or options serialization change requires rolling-upgrade handling.
  7. Shared definitions: no Thrift/protobuf definition changes; field-ID comparison is not applicable.
  8. Parallel paths: both FE and scanner logging sites and related range/predicate/native transport paths were inspected; no additional issue introduced by this patch was found.
  9. Conditions: production option/read-mode/null gates are unchanged. The scanner test includes missing query_id.
  10. Coverage: tests inspect INFO/DEBUG logging, unknown future parameters, signed-path tokens in scanner params, credential propagation and unchanged input maps. Native IO is mocked; no live SQL/integration coverage is claimed.
  11. Expected results: no regression output file changes. The new assertions match the inspected logging and option-mapping behavior; author-reported test passes were not rerun by this review.
  12. Observability: query ID and batch size retain scanner correlation context. This focused logging fix needs no new metric.
  13. Transactions and persistence: no commit/visibility, EditLog, replay or failover behavior changes.
  14. Writes and crash behavior: no data mutation, atomicity, resource-ownership or crash-recovery behavior changes.
  15. FE/BE variables: wire values and credential propagation are unchanged; query_id was already supplied by the BE runtime.
  16. Performance: removal of whole-map formatting reduces logging work; no new production loop, allocation pattern or heavy locked operation was introduced.
  17. Other concerns: logger identity/capture, native mocking, fresh-range population order, test dependency/discovery and default-versus-opt-in packaging were checked without finding a substantiated regression.

Validation limit: this was a static-only review. No builds, tests, native execution or end-to-end SQL scans were run, and repository source files were not modified. Review convergence does not assert runtime validation.

Review converged after 1 round(s).

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 100% (0/0) 🎉
Increment coverage report
Complete coverage report

@yiguolei
yiguolei merged commit f5905b5 into apache:branch-4.1 Oct 23, 2026
32 checks passed
yiguolei pushed a commit that referenced this pull request Oct 24, 2026
### What problem does this PR solve?

Issue Number: N/A (no public GitHub issue)

Related PR: #69473

Problem Summary:

Port the LakeSoul diagnostic logging fix to `branch-4.2`. The retained
scanner logs complete parameters at INFO, including object storage
credentials carried in `options`. The FE scan node also logs complete
catalog properties at DEBUG.

Log only the query ID and batch size when opening the scanner, and
remove the catalog-property dump. Authentication options continue to
reach the reader unchanged.

Add the same focused unit regressions and an explicit
`lakesoul-scanner-tests` Maven profile. The deprecated scanner remains
excluded from default builds, and normal LakeSoul catalog creation
remains disabled.

### Release note

Prevent sensitive LakeSoul scanner parameters and catalog properties
from being written to logs.

### Check List (For Author)

- Test
    - [ ] Regression test
    - [x] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:

Validation on branch-4.2 with JDK 17 and a matching Thrift 0.24.0
generator:
- `LakeSoulJniScannerTest`: 3 tests passed, covering INFO, DEBUG, and a
missing query ID; credentials still reach the native reader.
- `LakeSoulScanNodeTest`: 1 test passed, verifying credentials stay in
scan options and out of DEBUG logs.
- All 4 tests fail on credential-leak assertions with the original 4.2
production logging, and pass with this fix. Fresh Surefire XML reports
show no skipped tests or errors.
  - Targeted Checkstyle: 0 violations. `git diff --check` passed.

Native I/O is isolated with Mockito. No live LakeSoul SQL scan was run;
the branch rejects LakeSoul catalog creation.

- Behavior changed:
    - [ ] No.
- [x] Yes. Scanner startup logs contain the query ID and batch size
instead of the full parameters; FE no longer logs complete catalog
properties.

- Does this need documentation?
    - [x] No.
    - [ ] Yes.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants