Skip to content

[fix](hudi) Keep storage credentials out of scanner logs (branch-3.1) - #69481

Closed
OIiveirra wants to merge 1 commit into
apache:branch-3.1from
OIiveirra:fix/jira-sdbsec-256-3.1
Closed

OIiveirra wants to merge 1 commit into
apache:branch-3.1from
OIiveirra:fix/jira-sdbsec-256-3.1

Conversation

@OIiveirra

Copy link
Copy Markdown
Collaborator

What problem does this PR solve?

Issue Number: N/A

Related PR: #69478 (backport to branch-3.1)

Problem Summary: When a Hudi JNI scan uses explicit S3A credentials, enabling scanner DEBUG logging prints every parameter value, including access keys, secret keys and session tokens. Log parameter names without values and record the query ID separately so scans remain traceable. The Hadoop configuration used to read data is unchanged.

Add a regression test that captures the real scanner's DEBUG output, checks that parameter names and the query ID remain visible, rejects ordinary and bucket-specific credentials and custom tokens, and verifies that Hadoop properties retain their original values.

Release note

Hudi JNI scanner debug logs no longer include storage credential values and continue to include query IDs for diagnostics.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test
    • No need to test or manual test.

    Validation with JDK 17:

    • ./run-fe-ut.sh --run 'org.apache.doris.hudi.HadoopHudiJniScannerLogTest': 1 test passed.
    • The log regression test verifies query ID retention, credential-free DEBUG output and unchanged Hadoop properties.
    • The test captures this branch's Log4j/reload4j backends.
    • Maven validate / Checkstyle for be-java-extensions/hadoop-hudi-scanner and its dependencies, plus git diff --check: passed.
    • The logging regression is covered directly by JUnit. FE/BE + S3 end-to-end regression was not run locally; SQL planning, credential propagation and data-reading logic are unchanged.
  • Behavior changed:

    • No.
    • Yes. DEBUG logs retain parameter names and query IDs without printing other parameter values.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: Hudi JNI scanners logged every input parameter value at DEBUG,
including explicit S3A access keys, secret keys and session tokens. Log parameter
names and the query ID separately while preserving the Hadoop configuration used
to read data. Add a regression test that captures DEBUG output, checks that query
IDs remain available, rejects bucket-specific and custom credential values, and
verifies that the scanner retains the original Hadoop properties.

### Release note

Hudi JNI scanner debug logs no longer record storage credential values and
continue to include query IDs for diagnostics.

### Check List (For Author)

- Test: Unit Test - the targeted Hudi parameter-log test passes. Maven validate / Checkstyle and
  git diff --check pass.
- Behavior changed: Yes - scanner debug logs show parameter names and the query
  ID without storage credentials.
- Does this need documentation: No
@OIiveirra
OIiveirra requested a review from morrySnow as a code owner October 22, 2026 17:32
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

Copilot AI balanced review requested due to automatic review settings October 22, 2026 17:32

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@OIiveirra

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Static review of PR #69481 at a422a63: no actionable findings.

Reviewed both changed files and the related Hudi JNI parameter, authentication/configuration, logging dependency, and Maven/JUnit integration paths. The DEBUG parameter loop now logs names without values, while the generated query ID remains available for diagnostics. Hadoop properties, authentication inputs, and reader setup remain unchanged. The regression test checks ordinary and bucket-specific S3A credentials, session tokens, custom values, query/parameter-name visibility, and exact retention of the Hadoop property map.

No existing inline findings require carry-forward, and no additional user focus was supplied. The release note matches the change; no documentation change appears necessary. Validation was static only: builds, unit tests, and FE/BE plus S3 integration were not executed. No unresolved actionable candidate or uncovered changed-file scope remains.

Review converged after 2 round(s).

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

yiguolei pushed a commit that referenced this pull request Oct 23, 2026
### What problem does this PR solve?

Issue Number: N/A

Related PR: #69479 (branch-4.2), #69480 (branch-4.1), #69481
(branch-3.1)

Problem Summary: When a Hudi JNI scan uses explicit S3A credentials,
enabling scanner DEBUG logging prints every parameter value, including
access keys, secret keys and session tokens. Log parameter names without
values and record the query ID separately so scans remain traceable. The
Hadoop configuration used to read data is unchanged.

Add a regression test that captures the real scanner's DEBUG output,
checks that parameter names and the query ID remain visible, rejects
ordinary and bucket-specific credentials and custom tokens, and verifies
that Hadoop properties retain their original values.

### Release note

Hudi JNI scanner debug logs no longer include storage credential values
and continue to include query IDs for diagnostics.

### Check List (For Author)

- Test
    - [ ] Regression test
    - [x] Unit Test
    - [ ] Manual test
    - [ ] No need to test or manual test.

  Validation with JDK 17:
- `./run-fe-ut.sh --run
'org.apache.doris.hudi.HadoopHudiJniScannerLogTest,org.apache.doris.hudi.HadoopHudiPluginTest,org.apache.doris.hudi.HadoopHudiColumnValueTest'`:
16 tests passed.
- The new log test fails on the original code because parameter values
appear in DEBUG output and passes after the fix.
- The existing plugin test writes and reads a local Hudi table through
the Java scanner.
- Maven `validate` / Checkstyle for
`be-java-extensions/hadoop-hudi-scanner` and its dependencies, plus `git
diff --check`: passed.
- The logging regression is covered directly by JUnit. FE/BE + S3
end-to-end regression was not run locally; SQL planning, credential
propagation and data-reading logic are unchanged.

- Behavior changed:
    - [ ] No.
- [x] Yes. DEBUG logs retain parameter names and query IDs without
printing other parameter values.

- Does this need documentation?
    - [x] No.
    - [ ] Yes.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
@OIiveirra

Copy link
Copy Markdown
Collaborator Author

run feut

@OIiveirra

Copy link
Copy Markdown
Collaborator Author

run p0

@OIiveirra

Copy link
Copy Markdown
Collaborator Author

run external

@OIiveirra

Copy link
Copy Markdown
Collaborator Author

run nonConcurrent

@OIiveirra OIiveirra closed this Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants