Repository navigation
Conversation
### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: Hudi JNI scanners logged every input parameter value at DEBUG, including explicit S3A access keys, secret keys and session tokens. Log parameter names and the query ID separately while preserving the Hadoop configuration used to read data. Add a regression test that captures DEBUG output, checks that query IDs remain available, rejects bucket-specific and custom credential values, and verifies that the scanner retains the original Hadoop properties. ### Release note Hudi JNI scanner debug logs no longer record storage credential values and continue to include query IDs for diagnostics. ### Check List (For Author) - Test: Unit Test - the targeted Hudi parameter-log test passes. Maven validate / Checkstyle and git diff --check pass. - Behavior changed: Yes - scanner debug logs show parameter names and the query ID without storage credentials. - Does this need documentation: No
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
/review |
There was a problem hiding this comment.
Static review of PR #69481 at a422a63: no actionable findings.
Reviewed both changed files and the related Hudi JNI parameter, authentication/configuration, logging dependency, and Maven/JUnit integration paths. The DEBUG parameter loop now logs names without values, while the generated query ID remains available for diagnostics. Hadoop properties, authentication inputs, and reader setup remain unchanged. The regression test checks ordinary and bucket-specific S3A credentials, session tokens, custom values, query/parameter-name visibility, and exact retention of the Hadoop property map.
No existing inline findings require carry-forward, and no additional user focus was supplied. The release note matches the change; no documentation change appears necessary. Validation was static only: builds, unit tests, and FE/BE plus S3 integration were not executed. No unresolved actionable candidate or uncovered changed-file scope remains.
Review converged after 2 round(s).
FE UT Coverage ReportIncrement line coverage `` 🎉 |
### What problem does this PR solve? Issue Number: N/A Related PR: #69479 (branch-4.2), #69480 (branch-4.1), #69481 (branch-3.1) Problem Summary: When a Hudi JNI scan uses explicit S3A credentials, enabling scanner DEBUG logging prints every parameter value, including access keys, secret keys and session tokens. Log parameter names without values and record the query ID separately so scans remain traceable. The Hadoop configuration used to read data is unchanged. Add a regression test that captures the real scanner's DEBUG output, checks that parameter names and the query ID remain visible, rejects ordinary and bucket-specific credentials and custom tokens, and verifies that Hadoop properties retain their original values. ### Release note Hudi JNI scanner debug logs no longer include storage credential values and continue to include query IDs for diagnostics. ### Check List (For Author) - Test - [ ] Regression test - [x] Unit Test - [ ] Manual test - [ ] No need to test or manual test. Validation with JDK 17: - `./run-fe-ut.sh --run 'org.apache.doris.hudi.HadoopHudiJniScannerLogTest,org.apache.doris.hudi.HadoopHudiPluginTest,org.apache.doris.hudi.HadoopHudiColumnValueTest'`: 16 tests passed. - The new log test fails on the original code because parameter values appear in DEBUG output and passes after the fix. - The existing plugin test writes and reads a local Hudi table through the Java scanner. - Maven `validate` / Checkstyle for `be-java-extensions/hadoop-hudi-scanner` and its dependencies, plus `git diff --check`: passed. - The logging regression is covered directly by JUnit. FE/BE + S3 end-to-end regression was not run locally; SQL planning, credential propagation and data-reading logic are unchanged. - Behavior changed: - [ ] No. - [x] Yes. DEBUG logs retain parameter names and query IDs without printing other parameter values. - Does this need documentation? - [x] No. - [ ] Yes. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
|
run feut |
|
run p0 |
|
run external |
|
run nonConcurrent |
What problem does this PR solve?
Issue Number: N/A
Related PR: #69478 (backport to branch-3.1)
Problem Summary: When a Hudi JNI scan uses explicit S3A credentials, enabling scanner DEBUG logging prints every parameter value, including access keys, secret keys and session tokens. Log parameter names without values and record the query ID separately so scans remain traceable. The Hadoop configuration used to read data is unchanged.
Add a regression test that captures the real scanner's DEBUG output, checks that parameter names and the query ID remain visible, rejects ordinary and bucket-specific credentials and custom tokens, and verifies that Hadoop properties retain their original values.
Release note
Hudi JNI scanner debug logs no longer include storage credential values and continue to include query IDs for diagnostics.
Check List (For Author)
Test
Validation with JDK 17:
./run-fe-ut.sh --run 'org.apache.doris.hudi.HadoopHudiJniScannerLogTest': 1 test passed.validate/ Checkstyle forbe-java-extensions/hadoop-hudi-scannerand its dependencies, plusgit diff --check: passed.Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)