Repository navigation
Fix leading CLI --conf properties being consumed by Hadoop - #3866
Open
zhangxinyao88 wants to merge 1 commit into
Open
zhangxinyao88 wants to merge 1 commit into
zhangxinyao88 wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A leading
--conf key=valueis consumed by Hadoop'sToolRunneras an XML resource option, so the command can succeed without applying the property. For example,parquet --conf parquet.avro.write-parquet-uuid=true convert ...writes a string column instead of UUID storage.Normalize that alias to
--propertyonly within the leading Hadoop option prefix, then use the same entry path inMainand the shared CLI test helper. Preserve Hadoop's-confXML resources,-Dproperties and generic option values; leave command arguments untouched. Explicit properties still override configuration-file values.Fourteen regression cases cover configuration delivery, repeated properties, precedence, Hadoop options and argument values. Two launch the actual
Mainprocess and inspect the converted Parquet schema. Removing the fix produces ten assertion failures, including the UUID storage regression.Validation:
./mvnw -o -B -ntp -pl parquet-cli spotless:check testpassed on Java 17: 97 tests, 95 passed, 2 skipped; Spotless and RAT passed. Full-reactor and other-JDK checks were not run.Related to #3283 and #3289. This is independent of the test-only conversion coverage in #3861.