Repository navigation
[Iceberg CDC sink] SchemaTransform (ManagedIO) layer - #40229
Conversation
|
Assigning reviewers: R: @shunping for label python. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
| /** Rejects every set option that the selected mode does not support. */ | ||
| void validateModeOptions() { | ||
| Mode mode = mode(); | ||
| Map<String, @Nullable Object> values = new LinkedHashMap<>(); |
There was a problem hiding this comment.
We should derive this from the Configuration instead of maintaining a separate local map here and a SUPPORTED_MODES map above so that the two things don't drift in the future.
There was a problem hiding this comment.
These options are only a subset of Configuration. They represent the options that apply to only one of CDC or Append mode. Options that apply to both (e.g. table or catalog_properties) are not mentioned here
Will try to think of something better though
| if (!invalidOptions.isEmpty()) { | ||
| throw new IllegalArgumentException( | ||
| String.format( | ||
| "The following options are not supported by %s writes yet: %s", |
There was a problem hiding this comment.
Probably give a better hint regarding potential action to take in this error message based on the property. For example,
- set the cdc block
- these aren't implemented for CDC yet
(cherry picked from commit e3b3e7c)
Adds the final SchemaTransform layer for the cdc sink
Fixes #39979
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.