Repository navigation
Add WITH clause to Beam SQL to set connector properties - #40848
ahmedabu98 wants to merge 2 commits into
Conversation
There was a problem hiding this comment.
📋 Review Summary
This PR adds support for per-reference WITH ('key' = value, ...) options on SQL FROM table references and INSERT INTO targets, encoding the clause into the table identifier for Calcite resolution and wiring it into BeamSqlTable.forRead/forWrite and IcebergTable.forWrite (including merge-on-read CDC writes). The design is clean, well-documented, and thoroughly tested across parser edge cases, Calcite resolution, and Iceberg execution.
🔍 General Feedback
- The approach of folding canonical
TableClauseSQL into the finalSqlIdentifiercomponent and aliasing read references is a clever way to pass per-occurrence options through Calcite's validator andSqlToRelConverterwithout invasive changes to Calcite internals. IcebergTable.forWritecurrently does not validate mode-specific options (viaIcebergWriteSchemaTransformProvider.Configuration#validateModeOptions) untilbuildIOWriterruns, which is slightly inconsistent with the rest of the option checks happening during query parsing/validation.
💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.
| @Override | ||
| public BeamSqlTable forWrite(Map<String, Object> options) { | ||
| Map<String, Object> checked = | ||
| ManagedOptions.check(options, WRITE_CONFIG_SCHEMA, WRITE_OWNED_OPTIONS, "an Iceberg write"); |
There was a problem hiding this comment.
🟢 Low: ManagedOptions.check validates option names and types against WRITE_CONFIG_SCHEMA, but IcebergWriteSchemaTransformProvider.Configuration#validateModeOptions() (which checks that mode is valid and that mode-specific options like equality_columns or autosharding match the chosen mode) is only invoked later during Managed.write expansion in buildIOWriter. Consider validating mode compatibility in forWrite as well so invalid mode/option combinations fail fast during parseQuery like other invalid WITH options.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #40848 +/- ##
=============================================
+ Coverage 0 64.69% +64.69%
- Complexity 0 2457 +2457
=============================================
Files 0 324 +324
Lines 0 15884 +15884
Branches 0 2081 +2081
=============================================
+ Hits 0 10276 +10276
- Misses 0 4831 +4831
- Partials 0 777 +777
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Assigning reviewers: R: @chamikaramj for label iceberg. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
Adds per-statement table options to Beam SQL so users can set/override connector properties within a query. Example:
FROM t WITH ('key' = value, ...)INSERT INTO t WITH (...)Implementation adds new
forReadandforWritehooks in BeamSqlTable that table types can implement to support WITH options.Includes such an implementation for Managed Iceberg write options.