Skip to content

Add WITH clause to Beam SQL to set connector properties - #40848

Open
ahmedabu98 wants to merge 2 commits into
apache:masterfrom
ahmedabu98:sql-with-options
Open

ahmedabu98 wants to merge 2 commits into
apache:masterfrom
ahmedabu98:sql-with-options

Conversation

@ahmedabu98

Copy link
Copy Markdown
Contributor

Adds per-statement table options to Beam SQL so users can set/override connector properties within a query. Example:

  • FROM t WITH ('key' = value, ...)
  • INSERT INTO t WITH (...)

Implementation adds new forRead and forWrite hooks in BeamSqlTable that table types can implement to support WITH options.

Includes such an implementation for Managed Iceberg write options.

@github-actions github-actions Bot added java extensions sql iceberg Iceberg IO connector size/xl XL: >=1000 lines changed labels Oct 27, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📋 Review Summary

This PR adds support for per-reference WITH ('key' = value, ...) options on SQL FROM table references and INSERT INTO targets, encoding the clause into the table identifier for Calcite resolution and wiring it into BeamSqlTable.forRead/forWrite and IcebergTable.forWrite (including merge-on-read CDC writes). The design is clean, well-documented, and thoroughly tested across parser edge cases, Calcite resolution, and Iceberg execution.

🔍 General Feedback

  • The approach of folding canonical TableClause SQL into the final SqlIdentifier component and aliasing read references is a clever way to pass per-occurrence options through Calcite's validator and SqlToRelConverter without invasive changes to Calcite internals.
  • IcebergTable.forWrite currently does not validate mode-specific options (via IcebergWriteSchemaTransformProvider.Configuration#validateModeOptions) until buildIOWriter runs, which is slightly inconsistent with the rest of the option checks happening during query parsing/validation.

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

@Override
public BeamSqlTable forWrite(Map<String, Object> options) {
Map<String, Object> checked =
ManagedOptions.check(options, WRITE_CONFIG_SCHEMA, WRITE_OWNED_OPTIONS, "an Iceberg write");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Low: ManagedOptions.check validates option names and types against WRITE_CONFIG_SCHEMA, but IcebergWriteSchemaTransformProvider.Configuration#validateModeOptions() (which checks that mode is valid and that mode-specific options like equality_columns or autosharding match the chosen mode) is only invoked later during Managed.write expansion in buildIOWriter. Consider validating mode compatibility in forWrite as well so invalid mode/option combinations fail fast during parseQuery like other invalid WITH options.

@codecov

codecov Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 90.14085% with 28 lines in your changes missing coverage. Please review.
✅ Project coverage is 64.69%. Comparing base (2213e54) to head (0fef173).
⚠️ Report is 42 commits behind head on master.

Files with missing lines Patch % Lines
...ions/sql/meta/provider/iceberg/ManagedOptions.java 68.75% 12 Missing and 8 partials ⚠️
...am/sdk/extensions/sql/impl/parser/TableClause.java 94.93% 3 Missing and 5 partials ⚠️
Additional details and impacted files
@@              Coverage Diff              @@
##             master   #40848       +/-   ##
=============================================
+ Coverage          0   64.69%   +64.69%     
- Complexity        0     2457     +2457     
=============================================
  Files             0      324      +324     
  Lines             0    15884    +15884     
  Branches          0     2081     +2081     
=============================================
+ Hits              0    10276    +10276     
- Misses            0     4831     +4831     
- Partials          0      777      +777     
Flag Coverage Δ
java 64.69% <90.14%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @chamikaramj for label iceberg.
Backup reviewer: Abacn (if the main reviewer is unresponsive, you can escalate to them by tagging their username)

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant