Repository navigation
[YAML] Fix SQL-backed Combine generating invalid syntax when group_by is empty or omitted - #41049
vishalmore90 wants to merge 1 commit into
Conversation
… is empty or omitted (apache#41047)
There was a problem hiding this comment.
📋 Review Summary
This PR fixes a bug in YAML's SQL-backed Combine transform where omitting group_by (global aggregation) generated an invalid trailing GROUP BY clause in the SQL query. The query construction logic and unit test coverage for both PyJsYamlCombine and _SqlCombineTransform look solid.
🔍 General Feedback
InlineProvider.config_schema_from_callable(used bygenerate_yaml_docs.pyandper_transformschema validation) inspects parameter type annotations viatyping_to_runner_apiand setsnullable=Trueonly when a parameter is annotated withOptional[...]. Adjusting thegroup_byandcombinetype annotations inPyJsYamlCombine.__init__and_SqlCombineTransformwill ensuregroup_byis recognized as optional andcombineremains required in the generated schema and docs.
💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.
| self, | ||
| group_by: Iterable[str], | ||
| combine: Mapping[str, Mapping[str, Any]], | ||
| group_by: Iterable[str] = (), |
There was a problem hiding this comment.
🟡 Medium: InlineProvider.config_schema_from_callable derives config field nullability (used for required in beam_schema_to_json_schema and (Optional) in generate_yaml_docs.py) directly from Optional[...] in the type annotations via typing_to_runner_api, rather than from default values.
Annotating group_by as Iterable[str] leaves group_by marked as required (nullable=False) in the generated schema/docs and conflicts with passing group_by=None. Annotating group_by as Optional[Iterable[str]] = () (matching _PyJsMapToFields in yaml_mapping.py) and keeping combine non-Optional (e.g., combine: Mapping[str, Mapping[str, Any]] = {} # pylint: disable=dangerous-default-value) ensures group_by is marked optional while combine remains required in the config schema.
| group_by: Iterable[str] = (), | |
| group_by: Optional[Iterable[str]] = (), |
| pcoll, sql_transform_constructor, group_by, combine, language=None): | ||
| pcoll, | ||
| sql_transform_constructor, | ||
| group_by: Iterable[str] = (), |
There was a problem hiding this comment.
🟡 Medium: Similarly here for SqlBackedProvider.config_schema (and for type-checking the group_by=None call in test_sql_combine_query_generation), group_by should be annotated as Optional[Iterable[str]] = () and combine kept non-Optional (combine: Mapping[str, Any] = {} # pylint: disable=dangerous-default-value).
| group_by: Iterable[str] = (), | |
| group_by: Optional[Iterable[str]] = (), |
|
Assigning reviewers: R: @chamikaramj for label yaml. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
fixes #41047
Description
In Beam YAML, global aggregations across an entire PCollection can be specified without grouping keys (e.g.
group_by: []or omittinggroup_by).Previously:
Combine-sql,Combine-calcite, orlanguage: sql),_SqlCombineTransformunconditionally appendedGROUP BY %sformatted with', '.join(group_by). Whengroup_bywas empty ([]), this generated an invalid SQL query string ending with a danglingGROUP BY:SqlParseException: Encountered "<EOF>").group_bywas omitted entirely from the YAMLconfig,normalize_combinedid not defaultgroup_byto an empty list, causingset(group_by)in_SqlCombineTransformto raiseTypeError: 'NoneType' object is not iterableandPyJsYamlCombine.__init__to fail on missing positional argumentgroup_by.This PR fixes both issues:
_SqlCombineTransform: Defaultsgroup_byto()and only appends theGROUP BYclause if grouping columns are present.normalize_combine: Defaults missing/falsygroup_byconfigurations to[].PyJsYamlCombine: Defaultsgroup_byto()andcombineto{}.yaml_combine_test.pyverifying SQL query generation with grouping keys, withgroup_by: [], with omittedgroup_by, and YAML execution withoutgroup_by.CHANGES.md.