[dhanusharer]

Fixes #40709

Summary of Changes

• Dependency & Build: Enabled pandas>=2.2.2,<3.2 for Python >=3.11 while preserving pandas<2.4 for Python 3.10 using PEP 508 environment markers. Added py311-pandas-30 tox environment and Gradle task.
• Introspection Hardening: Added defensive hasattr() checks in frame_base.py decorators (@args_to_kwargs, @populate_defaults, @with_docs_from) to prevent import-time reflection crashes when pandas 3.0 removes deprecated methods or parameters. Handled permanent Copy-on-Write defaults.
• DataFrames API Updates:
  • Added removed_args=['method'] and avoided passing method to fillna() under pandas 3.0.
  • Added removed_args=['axis'] to groupby() and raised clean NotImplementedError for column-wise groupby.
  • Guarded first, last, and applymap with hasattr() checks.
  • Guarded GroupBy methods (fillna, first, last) with hasattr(DataFrameGroupBy, ...).
  • Removed method=None from _DeferredStringMethods.get_dummies fillna call.
• Typehint Compatibility: Added support for pandas 3.0 str, pd.StringDtype, and datetime dtypes in dtype_to_fieldtype().
• Tests: Updated test_applymap_na_action to skip on pandas 3.0+ and added pandas 3.0 ffill/bfill branch to test_fillna_columns.

[github-actions]

📋 Review Summary

This PR updates the DataFrame API, schema conversion utilities, and type compatibility test suites to support Pandas 3.0 (where infer_string is enabled by default) and adds py311-pandas-30 to tox.ini. While the core Pandas 3.0 compatibility fixes are solid, the setup.py package bounds and Gradle test tasks are not updated yet, and one schema conversion path still relies on dtype.kind == 'O' instead of pd.api.types.is_string_dtype.

🔍 General Feedback

• Dependency & CI Wiring (setup.py & build.gradle): sdks/python/setup.py still caps dataframe_dependency at 'pandas>=2.2.2,<2.4' (with # TODO(https://github.com/apache/beam/issues/40709): support pandas 3+) and docs/test extras at 'pandas<2.6.0'. Additionally, sdks/python/test-suites/tox/py311/build.gradle needs a testPy311pandas-30 task registered so the new py311-pandas-30 tox environment is actually executed in postCommitPyDep (and .github/trigger_files/beam_PostCommit_Python_Dependency.json can be bumped to trigger it on the PR).
• Remaining dtype.kind == 'O' check in schemas.py: sdks/python/apache_beam/dataframe/schemas.py (line 206 in element_type_from_dataframe) still checks if dtype.kind == 'O': return Any when inferring schema types from a proxy without element_type. Under Pandas 3.0, default string columns have pd.StringDtype(na_value=np.nan) (dtype.kind == 'U'), which bypasses that branch and falls through to dtype_to_fieldtype (raising TypeError: Unable to encounter corresponding Beam type for dtype=str). Updating that check to if dtype.kind == 'O' or pd.api.types.is_string_dtype(dtype): (or handling pd.api.types.is_string_dtype in pandas_type_compatibility.py) will keep string column fallback consistent.

---

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

💬 Inline Findings

• sdks/python/tox.ini (line 376): 🟠 High: py311-pandas-30 is added here in tox.ini, but two other places need to be updated in tandem:

1. sdks/python/test-suites/tox/py311/build.gradle needs a corresponding toxTask "testPy311pandas-30", "py311-pandas-30", "${posargs}" (added to test and postCommitPyDep) so that CI actually runs this tox environment.
2. sdks/python/setup.py still restricts dataframe_dependency to 'pandas>=2.2.2,<2.4' (and docs/test extras to 'pandas<2.6.0'). Since tox installs the package with extras = test,dataframe,..., pip will either report a version conflict in commands_pre (pip check) or prevent users from installing Pandas 3.0 with apache-beam[dataframe].

• sdks/python/apache_beam/dataframe/schemas_test.py (line 227): 🟡 Medium: When major >= 3, series_tests is reassigned instead of appended to (+=), which drops all the numeric and nullable integer Series test cases defined in lines 211–218 on Pandas 3+.

      series_tests += [(pd.Series(['a', 'b', 'c'], dtype=dtype), str)]

[dhanusharer]

/gemini review

[github-actions]

Assigning reviewers:

R: @claudevdm for label python.
Backup reviewer: jrmccluskey (if the main reviewer is unresponsive, you can escalate to them by tagging their username)

This pull request likely touches a core component ("core" label). Please review with scrutiny.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).