diff --git a/.github/workflows/README.md b/.github/workflows/README.md index 682b51699c09..a88297940601 100644 --- a/.github/workflows/README.md +++ b/.github/workflows/README.md @@ -372,11 +372,11 @@ PostCommit Jobs run in a schedule against master branch and generally do not get | [ PostCommit Python MongoDBIO IT ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_MongoDBIO_IT.yml) | N/A |`beam_PostCommit_Python_MongoDBIO_IT.json`| [![.github/workflows/beam_PostCommit_Python_MongoDBIO_IT.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_MongoDBIO_IT.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_MongoDBIO_IT.yml?query=event%3Aschedule) | | [ PostCommit Python Nexmark Direct ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_Nexmark_Direct.yml) | N/A |`beam_PostCommit_Python_Nexmark_Direct.json`| [![.github/workflows/beam_PostCommit_Python_Nexmark_Direct.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_Nexmark_Direct.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_Nexmark_Direct.yml?query=event%3Aschedule) | | [ PostCommit Python Tpcds Prism ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_Tpcds_Prism.yml) | N/A |`beam_PostCommit_Python_Tpcds_Prism.json`| [![.github/workflows/beam_PostCommit_Python_Tpcds_Prism.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_Tpcds_Prism.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_Tpcds_Prism.yml?query=event%3Aschedule) | -| [ PostCommit Python ValidatesContainer Dataflow ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesContainer_Dataflow.yml) | ['3.9','3.10','3.11','3.12'] |`beam_PostCommit_Python_ValidatesContainer_Dataflow.json`| [![.github/workflows/beam_PostCommit_Python_ValidatesContainer_Dataflow.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesContainer_Dataflow.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesContainer_Dataflow.yml?query=event%3Aschedule) | -| [ PostCommit Python ValidatesContainer Dataflow With RC ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesContainer_Dataflow_With_RC.yml) | ['3.9','3.10','3.11','3.12'] |`beam_PostCommit_Python_ValidatesContainer_Dataflow_With_RC.json`| [![.github/workflows/beam_PostCommit_Python_ValidatesContainer_Dataflow_With_RC.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesContainer_Dataflow_With_RC.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesContainer_Dataflow_With_RC.yml?query=event%3Aschedule) | -| [ PostCommit Python ValidatesRunner Dataflow ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesRunner_Dataflow.yml) | ['3.9','3.11','3.12'] |`beam_PostCommit_Python_ValidatesRunner_Dataflow.json`| [![.github/workflows/beam_PostCommit_Python_ValidatesRunner_Dataflow.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesRunner_Dataflow.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesRunner_Dataflow.yml?query=event%3Aschedule) | -| [ PostCommit Python ValidatesRunner Flink ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesRunner_Flink.yml) | ['3.9','3.11','3.12'] |`beam_PostCommit_Python_ValidatesRunner_Flink.json`| [![.github/workflows/beam_PostCommit_Python_ValidatesRunner_Flink.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesRunner_Flink.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesRunner_Flink.yml?query=event%3Aschedule) | -| [ PostCommit Python ValidatesRunner Spark ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesRunner_Spark.yml) | ['3.9','3.10','3.11','3.12'] |`beam_PostCommit_Python_ValidatesRunner_Spark.json`| [![.github/workflows/beam_PostCommit_Python_ValidatesRunner_Spark.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesRunner_Spark.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesRunner_Spark.yml?query=event%3Aschedule) | +| [ PostCommit Python ValidatesContainer Dataflow ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesContainer_Dataflow.yml) | ['3.11','3.12','3.13','3.14'] |`beam_PostCommit_Python_ValidatesContainer_Dataflow.json`| [![.github/workflows/beam_PostCommit_Python_ValidatesContainer_Dataflow.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesContainer_Dataflow.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesContainer_Dataflow.yml?query=event%3Aschedule) | +| [ PostCommit Python ValidatesContainer Dataflow With RC ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesContainer_Dataflow_With_RC.yml) | ['3.11','3.12','3.13','3.14'] |`beam_PostCommit_Python_ValidatesContainer_Dataflow_With_RC.json`| [![.github/workflows/beam_PostCommit_Python_ValidatesContainer_Dataflow_With_RC.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesContainer_Dataflow_With_RC.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesContainer_Dataflow_With_RC.yml?query=event%3Aschedule) | +| [ PostCommit Python ValidatesRunner Dataflow ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesRunner_Dataflow.yml) | ['3.11','3.12','3.13','3.14'] |`beam_PostCommit_Python_ValidatesRunner_Dataflow.json`| [![.github/workflows/beam_PostCommit_Python_ValidatesRunner_Dataflow.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesRunner_Dataflow.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesRunner_Dataflow.yml?query=event%3Aschedule) | +| [ PostCommit Python ValidatesRunner Flink ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesRunner_Flink.yml) | ['3.11','3.12','3.13','3.14'] |`beam_PostCommit_Python_ValidatesRunner_Flink.json`| [![.github/workflows/beam_PostCommit_Python_ValidatesRunner_Flink.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesRunner_Flink.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesRunner_Flink.yml?query=event%3Aschedule) | +| [ PostCommit Python ValidatesRunner Spark ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesRunner_Spark.yml) | ['3.11','3.12','3.13','3.14'] |`beam_PostCommit_Python_ValidatesRunner_Spark.json`| [![.github/workflows/beam_PostCommit_Python_ValidatesRunner_Spark.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesRunner_Spark.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesRunner_Spark.yml?query=event%3Aschedule) | | [ PostCommit Python Xlang Gcp Dataflow ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_Xlang_Gcp_Dataflow.yml) | N/A |`beam_PostCommit_Python_Xlang_Gcp_Dataflow.json`| [![.github/workflows/beam_PostCommit_Python_Xlang_Gcp_Dataflow.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_Xlang_Gcp_Dataflow.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_Xlang_Gcp_Dataflow.yml?query=event%3Aschedule) | | [ PostCommit Python Xlang Gcp Direct ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_Xlang_Gcp_Direct.yml) | N/A |`beam_PostCommit_Python_Xlang_Gcp_Direct.json`| [![.github/workflows/beam_PostCommit_Python_Xlang_Gcp_Direct.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_Xlang_Gcp_Direct.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_Xlang_Gcp_Direct.yml?query=event%3Aschedule) | | [ PostCommit Python Xlang IO Dataflow ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_Xlang_IO_Dataflow.yml) | N/A |`beam_PostCommit_Python_Xlang_IO_Dataflow.json`| [![.github/workflows/beam_PostCommit_Python_Xlang_IO_Dataflow.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_Xlang_IO_Dataflow.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_Xlang_IO_Dataflow.yml?query=event%3Aschedule) | @@ -392,7 +392,7 @@ PostCommit Jobs run in a schedule against master branch and generally do not get | [ PostCommit XVR PythonUsingJavaSQL Dataflow ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_XVR_PythonUsingJavaSQL_Dataflow.yml) | N/A |`beam_PostCommit_XVR_PythonUsingJavaSQL_Dataflow.json`| [![.github/workflows/beam_PostCommit_XVR_PythonUsingJavaSQL_Dataflow.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_XVR_PythonUsingJavaSQL_Dataflow.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_XVR_PythonUsingJavaSQL_Dataflow.yml?query=event%3Aschedule) | | [ PostCommit XVR Spark3 ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_XVR_Spark3.yml) | N/A |`beam_PostCommit_XVR_Spark3.json`| [![.github/workflows/beam_PostCommit_XVR_Spark3.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_XVR_Spark3.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_XVR_Spark3.yml?query=event%3Aschedule) | | [ PostCommit YAML Xlang Direct ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Yaml_Xlang_Direct.yml) | N/A |`beam_PostCommit_Yaml_Xlang_Direct.json`| [![.github/workflows/beam_PostCommit_Yaml_Xlang_Direct.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Yaml_Xlang_Direct.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Yaml_Xlang_Direct.yml?query=event%3Aschedule) | -| [ Python Validates Container Dataflow ARM ](https://github.com/apache/beam/actions/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml) | ['3.9','3.10','3.11','3.12'] |`beam_Python_ValidatesContainer_Dataflow_ARM.json`|[![.github/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml](https://github.com/apache/beam/actions/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml?query=event%3Aschedule) | +| [ Python Validates Container Dataflow ARM ](https://github.com/apache/beam/actions/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml) | ['3.11','3.12','3.13','3.14'] |`beam_Python_ValidatesContainer_Dataflow_ARM.json`|[![.github/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml](https://github.com/apache/beam/actions/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml?query=event%3Aschedule) | | [ PostCommit Java ValidatesRunner Dataflow Streaming TagEncodingV2 ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming_TagEncodingV2.yml) | N/A |`beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming_TagEncodingV2.json`| [![.github/workflows/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming_TagEncodingV2.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming_TagEncodingV2.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming_TagEncodingV2.yml?query=event%3Aschedule) | | [ PostCommit Java ValidatesRunner Dataflow Streaming Engine ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming_Engine.yml) | N/A |`beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming_Engine.json`| [![.github/workflows/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming_Engine.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming_Engine.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming_Engine.yml?query=event%3Aschedule) | | [ PostCommit Python Portable Flink ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_Portable_Flink.yml) | N/A |`beam_PostCommit_Python_Portable_Flink.json`| [![.github/workflows/beam_PostCommit_Python_Portable_Flink.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_Portable_Flink.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_Portable_Flink.yml?query=event%3Aschedule) | @@ -435,7 +435,7 @@ PostCommit Jobs run in a schedule against master branch and generally do not get | [ PerformanceTests TextIOIT ](https://github.com/apache/beam/actions/workflows/beam_PerformanceTests_TextIOIT.yml) | N/A | [![.github/workflows/beam_PerformanceTests_TextIOIT.yml](https://github.com/apache/beam/actions/workflows/beam_PerformanceTests_TextIOIT.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PerformanceTests_TextIOIT.yml?query=event%3Aschedule) | [ PerformanceTests TFRecordIOIT HDFS ](https://github.com/apache/beam/actions/workflows/beam_PerformanceTests_TFRecordIOIT_HDFS.yml) | N/A | [![.github/workflows/beam_PerformanceTests_TFRecordIOIT_HDFS.yml](https://github.com/apache/beam/actions/workflows/beam_PerformanceTests_TFRecordIOIT_HDFS.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PerformanceTests_TFRecordIOIT_HDFS.yml?query=event%3Aschedule) | [ PerformanceTests TFRecordIOIT ](https://github.com/apache/beam/actions/workflows/beam_PerformanceTests_TFRecordIOIT.yml) | N/A | [![.github/workflows/beam_PerformanceTests_TFRecordIOIT.yml](https://github.com/apache/beam/actions/workflows/beam_PerformanceTests_TFRecordIOIT.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PerformanceTests_TFRecordIOIT.yml?query=event%3Aschedule) -| [ PerformanceTests WordCountIT PythonVersions ](https://github.com/apache/beam/actions/workflows/beam_PerformanceTests_WordCountIT_PythonVersions.yml) | ['3.9','3.10','3.11','3.12'] | [![.github/workflows/beam_PerformanceTests_WordCountIT_PythonVersions.yml](https://github.com/apache/beam/actions/workflows/beam_PerformanceTests_WordCountIT_PythonVersions.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PerformanceTests_WordCountIT_PythonVersions.yml?query=event%3Aschedule) +| [ PerformanceTests WordCountIT PythonVersions ](https://github.com/apache/beam/actions/workflows/beam_PerformanceTests_WordCountIT_PythonVersions.yml) | ['3.11','3.12','3.13','3.14'] | [![.github/workflows/beam_PerformanceTests_WordCountIT_PythonVersions.yml](https://github.com/apache/beam/actions/workflows/beam_PerformanceTests_WordCountIT_PythonVersions.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PerformanceTests_WordCountIT_PythonVersions.yml?query=event%3Aschedule) | [ PerformanceTests XmlIOIT HDFS ](https://github.com/apache/beam/actions/workflows/beam_PerformanceTests_XmlIOIT_HDFS.yml) | N/A | [![.github/workflows/beam_PerformanceTests_XmlIOIT_HDFS.yml](https://github.com/apache/beam/actions/workflows/beam_PerformanceTests_XmlIOIT_HDFS.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PerformanceTests_XmlIOIT_HDFS.yml?query=event%3Aschedule) | [ PerformanceTests XmlIOIT ](https://github.com/apache/beam/actions/workflows/beam_PerformanceTests_XmlIOIT.yml) | N/A | [![.github/workflows/beam_PerformanceTests_XmlIOIT.yml](https://github.com/apache/beam/actions/workflows/beam_PerformanceTests_XmlIOIT.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PerformanceTests_XmlIOIT.yml?query=event%3Aschedule) | [ PerformanceTests xlang KafkaIO Python ](https://github.com/apache/beam/actions/workflows/beam_PerformanceTests_xlang_KafkaIO_Python.yml) | N/A | [![.github/workflows/beam_PerformanceTests_xlang_KafkaIO_Python.yml](https://github.com/apache/beam/actions/workflows/beam_PerformanceTests_xlang_KafkaIO_Python.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PerformanceTests_xlang_KafkaIO_Python.yml?query=event%3Aschedule) diff --git a/.github/workflows/beam_PostCommit_Yaml_Xlang_Direct.yml b/.github/workflows/beam_PostCommit_Yaml_Xlang_Direct.yml index 8317f71b803a..2111b9c40537 100644 --- a/.github/workflows/beam_PostCommit_Yaml_Xlang_Direct.yml +++ b/.github/workflows/beam_PostCommit_Yaml_Xlang_Direct.yml @@ -75,7 +75,7 @@ jobs: - name: run PostCommit Yaml Xlang Direct script uses: ./.github/actions/gradle-command-self-hosted-action with: - gradle-command: :sdks:python:postCommitYamlIntegrationTests -PyamlTestSet=${{ matrix.test_set }} -PbeamPythonExtra=p310_ml_test,yaml,gcp_test + gradle-command: :sdks:python:postCommitYamlIntegrationTests -PyamlTestSet=${{ matrix.test_set }} -PbeamPythonExtra=p311_ml_test,yaml,gcp_test - name: Archive Python Test Results uses: actions/upload-artifact@v7 if: failure() diff --git a/.github/workflows/beam_PreCommit_Yaml_Xlang_Direct.yml b/.github/workflows/beam_PreCommit_Yaml_Xlang_Direct.yml index 65fbc647915d..6f1742b79d7c 100644 --- a/.github/workflows/beam_PreCommit_Yaml_Xlang_Direct.yml +++ b/.github/workflows/beam_PreCommit_Yaml_Xlang_Direct.yml @@ -85,7 +85,7 @@ jobs: - name: run PreCommit Yaml Xlang Direct script uses: ./.github/actions/gradle-command-self-hosted-action with: - gradle-command: :sdks:python:yamlIntegrationTests -PbeamPythonExtra=p310_ml_test,yaml,gcp_test + gradle-command: :sdks:python:yamlIntegrationTests -PbeamPythonExtra=p311_ml_test,yaml,gcp_test - name: Archive Python Test Results uses: actions/upload-artifact@v7 if: failure() diff --git a/.github/workflows/build_release_candidate.yml b/.github/workflows/build_release_candidate.yml index a6a1164dfb8c..2d3a77995640 100644 --- a/.github/workflows/build_release_candidate.yml +++ b/.github/workflows/build_release_candidate.yml @@ -280,11 +280,10 @@ jobs: # tasks as they are added. images_to_publish: [ {"gradle_task": ":pushAllRunnersDockerImages", "include_skip_flags": "-Pinclude-ml -Pinclude-distroless"}, - {"gradle_task": ":sdks:python:container:push310", "include_skip_flags": "-Pinclude-ml -Pinclude-distroless"}, {"gradle_task": ":sdks:python:container:push311", "include_skip_flags": "-Pinclude-ml -Pinclude-distroless"}, {"gradle_task": ":sdks:python:container:push312", "include_skip_flags": "-Pinclude-ml -Pinclude-distroless"}, {"gradle_task": ":sdks:python:container:push313", "include_skip_flags": "-Pinclude-ml -Pinclude-distroless"}, - {"gradle_task": ":sdks:python:container:pushAll", "include_skip_flags": "-Pinclude-ml -Pinclude-distroless -Pskip-python-39-images -Pskip-python-310-images -Pskip-python-311-images -Pskip-python-312-images -Pskip-python-313-images"}, + {"gradle_task": ":sdks:python:container:pushAll", "include_skip_flags": "-Pinclude-ml -Pinclude-distroless -Pskip-python-311-images -Pskip-python-312-images -Pskip-python-313-images"}, {"gradle_task": ":pushAllSdkDockerImages", "include_skip_flags": "-Pskip-python-images"}, {"gradle_task": ":pushAllDockerImages", "include_skip_flags": "-Pskip-runner-images -Pskip-sdk-images"} ] diff --git a/.github/workflows/republish_released_docker_containers.yml b/.github/workflows/republish_released_docker_containers.yml index c857476f5cfa..f62a6bb57929 100644 --- a/.github/workflows/republish_released_docker_containers.yml +++ b/.github/workflows/republish_released_docker_containers.yml @@ -47,11 +47,10 @@ jobs: # tasks as they are added. images_to_publish: [ {"gradle_task": ":pushAllRunnersDockerImages", "include_skip_flags": "-Pinclude-ml -Pinclude-distroless"}, - {"gradle_task": ":sdks:python:container:push310", "include_skip_flags": "-Pinclude-ml -Pinclude-distroless"}, {"gradle_task": ":sdks:python:container:push311", "include_skip_flags": "-Pinclude-ml -Pinclude-distroless"}, {"gradle_task": ":sdks:python:container:push312", "include_skip_flags": "-Pinclude-ml -Pinclude-distroless"}, {"gradle_task": ":sdks:python:container:push313", "include_skip_flags": "-Pinclude-ml -Pinclude-distroless"}, - {"gradle_task": ":sdks:python:container:pushAll", "include_skip_flags": "-Pinclude-ml -Pinclude-distroless -Pskip-python-39-images -Pskip-python-310-images -Pskip-python-311-images -Pskip-python-312-images -Pskip-python-313-images"}, + {"gradle_task": ":sdks:python:container:pushAll", "include_skip_flags": "-Pinclude-ml -Pinclude-distroless -Pskip-python-311-images -Pskip-python-312-images -Pskip-python-313-images"}, {"gradle_task": ":pushAllSdkDockerImages", "include_skip_flags": "-Pskip-python-images"}, {"gradle_task": ":pushAllDockerImages", "include_skip_flags": "-Pskip-runner-images -Pskip-sdk-images"} ] diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 03be20b66347..9712d250ada7 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -53,8 +53,8 @@ These steps and instructions on getting started are outlined below as well. - For running test suites, however, you will need Python interpreters for all Python versions supported by Beam. Interpreters should be installed and available in shell via `python3.x` commands. See Python installation tips in [Developer Wiki](https://cwiki.apache.org/confluence/display/BEAM/Python+Tips#PythonTips-InstallingPythoninterpreters). - - If you encounter errors like `python3.10: command not found` or `Cannot run program "python3.10": No such file or directory`, your system may be missing some Python versions. - Workaround: install missing versions with `pyenv` (for example, `pyenv install 3.10` then `pyenv global 3.10 3.11 3.12 3.13` to make all available), or create symlinks in `/usr/local/bin/` pointing to installed Python binaries. + - If you encounter errors like `python3.11: command not found` or `Cannot run program "python3.11": No such file or directory`, your system may be missing some Python versions. + Workaround: install missing versions with `pyenv` (for example, `pyenv install 3.11` then `pyenv global 3.11 3.12 3.13 3.14` to make all available), or create symlinks in `/usr/local/bin/` pointing to installed Python binaries. - For large contributions, a signed [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf) (ICLA) to the Apache Software Foundation (ASF). diff --git a/contributor-docs/python-tips.md b/contributor-docs/python-tips.md index 90e2f2834988..790901591fe3 100644 --- a/contributor-docs/python-tips.md +++ b/contributor-docs/python-tips.md @@ -57,22 +57,22 @@ Installation steps may look as follows: 2. Install Python intepreter for each supported Python minor version. Leaving out the patch version will install the latest. ```bash -pyenv install 3.10 pyenv install 3.11 pyenv install 3.12 pyenv install 3.13 +pyenv install 3.14 ``` 3. Make installed interpreters available in your shell by first running: ```bash -pyenv global 3.10 3.11 3.12 3.13 +pyenv global 3.11 3.12 3.13 3.14 ``` 4. (**OPTIONAL**) Pyenv will sometimes [fail to make these interpreters directly available](https://github.com/pyenv/pyenv/issues/34) without a local configuration. If you see errors trying to use `python3.x`, then also run `pyenv local`   ```bash -pyenv local 3.10 3.11 3.12 3.13 +pyenv local 3.11 3.12 3.13 3.14 ``` After these steps, all `python3.x` interpreters should be available in your shell. The first version in the list passed to pyenv global will be used as default `python` / `python3` interpreter if the minor version is not specified. @@ -156,11 +156,11 @@ curl https://pyenv.run | bash # Run the outputted commands to initialize pyenv in .bashrc ``` -#### Example: How to Run Unit Tests with PyCharm Using Python 3.10.10 in a virtualenv -1. Install Python 3.10.10 and create a virtualenv +#### Example: How to Run Unit Tests with PyCharm Using Python 3.11.4 in a virtualenv +1. Install Python 3.11.4 and create a virtualenv ```bash -pyenv install 3.10.10 -pyenv virtualenv 3.10.10 ENV_NAME +pyenv install 3.11.4 +pyenv virtualenv 3.11.4 ENV_NAME pyenv activate ENV_NAME ``` @@ -171,7 +171,7 @@ pip install --upgrade pip setuptools 3. Set up PyCharm * Start by adding a new project interpreter (from the bottom right or in Settings). - * Select Existing environment and the interpreter, which should be under ~/.pyenv/versions/3.10.10/envs/ENV_NAME/bin/python or ~/.pyenv/versions/ENV_NAME/bin/python. + * Select Existing environment and the interpreter, which should be under ~/.pyenv/versions/3.11.4/envs/ENV_NAME/bin/python or ~/.pyenv/versions/ENV_NAME/bin/python. * Switch interpreters at the bottom right. #### Cleaning up environments @@ -519,7 +519,7 @@ NOTE for RELEASE MANAGERS: We should update dependencies at least once per relea You may see that the pip command will lead to segmentation fault as well. If this happens, remove the python version from pyenv, and reinstall the version like this. ```bash -CFLAGS="-O2" pyenv install 3.10.10 +CFLAGS="-O2" pyenv install 3.11.4 ``` There have been issues with older Python versions. See [here](https://github.com/pyenv/pyenv/issues/2046) for details. \ No newline at end of file diff --git a/dev-support/docker/pkglist b/dev-support/docker/pkglist index c8ab6bc6b6bc..2a9bc73a9719 100644 --- a/dev-support/docker/pkglist +++ b/dev-support/docker/pkglist @@ -32,10 +32,6 @@ time openjdk-11-jdk python3-setuptools python3-pip -python3.10 -python3.10-dev -python3.10-distutils -python3.10-venv python3.11 python3.11-dev python3.11-distutils diff --git a/release/src/main/python-release/python_release_automation_utils.sh b/release/src/main/python-release/python_release_automation_utils.sh index 3884b02c40f5..618450a859fd 100644 --- a/release/src/main/python-release/python_release_automation_utils.sh +++ b/release/src/main/python-release/python_release_automation_utils.sh @@ -82,9 +82,7 @@ function get_version() { ####################################### function download_files() { if [[ $1 = *"wheel"* ]]; then - if [[ $2 == "python3.10" ]]; then - BEAM_PYTHON_SDK_WHL="apache_beam-$VERSION*-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl" - elif [[ $2 == "python3.11" ]]; then + if [[ $2 == "python3.11" ]]; then BEAM_PYTHON_SDK_WHL="apache_beam-$VERSION*-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl" elif [[ $2 == "python3.12" ]]; then BEAM_PYTHON_SDK_WHL="apache_beam-$VERSION*-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl" diff --git a/release/src/main/scripts/github_actions_jobs.txt b/release/src/main/scripts/github_actions_jobs.txt index 6f200454afe0..d310c11c551e 100644 --- a/release/src/main/scripts/github_actions_jobs.txt +++ b/release/src/main/scripts/github_actions_jobs.txt @@ -19,7 +19,7 @@ Run Go PostCommit Dataflow ARM,beam_PostCommit_Go_Dataflow_ARM Run Java_Examples_Dataflow_ARM PostCommit 8,beam_PostCommit_Java_Examples_Dataflow_ARM Run Java_Examples_Dataflow_ARM PostCommit 11,beam_PostCommit_Java_Examples_Dataflow_ARM Run Java_Examples_Dataflow_ARM PostCommit 17,beam_PostCommit_Java_Examples_Dataflow_ARM -Run Python ValidatesContainer Dataflow ARM 3.8,beam_Python_ValidatesContainer_Dataflow_ARM -Run Python ValidatesContainer Dataflow ARM 3.9,beam_Python_ValidatesContainer_Dataflow_ARM -Run Python ValidatesContainer Dataflow ARM 3.10,beam_Python_ValidatesContainer_Dataflow_ARM Run Python ValidatesContainer Dataflow ARM 3.11,beam_Python_ValidatesContainer_Dataflow_ARM +Run Python ValidatesContainer Dataflow ARM 3.12,beam_Python_ValidatesContainer_Dataflow_ARM +Run Python ValidatesContainer Dataflow ARM 3.13,beam_Python_ValidatesContainer_Dataflow_ARM +Run Python ValidatesContainer Dataflow ARM 3.14,beam_Python_ValidatesContainer_Dataflow_ARM diff --git a/release/src/main/scripts/run_rc_validation.sh b/release/src/main/scripts/run_rc_validation.sh index 62f79c7ee626..a42fbe80793a 100755 --- a/release/src/main/scripts/run_rc_validation.sh +++ b/release/src/main/scripts/run_rc_validation.sh @@ -99,7 +99,7 @@ HUB_VERSION=2.12.0 HUB_ARTIFACTS_NAME=hub-linux-amd64-${HUB_VERSION} BACKUP_BASHRC=.bashrc_backup_$(date +"%Y%m%d%H%M%S") BACKUP_M2=settings_backup_$(date +"%Y%m%d%H%M%S").xml -declare -a PYTHON_VERSIONS_TO_VALIDATE=("python3.10") +declare -a PYTHON_VERSIONS_TO_VALIDATE=("python3.11") echo "" echo "====================Checking Environment & Variables=================" echo "PLEASE update RC_VALIDATE_CONFIGS in file script.config first." diff --git a/sdks/python/apache_beam/ml/inference/test_resources/vllm.dockerfile b/sdks/python/apache_beam/ml/inference/test_resources/vllm.dockerfile index f4022ae90160..42ee4019be78 100644 --- a/sdks/python/apache_beam/ml/inference/test_resources/vllm.dockerfile +++ b/sdks/python/apache_beam/ml/inference/test_resources/vllm.dockerfile @@ -24,29 +24,31 @@ ENV DEBIAN_FRONTEND=noninteractive \ TZ=Etc/UTC RUN apt-get update && \ + apt-get install -y --no-install-recommends software-properties-common && \ + add-apt-repository -y ppa:deadsnakes/ppa && \ + apt-get update && \ apt-get install -y --no-install-recommends \ curl \ tzdata \ - software-properties-common \ - python3.10-full \ - python3.10-distutils \ + python3.11-full \ + python3.11-distutils \ build-essential \ - python3.10-dev \ + python3.11-dev \ cython3 && \ ln -fs /usr/share/zoneinfo/$TZ /etc/localtime && \ dpkg-reconfigure --frontend noninteractive tzdata && \ rm -rf /var/lib/apt/lists/* -# 2) Symlink python3 to 3.10 -RUN ln -sf /usr/bin/python3.10 /usr/bin/python3 && \ - ln -sf /usr/bin/python3.10 /usr/bin/python +# 2) Symlink python3 to 3.11 +RUN ln -sf /usr/bin/python3.11 /usr/bin/python3 && \ + ln -sf /usr/bin/python3.11 /usr/bin/python # 3) Install pip, setuptools & wheel RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3 && \ python3 -m pip install --upgrade pip setuptools wheel # 4) Copy the Beam SDK harness (for Dataflow workers) -COPY --from=gcr.io/apache-beam-testing/beam-sdk/beam_python3.10_sdk:latest \ +COPY --from=gcr.io/apache-beam-testing/beam-sdk/beam_python3.11_sdk:latest \ /opt/apache/beam /opt/apache/beam # 5) Make sure the harness is discovered first diff --git a/sdks/python/apache_beam/runners/dataflow/internal/apiclient_test.py b/sdks/python/apache_beam/runners/dataflow/internal/apiclient_test.py index 368ac5dbe381..761ce0855f08 100644 --- a/sdks/python/apache_beam/runners/dataflow/internal/apiclient_test.py +++ b/sdks/python/apache_beam/runners/dataflow/internal/apiclient_test.py @@ -1189,7 +1189,7 @@ def test_interpreter_version_check_passes_with_experiment(self): @mock.patch( 'apache_beam.runners.dataflow.internal.apiclient.sys.version_info', - (3, 8, 2)) + (3, 10, 0)) @mock.patch( 'apache_beam.runners.dataflow.internal.apiclient.' 'beam_version.__version__', diff --git a/sdks/python/container/distroless/py310/build.gradle b/sdks/python/container/distroless/py310/build.gradle deleted file mode 100644 index 28894725165a..000000000000 --- a/sdks/python/container/distroless/py310/build.gradle +++ /dev/null @@ -1,28 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * License); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an AS IS BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -plugins { - id 'base' - id 'org.apache.beam.module' -} -applyDockerNature() -applyPythonNature() - -pythonVersion = '3.10' - -apply from: "../common.gradle" diff --git a/sdks/python/container/ml/py310/base_image_requirements.txt b/sdks/python/container/ml/py310/base_image_requirements.txt deleted file mode 100644 index 9b149a59a12b..000000000000 --- a/sdks/python/container/ml/py310/base_image_requirements.txt +++ /dev/null @@ -1,237 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# Autogenerated requirements file for Apache Beam py310 container image. -# Run ./gradlew :sdks:python:container:generatePythonRequirementsAll to update. -# Do not edit manually, adjust ../base_image_requirements_manual.txt or -# Apache Beam's setup.py instead, and regenerate the list. -# You will need Python interpreters for all versions supported by Beam, see: -# https://s.apache.org/beam-python-dev-wiki -# Reach out to a committer if you need help. - -absl-py==2.5.0 -aiohappyeyeballs==2.7.1 -aiohttp==3.14.3 -aiosignal==1.4.0 -annotated-types==0.8.0 -anyio==4.14.2 -asn1crypto==1.5.1 -astunparse==1.6.3 -async-timeout==5.0.1 -attrs==26.1.0 -backports.tarfile==1.2.0 -beartype==0.22.9 -beautifulsoup4==4.15.0 -bs4==0.0.2 -build==1.5.0 -cachetools==6.2.6 -certifi==2026.10.23 -cffi==2.1.1 -charset-normalizer==3.5.0 -click==8.4.2 -cloud-sql-python-connector==1.21.0 -crcmod==1.7 -cryptography==50.0.0 -Cython==3.2.9 -defusedxml==0.7.1 -dill==0.3.1.1 -distro==1.9.0 -dnspython==2.8.0 -docker==7.2.0 -docstring_parser==0.18.0 -exceptiongroup==1.3.1 -execnet==2.1.2 -fastavro==1.12.2 -fasteners==0.20 -filelock==3.32.3 -flatbuffers==25.12.19 -freezegun==1.5.5 -frozenlist==1.8.0 -fsspec==2026.7.0 -future==1.0.0 -gast==0.7.0 -google-api-core==2.34.0 -google-api-python-client==2.198.0 -google-apitools==0.5.31 -google-auth==2.56.3 -google-auth-httplib2==0.2.1 -google-cloud-aiplatform==1.164.0 -google-cloud-bigquery==3.43.0 -google-cloud-bigquery-storage==2.40.0 -google-cloud-bigtable==2.44.0 -google-cloud-build==3.38.1 -google-cloud-core==2.6.1 -google-cloud-dataflow-client==0.13.0 -google-cloud-datastore==2.26.0 -google-cloud-dlp==3.38.0 -google-cloud-kms==3.16.0 -google-cloud-language==2.21.0 -google-cloud-monitoring==2.31.0 -google-cloud-profiler==4.1.0 -google-cloud-pubsub==2.39.1 -google-cloud-recommendations-ai==0.10.18 -google-cloud-resource-manager==1.18.0 -google-cloud-secret-manager==2.30.0 -google-cloud-spanner==3.69.1 -google-cloud-storage==3.13.1 -google-cloud-videointelligence==2.20.0 -google-cloud-vision==3.15.0 -google-crc32c==1.8.0 -google-genai==2.18.1 -google-pasta==0.2.0 -google-resumable-media==2.10.1 -googleapis-common-protos==1.75.1 -greenlet==3.5.5 -grpc-google-iam-v1==0.14.5 -grpc-interceptor==0.15.4 -grpcio==1.85.0 -grpcio-status==1.85.0 -guppy3==3.1.7 -h11==0.16.0 -h2==4.4.1 -h5py==3.14.0 -hf-xet==1.6.0 -hpack==4.2.0 -httpcore==1.0.9 -httplib2==0.32.1 -httpx==0.28.1 -huggingface_hub==0.36.2 -hyperframe==6.1.0 -hypothesis==6.148.3 -idna==3.18 -importlib_metadata==9.0.0 -iniconfig==2.3.0 -jaraco.classes==3.4.0 -jaraco.context==6.1.2 -jaraco.functools==4.6.0 -jeepney==0.9.0 -Jinja2==3.1.6 -joblib==1.5.3 -jsonpickle==4.1.2 -jsonschema==4.26.0 -jsonschema-specifications==2025.9.1 -keras==3.12.4 -keyring==25.7.0 -keyrings.google-artifactregistry-auth==1.1.2 -libclang==18.1.1 -linkify-it-py==2.1.0 -markdown-it-py==4.2.0 -MarkupSafe==3.0.3 -mdit-py-plugins==0.6.1 -mdurl==0.1.2 -memray==1.19.3 -ml_dtypes==0.6.0 -mmh3==5.2.1 -mock==5.2.0 -more-itertools==11.1.0 -mpmath==1.3.0 -multidict==6.7.1 -namex==0.1.0 -networkx==3.4.2 -nltk==3.10.3 -numpy==2.2.6 -oauth2client==4.1.3 -objsize==0.7.1 -opentelemetry-api==1.44.0 -opentelemetry-resourcedetector-gcp==1.14.0 -opentelemetry-sdk==1.44.0 -opentelemetry-semantic-conventions==0.65b0 -opt_einsum==3.4.0 -optree==0.19.1 -oracledb==4.0.2 -orjson==3.12.0 -packaging==26.3 -pandas==2.3.3 -parameterized==0.9.0 -pg8000==1.31.5 -pillow==12.3.0 -pip==26.2.2 -platformdirs==4.11.3 -pluggy==1.6.0 -portalocker==3.2.0 -propcache==0.5.2 -proto-plus==1.28.3 -protobuf==6.33.6 -psycopg2-binary==2.9.12 -pyarrow==25.0.1 -pyasn1==0.6.4 -pyasn1_modules==0.4.2 -pycparser==3.0 -pydantic==2.13.4 -pydantic_core==2.46.4 -pydot==1.4.2 -Pygments==2.20.0 -PyHamcrest==2.1.0 -pymongo==4.17.0 -PyMySQL==1.2.0 -pyparsing==3.3.2 -pyproject_hooks==1.2.0 -pystack==1.7.1 -pytest==9.1.1 -pytest-timeout==2.4.0 -pytest-xdist==3.8.0 -python-dateutil==2.9.0.post0 -python-dotenv==1.2.2 -python-tds==1.17.1 -pytz==2026.3.post1 -PyYAML==6.0.3 -qdrant-client==1.19.0 -quickjs-ng==0.16.0.1 -referencing==0.37.0 -regex==2026.7.19 -requests==2.34.2 -requests-mock==1.12.1 -rich==15.0.0 -rpds-py==0.30.0 -rsa==4.9.1 -safetensors==0.8.0 -scikit-learn==1.7.2 -scipy==1.15.3 -scramp==1.4.17 -SecretStorage==3.5.0 -setuptools==84.0.0 -six==1.17.0 -sniffio==1.3.1 -sortedcontainers==2.4.0 -soupsieve==2.9.2 -SQLAlchemy==2.0.52 -sqlalchemy_pytds==1.0.2 -sqlparse==0.6.0 -sympy==1.14.0 -tenacity==9.1.4 -tensorflow==2.21.0 -tensorflow-cpu-aws==2.21.0;platform_machine=="aarch64" -termcolor==3.3.0 -testcontainers==4.15.0 -textual==8.2.8 -threadpoolctl==3.6.0 -tokenizers==0.21.4 -tomli==2.4.1 -torch==2.8.0+cpu -tqdm==4.70.0 -transformers==4.55.4 -typing-inspection==0.4.4 -typing_extensions==4.16.0 -tzdata==2026.3 -uc-micro-py==2.0.0 -uritemplate==4.2.0 -urllib3==2.7.0 -virtualenv-clone==0.5.7 -websockets==16.1.1 -wheel==0.48.0 -wrapt==2.3.0 -yarl==1.24.5 -zipp==4.1.0 -zstandard==0.25.0 diff --git a/sdks/python/container/ml/py310/build.gradle b/sdks/python/container/ml/py310/build.gradle deleted file mode 100644 index 28894725165a..000000000000 --- a/sdks/python/container/ml/py310/build.gradle +++ /dev/null @@ -1,28 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * License); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an AS IS BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -plugins { - id 'base' - id 'org.apache.beam.module' -} -applyDockerNature() -applyPythonNature() - -pythonVersion = '3.10' - -apply from: "../common.gradle" diff --git a/sdks/python/container/ml/py310/gpu_image_requirements.txt b/sdks/python/container/ml/py310/gpu_image_requirements.txt deleted file mode 100644 index cc09fd7ab598..000000000000 --- a/sdks/python/container/ml/py310/gpu_image_requirements.txt +++ /dev/null @@ -1,334 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# Autogenerated requirements file for Apache Beam py310 container image. -# Run ./gradlew :sdks:python:container:generatePythonRequirementsAll to update. -# Do not edit manually, adjust ../base_image_requirements_manual.txt or -# Apache Beam's setup.py instead, and regenerate the list. -# You will need Python interpreters for all versions supported by Beam, see: -# https://s.apache.org/beam-python-dev-wiki -# Reach out to a committer if you need help. - -absl-py==2.5.0 -aiohappyeyeballs==2.7.1 -aiohttp==3.14.3 -aiosignal==1.4.0 -annotated-doc==0.0.5 -annotated-types==0.8.0 -anyio==4.14.2 -asn1crypto==1.5.1 -astor==0.8.1 -astunparse==1.6.3 -async-timeout==5.0.1 -attrs==26.1.0 -backports.tarfile==1.2.0 -beartype==0.22.9 -beautifulsoup4==4.15.0 -blake3==1.0.9 -bs4==0.0.2 -build==1.5.0 -cachetools==6.2.6 -cbor2==6.1.4 -certifi==2026.10.23 -cffi==2.1.1 -charset-normalizer==3.5.0 -click==8.4.2 -cloud-sql-python-connector==1.21.0 -cloudpickle==3.1.2 -compressed-tensors==0.10.2 -crcmod==1.7 -cryptography==50.0.0 -cuda-bindings==13.3.1 -cuda-pathfinder==1.6.0 -cuda-toolkit==13.0.3.0 -cupy-cuda12x==14.1.1 -Cython==3.2.9 -defusedxml==0.7.1 -depyf==0.19.0 -detect-installer==0.1.0 -dill==0.3.1.1 -diskcache==5.6.3 -distro==1.9.0 -dnspython==2.8.0 -docker==7.2.0 -docstring_parser==0.18.0 -einops==0.8.2 -email-validator==2.3.0 -exceptiongroup==1.3.1 -execnet==2.1.2 -fastapi==0.141.1 -fastapi-cli==0.0.32 -fastapi-cloud-cli==0.23.0 -fastar==0.11.0 -fastavro==1.12.2 -fasteners==0.20 -filelock==3.32.3 -flatbuffers==25.12.19 -freezegun==1.5.5 -frozenlist==1.8.0 -fsspec==2026.7.0 -future==1.0.0 -gast==0.7.0 -gguf==0.19.0 -google-api-core==2.34.0 -google-api-python-client==2.198.0 -google-apitools==0.5.31 -google-auth==2.56.3 -google-auth-httplib2==0.2.1 -google-cloud-aiplatform==1.164.0 -google-cloud-bigquery==3.43.0 -google-cloud-bigquery-storage==2.40.0 -google-cloud-bigtable==2.44.0 -google-cloud-build==3.38.1 -google-cloud-core==2.6.1 -google-cloud-dataflow-client==0.13.0 -google-cloud-datastore==2.26.0 -google-cloud-dlp==3.38.0 -google-cloud-kms==3.16.0 -google-cloud-language==2.21.0 -google-cloud-monitoring==2.31.0 -google-cloud-profiler==4.1.0 -google-cloud-pubsub==2.39.1 -google-cloud-recommendations-ai==0.10.18 -google-cloud-resource-manager==1.18.0 -google-cloud-secret-manager==2.30.0 -google-cloud-spanner==3.69.1 -google-cloud-storage==3.13.1 -google-cloud-videointelligence==2.20.0 -google-cloud-vision==3.15.0 -google-crc32c==1.8.0 -google-genai==2.18.1 -google-pasta==0.2.0 -google-resumable-media==2.10.1 -googleapis-common-protos==1.75.1 -greenlet==3.5.5 -grpc-google-iam-v1==0.14.5 -grpc-interceptor==0.15.4 -grpcio==1.85.0 -grpcio-status==1.85.0 -guppy3==3.1.7 -h11==0.16.0 -h2==4.4.1 -h5py==3.14.0 -hf-xet==1.6.0 -hpack==4.2.0 -httpcore==1.0.9 -httplib2==0.32.1 -httptools==0.8.0 -httpx==0.28.1 -huggingface_hub==0.36.2 -hyperframe==6.1.0 -hypothesis==6.148.3 -idna==3.18 -importlib_metadata==9.0.0 -iniconfig==2.3.0 -interegular==0.3.3 -jaraco.classes==3.4.0 -jaraco.context==6.1.2 -jaraco.functools==4.6.0 -jeepney==0.9.0 -Jinja2==3.1.6 -jiter==0.16.0 -joblib==1.5.3 -jsonpickle==4.1.2 -jsonschema==4.26.0 -jsonschema-specifications==2025.9.1 -keras==3.12.4 -keyring==25.7.0 -keyrings.google-artifactregistry-auth==1.1.2 -lark==1.2.2 -libclang==18.1.1 -linkify-it-py==2.1.0 -llguidance==0.7.30 -llvmlite==0.44.0 -lm-format-enforcer==0.10.12 -markdown-it-py==4.2.0 -MarkupSafe==3.0.3 -mdit-py-plugins==0.6.1 -mdurl==0.1.2 -memray==1.19.3 -mistral_common==1.11.7 -ml_dtypes==0.6.0 -mmh3==5.2.1 -mock==5.2.0 -more-itertools==11.1.0 -mpmath==1.3.0 -msgpack==1.2.1 -msgspec==0.21.1 -multidict==6.7.1 -namex==0.1.0 -networkx==3.4.2 -ninja==1.13.0 -nltk==3.10.3 -numba==0.61.2 -numpy==2.2.6 -nvidia-cublas==13.1.1.3 -nvidia-cublas-cu12==12.6.4.1 -nvidia-cuda-cupti==13.0.85 -nvidia-cuda-cupti-cu12==12.6.80 -nvidia-cuda-nvrtc==13.0.88 -nvidia-cuda-nvrtc-cu12==12.6.77 -nvidia-cuda-runtime==13.0.96 -nvidia-cuda-runtime-cu12==12.6.77 -nvidia-cudnn-cu12==9.5.1.17 -nvidia-cudnn-cu13==9.20.0.48 -nvidia-cufft==12.0.0.61 -nvidia-cufft-cu12==11.3.0.4 -nvidia-cufile==1.15.1.6 -nvidia-cufile-cu12==1.11.1.6 -nvidia-curand==10.4.0.35 -nvidia-curand-cu12==10.3.7.77 -nvidia-cusolver==12.0.4.66 -nvidia-cusolver-cu12==11.7.1.2 -nvidia-cusparse==12.6.3.3 -nvidia-cusparse-cu12==12.5.4.2 -nvidia-cusparselt-cu12==0.6.3 -nvidia-cusparselt-cu13==0.8.1 -nvidia-nccl-cu12==2.26.2 -nvidia-nccl-cu13==2.29.7 -nvidia-nvjitlink==13.3.33 -nvidia-nvjitlink-cu12==12.6.85 -nvidia-nvshmem-cu13==3.4.5 -nvidia-nvtx==13.0.85 -nvidia-nvtx-cu12==12.6.77 -oauth2client==4.1.3 -objsize==0.7.1 -openai==1.107.1 -openai-harmony==0.0.8 -opencv-python-headless==5.0.0.93 -opentelemetry-api==1.44.0 -opentelemetry-resourcedetector-gcp==1.14.0 -opentelemetry-sdk==1.44.0 -opentelemetry-semantic-conventions==0.65b0 -opt_einsum==3.4.0 -optree==0.19.1 -oracledb==4.0.2 -orjson==3.12.0 -outlines_core==0.2.10 -packaging==26.3 -pandas==2.3.3 -parameterized==0.9.0 -partial-json-parser==0.2.1.1.post7 -pg8000==1.31.5 -pillow==12.3.0 -pip==26.2.2 -platformdirs==4.11.3 -pluggy==1.6.0 -portalocker==3.2.0 -prometheus-fastapi-instrumentator==8.1.0 -prometheus_client==0.26.0 -propcache==0.5.2 -proto-plus==1.28.3 -protobuf==6.33.6 -psutil==7.2.2 -psycopg2-binary==2.9.12 -py-cpuinfo==9.0.0 -pyarrow==25.0.1 -pyasn1==0.6.4 -pyasn1_modules==0.4.2 -pybase64==1.5.0 -pycountry==26.2.16 -pycparser==3.0 -pydantic==2.13.4 -pydantic-extra-types==2.11.2 -pydantic-settings==2.15.0 -pydantic_core==2.46.4 -pydot==1.4.2 -Pygments==2.20.0 -PyHamcrest==2.1.0 -pymongo==4.17.0 -PyMySQL==1.2.0 -pyparsing==3.3.2 -pyproject_hooks==1.2.0 -pystack==1.7.1 -pytest==9.1.1 -pytest-timeout==2.4.0 -pytest-xdist==3.8.0 -python-dateutil==2.9.0.post0 -python-dotenv==1.2.2 -python-json-logger==4.1.0 -python-multipart==0.0.32 -python-tds==1.17.1 -pytz==2026.3.post1 -PyYAML==6.0.3 -pyzmq==27.1.0 -qdrant-client==1.19.0 -ray==2.57.0 -referencing==0.37.0 -regex==2026.7.19 -requests==2.34.2 -requests-mock==1.12.1 -rich==15.0.0 -rich-toolkit==0.20.3 -rignore==0.8.1 -rpds-py==0.30.0 -rsa==4.9.1 -safetensors==0.8.0 -scikit-learn==1.7.2 -scipy==1.15.3 -scramp==1.4.17 -SecretStorage==3.5.0 -sentencepiece==0.2.2 -sentry-sdk==2.68.0 -setproctitle==1.3.7 -setuptools==84.0.0 -shellingham==1.5.4 -six==1.17.0 -sniffio==1.3.1 -sortedcontainers==2.4.0 -soundfile==0.14.0 -soupsieve==2.9.2 -soxr==1.1.0 -SQLAlchemy==2.0.52 -sqlalchemy_pytds==1.0.2 -sqlparse==0.6.0 -starlette==1.6.0 -sympy==1.14.0 -tenacity==9.1.4 -tensorflow==2.21.0 -tensorflow-cpu-aws==2.21.0;platform_machine=="aarch64" -termcolor==3.3.0 -testcontainers==4.15.0 -textual==8.2.8 -threadpoolctl==3.6.0 -tiktoken==0.13.0 -tokenizers==0.21.4 -tomli==2.4.1 -torch==2.7.1 -torchaudio==2.7.1 -torchvision==0.22.1 -tqdm==4.70.0 -transformers==4.55.4 -triton==3.3.1 -typer==0.27.1 -typing-inspection==0.4.4 -typing_extensions==4.16.0 -tzdata==2026.3 -uc-micro-py==2.0.0 -uritemplate==4.2.0 -urllib3==2.7.0 -uvicorn==0.52.3 -uvloop==0.22.1 -virtualenv-clone==0.5.7 -vllm==0.10.1.1 -watchfiles==1.2.0 -websockets==16.1.1 -wheel==0.48.0 -wrapt==2.3.0 -xformers==0.0.31 -xgrammar==0.1.21 -yarl==1.24.5 -zipp==4.1.0 -zstandard==0.25.0 diff --git a/sdks/python/container/py310/base_image_requirements.txt b/sdks/python/container/py310/base_image_requirements.txt deleted file mode 100644 index e1ac27a882b8..000000000000 --- a/sdks/python/container/py310/base_image_requirements.txt +++ /dev/null @@ -1,211 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# Autogenerated requirements file for Apache Beam py310 container image. -# Run ./gradlew :sdks:python:container:generatePythonRequirementsAll to update. -# Do not edit manually, adjust ../base_image_requirements_manual.txt or -# Apache Beam's setup.py instead, and regenerate the list. -# You will need Python interpreters for all versions supported by Beam, see: -# https://s.apache.org/beam-python-dev-wiki -# Reach out to a committer if you need help. - -aiohappyeyeballs==2.7.1 -aiohttp==3.14.3 -aiosignal==1.4.0 -annotated-types==0.8.0 -anyio==4.14.2 -asn1crypto==1.5.1 -async-timeout==5.0.1 -attrs==26.1.0 -backports.tarfile==1.2.0 -beartype==0.22.9 -beautifulsoup4==4.15.0 -bs4==0.0.2 -build==1.5.0 -cachetools==6.2.6 -certifi==2026.10.23 -cffi==2.1.1 -charset-normalizer==3.5.0 -click==8.4.2 -cloud-sql-python-connector==1.21.0 -crcmod==1.7 -cryptography==50.0.0 -Cython==3.2.9 -defusedxml==0.7.1 -dill==0.3.1.1 -distro==1.9.0 -dnspython==2.8.0 -docker==7.2.0 -docstring_parser==0.18.0 -exceptiongroup==1.3.1 -execnet==2.1.2 -fastavro==1.12.2 -fasteners==0.20 -freezegun==1.5.5 -frozenlist==1.8.0 -future==1.0.0 -google-api-core==2.34.0 -google-api-python-client==2.198.0 -google-apitools==0.5.31 -google-auth==2.56.3 -google-auth-httplib2==0.2.1 -google-cloud-aiplatform==1.164.0 -google-cloud-bigquery==3.43.0 -google-cloud-bigquery-storage==2.40.0 -google-cloud-bigtable==2.44.0 -google-cloud-build==3.38.1 -google-cloud-core==2.6.1 -google-cloud-dataflow-client==0.13.0 -google-cloud-datastore==2.26.0 -google-cloud-dlp==3.38.0 -google-cloud-kms==3.16.0 -google-cloud-language==2.21.0 -google-cloud-monitoring==2.31.0 -google-cloud-profiler==4.1.0 -google-cloud-pubsub==2.39.1 -google-cloud-recommendations-ai==0.10.18 -google-cloud-resource-manager==1.18.0 -google-cloud-secret-manager==2.30.0 -google-cloud-spanner==3.69.1 -google-cloud-storage==3.13.1 -google-cloud-videointelligence==2.20.0 -google-cloud-vision==3.15.0 -google-crc32c==1.8.0 -google-genai==2.18.1 -google-resumable-media==2.10.1 -googleapis-common-protos==1.75.1 -greenlet==3.5.5 -grpc-google-iam-v1==0.14.5 -grpc-interceptor==0.15.4 -grpcio==1.85.0 -grpcio-status==1.85.0 -guppy3==3.1.7 -h11==0.16.0 -h2==4.4.1 -hpack==4.2.0 -httpcore==1.0.9 -httplib2==0.32.1 -httpx==0.28.1 -hyperframe==6.1.0 -hypothesis==6.148.3 -idna==3.18 -importlib_metadata==9.0.0 -iniconfig==2.3.0 -jaraco.classes==3.4.0 -jaraco.context==6.1.2 -jaraco.functools==4.6.0 -jeepney==0.9.0 -Jinja2==3.1.6 -joblib==1.5.3 -jsonpickle==4.1.2 -jsonschema==4.26.0 -jsonschema-specifications==2025.9.1 -keyring==25.7.0 -keyrings.google-artifactregistry-auth==1.1.2 -linkify-it-py==2.1.0 -markdown-it-py==4.2.0 -MarkupSafe==3.0.3 -mdit-py-plugins==0.6.1 -mdurl==0.1.2 -memray==1.19.3 -mmh3==5.2.1 -mock==5.2.0 -more-itertools==11.1.0 -multidict==6.7.1 -nltk==3.10.3 -numpy==2.2.6 -oauth2client==4.1.3 -objsize==0.7.1 -opentelemetry-api==1.44.0 -opentelemetry-resourcedetector-gcp==1.14.0 -opentelemetry-sdk==1.44.0 -opentelemetry-semantic-conventions==0.65b0 -oracledb==4.0.2 -orjson==3.12.0 -packaging==26.3 -pandas==2.3.3 -parameterized==0.9.0 -pg8000==1.31.5 -pillow==12.3.0 -pip==26.2.2 -platformdirs==4.11.3 -pluggy==1.6.0 -portalocker==3.2.0 -propcache==0.5.2 -proto-plus==1.28.3 -protobuf==6.33.6 -psycopg2-binary==2.9.12 -pyarrow==25.0.1 -pyasn1==0.6.4 -pyasn1_modules==0.4.2 -pycparser==3.0 -pydantic==2.13.4 -pydantic_core==2.46.4 -pydot==1.4.2 -Pygments==2.20.0 -PyHamcrest==2.1.0 -pymongo==4.17.0 -PyMySQL==1.2.0 -pyparsing==3.3.2 -pyproject_hooks==1.2.0 -pystack==1.7.1 -pytest==9.1.1 -pytest-timeout==2.4.0 -pytest-xdist==3.8.0 -python-dateutil==2.9.0.post0 -python-dotenv==1.2.2 -python-tds==1.17.1 -pytz==2026.3.post1 -PyYAML==6.0.3 -qdrant-client==1.19.0 -quickjs-ng==0.16.0.1 -referencing==0.37.0 -regex==2026.7.19 -requests==2.34.2 -requests-mock==1.12.1 -rich==15.0.0 -rpds-py==0.30.0 -rsa==4.9.1 -scikit-learn==1.7.2 -scipy==1.15.3 -scramp==1.4.17 -SecretStorage==3.5.0 -setuptools==84.0.0 -six==1.17.0 -sniffio==1.3.1 -sortedcontainers==2.4.0 -soupsieve==2.9.2 -SQLAlchemy==2.0.52 -sqlalchemy_pytds==1.0.2 -sqlparse==0.6.0 -tenacity==9.1.4 -testcontainers==4.15.0 -textual==8.2.8 -threadpoolctl==3.6.0 -tomli==2.4.1 -tqdm==4.70.0 -typing-inspection==0.4.4 -typing_extensions==4.16.0 -tzdata==2026.3 -uc-micro-py==2.0.0 -uritemplate==4.2.0 -urllib3==2.7.0 -virtualenv-clone==0.5.7 -websockets==16.1.1 -wheel==0.48.0 -wrapt==2.3.0 -yarl==1.24.5 -zipp==4.1.0 -zstandard==0.25.0 diff --git a/sdks/python/container/py310/build.gradle b/sdks/python/container/py310/build.gradle deleted file mode 100644 index 0e645489af80..000000000000 --- a/sdks/python/container/py310/build.gradle +++ /dev/null @@ -1,28 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * License); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an AS IS BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -plugins { - id 'base' - id 'org.apache.beam.module' -} -applyDockerNature() -applyPythonNature() - -pythonVersion = '3.10' - -apply from: "../common.gradle" diff --git a/sdks/python/setup.py b/sdks/python/setup.py index ce9af613cd0c..9f9618e30fbc 100644 --- a/sdks/python/setup.py +++ b/sdks/python/setup.py @@ -353,7 +353,7 @@ def get_portability_package_data(): return files -python_requires = '>=3.10' +python_requires = '>=3.11' if sys.version_info.major == 3 and sys.version_info.minor >= 15: warnings.warn( @@ -626,9 +626,6 @@ def get_portability_package_data(): # which tensorflow-transform 1.21 requires. 'tf2onnx>=1.17.0,<1.18', ] + ml_base_core, - 'p310_ml_test': [ - 'datatable', - ] + ml_base + qdrant_dependency, # xgboost_dependency includes datatable, which only has wheels for # Python <=3.12, so it is limited to the 3.11 and 3.12 test extras. 'p311_ml_test': ml_base + xgboost_dependency + qdrant_dependency, diff --git a/sdks/python/test-suites/dataflow/py310/build.gradle b/sdks/python/test-suites/dataflow/py310/build.gradle deleted file mode 100644 index e3b96494dfcd..000000000000 --- a/sdks/python/test-suites/dataflow/py310/build.gradle +++ /dev/null @@ -1,24 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * License); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an AS IS BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -apply plugin: org.apache.beam.gradle.BeamModulePlugin -applyPythonNature() - -// Required to setup a Python 3 virtualenv and task names. -pythonVersion = '3.10' -apply from: "../common.gradle" diff --git a/sdks/python/test-suites/direct/py310/build.gradle b/sdks/python/test-suites/direct/py310/build.gradle deleted file mode 100644 index d1727740d060..000000000000 --- a/sdks/python/test-suites/direct/py310/build.gradle +++ /dev/null @@ -1,24 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * License); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an AS IS BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -plugins { id 'org.apache.beam.module' } -applyPythonNature() - -// Required to setup a Python 3 virtualenv and task names. -pythonVersion = '3.10' -apply from: '../common.gradle' diff --git a/sdks/python/test-suites/portable/py310/build.gradle b/sdks/python/test-suites/portable/py310/build.gradle deleted file mode 100644 index e1e2e068ac28..000000000000 --- a/sdks/python/test-suites/portable/py310/build.gradle +++ /dev/null @@ -1,26 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * License); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an AS IS BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -apply plugin: org.apache.beam.gradle.BeamModulePlugin -applyPythonNature() - -addPortableWordCountTasks() - -// Required to setup a Python 3 virtualenv and task names. -pythonVersion = '3.10' -apply from: "../common.gradle" diff --git a/sdks/python/test-suites/tox/py310/build.gradle b/sdks/python/test-suites/tox/py310/build.gradle deleted file mode 100644 index 482563cc3522..000000000000 --- a/sdks/python/test-suites/tox/py310/build.gradle +++ /dev/null @@ -1,216 +0,0 @@ -/* - * Licensed to the Apache Software Foundation (ASF) under one - * or more contributor license agreements. See the NOTICE file - * distributed with this work for additional information - * regarding copyright ownership. The ASF licenses this file - * to you under the Apache License, Version 2.0 (the - * License); you may not use this file except in compliance - * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an AS IS BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ - -/** - * Unit tests for Python 3.10 - */ - -plugins { id 'org.apache.beam.module' } -applyPythonNature() - -// Required to setup a Python 3 virtualenv and task names. -pythonVersion = '3.10' - -def posargs = project.findProperty("posargs") ?: "" - -apply from: "../common.gradle" - -toxTask "testPy310CloudCoverage", "py310-cloudcoverage", "${posargs}" -test.dependsOn "testPy310CloudCoverage" -project.tasks.register("preCommitPyCoverage") { - dependsOn = ["testPy310CloudCoverage"] -} - -// Dep Postcommit runs test suites that evaluate compatibility of particular -// dependencies. Each suite is exercised on at most one python version. -// -// Should still leave at least one version in PreCommit unless the marked tests -// are also exercised by existing PreCommit -// e.g. pyarrow and pandas also run on PreCommit Dataframe and Coverage -project.tasks.register("postCommitPyDep") {} - -// Create a test task for supported major versions of pyarrow. -// Keep in sync with [testenv:py{310,311}-pyarrow-...] in tox.ini -// (versions released in roughly the last 1-2 years). - -toxTask "testPy310pyarrow-17", "py310-pyarrow-17", "${posargs}" -test.dependsOn "testPy310pyarrow-17" -postCommitPyDep.dependsOn "testPy310pyarrow-17" - -toxTask "testPy310pyarrow-18", "py310-pyarrow-18", "${posargs}" -test.dependsOn "testPy310pyarrow-18" -postCommitPyDep.dependsOn "testPy310pyarrow-18" - -toxTask "testPy310pyarrow-19", "py310-pyarrow-19", "${posargs}" -test.dependsOn "testPy310pyarrow-19" -postCommitPyDep.dependsOn "testPy310pyarrow-19" - -toxTask "testPy310pyarrow-20", "py310-pyarrow-20", "${posargs}" -test.dependsOn "testPy310pyarrow-20" -postCommitPyDep.dependsOn "testPy310pyarrow-20" - -toxTask "testPy310pyarrow-21", "py310-pyarrow-21", "${posargs}" -test.dependsOn "testPy310pyarrow-21" -postCommitPyDep.dependsOn "testPy310pyarrow-21" - -toxTask "testPy310pyarrow-22", "py310-pyarrow-22", "${posargs}" -test.dependsOn "testPy310pyarrow-22" -postCommitPyDep.dependsOn "testPy310pyarrow-22" - -toxTask "testPy310pyarrow-23", "py310-pyarrow-23", "${posargs}" -test.dependsOn "testPy310pyarrow-23" -postCommitPyDep.dependsOn "testPy310pyarrow-23" - -toxTask "testPy310pyarrow-24", "py310-pyarrow-24", "${posargs}" -test.dependsOn "testPy310pyarrow-24" -postCommitPyDep.dependsOn "testPy310pyarrow-24" - -toxTask "testPy310pyarrow-25", "py310-pyarrow-25", "${posargs}" -test.dependsOn "testPy310pyarrow-25" -postCommitPyDep.dependsOn "testPy310pyarrow-25" - -// Create a test task for each supported minor version of pandas -toxTask "testPy310pandas-14", "py310-pandas-14", "${posargs}" -test.dependsOn "testPy310pandas-14" -postCommitPyDep.dependsOn "testPy310pandas-14" - -toxTask "testPy310pandas-15", "py310-pandas-15", "${posargs}" -test.dependsOn "testPy310pandas-15" -postCommitPyDep.dependsOn "testPy310pandas-15" - -toxTask "testPy310pandas-20", "py310-pandas-20", "${posargs}" -test.dependsOn "testPy310pandas-20" -postCommitPyDep.dependsOn "testPy310pandas-20" - -// TODO(https://github.com/apache/beam/issues/31192): Add below suites -// after dependency compat tests suite switches to Python 3.9 or we add -// Python 2.2 support. - -// toxTask "testPy310pandas-21", "py310-pandas-21", "${posargs}" -// test.dependsOn "testPy310pandas-21" -// postCommitPyDep.dependsOn "testPy310pandas-21" - -// toxTask "testPy310pandas-22", "py310-pandas-22", "${posargs}" -// test.dependsOn "testPy310pandas-22" -// postCommitPyDep.dependsOn "testPy310pandas-22" - -// TODO(https://github.com/apache/beam/issues/30908): Revise what are we testing - -// Create a test task for each minor version of pytorch - -toxTask "testPy310pytorch-111", "py310-pytorch-111", "${posargs}" -test.dependsOn "testPy310pytorch-111" -postCommitPyDep.dependsOn "testPy310pytorch-111" - -toxTask "testPy310pytorch-112", "py310-pytorch-112", "${posargs}" -test.dependsOn "testPy310pytorch-112" -postCommitPyDep.dependsOn "testPy310pytorch-112" - -toxTask "testPy310pytorch-113", "py310-pytorch-113", "${posargs}" -test.dependsOn "testPy310pytorch-113" -postCommitPyDep.dependsOn "testPy310pytorch-113" - -// run on precommit -toxTask "testPy310pytorch-200", "py310-pytorch-200", "${posargs}" -test.dependsOn "testPy310pytorch-200" -postCommitPyDep.dependsOn "testPy310pytorch-200" - -toxTask "testPy310tft-113", "py310-tft-113", "${posargs}" -test.dependsOn "testPy310tft-113" -postCommitPyDep.dependsOn "testPy310tft-113" - -// TODO(https://github.com/apache/beam/issues/25796) - uncomment onnx tox task once onnx supports protobuf 4.x.x -// Create a test task for each minor version of onnx -// toxTask "testPy310onnx-113", "py310-onnx-113", "${posargs}" -// test.dependsOn "testPy310onnx-113" -// postCommitPyDep.dependsOn "testPy310onnx-113" - -// Create a test task for each minor version of tensorflow -toxTask "testPy310tensorflow-212", "py310-tensorflow-212", "${posargs}" -test.dependsOn "testPy310tensorflow-212" -postCommitPyDep.dependsOn "testPy310tensorflow-212" - -// Create a test task for each minor version of transformers -toxTask "testPy310transformers-428", "py310-transformers-428", "${posargs}" -test.dependsOn "testPy310transformers-428" -postCommitPyDep.dependsOn "testPy310transformers-428" - -toxTask "testPy310transformers-447", "py310-transformers-447", "${posargs}" -test.dependsOn "testPy310transformers-447" -postCommitPyDep.dependsOn "testPy310transformers-447" - -toxTask "testPy310transformers-latest", "py310-transformers-latest", "${posargs}" -test.dependsOn "testPy310transformers-latest" -postCommitPyDep.dependsOn "testPy310transformers-latest" - -toxTask "testPy310embeddingsMLTransform", "py310-embeddings", "${posargs}" -test.dependsOn "testPy310embeddingsMLTransform" -postCommitPyDep.dependsOn "testPy310embeddingsMLTransform" - -// Part of MLTransform embeddings test suite but requires tensorflow hub, which we need to test on -// mutliple versions so keeping this suite separate. -toxTask "testPy310TensorflowHubEmbeddings-014", "py310-TFHubEmbeddings-014", "${posargs}" -test.dependsOn "testPy310TensorflowHubEmbeddings-014" -postCommitPyDep.dependsOn "testPy310TensorflowHubEmbeddings-014" - -toxTask "testPy310TensorflowHubEmbeddings-015", "py310-TFHubEmbeddings-015", "${posargs}" -test.dependsOn "testPy310TensorflowHubEmbeddings-015" -postCommitPyDep.dependsOn "testPy310TensorflowHubEmbeddings-015" - -toxTask "whitespacelint", "whitespacelint", "${posargs}" - -task archiveFilesToLint(type: Zip) { - archiveFileName = "files-to-whitespacelint.zip" - destinationDirectory = file("$buildDir/dist") - - from ("$rootProject.projectDir") { - include "**/*.md" - include "**/build.gradle" - include '**/build.gradle.kts' - exclude '**/build/**' // intermediate build directory - exclude 'website/www/site/themes/docsy/**' // fork to google/docsy - exclude "**/node_modules/*" - exclude "**/.gogradle/*" - } -} - -task unpackFilesToLint(type: Copy) { - from zipTree("$buildDir/dist/files-to-whitespacelint.zip") - into "$buildDir/files-to-whitespacelint" -} - -whitespacelint.dependsOn archiveFilesToLint, unpackFilesToLint -unpackFilesToLint.dependsOn archiveFilesToLint -archiveFilesToLint.dependsOn cleanPython - -toxTask "jest", "jest", "${posargs}" - -toxTask "eslint", "eslint", "${posargs}" - -task copyTsSource(type: Copy) { - from ("$rootProject.projectDir") { - include "sdks/python/apache_beam/runners/interactive/extensions/**/*" - exclude "sdks/python/apache_beam/runners/interactive/extensions/**/lib/*" - exclude "sdks/python/apache_beam/runners/interactive/extensions/**/node_modules/*" - } - into "$buildDir/ts" -} - -jest.dependsOn copyTsSource -eslint.dependsOn copyTsSource -copyTsSource.dependsOn cleanPython diff --git a/sdks/python/tox.ini b/sdks/python/tox.ini index c55a989ec7d2..c7a7016a391d 100644 --- a/sdks/python/tox.ini +++ b/sdks/python/tox.ini @@ -533,9 +533,7 @@ commands = deps = sentence-transformers==3.3.1 accelerate>=1.6.0 - # Use Python version-specific transformers constraints to avoid union type syntax issues - transformers>=4.28.0,<4.55.0; python_version < "3.10" - transformers>=4.28.0,<4.56.0; python_version >= "3.10" + transformers>=4.28.0,<4.56.0 passenv = HF_INFERENCE_TOKEN extras = test,gcp commands = diff --git a/settings.gradle.kts b/settings.gradle.kts index 612f99259040..0d16eb1db1ca 100644 --- a/settings.gradle.kts +++ b/settings.gradle.kts @@ -312,44 +312,37 @@ include(":sdks:python:apache_beam:testing:load_tests") include(":sdks:python:apache_beam:testing:benchmarks:nexmark") include(":sdks:python:apache_beam:testing:benchmarks:tpcds") include(":sdks:python:container") -include(":sdks:python:container:py310") include(":sdks:python:container:py311") include(":sdks:python:container:py312") include(":sdks:python:container:py313") include(":sdks:python:container:py314") include(":sdks:python:container:distroless") -include(":sdks:python:container:distroless:py310") include(":sdks:python:container:distroless:py311") include(":sdks:python:container:distroless:py312") include(":sdks:python:container:distroless:py313") include(":sdks:python:container:distroless:py314") include(":sdks:python:container:ml") -include(":sdks:python:container:ml:py310") include(":sdks:python:container:ml:py311") include(":sdks:python:container:ml:py312") include(":sdks:python:container:ml:py313") include(":sdks:python:container:ml:py314") include(":sdks:python:expansion-service-container") include(":sdks:python:test-suites:dataflow") -include(":sdks:python:test-suites:dataflow:py310") include(":sdks:python:test-suites:dataflow:py311") include(":sdks:python:test-suites:dataflow:py312") include(":sdks:python:test-suites:dataflow:py313") include(":sdks:python:test-suites:dataflow:py314") include(":sdks:python:test-suites:direct") -include(":sdks:python:test-suites:direct:py310") include(":sdks:python:test-suites:direct:py311") include(":sdks:python:test-suites:direct:py312") include(":sdks:python:test-suites:direct:py313") include(":sdks:python:test-suites:direct:py314") include(":sdks:python:test-suites:direct:xlang") -include(":sdks:python:test-suites:portable:py310") include(":sdks:python:test-suites:portable:py311") include(":sdks:python:test-suites:portable:py312") include(":sdks:python:test-suites:portable:py313") include(":sdks:python:test-suites:portable:py314") include(":sdks:python:test-suites:tox:pycommon") -include(":sdks:python:test-suites:tox:py310") include(":sdks:python:test-suites:tox:py311") include(":sdks:python:test-suites:tox:py312") include(":sdks:python:test-suites:tox:py313") diff --git a/website/www/site/content/en/documentation/programming-guide.md b/website/www/site/content/en/documentation/programming-guide.md index 3f37f45bace6..1ee679d30635 100644 --- a/website/www/site/content/en/documentation/programming-guide.md +++ b/website/www/site/content/en/documentation/programming-guide.md @@ -35,7 +35,7 @@ programming guide, take a look at the {{< language-switcher java py go typescript yaml >}} {{< paragraph class="language-py" >}} -The Python SDK supports Python 3.8, 3.9, 3.10, 3.11, and 3.12. +The Python SDK supports Python 3.11, 3.12, 3.13, and 3.14. {{< /paragraph >}} {{< paragraph class="language-go">}} diff --git a/website/www/site/content/en/documentation/sdks/python.md b/website/www/site/content/en/documentation/sdks/python.md index 0dba9258856e..5e224c6254fa 100644 --- a/website/www/site/content/en/documentation/sdks/python.md +++ b/website/www/site/content/en/documentation/sdks/python.md @@ -89,7 +89,7 @@ Some common errors can occur during worker start-up and prevent jobs from starti 3.10 - ≥ 2.43.0 + 2.43.0 - 2.77.0 3.9 diff --git a/website/www/site/content/en/get-started/quickstart-py.md b/website/www/site/content/en/get-started/quickstart-py.md index e544a1fcc692..6b3c6e80432d 100644 --- a/website/www/site/content/en/get-started/quickstart-py.md +++ b/website/www/site/content/en/get-started/quickstart-py.md @@ -23,7 +23,7 @@ If you're interested in contributing to the Apache Beam Python codebase, see the {{< toc >}} -The Python SDK supports Python 3.10, 3.11, 3.12, 3.13 and 3.14. Beam 2.69.0 was the last release with support for Python 3.9. Beam 2.60.0 was the last release with support for Python 3.8. +The Python SDK supports Python 3.11, 3.12, 3.13 and 3.14. Beam 2.77.0 was the last release with support for Python 3.10. Beam 2.69.0 was the last release with support for Python 3.9. Beam 2.60.0 was the last release with support for Python 3.8. ## Set up your environment diff --git a/website/www/site/content/en/roadmap/_index.md b/website/www/site/content/en/roadmap/_index.md index d59ccc60fba4..0af2609879bb 100644 --- a/website/www/site/content/en/roadmap/_index.md +++ b/website/www/site/content/en/roadmap/_index.md @@ -68,12 +68,12 @@ Learn more from the [Beam Element Extended Metadata Design Doc](https://s.apache The Go SDK is not actively being developed beyond bugfixes due to lack of contributors. If you'd like to help move this forward again, see the [Go SDK's Roadmap](/roadmap/go-sdk) -## Python 3 support +## Python SDK -As of Apache Beam 2.75.0, we support python version from 3.10 uptil Python 3.14. Supporting new Python 3 minor version is in our roadmap. +As of Apache Beam 2.78.0, we support Python 3.11, 3.12, 3.13 and 3.14. See details on -the [Python SDK's Roadmap](/roadmap/python-sdk/#python-3-support). +the [Python SDK's Roadmap](/roadmap/python-sdk/). ## Java support diff --git a/website/www/site/content/en/roadmap/python-sdk.md b/website/www/site/content/en/roadmap/python-sdk.md index d15e97d653fe..e15d35b7539a 100644 --- a/website/www/site/content/en/roadmap/python-sdk.md +++ b/website/www/site/content/en/roadmap/python-sdk.md @@ -23,7 +23,7 @@ _Last updated on Aug 2026._ ## Python Version Support -Apache Beam 2.73.0 and higher support Python 3.10, 3.11, 3.12, 3.13 and 3.14. Beam aims to support new Python minor versions in a timely manner: +Apache Beam supports Python 3.11, 3.12, 3.13 and 3.14. Beam aims to support new Python minor versions in a timely manner: - [Python Version Compatibility matrix](/documentation/sdks/python/#python-version-compatibility)