Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions .github/workflows/README.md

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion .github/workflows/beam_PostCommit_Yaml_Xlang_Direct.yml
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ jobs:
- name: run PostCommit Yaml Xlang Direct script
uses: ./.github/actions/gradle-command-self-hosted-action
with:
gradle-command: :sdks:python:postCommitYamlIntegrationTests -PyamlTestSet=${{ matrix.test_set }} -PbeamPythonExtra=p310_ml_test,yaml,gcp_test
gradle-command: :sdks:python:postCommitYamlIntegrationTests -PyamlTestSet=${{ matrix.test_set }} -PbeamPythonExtra=p311_ml_test,yaml,gcp_test
- name: Archive Python Test Results
uses: actions/upload-artifact@v7
if: failure()
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/beam_PreCommit_Yaml_Xlang_Direct.yml
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ jobs:
- name: run PreCommit Yaml Xlang Direct script
uses: ./.github/actions/gradle-command-self-hosted-action
with:
gradle-command: :sdks:python:yamlIntegrationTests -PbeamPythonExtra=p310_ml_test,yaml,gcp_test
gradle-command: :sdks:python:yamlIntegrationTests -PbeamPythonExtra=p311_ml_test,yaml,gcp_test
- name: Archive Python Test Results
uses: actions/upload-artifact@v7
if: failure()
Expand Down
3 changes: 1 addition & 2 deletions .github/workflows/build_release_candidate.yml
Original file line number Diff line number Diff line change
Expand Up @@ -280,11 +280,10 @@ jobs:
# tasks as they are added.
images_to_publish: [
{"gradle_task": ":pushAllRunnersDockerImages", "include_skip_flags": "-Pinclude-ml -Pinclude-distroless"},
{"gradle_task": ":sdks:python:container:push310", "include_skip_flags": "-Pinclude-ml -Pinclude-distroless"},
{"gradle_task": ":sdks:python:container:push311", "include_skip_flags": "-Pinclude-ml -Pinclude-distroless"},
{"gradle_task": ":sdks:python:container:push312", "include_skip_flags": "-Pinclude-ml -Pinclude-distroless"},
{"gradle_task": ":sdks:python:container:push313", "include_skip_flags": "-Pinclude-ml -Pinclude-distroless"},
{"gradle_task": ":sdks:python:container:pushAll", "include_skip_flags": "-Pinclude-ml -Pinclude-distroless -Pskip-python-39-images -Pskip-python-310-images -Pskip-python-311-images -Pskip-python-312-images -Pskip-python-313-images"},
{"gradle_task": ":sdks:python:container:pushAll", "include_skip_flags": "-Pinclude-ml -Pinclude-distroless -Pskip-python-311-images -Pskip-python-312-images -Pskip-python-313-images"},
{"gradle_task": ":pushAllSdkDockerImages", "include_skip_flags": "-Pskip-python-images"},
{"gradle_task": ":pushAllDockerImages", "include_skip_flags": "-Pskip-runner-images -Pskip-sdk-images"}
]
Expand Down
3 changes: 1 addition & 2 deletions .github/workflows/republish_released_docker_containers.yml
Original file line number Diff line number Diff line change
Expand Up @@ -47,11 +47,10 @@ jobs:
# tasks as they are added.
images_to_publish: [
{"gradle_task": ":pushAllRunnersDockerImages", "include_skip_flags": "-Pinclude-ml -Pinclude-distroless"},
{"gradle_task": ":sdks:python:container:push310", "include_skip_flags": "-Pinclude-ml -Pinclude-distroless"},
{"gradle_task": ":sdks:python:container:push311", "include_skip_flags": "-Pinclude-ml -Pinclude-distroless"},
{"gradle_task": ":sdks:python:container:push312", "include_skip_flags": "-Pinclude-ml -Pinclude-distroless"},
{"gradle_task": ":sdks:python:container:push313", "include_skip_flags": "-Pinclude-ml -Pinclude-distroless"},
{"gradle_task": ":sdks:python:container:pushAll", "include_skip_flags": "-Pinclude-ml -Pinclude-distroless -Pskip-python-39-images -Pskip-python-310-images -Pskip-python-311-images -Pskip-python-312-images -Pskip-python-313-images"},
{"gradle_task": ":sdks:python:container:pushAll", "include_skip_flags": "-Pinclude-ml -Pinclude-distroless -Pskip-python-311-images -Pskip-python-312-images -Pskip-python-313-images"},
{"gradle_task": ":pushAllSdkDockerImages", "include_skip_flags": "-Pskip-python-images"},
{"gradle_task": ":pushAllDockerImages", "include_skip_flags": "-Pskip-runner-images -Pskip-sdk-images"}
]
Expand Down
4 changes: 2 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,8 +53,8 @@ These steps and instructions on getting started are outlined below as well.
- For running test suites, however, you will need Python interpreters for all Python versions supported by Beam.
Interpreters should be installed and available in shell via `python3.x` commands.
See Python installation tips in [Developer Wiki](https://cwiki.apache.org/confluence/display/BEAM/Python+Tips#PythonTips-InstallingPythoninterpreters).
- If you encounter errors like `python3.10: command not found` or `Cannot run program "python3.10": No such file or directory`, your system may be missing some Python versions.
Workaround: install missing versions with `pyenv` (for example, `pyenv install 3.10` then `pyenv global 3.10 3.11 3.12 3.13` to make all available), or create symlinks in `/usr/local/bin/` pointing to installed Python binaries.
- If you encounter errors like `python3.11: command not found` or `Cannot run program "python3.11": No such file or directory`, your system may be missing some Python versions.
Workaround: install missing versions with `pyenv` (for example, `pyenv install 3.11` then `pyenv global 3.11 3.12 3.13 3.14` to make all available), or create symlinks in `/usr/local/bin/` pointing to installed Python binaries.
- For large contributions, a signed [Individual Contributor License
Agreement](https://www.apache.org/licenses/icla.pdf) (ICLA) to the Apache
Software Foundation (ASF).
Expand Down
18 changes: 9 additions & 9 deletions contributor-docs/python-tips.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,22 +57,22 @@ Installation steps may look as follows:
2. Install Python intepreter for each supported Python minor version. Leaving out the patch version will install the latest.

```bash
pyenv install 3.10
pyenv install 3.11
pyenv install 3.12
pyenv install 3.13
pyenv install 3.14
```

3. Make installed interpreters available in your shell by first running:

```bash
pyenv global 3.10 3.11 3.12 3.13
pyenv global 3.11 3.12 3.13 3.14
```

4. (**OPTIONAL**) Pyenv will sometimes [fail to make these interpreters directly available](https://github.com/pyenv/pyenv/issues/34) without a local configuration. If you see errors trying to use `python3.x`, then also run `pyenv local`  

```bash
pyenv local 3.10 3.11 3.12 3.13
pyenv local 3.11 3.12 3.13 3.14
```

After these steps, all `python3.x` interpreters should be available in your shell. The first version in the list passed to pyenv global will be used as default `python` / `python3` interpreter if the minor version is not specified.
Expand Down Expand Up @@ -156,11 +156,11 @@ curl https://pyenv.run | bash
# Run the outputted commands to initialize pyenv in .bashrc
```

#### Example: How to Run Unit Tests with PyCharm Using Python 3.10.10 in a virtualenv
1. Install Python 3.10.10 and create a virtualenv
#### Example: How to Run Unit Tests with PyCharm Using Python 3.11.4 in a virtualenv
1. Install Python 3.11.4 and create a virtualenv
```bash
pyenv install 3.10.10
pyenv virtualenv 3.10.10 ENV_NAME
pyenv install 3.11.4
pyenv virtualenv 3.11.4 ENV_NAME
pyenv activate ENV_NAME
```

Expand All @@ -171,7 +171,7 @@ pip install --upgrade pip setuptools

3. Set up PyCharm
* Start by adding a new project interpreter (from the bottom right or in Settings).
* Select Existing environment and the interpreter, which should be under ~/.pyenv/versions/3.10.10/envs/ENV_NAME/bin/python or ~/.pyenv/versions/ENV_NAME/bin/python.
* Select Existing environment and the interpreter, which should be under ~/.pyenv/versions/3.11.4/envs/ENV_NAME/bin/python or ~/.pyenv/versions/ENV_NAME/bin/python.
* Switch interpreters at the bottom right.

#### Cleaning up environments
Expand Down Expand Up @@ -519,7 +519,7 @@ NOTE for RELEASE MANAGERS: We should update dependencies at least once per relea
You may see that the pip command will lead to segmentation fault as well. If this happens, remove the python version from pyenv, and reinstall the version like this.

```bash
CFLAGS="-O2" pyenv install 3.10.10
CFLAGS="-O2" pyenv install 3.11.4
```

There have been issues with older Python versions. See [here](https://github.com/pyenv/pyenv/issues/2046) for details.
4 changes: 0 additions & 4 deletions dev-support/docker/pkglist
Original file line number Diff line number Diff line change
Expand Up @@ -32,10 +32,6 @@ time
openjdk-11-jdk
python3-setuptools
python3-pip
python3.10
python3.10-dev
python3.10-distutils
python3.10-venv
python3.11
python3.11-dev
python3.11-distutils
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -82,9 +82,7 @@ function get_version() {
#######################################
function download_files() {
if [[ $1 = *"wheel"* ]]; then
if [[ $2 == "python3.10" ]]; then
BEAM_PYTHON_SDK_WHL="apache_beam-$VERSION*-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl"
elif [[ $2 == "python3.11" ]]; then
if [[ $2 == "python3.11" ]]; then
BEAM_PYTHON_SDK_WHL="apache_beam-$VERSION*-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl"
elif [[ $2 == "python3.12" ]]; then
BEAM_PYTHON_SDK_WHL="apache_beam-$VERSION*-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl"
Expand Down
6 changes: 3 additions & 3 deletions release/src/main/scripts/github_actions_jobs.txt
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ Run Go PostCommit Dataflow ARM,beam_PostCommit_Go_Dataflow_ARM
Run Java_Examples_Dataflow_ARM PostCommit 8,beam_PostCommit_Java_Examples_Dataflow_ARM
Run Java_Examples_Dataflow_ARM PostCommit 11,beam_PostCommit_Java_Examples_Dataflow_ARM
Run Java_Examples_Dataflow_ARM PostCommit 17,beam_PostCommit_Java_Examples_Dataflow_ARM
Run Python ValidatesContainer Dataflow ARM 3.8,beam_Python_ValidatesContainer_Dataflow_ARM
Run Python ValidatesContainer Dataflow ARM 3.9,beam_Python_ValidatesContainer_Dataflow_ARM
Run Python ValidatesContainer Dataflow ARM 3.10,beam_Python_ValidatesContainer_Dataflow_ARM
Run Python ValidatesContainer Dataflow ARM 3.11,beam_Python_ValidatesContainer_Dataflow_ARM
Run Python ValidatesContainer Dataflow ARM 3.12,beam_Python_ValidatesContainer_Dataflow_ARM
Run Python ValidatesContainer Dataflow ARM 3.13,beam_Python_ValidatesContainer_Dataflow_ARM
Run Python ValidatesContainer Dataflow ARM 3.14,beam_Python_ValidatesContainer_Dataflow_ARM
2 changes: 1 addition & 1 deletion release/src/main/scripts/run_rc_validation.sh
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ HUB_VERSION=2.12.0
HUB_ARTIFACTS_NAME=hub-linux-amd64-${HUB_VERSION}
BACKUP_BASHRC=.bashrc_backup_$(date +"%Y%m%d%H%M%S")
BACKUP_M2=settings_backup_$(date +"%Y%m%d%H%M%S").xml
declare -a PYTHON_VERSIONS_TO_VALIDATE=("python3.10")
declare -a PYTHON_VERSIONS_TO_VALIDATE=("python3.11")
echo ""
echo "====================Checking Environment & Variables================="
echo "PLEASE update RC_VALIDATE_CONFIGS in file script.config first."
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,29 +24,31 @@ ENV DEBIAN_FRONTEND=noninteractive \
TZ=Etc/UTC

RUN apt-get update && \
apt-get install -y --no-install-recommends software-properties-common && \
add-apt-repository -y ppa:deadsnakes/ppa && \
apt-get update && \
apt-get install -y --no-install-recommends \
curl \
tzdata \
software-properties-common \
python3.10-full \
python3.10-distutils \
python3.11-full \
python3.11-distutils \
build-essential \
python3.10-dev \
python3.11-dev \
cython3 && \
ln -fs /usr/share/zoneinfo/$TZ /etc/localtime && \
dpkg-reconfigure --frontend noninteractive tzdata && \
rm -rf /var/lib/apt/lists/*

# 2) Symlink python3 to 3.10
RUN ln -sf /usr/bin/python3.10 /usr/bin/python3 && \
ln -sf /usr/bin/python3.10 /usr/bin/python
# 2) Symlink python3 to 3.11
RUN ln -sf /usr/bin/python3.11 /usr/bin/python3 && \
ln -sf /usr/bin/python3.11 /usr/bin/python

# 3) Install pip, setuptools & wheel
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3 && \
python3 -m pip install --upgrade pip setuptools wheel

# 4) Copy the Beam SDK harness (for Dataflow workers)
COPY --from=gcr.io/apache-beam-testing/beam-sdk/beam_python3.10_sdk:latest \
COPY --from=gcr.io/apache-beam-testing/beam-sdk/beam_python3.11_sdk:latest \
/opt/apache/beam /opt/apache/beam

# 5) Make sure the harness is discovered first
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1189,7 +1189,7 @@ def test_interpreter_version_check_passes_with_experiment(self):

@mock.patch(
'apache_beam.runners.dataflow.internal.apiclient.sys.version_info',
(3, 8, 2))
(3, 10, 0))
@mock.patch(
'apache_beam.runners.dataflow.internal.apiclient.'
'beam_version.__version__',
Expand Down
28 changes: 0 additions & 28 deletions sdks/python/container/distroless/py310/build.gradle

This file was deleted.

Loading
Loading