The PostCommit Python Arm job is flaky #30760

Activity

1. github-actions

   added

   bug

   flaky_test

   P1

   workflow_id: 70955092

   on Mar 27, 2024
2. chamikaramj

   assigned

   tvalentyn

   on Apr 23, 2024
3. chamikaramj commented on Apr 23, 2024

   

   chamikaramj

   on Apr 23, 2024

   Contributor

   More actions

   @tvalentyn do we have a good owner for this ?
4. ahmedabu98 commented on Apr 24, 2024

   

   ahmedabu98

   on Apr 24, 2024

   Contributor

   More actions

   I actually can't find a single green run since this test suite was created (back in September)
5. tvalentyn commented on Apr 24, 2024

   

   tvalentyn

   on Apr 24, 2024

   Contributor

   More actions

   You may be right, thanks for correction, @ahmedabu98

   2024-04-24T12:03:53.0963029Z Please verify that you have permissions to write to the parent directory..
   2024-04-24T12:03:53.0964903Z The configuration directory may not be writable. To learn more, see https://cloud.google.com/sdk/docs/configurations#creating_a_configuration
   2024-04-24T12:03:53.0968080Z ERROR: (gcloud.auth.docker-helper) Could not create directory [/var/lib/kubelet/pods/573a1844-124b-4e12-bb0f-0325d0f3c3aa/volumes/kubernetes.io~empty-dir/gcloud]: Permission denied.
   2024-04-24T12:03:53.0969612Z 
   2024-04-24T12:03:53.0970063Z Please verify that you have permissions to write to the parent directory.
   2024-04-24T12:03:53.3953756Z #29 pushing layers 1.4s done
   2024-04-24T12:03:53.3956208Z #29 ERROR: failed to push us.gcr.io/apache-beam-testing/github-actions/beam_python3.8_sdk:2.57.0-SNAPSHOT: error getting credentials - err: exit status 1, out: ``
   2024-04-24T12:03:53.8953735Z ------
   

   cc: @damccorm - do you remember if this suite never worked or the above error is an artifact of GHA migration?

   We can reclassify this as part part of ARM backlog work.
6. tvalentyn

   added

   P2

   and removed

   P1

   on Apr 24, 2024
7. damccorm commented on Apr 24, 2024

   

   damccorm

   on Apr 24, 2024

   Contributor

   More actions

   This was working last month - https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_Arm.yml?query=is%3Asuccess+branch%3Amaster+event%3Aschedule
8. damccorm commented on Apr 24, 2024

   

   damccorm

   on Apr 24, 2024

   Contributor

   More actions

   Looks like it went flaky then permared around then
9. ahmedabu98 commented on Apr 24, 2024

   

   ahmedabu98

   on Apr 24, 2024

   Contributor

   More actions

   Ahh my apologies, I was looking at it through a is:failure filter

   👍React with 👍1Reacted by Danny McCormick
10. volatilemolotov commented on Apr 25, 2024

    

    volatilemolotov

    on Apr 25, 2024

    Contributor

    More actions

    So by removing
    https://github.com/apache/beam/blob/master/.github/workflows/beam_PostCommit_Python_Arm.yml#L113

    I get the test to move along but its still failing on my fork due to some permission with the Healthcare api.
    Oauth scope is wrong or something:
    https://github.com/volatilemolotov/beam/actions/runs/8820257015/job/24213449686#step:13:13113
11. damccorm commented on Apr 25, 2024

    

    damccorm

    on Apr 25, 2024

    Contributor

    More actions

    @volatilemolotov could you put up a PR to make that change? Definitely seems like it is getting further.

    @svetakvsundhar do you know what scope is missing? Given the normal postcommit python isn't failing, it might just be an issue with your service account specifically?
12. 120 remaining items

    Load more

13. github-actions commented on Aug 22, 2026

    

    github-actionsBot

    on Aug 22, 2026 – with GitHub Actions

    ContributorAuthor

    More actions

    Reopening since the workflow is still flaky
14. Amar3tto commented on Aug 24, 2026

    

    Amar3tto

    on Aug 24, 2026

    Collaborator

    More actions

    Stabilized
15. Amar3tto

    closed this as completedon Aug 24, 2026
16. github-actions

    modified the milestones: 2.76.0 Release, 2.77.0 Release on Aug 24, 2026
17. github-actions

    reopened this on Sep 5, 2026
18. github-actions commented on Sep 5, 2026

    

    github-actionsBot

    on Sep 5, 2026 – with GitHub Actions

    ContributorAuthor

    More actions

    Reopening since the workflow is still flaky
19. shunping

    closed this as completedin #40042on Sep 9, 2026
20. github-actions

    reopened this on Sep 23, 2026
21. github-actions commented on Sep 23, 2026

    

    github-actionsBot

    on Sep 23, 2026 – with GitHub Actions

    ContributorAuthor

    More actions

    Reopening since the workflow is still flaky
22. aIbrahiim commented on Sep 25, 2026

    

    aIbrahiim

    on Sep 25, 2026

    Contributor

    More actions

    The workflow is currently failing consistently on two integration tests because the CI service account is missing a couple of GCP IAM permissions:

    • VertexAIModelMonitoringV2IntegrationTest fails because the account does not have aiplatform.models.upload
    • RecommendationAIIT.test_predict passes, but its teardown errors because the account cannot run automlrecommendations.catalogItems.list

    Log:

    google.api_core.exceptions.PermissionDenied: 403
    Permission aiplatform.models.upload denied on resource
    //aiplatform.googleapis.com/projects/apache-beam-testing/locations/us-central1

    cc: @damccorm @Abacn @shunping
23. Abacn commented on Sep 25, 2026

    

    Abacn

    on Sep 25, 2026

    Contributor

    More actions

    ack, fixing
24. damccorm

    removed this from the 2.77.0 Release milestone on Sep 28, 2026
25. Abacn

    closed this as completedin #40559on Oct 15, 2026
26. github-actions

    added this to the 2.78.0 Release milestone on Oct 15, 2026