Skip to content

ci: reduce iOS functional test flakiness - #1373

Open
Dor-bl wants to merge 1 commit into
appium:masterfrom
Dor-bl:ccr-08794045-91mrk3-ios-upstream
Open

Dor-bl wants to merge 1 commit into
appium:masterfrom
Dor-bl:ccr-08794045-91mrk3-ios-upstream

Conversation

@Dor-bl

@Dor-bl Dor-bl commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Makes the iOS functional test jobs (ios_test, flutter_e2e_test / flutter-ios) less flaky.

What was failing

In 13 failed runs of this workflow (on a fork running the same workflow file), the Android jobs passed every time, and at least one iOS job failed in every run:

  • First session hangs (12 failures): the first webdriver.Remote on a freshly booted simulator either hit the client's 600 s read timeout (urllib3.exceptions.ReadTimeoutError on POST /session), or WDA failed with XCTDaemonErrorDomain Code=5 "Timed out attempting to launch app". Only the first session was affected; the next test's session usually started fine.
  • Simulator boot never finished (3 failures): xcrun simctl bootstatus failed 4 of 4 attempts after about 30 min.
  • setup-uv timed out (1 failure) fetching the "latest" version manifest.

Changes

  • Set up Python, uv and uv sync before booting the simulator, and pin the uv version (UV_VERSION) in all jobs.
  • Open Simulator.app after the boot instead of before it.
  • Cap the simulator boot step at 20 minutes, so a stuck boot fails fast.
  • Warm-up steps before the tests: install the prebuilt WDA and launch Safari once, then start and quit one throwaway Appium session (test/functional/ios/helper/warm_up.py), so WDA's slow first launch happens there rather than in a test. The warm-up makes a single attempt with the tests' 600 s timeout and gives quit() a 15 s budget, so it normally finishes within its 12-minute step; the step is continue-on-error.
  • Retry once on WDA startup errors in the iOS and Flutter fixtures (create_session_with_wda_retry), for "Unable to launch WebDriverAgent" / "Unable to start WebDriverAgent session". The XCUITest driver ignores wdaStartupRetries with usePreinstalledWDA, and deletes the failed session before raising, so a client-side retry is safe. A client-side ReadTimeoutError is deliberately not retried: the server may still be creating that session.
  • Raise wdaLaunchTimeout to 240 s, to give WDA more time to launch on a slow simulator.
  • Keep one Safari session per test, so a session that fails to start only costs that one test.
  • Add a read-only permissions: contents: read block and quote the python-version values.

Testing

  • Unit tests: 268 passed. ruff check and ruff format --check are clean, the workflow YAML parses, and pytest collects the 14 iOS and Flutter functional tests.
  • The functional jobs themselves haven't run with these changes before this PR; this PR's CI run is the first real check.

iOS functional test jobs fail often on hosted macOS runners: the first
Appium session on a freshly booted simulator times out, the simulator
boot sometimes never finishes, and setup-uv can time out fetching its
version manifest.

- Set up Python, uv and dependencies before booting the simulator, and
  pin the uv version in every job.
- Open Simulator.app after the boot, and cap the boot step at 20 min.
- Warm up the simulator (install WDA, launch Safari once) and WDA (one
  throwaway Appium session) before the tests; the WDA warm-up never
  fails the job.
- Retry session creation once in the iOS and Flutter fixtures when WDA
  fails to start. The XCUITest driver ignores wdaStartupRetries with
  usePreinstalledWDA, and deletes the failed session before raising.
- Raise wdaLaunchTimeout to 240 s for the Safari tests.
- Use read-only workflow token permissions; quote python-version values.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant