Skip to content

fix(state): retry timed-out state reads once and say when we are waiting - #507

Merged
Timo972 merged 1 commit into
developfrom
timo/droid-305-state-timeout-retries
Oct 23, 2026
Merged

Timo972 merged 1 commit into
developfrom
timo/droid-305-state-timeout-retries

Conversation

@Timo972

@Timo972 Timo972 commented Oct 23, 2026

Copy link
Copy Markdown
Collaborator

Why

A user reported that mobilerun device ui --ios with TikTok open on an iPhone X running iOS 16 "does nothing and just hangs, no log lines". Their output eventually showed:

get_state attempt 1 failed: ReadTimeout (retrying in 1s)
get_state attempt 2 failed: ReadTimeout (retrying in 2s)
get_state attempt 3 failed: ReadTimeout (retrying in 3s)

fetch_state_with_retry made 7 attempts with 1/2/3/5/8/10s backoff. Each attempt was a GET /state_full with a 30s httpx timeout. Worst case was about 4 minutes, with nothing printed for the first 30s. A timeout means the device side is still working on the request we abandoned. The portal (backed by WDA) and Android Portal both serve state one request at a time. So each retry queued another snapshot behind the app that was already stuck.

The root cause is fixed on the device side in droidrun/WebDriverAgent#43, released in v1.3.5 and v1.3.6. When an app stops answering accessibility, WDA now responds within about 15s with the foreground package and a partial tree marked truncated. Through the portal that takes 15–19s, 23s at worst, which fits inside the 30s driver timeout. Older WDA builds can still go past 30s. This PR is the client-side follow-up. A companion mobilerun-ios change lowers the portal's WDA state deadline below 30s.

Refs DROID-305, #484 (both already closed).

Changes

mobilerun/tools/ui/provider.py: retry policy

  • A timed-out read (TimeoutError or httpx.ReadTimeout) is retried once. The recovery callback runs before that retry. All other errors keep the existing 7-attempt backoff, the late recovery after 5 failures, and the immediate recovery when the accessibility service is gone. Timeouts are counted across the whole call, so a timeout, then another error, then a timeout gives up after 3 attempts.
  • httpx.ConnectTimeout and PoolTimeout are not treated as a busy device. Those requests never reached it, so retrying them doesn't pile up work, and they keep the normal schedule.
  • The final error now reports the actual number of attempts ("after 2 attempts") instead of max_retries.

Why the timeout rule is right for every driver. The loop is shared by every driver that uses AndroidStateProvider:

  • iOS portal (IOSPortalHttpDriver) and Android Portal HTTP: a ReadTimeout means the single-threaded server is still busy. Each extra retry just adds to the queue.
  • Android over ADB: the TCP path already falls back to the content provider internally. A timeout reaches this loop only as TimeoutError from a wedged on-device serializer, after 15s. Before this PR, recovery (restarting the accessibility service) ran only after 5 such timeouts, roughly 75s or more of waiting. Now it runs right after the first timeout, so Android recovers sooner.
  • iOS: _recover_portal does nothing for non-ADB drivers, so the single retry is a plain second read.

Worst case with timeouts: about 30s + 1s + 30s ≈ 61s through the portal, down from about 4 minutes. On Android ADB it is about 15s + recovery + 15s. With WDA ≥ v1.3.5 the first read normally succeeds in 15–23s.

fetch_state_with_retry: visible wait. When a fetch is still running after 3s, it logs Waiting for the UI state from the device... at INFO, once per call. It doesn't repeat on the retry. If the caller is cancelled, the fetch is cancelled too.

mobilerun/cli/device_commands.py: making that INFO line visible. device ui never set up logging, so Python's last-resort handler dropped everything below WARNING. That's why the user saw nothing for 30s. device ui now attaches a plain stderr handler at INFO when the mobilerun logger has none. stdout still carries only the tree, so scripts that parse it are unaffected. A logger that is already set up (by the SDK or the agent CLI) is left alone.

Left out: The 30s per-request driver timeout (in mobilerun-core-local) is unchanged. The new WDA-plus-portal answer fits inside it. The cloud driver's APITimeoutError is not classified as a timeout here: that driver already retries its own reads.

Tests

  • New tests/test_state_fetch_retry_policy.py (12 tests):
    • one retry with recovery for ReadTimeout and TimeoutError
    • timeout followed by success
    • timeouts counted across mixed errors
    • other errors keep the full schedule and late recovery
    • ConnectTimeout keeps the full schedule
    • worst-case backoff with timeouts is a single 1s sleep
    • waiting notice: logged once, not on fast fetches, not repeated on retry
    • cancelling the caller cancels the fetch
    • the device ui stderr handler, including leaving an existing one alone
  • pytest tests: 1342 passed, 41 subtests passed
  • black==26.10.1 --check . (CI pin): 203 files unchanged
  • ruff check on the changed files: passed

No device testing.

🤖 Generated with Claude Code

A timed-out get_ui_tree means the device side is still busy with the
request we abandoned. Portal and the mobilerun-ios portal (WDA) serve
state one request at a time, so the old 7-attempt schedule queued six
more reads behind a stuck app: up to ~4 minutes of `device ui --ios`
with nothing printed for the first 30s (DROID-305, mobilerun#484).

- Timeouts (TimeoutError, httpx.ReadTimeout) now get one retry, preceded
  by the recovery callback; other errors keep the full backoff schedule
  and late recovery. Connect timeouts are not treated as a busy device.
- Log "Waiting for the UI state from the device..." at INFO once when a
  fetch runs past 3s.
- `mobilerun device ui` sends INFO progress to stderr when no handler is
  configured, so the notice is visible and stdout stays the tree.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Timo972
Timo972 requested a review from RasulOs October 23, 2026 15:01
@Timo972
Timo972 merged commit 023be62 into develop Oct 23, 2026
7 checks passed
@Timo972
Timo972 deleted the timo/droid-305-state-timeout-retries branch October 23, 2026 22:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants