From 91792909e20739fb012116792b11d0489b1aa0e4 Mon Sep 17 00:00:00 2001 From: Bernard Ladenthin Date: Sun, 11 Oct 2026 17:31:33 +0200 Subject: [PATCH] CI: resolve sdkmanager properly -- the emulator pre-install never worked The step that pre-installs the Android emulator package calls sdkmanager bare, and it is not on PATH on the hosted runner: the SDK sits at ANDROID_HOME with the command under cmdline-tools//bin. So it failed four times with sdkmanager: command not found, slept 20 plus 40 plus 60 seconds in between, and then printed that the package could not be installed because the dl.google.com download or archive was broken -- a cause with nothing to do with it. Two consequences worth naming. The step was added to make a corrupt-archive failure diagnosable, and instead it converted a job that was only occasionally flaky into one that is always red: the Android emulator jobs have failed in every run since, 38146809329, 38148401395 and 38149667195. And the misleading message is what kept that invisible -- a reader sees a download problem at Google and waits for it to pass rather than looking at the step. Fixed by resolving the command (PATH first, then ANDROID_HOME, ANDROID_SDK_ROOT and the image's /usr/local/lib/android/sdk, taking the newest cmdline-tools version) and by keeping the two failure modes apart: not-found now says it is a runner-image change and not a download problem, and the download failure names the binary it used. Both copies of the step are fixed identically. What this does not prove: the emulator jobs have not been green since the step was added, so the corrupt archive it was built to survive is still unverified. Recorded in TODO.md rather than claimed. check-run-scripts.py 160 scripts clean, check-release-gate.py 0 violations, check-natives.py 0 disagreements. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01E2h8gXyyE5UeimkL9vQv9G --- .github/workflows/publish.yml | 50 ++++++++++++++++++++++++++++++----- TODO.md | 23 +++++++++++----- 2 files changed, 60 insertions(+), 13 deletions(-) diff --git a/.github/workflows/publish.yml b/.github/workflows/publish.yml index d7c4e806a..047948963 100644 --- a/.github/workflows/publish.yml +++ b/.github/workflows/publish.yml @@ -1178,17 +1178,36 @@ jobs: # action's "adb emu kill" exit code 1. shell: bash run: | + # sdkmanager is NOT on PATH on the hosted runner: the SDK lives at $ANDROID_HOME and the + # command sits in cmdline-tools//bin. Calling it bare made this step fail four + # times with "sdkmanager: command not found" and then report "dl.google.com download or + # archive broken" -- the wrong cause, which is worse than no message, and it turned the + # emulator jobs from occasionally flaky into always red (runs 38146809329, 38148401395, + # 38149667195). Resolve it, and keep the two failure modes apart. + SDKMANAGER="$(command -v sdkmanager || true)" + if [ -z "$SDKMANAGER" ]; then + for root in "${ANDROID_HOME:-}" "${ANDROID_SDK_ROOT:-}" /usr/local/lib/android/sdk; do + [ -n "$root" ] || continue + candidate="$(find "$root/cmdline-tools" -maxdepth 2 -name sdkmanager -type f 2>/dev/null | sort | tail -1)" + if [ -n "$candidate" ]; then SDKMANAGER="$candidate"; break; fi + done + fi + if [ -z "$SDKMANAGER" ]; then + echo "::error::sdkmanager not found -- looked on PATH and under ANDROID_HOME='${ANDROID_HOME:-}', ANDROID_SDK_ROOT='${ANDROID_SDK_ROOT:-}' and /usr/local/lib/android/sdk. This is a runner-image change, not a download problem." + exit 1 + fi + echo "using sdkmanager: $SDKMANAGER" set +o pipefail # `yes` dies of SIGPIPE once sdkmanager stops reading; that is not a failure - yes | sdkmanager --licenses >/dev/null 2>&1 || true + yes | "$SDKMANAGER" --licenses >/dev/null 2>&1 || true for attempt in 1 2 3 4; do - if sdkmanager --install emulator --channel=0 >/dev/null; then + if "$SDKMANAGER" --install emulator --channel=0 >/dev/null; then echo "emulator package installed (attempt ${attempt})" exit 0 fi echo "sdkmanager --install emulator failed (attempt ${attempt} of 4)" sleep $((attempt * 20)) done - echo "::error::the Android emulator package could not be installed in 4 attempts (dl.google.com download or archive broken)" + echo "::error::the Android emulator package could not be installed in 4 attempts with '$SDKMANAGER' (dl.google.com download or archive broken)" exit 1 - name: Run on-emulator instrumentation (connectedDebugAndroidTest) uses: reactivecircus/android-emulator-runner@v2 @@ -1349,17 +1368,36 @@ jobs: # action's "adb emu kill" exit code 1. shell: bash run: | + # sdkmanager is NOT on PATH on the hosted runner: the SDK lives at $ANDROID_HOME and the + # command sits in cmdline-tools//bin. Calling it bare made this step fail four + # times with "sdkmanager: command not found" and then report "dl.google.com download or + # archive broken" -- the wrong cause, which is worse than no message, and it turned the + # emulator jobs from occasionally flaky into always red (runs 38146809329, 38148401395, + # 38149667195). Resolve it, and keep the two failure modes apart. + SDKMANAGER="$(command -v sdkmanager || true)" + if [ -z "$SDKMANAGER" ]; then + for root in "${ANDROID_HOME:-}" "${ANDROID_SDK_ROOT:-}" /usr/local/lib/android/sdk; do + [ -n "$root" ] || continue + candidate="$(find "$root/cmdline-tools" -maxdepth 2 -name sdkmanager -type f 2>/dev/null | sort | tail -1)" + if [ -n "$candidate" ]; then SDKMANAGER="$candidate"; break; fi + done + fi + if [ -z "$SDKMANAGER" ]; then + echo "::error::sdkmanager not found -- looked on PATH and under ANDROID_HOME='${ANDROID_HOME:-}', ANDROID_SDK_ROOT='${ANDROID_SDK_ROOT:-}' and /usr/local/lib/android/sdk. This is a runner-image change, not a download problem." + exit 1 + fi + echo "using sdkmanager: $SDKMANAGER" set +o pipefail # `yes` dies of SIGPIPE once sdkmanager stops reading; that is not a failure - yes | sdkmanager --licenses >/dev/null 2>&1 || true + yes | "$SDKMANAGER" --licenses >/dev/null 2>&1 || true for attempt in 1 2 3 4; do - if sdkmanager --install emulator --channel=0 >/dev/null; then + if "$SDKMANAGER" --install emulator --channel=0 >/dev/null; then echo "emulator package installed (attempt ${attempt})" exit 0 fi echo "sdkmanager --install emulator failed (attempt ${attempt} of 4)" sleep $((attempt * 20)) done - echo "::error::the Android emulator package could not be installed in 4 attempts (dl.google.com download or archive broken)" + echo "::error::the Android emulator package could not be installed in 4 attempts with '$SDKMANAGER' (dl.google.com download or archive broken)" exit 1 - name: Run LLM Service UI test on emulator (connectedDebugAndroidTest) uses: reactivecircus/android-emulator-runner@v2 diff --git a/TODO.md b/TODO.md index 072aa3c6a..5b6238f03 100644 --- a/TODO.md +++ b/TODO.md @@ -949,13 +949,22 @@ for everything else. `sun.misc.Unsafe`, or `jshell` writing to address 0) on a branch run, then look for the dump in the artifact -- and if none appears, replace the WER route with `procdump -ma -e` around the surefire run. -- **The emulator package pre-install is unproven.** Both emulator jobs died in one minute inside the - action's own `sdkmanager --install emulator --channel=0` ("Error on ZipFile unknown archive", run - 38069595101) an hour after being green; #506 installs the package in a step before the action, - four attempts, so the action's install is a no-op. Neither job has run since. If the download - keeps flaking, the sturdier form is `actions/cache` over `$ANDROID_HOME/emulator` keyed on the - emulator package version `sdkmanager --list` reports, so dl.google.com is reached once per - version instead of once per job. +- ~~**The emulator package pre-install is unproven.**~~ **Diagnosed and fixed 2026-10-11: it never + worked, and it reported the wrong cause.** The step calls `sdkmanager` bare, but it is **not on + `PATH`** on the hosted runner -- the SDK sits at `$ANDROID_HOME` with the command under + `cmdline-tools//bin`. So every run failed four times with `sdkmanager: command not found`, + slept 20+40+60 seconds between attempts, and then printed + `the Android emulator package could not be installed in 4 attempts (dl.google.com download or + archive broken)` -- a cause that had nothing to do with it. **The step therefore converted a job + that was only occasionally flaky into one that is always red** (runs 38146809329, 38148401395, + 38149667195), and the misleading message is what kept that invisible: a reader sees a download + problem at Google and waits for it to pass. + Fixed by resolving the command (PATH, then `$ANDROID_HOME`, `$ANDROID_SDK_ROOT` and the image's + `/usr/local/lib/android/sdk`, newest `cmdline-tools` version last) and by **keeping the two failure + modes apart**: "not found" now says it is a runner-image change and not a download problem. Both + copies of the step are fixed identically. **Still to verify in CI** -- the emulator jobs have not + been green since the step was added, so the thing the step was built to guard against (the corrupt + archive of run 38069595101) also remains unproven. - **Everything behind `package` has never run on the modular natives.** Run 38069595101 skipped `package` (it `needs` the Windows clang test job, red with the AMX crash); this branch's run 38112603436 reaches it and fails on the Windows OpenVINO module jar (the blocker at the top of