|
| 1 | +# UI Tests CI Hang Plan |
| 2 | + |
| 3 | +## Goal |
| 4 | + |
| 5 | +Fix the `DotPilot.UITests` GitHub Actions execution path so the required UI test job returns a real completed result instead of hanging or being canceled while `Run UI Tests` is still active. |
| 6 | + |
| 7 | +## Scope |
| 8 | + |
| 9 | +### In Scope |
| 10 | + |
| 11 | +- `DotPilot.UITests` harness code and supporting configuration |
| 12 | +- GitHub Actions validation behavior needed to expose or validate the harness fix |
| 13 | +- Durable notes for any repo-level CI implication discovered during the fix |
| 14 | + |
| 15 | +### Out Of Scope |
| 16 | + |
| 17 | +- New product functionality unrelated to UI test stability |
| 18 | +- Weakening, skipping, or conditionally bypassing the UI suite |
| 19 | +- Release workflow changes unrelated to the UI test blocker |
| 20 | + |
| 21 | +## Constraints And Risks |
| 22 | + |
| 23 | +- The UI test suite is mandatory and must stay in the normal validation workflow. |
| 24 | +- A hang, timeout, or canceled run counts as a failing harness outcome. |
| 25 | +- Prefer a deterministic harness fix over workflow-level retries or skip logic. |
| 26 | +- Keep the fix aligned with the current browserwasm test-launch path. |
| 27 | + |
| 28 | +## Testing Methodology |
| 29 | + |
| 30 | +- GitHub baseline: |
| 31 | + - inspect the active PR run and UI Tests job logs for the actual failure point |
| 32 | +- Local validation: |
| 33 | + - `dotnet test DotPilot.UITests/DotPilot.UITests.csproj` |
| 34 | + - any focused repro command needed to exercise the harness shutdown path |
| 35 | +- Broader validation: |
| 36 | + - `dotnet test DotPilot.slnx` |
| 37 | + - `dotnet format DotPilot.slnx --verify-no-changes` |
| 38 | +- Quality bar: |
| 39 | + - UI tests must complete with a terminal pass/fail result locally |
| 40 | + - the fix must directly address the hang/cancel path instead of hiding it |
| 41 | + |
| 42 | +## Ordered Plan |
| 43 | + |
| 44 | +- [x] Step 1: Capture the current failing GitHub UI test job details and relevant local baseline. |
| 45 | + Verification: |
| 46 | + - inspect the active PR workflow run and UI Tests job |
| 47 | + - run the relevant local UI test command |
| 48 | + Done when: the concrete hang symptom and likely failing code path are documented below. |
| 49 | + |
| 50 | +- [x] Step 2: Trace the harness launch and shutdown flow in `DotPilot.UITests` to isolate the blocking operation. |
| 51 | + Verification: |
| 52 | + - identify the exact code path used during CI teardown or cancellation |
| 53 | + - document the likely root cause before editing |
| 54 | + Done when: the intended fix path is clear. |
| 55 | + |
| 56 | +- [x] Step 3: Implement the harness fix and any needed regression coverage. |
| 57 | + Verification: |
| 58 | + - changed code remains within repo maintainability limits |
| 59 | + - local UI test execution still returns a real result |
| 60 | + Done when: the blocking behavior is removed from the identified path. |
| 61 | + |
| 62 | +- [ ] Step 4: Run final validation and record outcomes. |
| 63 | + Verification: |
| 64 | + - `dotnet test DotPilot.UITests/DotPilot.UITests.csproj` |
| 65 | + - `dotnet test DotPilot.slnx` |
| 66 | + - `dotnet format DotPilot.slnx --verify-no-changes` |
| 67 | + - GitHub Actions `UI Test Suite` returns a real completed result with the instrumented harness logs available |
| 68 | + Done when: required checks are green locally and the GitHub UI test job returns a real completed result or a concrete failing signal. |
| 69 | + |
| 70 | +## Full-Test Baseline Step |
| 71 | + |
| 72 | +- [x] Run the relevant baseline commands after the plan is prepared: |
| 73 | + - inspect the active GitHub UI Tests job |
| 74 | + - `dotnet test DotPilot.UITests/DotPilot.UITests.csproj` |
| 75 | + Done when: the failing symptom is recorded below. |
| 76 | + |
| 77 | +## Failing Tests And Checks Tracker |
| 78 | + |
| 79 | +- [x] `GitHub Actions job: UI Tests` |
| 80 | + Failure symptom: the PR validation run enters `Run UI Tests`, then fails to produce a terminal result and may later surface `Attempting to cancel the build... Error: The operation was canceled.` |
| 81 | + Suspected cause: the UI test harness had unbounded teardown calls for `_browserApp.Dispose()` and `BrowserTestHost.Stop()`, so a Windows-specific cleanup hang could leave the job running until GitHub eventually canceled it. |
| 82 | + Intended fix path: bound teardown cleanup with explicit timeouts and convert cleanup hangs into deterministic test failures instead of infinite jobs. |
| 83 | + Status: bounded cleanup is fixed locally; CI still needs instrumented logs to confirm whether any remaining hang is in driver/bootstrap, browser startup, host readiness, or teardown. |
| 84 | + |
| 85 | +## Baseline Notes |
| 86 | + |
| 87 | +- PR `#10` currently points at GitHub Actions run `23041255695`, where `Build`, `Unit Tests`, and `Coverage` complete successfully but `UI Tests` stays in progress inside `Run UI Tests`. |
| 88 | +- The affected job is `66920240879`, which started `Run UI Tests` at `2026-03-13T07:50:38Z` and did not produce a terminal result while the other validation jobs finished. |
| 89 | +- A later PR validation run, `23043020124`, shows the same shape so far: `Quality Gate`, `Unit Test Suite`, and `Coverage Suite` completed while `UI Test Suite` job `66925873162` remained `in_progress` inside `Run UI Tests` after starting at `2026-03-13T08:47:49Z`. |
| 90 | +- Local `dotnet test DotPilot.UITests/DotPilot.UITests.csproj` passed before the fix, which indicates the main failure mode is CI-specific hang behavior rather than a consistently failing test case. |
| 91 | +- Local `dotnet test ./DotPilot.UITests/DotPilot.UITests.csproj --logger GitHubActions --blame-crash` also passed before the fix on macOS, which points further toward a Windows-specific teardown or process-cleanup issue. |
| 92 | + |
| 93 | +## Validation Notes |
| 94 | + |
| 95 | +- Added bounded cleanup execution in `DotPilot.UITests` so teardown now fails fast if app disposal or browser-host shutdown hangs. |
| 96 | +- Added focused regression tests for the bounded-cleanup helper to prove success, exception, and timeout behavior. |
| 97 | +- Added harness logging around browser binary resolution, ChromeDriver resolution, host startup, setup, and cleanup so the next GitHub run exposes the exact blocking stage instead of sitting silent inside `Run UI Tests`. |
| 98 | +- `dotnet test DotPilot.UITests/DotPilot.UITests.csproj --filter BoundedCleanupTests` passed. |
| 99 | +- `dotnet test DotPilot.UITests/DotPilot.UITests.csproj` passed with `9` tests green. |
| 100 | +- `dotnet test ./DotPilot.UITests/DotPilot.UITests.csproj --logger GitHubActions --blame-crash` passed with `9` tests green. |
| 101 | +- `dotnet test DotPilot.slnx` passed. |
| 102 | +- `dotnet format DotPilot.slnx --verify-no-changes` passed. |
| 103 | + |
| 104 | +## Final Validation Skills |
| 105 | + |
| 106 | +1. `gh-fix-ci` |
| 107 | +Reason: inspect the failing GitHub Actions run and extract the real failure signal. |
| 108 | + |
| 109 | +2. `mcaf-testing` |
| 110 | +Reason: keep the UI suite mandatory and verified through real execution. |
| 111 | + |
| 112 | +3. `mcaf-dotnet` |
| 113 | +Reason: ensure the harness fix stays aligned with the repo’s .NET toolchain and quality path. |
0 commit comments