Add Detect-UITestCategories script and surface results in AI Summary comment by kubaflo · Pull Request #34856 · dotnet/maui

kubaflo · 2026-04-07T18:29:25Z

Note

Are you waiting for the changes in this PR to be merged?
It would be very helpful if you could test the resulting artifacts from this PR and let us know in a comment if this change resolves your issue. Thank you!

Summary

Adds a deterministic PowerShell detector that infers which UI test categories and device test categories a PR is likely to exercise, and surfaces the results inside the unified AI Summary PR comment posted by Review-PR.ps1. The output includes the exact parameter values reviewers can paste into the manual pipeline-trigger UI for maui-pr-uitests / maui-pr-devicetests, so partial test runs against a PR no longer require guessing categories by hand.

This PR contains only the detection + comment-posting tooling. The matching pipeline-side filtering changes live in a companion branch (see below).

What changed

.github/scripts/Detect-UITestCategories.ps1 (new, ~250 lines) — Core detector. Accepts -PrNumber (uses gh CLI), -DiffFile, or -BaseRef; emits text or JSON via -Json. Two detection paths:
1. Regex over added (+) diff lines for [Category(UITestCategories.X)], nameof(...), [Category("X")] (UI tests) and TestCategory.X (device tests).
2. Path-based inference rules when no test-attribute lines were touched (e.g. changes under src/Controls/.../Button/... → Button category).
Attribute hits take priority over path inference. Validated against Fix CollectionView not scrolling to top on iOS status bar tap #34687, Fix Flyout memory leak #34485, Add regression test for #34713: Binding with Converter and x:DataType is compiled #34717, [Windows] Fix RefreshView IsRefreshing property not working while binding #34845, Support xmlns prefixes and attached bindable properties in C# expressions #35007, [Windows] Fix Image layout inconsistency caused by async decode race in GetDesiredSize #34699.
.github/scripts/post-test-categories-comment.ps1 (new) — Wraps the detector in JSON mode and injects a  block into the AI Summary comment. Mirrors the established post-pr-finalize-comment.ps1 pattern: discover-or-search the comment, replace the section if present, fall back to a standalone post on permission errors, idempotent on re-run. Supports -DryRun, -PreviewFile, -ExistingCommentId.
.github/scripts/Review-PR.ps1 (modified) — New Step 3c, immediately after the existing Step 3b finalize block. Invokes the posting script with -PRNumber and the AI Summary comment ID captured in Step 3a (-ExistingCommentId) to avoid GitHub API eventual-consistency races.
.github/scripts/post-ai-summary-comment.ps1 (modified) — Adds "TEST-CATEGORIES" to the $sectionTypes preservation array (~line 755) so the section survives when the PR-REVIEW section is rebuilt on subsequent runs.

How it integrates with `Review-PR.ps1`

Review-PR.ps1 already posts a single multi-section AI Summary comment (sections delimited by  /  markers). Step 3a captures the comment ID after the summary is posted; Step 3b runs the existing pr-finalize update against that comment; the new Step 3c runs the test-categories detector and updates/inserts its section into the same comment using that captured ID. Re-running Review-PR.ps1 re-renders only the relevant sections and leaves the rest of the comment untouched.

Companion PR

The pipeline-side changes that consume the parameter values printed by this script have been split out and are not part of this PR. They live on the branch feature/pipeline-category-filtering (commit 64bf18b2bf) and will be opened as their own PR. That branch contains the edits to:

eng/pipelines/common/ci-uitests.yml
eng/pipelines/common/ci-device-tests.yml
eng/pipelines/common/ui-tests.yml
eng/pipelines/arcade/stage-device-tests.yml
eng/helix_xharness.proj

…which add categoryGroupsToTest filtering, prNumber checkout, and DeviceTestCategoryFilter plumbing — i.e. the pipeline parameters that the values printed by this script will plug into.

Verification

Detector run against real PRs: Fix CollectionView not scrolling to top on iOS status bar tap #34687, Fix Flyout memory leak #34485, Add regression test for #34713: Binding with Converter and x:DataType is compiled #34717, [Windows] Fix RefreshView IsRefreshing property not working while binding #34845, Support xmlns prefixes and attached bindable properties in C# expressions #35007, [Windows] Fix Image layout inconsistency caused by async decode race in GetDesiredSize #34699 — output matched the categories actually changed in each PR.
End-to-end posting verified on a live AI Summary comment in [Windows] Fix Image layout inconsistency caused by async decode race in GetDesiredSize #34699: [Windows] Fix Image layout inconsistency caused by async decode race in GetDesiredSize #34699 (comment)

Adds a GitHub Agentic Workflow (copilot-detect-categories) that automatically detects UI test categories from PR diffs and posts a comment listing which categories should run on CI pipelines. Triggers: - pull_request (opened/sync/reopen/ready) — no paths filter, uses gate step - issue_comment (/detect-categories) - workflow_dispatch (manual with PR number) Security: same sandbox model as copilot-evaluate-tests — safe-outputs, fork guard, Checkout-GhAwPr.ps1 for workflow_dispatch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-04-07T18:29:36Z

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.sh | bash -s -- 34856

Or

Run remotely in PowerShell:

iex "& { $(irm https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.ps1) } 34856"

Copilot

Pull request overview

Adds a new gh-aw (GitHub Agentic Workflows) workflow to detect UI test categories from PR diffs and post a PR comment describing which UI test categories were detected (and what to run).

Changes:

Added .github/workflows/copilot-detect-categories.md defining triggers, gating, diff parsing, and the agent prompt/comment format.
Added compiled workflow .github/workflows/copilot-detect-categories.lock.yml.
Updated .github/aw/actions-lock.json to include the new gh-aw setup action version/sha.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
.github/workflows/copilot-detect-categories.md	New gh-aw workflow source: gates on UI-test file changes, extracts `[Category(...)]` additions from PR diff, and instructs agent to post a summary comment.
.github/workflows/copilot-detect-categories.lock.yml	Auto-generated compiled workflow for the new gh-aw workflow.
.github/aw/actions-lock.json	Adds the pinned action entry needed by the compiled workflow.

Copilot · 2026-04-07T18:33:47Z

+    run: |
+      TEST_FILES=$(gh pr diff "$PR_NUMBER" --repo "$GITHUB_REPOSITORY" --name-only \
+        | grep -E '\.(cs|xaml)$' \
+        | grep -iE '(TestCases\.Shared\.Tests|TestCases\.HostApp|UITest|Xaml\.UnitTests)' \


The gate’s path filter misses several existing MAUI UI test project paths (e.g., src/Controls/tests/TestCases.iOS.Tests, .WinUI.Tests, .Android.Tests, .Mac.Tests). As a result, PRs that only touch those tests will be incorrectly skipped. Consider broadening this to match TestCases\. or TestCases (similar to copilot-evaluate-tests.md) or explicitly include all TestCases.*.Tests variants.

Suggested change

| grep -iE '(TestCases\.Shared\.Tests|TestCases\.HostApp|UITest|Xaml\.UnitTests)' \

| grep -iE '(TestCases(\.[^/]+\.Tests)?|TestCases\.HostApp|UITest|Xaml\.UnitTests)' \

Copilot · 2026-04-07T18:33:47Z

+      # Get list of changed test files
+      CHANGED_FILES=$(gh pr diff "$PR_NUMBER" --repo "$GITHUB_REPOSITORY" --name-only 2>/dev/null \
+        | grep -E '\.(cs|xaml)$' \
+        | grep -iE '(TestCases\.Shared\.Tests|TestCases\.HostApp|UITest|Xaml\.UnitTests)' \


Copilot · 2026-04-07T18:33:48Z

+| Category | Pipeline Filter |
+|----------|----------------|
+| {CategoryName} | `Category={CategoryName}` |
+| ... | ... |
+
+**To run only these categories on CI**, use the following filter:
+
+```
+{comma-separated category list}
+```


The suggested pipeline filter syntax here appears incorrect for MAUI UI tests. eng/pipelines/common/ui-tests-steps.yml builds the --test-filter expression using TestCategory=<name> (not Category=<name>), after splitting a comma-separated category list. The comment format should reflect TestCategory={CategoryName} and/or explain that the pipeline input expects a comma-separated list (which becomes TestCategory=A|TestCategory=B).

- eng/pipelines/ci-regression-detection.yml: New pipeline that runs targeted UI tests (by category) and optional device tests (Helix). Accepts uiTestCategories parameter (comma-separated) and runDeviceTests boolean. Manual trigger only. - Updated copilot-detect-categories workflow to: - Detect BOTH UI test categories (UITestCategories.X) and device test categories (TestCategory.X) from PR diffs - Infer categories from source file paths when no test changes exist - Broader gate: any src/ .cs/.xaml file triggers analysis (not just tests) - Post comment with pipeline trigger instructions for maui-pr-regression-detection Note: Pipeline registration in AzDO requires admin permissions. The YAML is ready — needs to be registered as 'maui-pr-regression-detection' under \dotnet\maui. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Remove ci-regression-detection.yml and instead add category filtering parameters to the existing pipelines: - ci-uitests.yml: Add 'uiTestCategories' parameter. When set (comma- separated), overrides categoryGroupsToTest to run only specified categories. Empty (default) runs all categories as before. - ci-device-tests.yml: Add 'deviceTestCategories' parameter. When set (semicolon-separated), passes DeviceTestCategoryFilter to Helix. - stage-device-tests.yml: Wire extraHelixArguments through to all HelixProjectArguments (iOS, Catalyst, Android, Android CoreCLR, Windows). Previously declared but never used. - helix_xharness.proj: Add DeviceTestCategoryFilter support. When set, disables iOS category splitting and applies the filter to all Apple work items via CustomCommands and to Android via instrumentation args. - Updated gh-aw workflow to reference existing pipelines in the comment. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

When manually triggering maui-pr-uitests or maui-pr-devicetests, set prNumber to check out a specific PR branch before building. This enables testing any PR without merging to main. The checkout step fetches refs/pull/<N>/head and switches to it before the build starts. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

The previous gh-aw workflow used an LLM to detect [Category(...)] attributes in PR diffs and infer categories from changed file paths. Both tasks are deterministic, so an LLM is unnecessary overhead. Detect-UITestCategories.ps1 does the same work as a plain script: - Parses [Category(UITestCategories.X)], nameof(...), quoted, and TestCategory.X forms from added diff lines - Falls back to path-based inference when no Category attribute is detected on changed test files - Supports three input modes: -PrNumber (via gh CLI), -DiffFile, or -BaseRef (defaults to origin/main) - Outputs human-readable text or JSON (-Json), with optional file write Removes: - .github/workflows/copilot-detect-categories.md (gh-aw source) - .github/workflows/copilot-detect-categories.lock.yml (compiled) - .github/aw/actions-lock.json entry for github/gh-aw-actions/setup@v0.62.5 Pipeline changes (uiTestCategories / deviceTestCategories / prNumber parameters in ci-uitests.yml, ci-device-tests.yml, common/ui-tests.yml, arcade/stage-device-tests.yml, helix_xharness.proj) are kept unchanged. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Wires Detect-UITestCategories.ps1 into the existing PR review flow: - New post-test-categories-comment.ps1 runs the detector in JSON mode and injects a  block into the unified AI Summary comment, mirroring post-pr-finalize-comment.ps1 (replace if section present, idempotent on re-run, fallback standalone post). - Review-PR.ps1 invokes the new posting script as Step 3c, passing the AI Summary comment ID from Step 3a to avoid eventual-consistency races. - post-ai-summary-comment.ps1 preserves TEST-CATEGORIES in sectionTypes so the section survives subsequent rebuilds. The section lists detected UI/device categories with detection source (attribute hit vs path inference) and the exact pipeline parameter values for triggering targeted maui-pr-uitests / maui-pr-devicetests runs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

This PR is now scoped to ONLY the detection script and its integration into Review-PR.ps1. The pipeline changes that enable category filtering (uiTestCategories / deviceTestCategories parameters, helix_xharness.proj DeviceTestCategoryFilter, extraHelixArguments wiring, prNumber checkout) will be submitted as a separate PR so each can be reviewed and merged independently. Reverted files (back to merge-base): - eng/helix_xharness.proj - eng/pipelines/arcade/stage-device-tests.yml - eng/pipelines/ci-device-tests.yml - eng/pipelines/ci-uitests.yml - eng/pipelines/common/ui-tests.yml Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Allows manual triggering of maui-pr-uitests and maui-pr-devicetests to run only a targeted subset of test categories — useful for fast regression checks when a PR is known to affect specific controls. Companion to #34856 (Detect-UITestCategories.ps1), which detects the relevant categories for a PR and surfaces them in the AI Summary comment. Reviewers can copy the suggested values directly into the manual pipeline-trigger UI. ci-uitests.yml - New `uiTestCategories` parameter (comma-separated, e.g. "Button,Label,Shell"). When set, overrides categoryGroupsToTest with a single matrix entry. When empty (default), the full matrix runs as before. - New `prNumber` parameter — when set, the build checks out refs/pull/<N>/head before building, so any PR can be tested without merging. ci-device-tests.yml - New `deviceTestCategories` parameter (semicolon-separated, e.g. "Button;Label;Shell"). Forwarded to helix_xharness.proj as `/p:DeviceTestCategoryFilter=...` via extraHelixArguments. - New `prNumber` parameter (same behavior as ci-uitests.yml). stage-device-tests.yml - Wires the existing `extraHelixArguments` parameter through to all HelixProjectArguments invocations (iOS, MacCatalyst, Android, Android CoreCLR, Windows). Previously declared but never plumbed. helix_xharness.proj - New `DeviceTestCategoryFilter` property. When set, disables iOS category splitting (so all categories run in a single filtered work item) and applies the filter via `--set-env=TestFilter=Category=...` on Apple work items and `--arg TestFilter=...` on Android. common/ui-tests.yml - New `prNumber` parameter that injects a PR-checkout step before the Mono and CoreCLR build legs. Inspired by #33176, which explored deeper provisioning optimizations (skip provisioning when no matching categories). This PR keeps the scope smaller — just expose the parameters; the optimization of skipping provisioning entirely can land in a follow-up. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Conflicts resolved: - .github/scripts/Review-PR.ps1 - Doc-comment header: adopted main's restructured Step 1 (Gate) / Step 2 (3-phase pr-review) wording, extended Step 3 description to mention "review + test categories". - Step 3b/3c block: main intentionally removed the post-pr-finalize invocation, so dropped my 3b. Kept only 3c (post-test-categories) renumbered to 3b. - .github/scripts/post-ai-summary-comment.ps1 - Adopted main's session-based Merge-Sessions structure (replaces the older "preserve other sections" loop). - Extended the standalone-section preservation from PR-FINALIZE only to a list-driven loop that also preserves TEST-CATEGORIES so it survives subsequent rebuilds. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

kubaflo · 2026-04-19T13:52:33Z

Closing in favor of #33176, which implements the same UI test category detection (and goes further by also gating provisioning per category). Conflicts on #33176 have been resolved and pushed — driving that one to merge instead.

Copilot AI review requested due to automatic review settings April 7, 2026 18:29

Copilot started reviewing on behalf of kubaflo April 7, 2026 18:30 View session

github-actions Bot mentioned this pull request Apr 7, 2026

[aw] Detect UI Test Categories failed #34857

Closed

Copilot AI reviewed Apr 7, 2026

View reviewed changes

github-actions Bot mentioned this pull request Apr 7, 2026

[aw] Detect Test Categories for Regression Detection failed #34858

Closed

kubaflo and others added 5 commits April 7, 2026 22:17

kubaflo mentioned this pull request Apr 19, 2026

Add category-filter and prNumber parameters to UI/device test pipelines #35033

Closed

kubaflo changed the title ~~Add gh-aw workflow for UI test category detection~~ Add Detect-UITestCategories script and surface results in AI Summary comment Apr 19, 2026

kubaflo and others added 2 commits April 19, 2026 14:16

Remove accidentally committed .copilot-tmp/ scratch file

2d8ff8a

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

kubaflo closed this Apr 19, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Detect-UITestCategories script and surface results in AI Summary comment#34856

Add Detect-UITestCategories script and surface results in AI Summary comment#34856
kubaflo wants to merge 9 commits intomainfrom
feature/copilot-detect-categories

kubaflo commented Apr 7, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Apr 7, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Apr 7, 2026

Uh oh!

Copilot AI Apr 7, 2026

Uh oh!

Copilot AI Apr 7, 2026

Uh oh!

kubaflo commented Apr 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	\| grep -iE '(TestCases\.Shared\.Tests\|TestCases\.HostApp\|UITest\|Xaml\.UnitTests)' \
	\| grep -iE '(TestCases(\.[^/]+\.Tests)?\|TestCases\.HostApp\|UITest\|Xaml\.UnitTests)' \

	\| grep -iE '(TestCases\.Shared\.Tests\|TestCases\.HostApp\|UITest\|Xaml\.UnitTests)' \
	\| grep -iE '(TestCases\.(Android\|iOS\|Mac\|WinUI\|Shared)\.Tests\|TestCases\.HostApp\|UITest\|Xaml\.UnitTests)' \

Conversation

kubaflo commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What changed

How it integrates with Review-PR.ps1

Companion PR

Verification

Uh oh!

github-actions Bot commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Apr 7, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 7, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 7, 2026

Choose a reason for hiding this comment

Uh oh!

kubaflo commented Apr 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kubaflo commented Apr 7, 2026 •

edited

Loading

How it integrates with `Review-PR.ps1`

github-actions Bot commented Apr 7, 2026 •

edited

Loading