fix(ci): make AI failure analysis reliable

[mkoura]

The nightly AI failure-analysis step produced no analysis: the prompts instructed the agent to run compound shell snippets (pipes, command substitution) for counting test results, but the CI allowlist permits only simple command prefixes. Every attempt was denied and the agent burned all 40 turns on workarounds without writing the analysis file.

Counting and failed/broken test enumeration:

• Add standalone scripts/count_test_results.py, allowlist it in both reusable workflows and point the regression and upgrade prompts at it. It groups allure result files per test (by historyId) and takes the newest real result of each test as authoritative. The per-file grep arithmetic used before could produce nonsense (Total lower than Passed, negative Skipped): pytest evaluates skipif markers before plain skip markers, so skipif-decorated tests are registered by the --skipall pass under their own skip reason and get a second result file from the real run, and the --skipall early return in conftest skips dynamic marker additions, so MARKEXPR can select different test sets in the two passes (the conftest issues will be addressed separately). Verified on a real results dir: grep arithmetic gave Total 815/Skipped -9, the script gives the true Total 948/ Skipped 124.
• Unreadable result files produce partial counts, an explicit warning and a non-zero exit code. Missing and empty directories are reported and skipped. Tests that have only the skipall registration and no real result are called out - they indicate an interrupted testrun.
• The failed/broken enumeration is capped at 40 lines with a '+N more' tail, so a mass failure cannot flood the agent tool output (which gets truncated around 30kB).

Guarantee at least a partial analysis:

• Double max_turns to 80 and instruct the agent (CI constraints prompt) to write a first draft of failure_analysis.md as soon as counts and failure list are known, so a cut-off run still leaves output on disk.
• Add timeout-minutes to the analyze step as insurance against action hangs - a job-level timeout would cancel artifact upload and mail.

Make failed analysis debuggable, never fail the testrun job:

• Copy the Claude execution log into run_workdir (created if the testing step died early) and include it in the testrun-files artifact; warn when no execution log is available.
• Mark the analysis as possibly incomplete in the mail, step summary and log when the analyze step outcome is not success (a partial early draft would otherwise read as a complete analysis).
• Escalate the missing-analysis annotation from warning to error and include the analyze step outcome in it and in the mail sentinel. All annotations are non-failing; the analyze step keeps continue-on-error, so artifact uploads and the failure mail run regardless of how the analysis ends.

[Copilot]

Pull request overview

This PR improves the reliability of the nightly CI “AI failure analysis” by replacing fragile shell-based test counting with a standalone Python summarizer, updating the AI prompts to use it under CI tool allowlisting constraints, and enhancing CI workflows to better preserve partial/debuggable analysis output.

Changes:

• Add scripts/count_test_results.py to compute per-test status counts and a capped failed/broken list from Allure results directories.
• Update failure-analysis prompts to call the new script as a single allowlisted command and instruct early writing of failure_analysis.md.
• Update regression/upgrade reusable workflows to increase AI max turns, add step-level timeout, mark partial analyses, and attach the Claude execution log into artifacts.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
scripts/count_test_results.py	New standalone summarizer for Allure result counting and capped failure enumeration.
agent_docs/upgrade_failure_analysis_prompt.md	Switch upgrade analysis instructions to use the new summarizer command.
agent_docs/failure_analysis_prompt.md	Switch regression analysis instructions to use the new summarizer command.
agent_docs/ci_analysis_prompt.md	Instruct the agent to write an early draft of failure_analysis.md to preserve partial output.
.github/workflows/upgrade_reusable.yaml	Increase max turns, allowlist the new script, add timeout, surface partial-analysis notes, and upload execution log.
.github/workflows/regression_reusable.yaml	Same CI analysis reliability improvements as upgrade reusable workflow.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.