Skip to content

[verifier] improve breakage visibility in the CI - #9130

Open
pq wants to merge 4 commits into
flutter:mainfrom
pq:verifier_CI_visibility
Open

pq wants to merge 4 commits into
flutter:mainfrom
pq:verifier_CI_visibility

Conversation

@pq

@pq pq commented Sep 18, 2026 •

Copy link
Copy Markdown
Collaborator

Ports flutter/dart-intellij-third-party#670. Improves visibility of new failures and ensures all verifications run (even after a failure).


LLM summary: VERIFY_BOT runs four verifications and then diffs the reports against committed baselines. Two things went wrong whenever something actually broke:

  • The bot runs under set -e, so the first failing verification aborted the script. The baseline diff — the step that explains which issues are new — was skipped precisely when it was needed.
  • Everything was log-only. Finding out what broke meant scrolling the job log.

Changes

  • Add tool/check_verifier_baselines.sh: one script shared by CI (check) and local re-baselining (update), so the two cannot drift.
  • VERIFY_BOT records each verification's status instead of aborting, and always runs the baseline diff.
  • New issues are surfaced as job annotations and a job summary table.
  • Add guards for cases that previously passed silently: an unparseable report, a verified IDE branch with no committed baseline, and a note when the baseline was recorded against a different IDE build.
  • tool/update_baselines.sh delegates to the same script.
  • Document how verification is gated in CONTRIBUTING.md.

Baseline format

Baselines gain a provenance header and a section tag:

# Plugin verifier baseline for IDE branch 253.
# Generated from AI-253.32098.37.2534.15232325 by tool/update_baselines.sh -- do not edit by hand.
Deprecated API usages<TAB>Deprecated class com.intellij.ui.ComboboxWithBrowseButton is referenced in ...

The section keeps the kind of finding visible in the diff. The issue text itself is unchanged: regenerating produces exactly the same set of issues as the committed baselines, only reformatted.

Notes

  • verifyPlugin's exit status stays ignored (|| true). failureLevel judges severity with no notion of what we have already accepted, so a baselined issue in an enabled category would fail every run with no way to suppress it. Novelty remains the baseline check's job, and a problem serious enough to matter still lands in the report, so it is still caught if it is new.
  • The parameter-name normalization for DartTemplateContextType.<init> (local builds emit names, CI does not) now applies only when comparing, never when writing, so the committed file stays a faithful record of the report.

Review the contribution guidelines below:

  • I’ve reviewed the contributor guide and applied the relevant portions to this PR.
  • I've included the required information in the description above.
  • My up-to-date information is in the AUTHORS file.
  • I've updated CHANGELOG.md if appropriate.
Contribution guidelines:
  • See
    our contributor guide and
    the Flutter organization contributor guide
    for general expectations for PRs.
  • Larger or significant changes should be discussed in an issue before creating a PR.
  • Dart contributions to our repos should follow the Dart style guide and use
    dart format.
  • Java and Kotlin contributions should strive to follow Java and Kotlin best
    practices (discussion).

@pq
pq requested a review from helin24 as a code owner September 18, 2026 04:07

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the IntelliJ plugin verification process by introducing a centralized bash script, tool/check_verifier_baselines.sh, to normalize and compare verifier reports against committed baselines. It updates the baseline files for IDE branches 252, 253, and 261 to a new tab-separated format, simplifies the CI workflow in tool/github.sh, and adds detailed documentation to CONTRIBUTING.md. The review feedback recommends dynamically discovering baseline versions in tool/github.sh to prevent version drift and guarding the mktemp call in the new script to handle failures robustly.

Comment thread tool/github.sh Outdated
Comment thread tool/check_verifier_baselines.sh Outdated
pq and others added 3 commits September 24, 2026 08:04
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
@pq

pq commented Sep 24, 2026

Copy link
Copy Markdown
Collaborator Author

@helin24: this one is ready for review 🦅

@helin24 helin24 left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Do you want to add a new verifier baseline for 2026.2? Android Studio has it in preview version now I think. (Edit: Maybe as a follow up. Or let me know if you'd rather I did it to try out new verification)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants