feat: add profiling regression proposal skill - #179
Merged
Merged
Conversation
Adds an evidence-only performance skill that interprets classified Hermes CPU-profile regressions without re-deciding whether a regression exists. Co-authored-by: javiergarciavera <javiergarciavera@users.noreply.github.com>
10 tasks
javiergarciavera
enabled auto-merge (squash)
September 23, 2026 11:49
racitores
approved these changes
Sep 23, 2026
javiergarciavera
deleted the
mmqa/profiling-regression-proposal-db05
branch
September 23, 2026 12:04
pull Bot
pushed a commit
to AmirulAndalib/metamask-mobile
that referenced
this pull request
Sep 23, 2026
<!-- CURSOR_AGENT_PR_BODY_BEGIN --> Follow-up to MetaMask#36398. Performance E2E already captures Hermes CPU profiles, but there was no durable history and no Slack signal between full weeks. A one-off slow BrowserStack run also looked like several scenario regressions. This PR: 1. After each **scheduled** 6-hour performance suite, analyzes that run, stores `report.json` for later weeks, and compares each scenario to the **median of scheduled runs from the previous 7 days** (not the previous run). A scenario is flagged only if it has at least **3** baseline observations and reaches **1.5×** that median. Several scenarios crossing together are one slow-run card. 2. Posts Slack on **every** scheduled analysis: either the findings, or a one-line **nothing to action** so a quiet channel means the job stopped. Scenarios still short of 3 runs are counted as unchecked, not clean. 3. When there are findings, lists at most **5** PRs merged since the previous scheduled run, keeps only files that already appear as hot frames in `report.json`, and asks Claude using the pinned `mms-profiling-regression-proposal` skill ([MetaMask/skills#179](MetaMask/skills#179)). The model sees Evidence JSON only; it does not set the 1.5× threshold. A missing skill or API key does not hide the deterministic exception. 4. Keeps **Monday 09:00 UTC** week-over-week conclusions (completed UTC week vs the same weekdays the week before). Recovered spikes are named for the record and are not findings. 5. On `main`, Slack goes to channel `C0C3WSWNKS5`. A dispatch from a branch still goes to the personal DM. 6. Names owning teams as plain text (`owner team-handle`). No Slack user-group pings. 7. Manual dispatch still produces the full per-run digest (notes, per-scenario artifacts and download links). Slack splits past 40k into a thread instead of truncating. 8. Durations are Hermes JS work (sampled JS self time), not test wall clock; compared numbers share one unit. ## **Changelog** CHANGELOG entry: null ## **Related issues** Fixes: Refs: MetaMask#36398 ## **Manual testing steps** ```gherkin Feature: scheduled Hermes profiling Slack Scenario: a clean scheduled run posts an all-clear Given Analyze App Profiling chained from a successful scheduled performance suite And no scenario reaches 1.5× its 7-day median with at least 3 baseline runs When Slack is posted Then the message says nothing to action And it names how many scenarios were compared versus still building a baseline Scenario: a scheduled run that crosses the threshold posts findings Given a scenario has at least 3 observations in the previous 7 days And this run reaches at least 1.5× that median JS work When Slack is posted on main Then the target is channel C0C3WSWNKS5 And the owner is named without a Slack user-group mention Scenario: several scenarios spike on the same run Given two or more scenarios cross the threshold in that run When the exception is built Then it is described as one run-level anomaly Scenario: a branch dispatch does not page the channel Given Analyze App Profiling is running for a feature-branch performance suite When Slack is posted Then the target is the personal DM Scenario: findings get an evidence-only AI proposal Given a scheduled run crossed the 1.5× threshold And the pinned mms-profiling-regression-proposal skill is installed When Slack is posted Then a thread card may name a merged PR only if a hot-frame path from report.json is in that PR Scenario: Monday still reports the completed week Given it is Monday 09:00 UTC When the weekly job runs Then Slack is the week-over-week exception report ``` ```bash node --test '.github/scripts/qa-automation/performance-tests/*.test.mjs' NODE_OPTIONS=--max-old-space-size=8192 yarn jest --coverage=false --runInBand .github/scripts/qa-automation/performance-tests/analyze-app-profiling-workflow.test.ts ``` ## **Screenshots/Recordings** ### **Before** N/A — CI-only. Scheduled runs either stayed silent or only reported on Monday. ### **After** N/A — every scheduled run posts. Example all-clear: ```text *Hermes CPU-profile run check* · nothing to action _Run:_ <…|35745913189> _Checked:_ 3/4 scenarios against their median of the previous 168h · 1 still building a baseline of 3 runs _Result:_ no scenario reached 1.5× its recent median JS work. ``` Example finding: ```text *Possible regression:* one scenario exceeded 1.5× its recent median. _Owners are named for routing; no team is notified._ • *Perps add funds* — JS work 12.6 s vs 3.0 s recent median (4.2× across 3 baseline runs) · owner mm-perps-engineering-team ``` ## **Pre-merge author checklist** - [x] I've followed [MetaMask Contributor Docs](https://github.com/MetaMask/contributor-docs) and [MetaMask Mobile Coding Standards](https://github.com/MetaMask/metamask-mobile/blob/main/.github/guidelines/CODING_GUIDELINES.md). - [x] I've completed the PR template to the best of my ability - [x] I've included tests if applicable - [x] I've documented my code using [JSDoc](https://jsdoc.app/) format if applicable - [x] I've applied the right labels on the PR (see [labeling guidelines](https://github.com/MetaMask/metamask-mobile/blob/main/.github/guidelines/LABELING_GUIDELINES.md)). Not required for external contributors. #### Performance checks (if applicable) - [x] I've tested on Android - Ideally on a mid-range device; emulator is acceptable - [x] I've tested with a power user scenario - Use these [power-user SRPs](https://consensyssoftware.atlassian.net/wiki/spaces/TL1/pages/edit-v2/401401446401?draftShareId=9d77e1e1-4bdc-4be1-9ebb-ccd916988d93) to import wallets with many accounts and tokens - [x] I've instrumented key operations with Sentry traces for production performance metrics - See [`trace()`](/app/util/trace.ts) for usage and [`addToken`](/app/components/Views/AddAsset/components/AddCustomToken/AddCustomToken.tsx#L274) for an example For performance guidelines and tooling, see the [Performance Guide](https://consensyssoftware.atlassian.net/wiki/spaces/TL1/pages/400085549067/Performance+Guide+for+Engineers). ## **Pre-merge reviewer checklist** - [ ] I've manually tested the PR (e.g. pull and build branch, run the app, test code being changed). - [ ] I confirm that this PR addresses all acceptance criteria described in the ticket it closes and includes the necessary testing evidence such as recordings and or screenshots. <!-- CURSOR_AGENT_PR_BODY_END --> <div><a href="https://cursor.com/agents/bc-691c5ae7-1eaf-478c-a7a1-04f3bb65ca15?cursor_ref=pr_footer&cursor_cta=open_in_web"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/open-in-web-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/open-in-web-light.png"><img alt="Open in Web" width="114" height="28" src="https://cursor.com/assets/images/open-in-web-dark.png"></picture></a> <a href="https://cursor.com/background-agent?bcId=bc-691c5ae7-1eaf-478c-a7a1-04f3bb65ca15&cursor_ref=pr_footer&cursor_cta=open_in_cursor"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/open-in-cursor-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/open-in-cursor-light.png"><img alt="Open in Cursor" width="131" height="28" src="https://cursor.com/assets/images/open-in-cursor-dark.png"></picture></a> </div> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com> Co-authored-by: javiergarciavera <javiergarciavera@users.noreply.github.com>
![https://cursor.com/assets/images/open-in-web-dark.png"><source](https://cursor.com/assets/images/open-in-web-dark.png"><source){kind=link}
![https://cursor.com/assets/images/open-in-web-light.png"><img](https://cursor.com/assets/images/open-in-web-light.png"><img){kind=link}
![https://cursor.com/assets/images/open-in-web-dark.png"></picture></a> <a](https://cursor.com/assets/images/open-in-web-dark.png"></picture></a> <a){kind=link}
![https://cursor.com/assets/images/open-in-cursor-dark.png"><source](https://cursor.com/assets/images/open-in-cursor-dark.png"><source){kind=link}
![https://cursor.com/assets/images/open-in-cursor-light.png"><img](https://cursor.com/assets/images/open-in-cursor-light.png"><img){kind=link}
![https://cursor.com/assets/images/open-in-cursor-dark.png"></picture></a> </div](https://cursor.com/assets/images/open-in-cursor-dark.png"></picture></a> </div){kind=link}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds
performance/profiling-regression-proposalfor MetaMask Mobile's scheduledHermes CPU-profile job. Deterministic CI already decides whether a run crossed
the 1.5× threshold and which merged PR files overlap hot frames. This skill is
the system prompt for the optional model pass: it may only use that Evidence
JSON, must not reclassify, and must emit a short Slack proposal.
Type of Change
Skill Details (if adding a new skill)
Provider Name: MetaMask
Skill Name: profiling-regression-proposal
Brief Description: Evidence-only follow-up proposals after a classified Hermes CPU-profile regression.
Checklist
Testing
node .github/scripts/lint-skill-entry.mjs(new skill has When To Use + Workflow)tools/install --repo metamask-mobile --include performance/profiling-regression-proposalwritesmms-profiling-regression-proposalAdditional Context
Consumer: MetaMask/metamask-mobile PR around scheduled app-profiling analysis.
Mobile will pin this skill SHA and pass Evidence as the user message; classification stays in Mobile CI.