Repository navigation
fix(test): the local verify gate fails a passing suite that ran past its time share on a loaded host #10847
Description
Refs: #8757
Problem
The local verify gate that /source-control:pull-request ready runs (scripts/check-affected.sh --tests, which runs scripts/run-plugin-tests.sh with --budget 120) fails a test suite that passes, because the suite took longer than its time share. A failure recorded by that gate stops the draft-to-ready flip, and an unattended run cannot skip verify. So a passing test can hold a PR in draft until the host's load drops.
Measured on the dev WSL host for #10842, where the diff does not touch the suite:
scripts/post-merge-demotion-alert.test.shreported PASS=173 FAIL=0 every time and took 32-44s, against a 30s share. The load average was between 5 and 16.- The time does not depend on the diff: 34s on the tree ci: count every GitHub request the Node clients send, GraphQL split out #10811 merged from, 32s on the fix(pr-pipeline): plan refire-refused's job-token client to github-token #10842 branch, run back to back.
- Six verify retries, each started with the 1-minute load under 8, all failed for this reason alone.
The share is suite_share in scripts/run-plugin-tests.sh: the suite's median in scripts/suite-seconds.txt times PLUGIN_TEST_SHARE_MARGIN (3), floored at PLUGIN_TEST_SHARE_DEFAULT (30). This suite has no line in suite-seconds.txt, so it gets the 30s floor, whatever it costs and however loaded the host is.
Fix
The share should come from the suite's own baseline and the host's current load, so that a passing suite never blocks a flip:
- Give every selected suite a measured baseline: add
post-merge-demotion-alert.test.shtosuite-seconds.txt, and have the selection report a suite with no median instead of applying the floor without saying so. - Scale the share by measured load, for example by
max(1, loadavg / nproc), read when the run starts. Or, in the local gate only, report a passing suite over its share as a warning and keep the failure for suites that fail. CI's legs, where the budget protects the lanes, keep it as a failure.
Verification
- A stay-quiet test in
scripts/run-plugin-tests.test.sh: a suite that passes but runs past its fixed share on a simulated loaded host is not failed by the local gate. Before the fix this test fails. check-affected.sh --testson fix(pr-pipeline): plan refire-refused's job-token client to github-token #10842's diff passes on the dev WSL host at load 5-16.
Related
- perf(test): cut the local affected-test gate's shell suites to a budget #8757 (introduced the per-suite share)
- test-perf: diagnose and cut the wall clock of the local test and gate corpus, especially on Windows Git Bash #3716 (local gate wall clock)
- fix(pr-pipeline): plan refire-refused's job-token client to github-token #10842 (held in draft by this gate)
🤖 Generated with Claude Code
Activity
github-actions commentedon Oct 29, 2026
Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.
Size: Small
Complexity: Low
Uncertainty: Known
Flags: none
Blocked by: none
I understood the issue as follows. The local verify gate fails a passing suite that runs past its time share. The recorded decision settles every open question. The local gate warns instead of failing, and an env var set by
check-affected.shselects that behavior. Suites with no median get a stderr note. A stay-quiet test is added. No open questions remain from this issue's text. I found no duplicates among the open issues.Automated triage, applied by the lanes App.
melodic-automation-lanes commentedon Oct 29, 2026
Not admitted to the issue lane: no trusted-human acceptance criteria: a listed human must put the confirmed
- [ ]criteria in the body or post them as their own comment, or, on an issue a listed human filed, intake's newest refinement comment must hold them unedited. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).melodic-automation-lanes commentedon Oct 29, 2026
The latest triage verdict holds nothing: the issue is at
status: ready.melodic-automation-lanes commentedon Oct 29, 2026
Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. On an issue a listed human filed, it counts as the issue's spec once posted, while unedited; it is still model text, so check it.
Intended outcome: The local verify gate stops failing a passing suite that runs past its share on a loaded host, the demotion-alert suite gets a baseline, and a suite with no median is reported by name.
Related work
Open questions
RECOMMENDED: Downgrade to a warning, because a passing suite should never block a flip. Load scaling needs a load read that Git Bash on Windows lacks (/proc/loadavg), and the measured loads of 5-16 could still leave 30s shares too tight. Basis: scripts/run-plugin-tests.sh:437-446 fails any passing suite over its share. The issue lists the two options with 'or'.
RECOMMENDED: Use about 15, an estimate of the CI median, which gives a 45s share with the margin of 3. This needs the owner to confirm it from a CI timing. Basis: judgment; scripts/run-plugin-tests.sh:44-48 says the medians are CI runner measurements, and the suite has no line in scripts/suite-seconds.txt.
RECOMMENDED: Add a flag or env var such as PLUGIN_TEST_SHARE_WARN_ONLY=1 that check-affected.sh sets, and keep the default as a failure for other --budget callers. Basis: .github/workflows/maintenance-audit-test-selection.yml:140; scripts/check-affected.sh:217,687
RECOMMENDED: A one-line stderr note per unlisted suite that names it and the default share applied, and no failure. Basis: scripts/run-plugin-tests.sh:361-365 applies the floor silently.
Sources read
To answer, post a comment that starts with this line:
Automated refinement by the lanes App. Answer with a comment whose first line is
**Decision recorded.**.kyle-sexton commentedon Oct 29, 2026
Cause found: scripts/post-merge-demotion-alert.test.sh (added in #10730) has no row in scripts/suite-seconds.txt, so it gets the 30 s default share. It runs 30-31 s even alone on an idle host. It failed the local verify gate in 13 of 18 tries for one builder and 10 of 11 for another tonight (#10876, #10842). Smallest fix: add its measured median to suite-seconds.txt. The load-scaling proposal above still covers the general case.
kyle-sexton commentedon Oct 30, 2026
Decision recorded. The refinement's four questions, answered from the code on main and the 2026-10-29 cause comment (#10847 (comment)):
scripts/run-plugin-tests.sh:490-493fails a passing suite on time alone. Load scaling needs a load reading the Windows Git Bash host lacks. That comment measured 30-31 s for this suite alone on an idle host, so the failure is not tied to load.scripts/suite-seconds.txt:793givespost-merge-demotion-alert.test.sha median of 100, measured from whole-tree run 40892244877 (header, lines 19-20). This was that comment's "Smallest fix".scripts/check-affected.shsets an env var (for examplePLUGIN_TEST_SHARE_WARN_ONLY=1), andrun-plugin-tests.shkeeps failure as the default. The other--budgetcaller,maintenance-audit-test-selection.yml:140, keeps today's behavior, and no workflow file changes.suite-seconds.txt, naming the suite and the default share applied. No failure. This replaces the silent floor atrun-plugin-tests.sh:411-414.Dropped: "
check-affected.sh --testson #10842's diff passes on the dev WSL host at load 5-16". #10842 has merged, a lane cannot run on that host, and the stay-quiet test covers the case.Since this was filed,
/source-control:pull-request readystopped running the local verify step in this repository (docs/conventions/source-control.yaml:31-38). The pre-pushcheck-affectedrun still uses the share, so the defect still applies there.post-merge-demotion-alert.test.shis insuite-seconds.txt, and the selection reports a suite with no median instead of applying the floor silently. (body, amended: the baseline row has landed on main, and the unlisted-suite report remains)--budgetcallers keep the failure. (body, amended: this picks the body's second option, "Or, in the local gate only...")scripts/run-plugin-tests.test.sh: a suite that passes but runs past its fixed share on a simulated loaded host is not failed by the local gate. The test fails before the fix. (body, unchanged)