Skip to content

perf(test): enforce a per-suite wall-clock budget in CI, warn then fail #10230

Open

Description

@kyle-sexton

Parent: #3716 (slice 3 of 3). Runs in parallel with slice 1.
Runs in: lane.

Problem

#3716 criterion 6: a documented per-suite wall-clock budget that warns above a threshold and fails above a ceiling, so a suite cannot grow past what a contributor can run. Today nothing enforces a per-suite time: scripts/plan-test-lanes.sh uses scripts/suite-seconds.txt only to size and balance CI legs (ceil(seconds / budget)), and fleet-state.test.sh once passed the 600 s tool maximum on its own (#3716 comment). #8757 covers the separate local affected-run budget; this slice is the per-suite CI check.

Fix

  1. Add a CI check that reads each suite's measured wall seconds from the test legs of the run and compares it with a documented threshold and ceiling: above the threshold prints a warning naming the suite; above the ceiling fails. Seed the threshold and ceiling from the current scripts/suite-seconds.txt distribution so no suite on main fails on day one; state the numbers and their basis where the check is documented.
  2. A suite that legitimately needs more gets a named per-suite exception in one listed file, so an exception is visible in review.
  3. Document the budget in the scripts' docs where contributors read how to run tests.

Done when

  • A test for the check shows a suite above the threshold produces a warning and exit 0, and a suite above the ceiling exits non-zero
  • The check runs on pull requests and is green on main's current suite times
  • The threshold, ceiling and their basis are written in one documented place, and the exception list is one file
  • ci-status is green on the PR head

Split by the backlog HITL session (2026-10-25, owner rule Q93: split large work before a run picks it up). This was generated by AI.

Activity

  1. added
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Oct 26, 2026
  2. github-actions commented on Oct 26, 2026

    @github-actions
    Contributor

    Proposed work class: work-class: scoped
    Basis: automated intake triage; the lanes App applied this class.

    Size: Medium
    Complexity: Medium
    Uncertainty: Known
    Flags: none
    Blocked by: none

    Understood: add a CI check that reads each suite's measured wall seconds from the test legs. It warns above a documented threshold and fails above a ceiling. The numbers are seeded from scripts/suite-seconds.txt so main stays green. A single listed file holds the per-suite exceptions, and the budget is documented where contributors read how to run tests. The done-when items are specific, so nothing is missing. Related, not duplicate: #10200 (per-suite timing option and baseline doc), which looks like the sibling slice that runs in parallel.

    Automated triage, applied by the lanes App.

  3. added
    priority: mediumReal value, no hard deadline; normal backlog flow.
    and removed
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    priority: mediumReal value, no hard deadline; normal backlog flow.status: readyTriaged, unblocked, and fully specified; eligible to pick up.work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.

    Type

    Fields

    No fields configured for Task.

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions