Skip to content

fix(ci): a contract-only ci-status that races the full run stays red #9113

Closed

Description

@kyle-sexton

Problem

A contract-only ci-status run that starts while the full run on the same SHA is still running fails ("no successful ci-lanes status on ... A full run that records ci-lanes=success re-runs this run"), and the promised re-run does not happen or races the full run, so the stale red stays the latest ci-status and the PR sits BLOCKED until someone re-runs it by hand. Seen 2026-10-22 on:

  • claude-code-plugins#9012: run 39791980285 (re-run hit the gate-job lookup failure during the Actions incident)
  • standards#845: run 39810529957 (attempt 1, never re-run automatically)
  • claude-code-plugins#9076: run 39813986263 attempt 2 (re-run raced the full run, which recorded ci-lanes=success at 16:29:56Z; the contract-only check failed at 16:30:04Z)

Fix

Make the contract-only run wait (bounded, well under its timeout) for an in-flight full run on the same SHA to record ci-lanes, or have the full run re-run the contract-only run after it records success even when that run is already completed. Either way, the latest ci-status on a SHA whose full run passed must not stay red.

Related

  • pr-require-checks.yml ci-status job

Activity

  1. added
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Oct 22, 2026
  2. github-actions commented on Oct 22, 2026

    @github-actions
    Contributor

    Proposed work class: work-class: scoped
    Basis: automated intake triage; the lanes App applied this class.

    Size: Medium
    Complexity: Medium
    Uncertainty: Known
    Flags: none
    Blocked by: none

    Decision recorded. The owner answered in the Refinement session (2026-10-25), deciding the open items below on the facts:

    go with whatever the facts/data/metrics + /discovery:research says and ensure we're aligned with SOLID, DRY/SSOT principles

    1. (intake Q1) Where does the fix land? Cross-repo.
      Decided: In melodic-software/ci-workflows (resolve_own_gate in aggregate-results run.sh), then a pin bump here.
    2. Decided: Lengthen the own-job lookup, retrying 5xx and lag, and make its failure name the own-job lookup.
    3. Decided: About 120 s instead of about 45 s, and raise GH_READ_MARGIN_SECONDS and the timeout comment by the same 75 s.

    [quote truncated; the full decision is at the link]

    #9113 (comment)

    The decision settles every open question. The work is a ci-workflows change to the own-job lookup (about 120 s retry, a named error), then a pin bump and read-margin raise in pr-require-checks.yml. Confirming the latest ci-status stays green on live PRs after the bump is a person's check.

    Automated triage, applied by the lanes App.

  3. added
    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.
    and removed
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Oct 22, 2026
  4. kyle-sexton commented on Oct 22, 2026

    @kyle-sexton
    MemberAuthor

    Root cause found for #9076 (run 39813986263 attempt 3, 16:45Z): the status filter matched the ci-lanes success at 16:29:56Z correctly. The step failed closed in resolve_own_gate (ci-workflows .github/actions/pr-require-checks/aggregate-results/run.sh:476,498,505-585 at 5e18f3d6): GET actions/jobs/<own job> did not show the running job as in_progress within its 5 reads (~15 s), logged "no in-progress job matched job.check_run_id ... after 5 reads", then reported the generic "no successful ci-lanes status" at run.sh:979, which hides the real cause. Same "no in-progress job matched" line as #9012's run 39791980285 during the Actions incident.

    Fix candidates:

    • Lengthen the own-job lookup's backoff (it is a read inside a running job; the job's own timeout bounds it), retrying 5xx and lag the way the other readers do.
    • When the own-job lookup fails, say so in the final error instead of "no successful ci-lanes status".
  5. removed
    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.
    on Oct 25, 2026
  6. kyle-sexton commented on Oct 25, 2026

    @kyle-sexton
    MemberAuthor

    Released for re-triage under the owner's 2026-10-24 rule: an issue stops for a person only for an intent or vision question, a security call, a host or attended run, or spend outside prepaid credit. The hold label came off and intake triage runs again now; it re-applies a hold, with a numbered question, when one of those applies.

    To hold this issue again, add needs-human (or status: needs-decision) and say why in a comment.

  7. added
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    priority: highSignificant impact, or blocks an imminent release; staff this cycle.
    and removed
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Oct 25, 2026
  8. added
    status: readyTriaged, unblocked, and fully specified; eligible to pick up.
    work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.
    status: needs-decisionAwaiting a human or maintainer judgment call.
    and removed
    status: readyTriaged, unblocked, and fully specified; eligible to pick up.
    on Oct 25, 2026
  9. 1 remaining item

  10. melodic-automation-lanes commented on Oct 25, 2026

    @melodic-automation-lanes
    Contributor

    The latest triage verdict holds nothing: the issue is at status: ready.

  11. melodic-automation-lanes commented on Oct 25, 2026

    @melodic-automation-lanes
    Contributor

    Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. On an issue a listed human filed, it counts as the issue's spec once posted, while unedited; it is still model text, so check it.

    Intended outcome: A contract-only ci-status on a SHA whose full run recorded ci-lanes=success no longer fails closed on the own-job lookup, and a failed lookup reports its real cause.

    Related work

    Open questions

    1. Where does the fix land? resolve_own_gate in aggregate-results run.sh is in melodic-software/ci-workflows, and this repo only pins it at v0.58.0 (5e18f3d6). Is this a ci-workflows change plus a pin bump here, or a change to the ci-status job alone?
      RECOMMENDED: A ci-workflows change (longer own-job lookup backoff retrying 5xx and lag, and a final error naming the own-job lookup), then a pin bump here. An unattended run in this repo cannot edit ci-workflows. Basis: .github/workflows/pr-require-checks.yml:4771
    2. Which fix does the owner want: a longer own-job lookup (the 2026-10-22 comment's root cause), a bounded wait for an in-flight full run (the issue body), or a full-run re-run of the completed contract-only run?
      RECOMMENDED: Lengthen the own-job lookup and make its failure message name it, since the comment shows the ci-lanes status matched and only the own-job read failed. Basis: judgment
    3. Is this the same work as #​8482 (stop the contract-only yield from posting red)? If so, which issue stays open?
      RECOMMENDED: Keep this one for the own-job lookup fix and link #​8482 as related, unless #​8482 turns out to ask for the same change. Basis: 8482
    4. What are the numeric limits? The ci-status job has timeout-minutes 11 and budgets 45 s for own-gate retries now; how long may the lookup retry?
      RECOMMENDED: Raise the own-job retry budget from about 45 s to about 120 s, and keep the total within the 11-minute timeout. Basis: .github/workflows/pr-require-checks.yml:3120-3133

    Sources read

    • context.json: issue 9113 body, 2026-10-22 comment naming resolve_own_gate (run.sh:476,498,505-585,979 at 5e18f3d6) as the cause
    • /home/runner/work/claude-code-plugins/claude-code-plugins/.base/.github/workflows/pr-require-checks.yml:73-81: a contract-only run waits for nothing and reads the ci-lanes status once
    • /home/runner/work/claude-code-plugins/claude-code-plugins/.base/.github/workflows/pr-require-checks.yml:3120-3142: ci-status timeout is 11 minutes and budgets 45 s of own-gate retries
    • /home/runner/work/claude-code-plugins/claude-code-plugins/.base/.github/workflows/pr-require-checks.yml:4771: aggregate-results is pinned to ci-workflows v0.58.0 (5e18f3d6); the code is not in this checkout

    Automated refinement by the lanes App. Answer with a comment whose first line is **Decision recorded.**.

  12. kyle-sexton commented on Oct 26, 2026

    @kyle-sexton
    MemberAuthor

    Decision recorded. The owner answered in the Refinement session (2026-10-25), deciding the open items below on the facts:

    go with whatever the facts/data/metrics + /discovery:research says and ensure we're aligned with SOLID, DRY/SSOT principles

    I should only ever be in the loop ONLY when necessary (intent/vision changed, something came up during implementation that impacts trajectory/alignment, OR a high security risk) - I've covered this all before

    Question asked: Fix the own-job lookup in ci-workflows (longer retry, named error) and bump the pin here, rather than a wait or re-run in this repo?

    1. (intake Q1) Where does the fix land? Cross-repo.
      Decided: In melodic-software/ci-workflows (resolve_own_gate in aggregate-results run.sh), then a pin bump here. Basis: kept; the pin is now at .github/workflows/pr-require-checks.yml:4829 (aggregate-results@5e18f3d6... # v0.58.0), moved from intake's :4771.
    2. (intake Q2) Which fix: longer own-job lookup, bounded wait for the full run (body), or a re-run of the completed contract-only run?
      Decided: Lengthen the own-job lookup, retrying 5xx and lag, and make its failure name the own-job lookup. The 2026-10-22 comment shows the ci-lanes status matched and only the own-job read failed. Basis: kept; fix(ci): a contract-only ci-status that races the full run stays red #9113 (comment) (a diagnosis, not an owner decision).
    3. (intake Q3) Same work as ci: stop the contract-only ci-status yield from posting red #8482?
      Decided: Keep this issue for the own-job lookup and link ci: stop the contract-only ci-status yield from posting red #8482 as related. Basis: kept, judgment; ci: stop the contract-only ci-status yield from posting red #8482's text was not read this session.
    4. (intake Q4) How long may the lookup retry?
      Decided: About 120 s instead of about 45 s, and raise GH_READ_MARGIN_SECONDS and the timeout comment by the same 75 s, since the 300 s margin already holds the 45 s. Basis: corrected (intake left the margin out); .github/workflows/pr-require-checks.yml:3120-3142.

    Acceptance criteria:

    • The pinned aggregate-results retries the own-job lookup for about 120 s, retrying 5xx and lag.
    • A failed own-job lookup's final error names the own-job lookup, not "no successful ci-lanes status".
    • pr-require-checks.yml pins that ci-workflows release, and its read margin covers the longer retry within timeout-minutes: 11.

    Checks a person runs:

    • The latest ci-status on a SHA whose full run passed does not stay red (the body's Fix outcome, wording unchanged), seen on live PRs after the pin bump.
  13. added
    admission-deferredAdmitted but waiting for WIP room; the hourly work-unblock sweep labels it agent-ready.
    on Oct 28, 2026
  14. added
    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.
    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.
    and removed
    admission-deferredAdmitted but waiting for WIP room; the hourly work-unblock sweep labels it agent-ready.
    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.
    on Oct 28, 2026
  15. melodic-automation-lanes commented on Oct 28, 2026

    @melodic-automation-lanes
    Contributor

    The work-resolve-issue lane stopped on this issue (patch-protected-path): the change needs a path the issue lane may not push, which a maintainer session builds by hand.
    It added host-run and removed agent-ready: an agent run on a host builds it, and it waits on no decision from a person.

    In the run's words:

    The run stopped early (patch-protected-path): its change needs a path the issue lane may not push, which a person changes instead. In the run's words: patch-protected-path: .github/workflows/pr-require-checks.yml. The recorded decision (issuecomment-6409831514) lands the fix in melodic-software/ci-workflows (resolve_own_gate in .github/actions/pr-require-checks/aggregate-results/run.sh: a lookup that retries for about 120 s, covering 5xx and lag, and a final error that names the own-job lookup). It then needs a pin bump here at .github/workflows/pr-require-checks.yml:5456 (aggregate-results81a603a6... # v0.65.17), plus GH_READ_MARGIN_SECONDS (line 3280, now 300) and its timeout comment (lines 3275-3280) raised by 75 s. This lane cannot patch that workflow file, and it cannot edit the ci-workflows repository at all.

    What the run says it did not do:

    • ci-workflows change to resolve_own_gate (about 120 s retry covering 5xx and lag, plus a final error that names the own-job lookup): that repository is outside this checkout
    • Pin bump and GH_READ_MARGIN_SECONDS +75 s in .github/workflows/pr-require-checks.yml: protected path the lane cannot patch
    • Repository checks (check-changelog-fragments, check-affected), tidy, review, verify: skipped because nothing changed

    Run: https://github.com/melodic-software/claude-code-plugins/actions/runs/40789984146

    To hand it back to the lane, remove host-run and add agent-ready again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.priority: highSignificant impact, or blocks an imminent release; staff this cycle.status: readyTriaged, unblocked, and fully specified; eligible to pick up.work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.

    Type

    Fields

    No fields configured for Bug.

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions