Skip to content

feat(pr-pipeline): trust reconciler re-runs of lost review, refine and feedback-dispatch runs #8524

Description

@kyle-sexton

Refs: #8468

Problem

The lost-run reconciler (.github/workflows/maintenance-reconcile-lost-runs.yml,
.github/scripts/maintenance-reconcile-lost-runs/reconcile.mjs) re-runs lost CI runs and, through
the reconciler-workflow input of check-trusted-trigger, lost pr-verify runs. Every other lane
run an outage loses is only listed in the run summary for a person, because a re-run that
github-actions[bot] starts is refused further down:

  • Review lanes (.github/workflows/pr-review-hosted.yml,
    .github/workflows/pr-review-security-hosted.yml) call the reusable workflows
    melodic-software/ci-workflows/.github/workflows/pr-review.yml and pr-review-security.yml
    (pinned v0.48.0). Their trigger gate lives in ci-workflows, which has no reconciler-workflow
    input to pass through.
  • Refine (.github/workflows/pr-refine.yml -> .github/workflows/pr-run-activity-write.yml)
    gets its lane App token from the lanes token broker through
    .github/actions/request-lane-token. The broker (Azure Function in melodic-software/azure-iac,
    ADR 0058, github-iac ADR 0015) refuses a mint for a run github-actions[bot] re-ran; it trusts
    a github-actions[bot] run only as a recorded pr-unblock-dispatch.yml dispatch
    (dispatcher_run_id).
  • Feedback dispatch (.github/workflows/pr-feedback-dispatch.yml) writes the
    pr-feedback-dispatched-* records that check-trusted-trigger's dispatcher check reads; that
    check accepts a dispatcher run only at attempt 1 or re-run by a listed human, so a
    github-actions[bot] re-run of it dispatches nothing the lanes trust.

Proposed change

Extend the pr-verify rule (reconciler record artifact reconcile-rerun-<run id>-2-<head sha>,
judged by the run GitHub attributes it to, never its contents) to:

  1. ci-workflows: add a reconciler-workflow input to pr-review.yml and pr-review-security.yml
    and pass it to their trigger check; bump the pin in both caller workflows here.
  2. Broker (melodic-software/azure-iac): accept a github-actions[bot] re-run at attempt 2 when
    the same reconciler record exists, with the same run, branch, event, actor, tag and ancestry
    checks as verifyReconcilerRerun in
    .github/actions/check-trusted-trigger/check-trusted-trigger.mjs; send the needed field from
    .github/actions/request-lane-token/action.yml.
  3. check-trusted-trigger dispatcher check: accept a dispatcher run at attempt 2 started by
    github-actions[bot] with a reconciler record, or keep feedback dispatch person-only and say
    so.
  4. Add each workflow to RECORDED_LANES in reconcile.mjs, and give each a "work already done"
    test like pr-verify's verdict-upload-step check, so a cancelled run that already acted (posted
    a review, pushed a refine commit, dispatched) is never re-run.

Acceptance

  • A lost review, refine or feedback-dispatch run is re-run once by the reconciler and its gate
    passes; a github-actions[bot] re-run without a record still stops.
  • A run that already reached its side effect is left to a person.
  • Broker and ci-workflows changes land and are pinned before this repo turns them on.

Activity

  1. added
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Oct 20, 2026
  2. github-actions commented on Oct 20, 2026

    @github-actions
    Contributor

    Proposed work class: work-class: structural
    Basis: automated intake triage; the lanes App applied this class.

    Size: Medium
    Complexity: Medium
    Uncertainty: Known
    Flags: none
    Blocked by: #9073, #10142

    Decision recorded. The owner answered in the Refinement and Issue Triage sessions (2026-10-25), deciding the open items below on the facts:

    go with whatever the facts/data/metrics + /discovery:research says and ensure we're aligned with SOLID, DRY/SSOT principles

    Decided with no trust widening: no change to the ci-workflows trigger gate, the azure-iac broker, or check-trusted-trigger's dispatcher check. The facts show each kind of lost run can be recovered without one, so the body's items 1 to 3 are not built.

    [quote truncated; the full decision is at the link]

    #8524 (comment)

    The decision settles the open questions. I read the work as: add lost conclusions to the babysit merge gate's re-run triggers for review runs, and add a sweep recovery for lost feedback-dispatch and refine runs. It also needs tests for each side-effect-already-done case and a README note. It depends on #9073 and #10142.

    Automated triage, applied by the lanes App.

  3. added
    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.
    and removed
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Oct 20, 2026
  4. removed
    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.
    on Oct 25, 2026
  5. kyle-sexton commented on Oct 25, 2026

    @kyle-sexton
    MemberAuthor

    Released for re-triage under the owner's 2026-10-24 rule: an issue stops for a person only for an intent or vision question, a security call, a host or attended run, or spend outside prepaid credit. The hold label came off and intake triage runs again now; it re-applies a hold, with a numbered question, when one of those applies.

    To hold this issue again, add needs-human (or status: needs-decision) and say why in a comment.

  6. added
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    priority: mediumReal value, no hard deadline; normal backlog flow.
    status: needs-decisionAwaiting a human or maintainer judgment call.
    on Oct 25, 2026
  7. melodic-automation-lanes commented on Oct 25, 2026

    @melodic-automation-lanes
    Contributor

    The latest triage verdict holds nothing: the issue is at status: ready.

  8. added
    work-class: structuralRefactors, migrations, contract changes; cross-cutting and hard to reverse.
    and removed
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Oct 25, 2026
  9. melodic-automation-lanes commented on Oct 25, 2026

    @melodic-automation-lanes
    Contributor

    Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. On an issue a listed human filed, it counts as the issue's spec once posted, while unedited; it is still model text, so check it.

    Intended outcome: Extend the pr-verify reconciler rule so lost review and refine runs are re-run once by the reconciler and pass their gates, with feedback dispatch likely kept person-only, landing ci-workflows, azure-iac and this repo as separate changes.

    Dependencies

    • melodic-software/ci-workflows: reconciler-workflow input on pr-review.yml and pr-review-security.yml must be released before the pin here is bumped
    • melodic-software/azure-iac: broker must accept a recorded github-actions[bot] attempt-2 re-run before refine is added to RECORDED_LANES

    Related work

    Open questions

    1. Should the check-trusted-trigger dispatcher check accept an attempt-2 feedback-dispatch run started by github-actions[bot] with a reconciler record, or keep feedback dispatch person-only?
      RECOMMENDED: Keep feedback dispatch person-only, say so in the check-trusted-trigger README, and leave it out of RECORDED_LANES. Basis: judgment: intake triage recommendation; the records it writes are what other lanes trust, so this is the largest trust widening
    2. Does the owner approve the broker trust change letting a github-actions[bot] attempt-2 re-run mint a lane token when a reconciler record exists?
      RECOMMENDED: Yes, with the same run, branch, event, actor, tag and ancestry checks as verifyReconcilerRerun. Basis: judgment: security call about token minting that lives in melodic-software/azure-iac, outside this repo
    3. Should this issue be split into one issue per repo (ci-workflows, azure-iac, this repo) since a lane run here can only change this repo?
      RECOMMENDED: Yes: a ci-workflows issue, an azure-iac issue, and a this-repo issue that waits on both and then edits RECORDED_LANES (.github/scripts/maintenance-reconcile-lost-runs/reconcile.mjs:94), the pins and the work-already-done tests. Basis: .github/scripts/maintenance-reconcile-lost-runs/reconcile.mjs:94 lists only pr-verify.yml

    Sources read

    • .github/scripts/maintenance-reconcile-lost-runs/reconcile.mjs:94: RECORDED_LANES contains only .github/workflows/pr-verify.yml
    • .github/workflows/pr-verify.yml:117: passes reconciler-workflow to check-trusted-trigger
    • .github/actions/check-trusted-trigger/README.md:141: documents the reconciler re-run rule at attempt 2

    Automated refinement by the lanes App. Answer with a comment whose first line is **Decision recorded.**.

  10. kyle-sexton commented on Oct 26, 2026

    @kyle-sexton
    MemberAuthor

    Decision recorded. The owner answered in the Refinement and Issue Triage sessions (2026-10-25), deciding the open items below on the facts:

    go with whatever the facts/data/metrics + /discovery:research says and ensure we're aligned with SOLID, DRY/SSOT principles

    I should only ever be in the loop ONLY when necessary (intent/vision changed, something came up during implementation that impacts trajectory/alignment, OR a high security risk) - I've covered this all before

    No owner ledger (.work/*/interview-checklist.md, .work/gh-monitor-owner-decisions/) names this issue.

    Decided with no trust widening: no change to the ci-workflows trigger gate, the azure-iac broker, or check-trusted-trigger's dispatcher check. The facts show each kind of lost run can be recovered without one, so the body's items 1 to 3 are not built.

    Questions from intake's refinement (#8524 (comment)), plus the review-lane premise:

    1. Do the review lanes need a ci-workflows reconciler-workflow input, or a reconciler re-run?
      Decided: Neither. Both callers now use pr-review-cli.yml (.github/workflows/pr-review-hosted.yml:172, .github/workflows/pr-review-security-hosted.yml:177), so the body's v0.48.0 premise is out of date. The reconciler stays out of review lanes: its own record says they are "listed in the summary for a person: their gates or the lanes token broker refuse a re-run github-actions[bot] starts" (.github/scripts/maintenance-reconcile-lost-runs/reconcile.mjs:43-45), and making github-actions[bot] a re-run actor for them would widen a trust gate. A lost review run is recovered through the one path that already re-runs AI review checks: the babysit merge gate, which re-runs under the owner's token on the pinned live head, attempt 1 only, at most once per check per head (plugins/source-control/skills/babysit-prs/reference/safety.md:847-858). The lost conclusions (cancelled, timed_out, startup_failure, the reconciler's LOST set at reconcile.mjs:89) join that gate's re-run triggers, beside rate-limit, publish rate-limit and pr-review: the bubblewrap install step times out at 1 minute and fails review lanes with no outcome #9073's install-failed, read from the one definition pr-review: the bubblewrap install step times out at 1 minute and fails review lanes with no outcome #9073 creates. This changes no actor, token or gate.
    2. Feedback dispatch: accept a github-actions[bot] attempt-2 dispatcher?
      Decided: No. It stays person-only: check-trusted-trigger accepts a dispatcher at attempt 1 or re-run by a listed human (.github/actions/check-trusted-trigger/check-trusted-trigger.mjs:732), unchanged. A lost pr-feedback-dispatch.yml run is recovered by the half-hourly sweep, a fresh attempt-1 run the gate already trusts (.github/scripts/pr-unblock/feedback-dispatch.mjs:52-62). The sweep looks for recovery only when its run listing shows a lost pr-feedback-dispatch.yml run, using the reconciler's definitions: a LOST conclusion with at least one job started (reconcile.mjs:89, :123-126), within its lookback window (LOOKBACK_HOURS = 24, .github/scripts/maintenance-reconcile-lost-runs/infra-failure.mjs:36), so a run replaced in its concurrency group (.github/workflows/pr-feedback-dispatch.yml:63-66) does not count and one failed tick does not lose the recovery. In the common case that is one call per tick, which keeps within the owner's rate-limit constraint ("within our environmental, github, rate limit, etc. constraints", owner-quotes.md, Issue Triage line). It then dispatches a head that dispatchFeedback would dispatch. dispatchFeedback already skips a head with an address-feedback check run or a queued or running address-feedback run (feedback-dispatch.mjs:29-34, :301-318), so a lost run that already dispatched is not dispatched twice. pr-pipeline: react to pull request comments without waiting for the next review run #8440 extends the same sweep for fix requests; both use one selection function.
    3. Broker: let a github-actions[bot] attempt-2 refine re-run mint a token?
      Decided: No. Refine is a workflow_dispatch from the dispatcher (.github/workflows/pr-refine.yml, input dispatcher-run-id), and fix-docs-dispatch.mjs runs in that dispatcher, so "the trigger gate and the broker check it as they check a resolve-conflicts dispatch" (.github/scripts/pr-unblock/fix-docs-dispatch.mjs:6-8). A lost refine run is recovered by one fresh dispatch on that path, which the broker already trusts. The rule is defined once, in pr-unblock: derive fix-ci-subscription's start and the restart cap from trusted values #8890's record: a fix-docs run lost to an outage with no check posted gets one more dispatch, and a completed run without a check otherwise makes the PR handled.
    4. Split per repository?
      Decided: Not needed. With no ci-workflows or azure-iac change, every edit is in this repository (question 0, reconcile.mjs:89-98).

    Converted from the body's "Acceptance" list, wording unchanged except: "re-run once by the reconciler" becomes a re-run by the babysit merge gate for review runs, and refine and feedback-dispatch runs get one fresh dispatch on their trusted path (questions 1 and 2), and the "without a record" clause applies to those two only, since a review re-run passes its unchanged gate (question 0). The third bullet ("Broker and ci-workflows changes land and are pinned ...") is dropped, since neither repository changes.

    Acceptance criteria:

    • A lost review run (cancelled, timed_out or startup_failure) is re-run once by the babysit merge gate under its existing head, attempt and job-binding rules, read from the single re-run trigger definition pr-review: the bubblewrap install step times out at 1 minute and fails review lanes with no outcome #9073 creates; a lost refine or feedback-dispatch run gets one fresh dispatch on its trusted path; a github-actions[bot] re-run of a refine or feedback-dispatch run still stops.
    • A run that already reached its side effect is left to a person: a review run whose lane posted its review or reviewed-head marker on that head, a refine run after which a lanes App commit landed on the PR, a feedback-dispatch run that already dispatched. A test covers each.
    • The babysit merge gate re-runs a review run on a lost conclusion and never on a failure outside its listed classes; a test covers a plain failure that is not re-run.
    • The reconciler still lists lost review runs for a person and re-runs none (reconcile.mjs review-lane exclusion unchanged).
    • The sweep's recovery path makes one run-listing call on a tick with no lost pr-feedback-dispatch.yml run in the lookback window, and a run with no started job does not count as lost. A test pins each.
    • RECORDED_LANES and check-trusted-trigger.mjs are unchanged, and nothing changes in melodic-software/ci-workflows or melodic-software/azure-iac.
    • .github/actions/check-trusted-trigger/README.md states that feedback dispatch stays person-only and that the sweep recovers its lost runs.

    Depends on #9073's single re-run trigger definition, and on #10142's rerun_binds so attempt 2 binds.

  11. melodic-automation-lanes commented on Oct 26, 2026

    @melodic-automation-lanes
    Contributor

    Not admitted to the issue lane yet: blocker #9073 is not closed by a listed human or a merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    priority: mediumReal value, no hard deadline; normal backlog flow.status: readyTriaged, unblocked, and fully specified; eligible to pick up.work-class: structuralRefactors, migrations, contract changes; cross-cutting and hard to reverse.

    Fields

    No fields configured for Feature.

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions