Repository navigation
feat(pr-pipeline): trust reconciler re-runs of lost review, refine and feedback-dispatch runs #8524
Copy link
Copy link
Open
Open
Feature
Copy link
Labels
priority: mediumReal value, no hard deadline; normal backlog flow.Real value, no hard deadline; normal backlog flow.status: readyTriaged, unblocked, and fully specified; eligible to pick up.Triaged, unblocked, and fully specified; eligible to pick up.work-class: structuralRefactors, migrations, contract changes; cross-cutting and hard to reverse.Refactors, migrations, contract changes; cross-cutting and hard to reverse.
Description
Refs: #8468
Problem
The lost-run reconciler (.github/workflows/maintenance-reconcile-lost-runs.yml,
.github/scripts/maintenance-reconcile-lost-runs/reconcile.mjs) re-runs lost CI runs and, through
the reconciler-workflow input of check-trusted-trigger, lost pr-verify runs. Every other lane
run an outage loses is only listed in the run summary for a person, because a re-run that
github-actions[bot] starts is refused further down:
- Review lanes (
.github/workflows/pr-review-hosted.yml,
.github/workflows/pr-review-security-hosted.yml) call the reusable workflows
melodic-software/ci-workflows/.github/workflows/pr-review.ymlandpr-review-security.yml
(pinned v0.48.0). Their trigger gate lives in ci-workflows, which has noreconciler-workflow
input to pass through. - Refine (
.github/workflows/pr-refine.yml->.github/workflows/pr-run-activity-write.yml)
gets its lane App token from the lanes token broker through
.github/actions/request-lane-token. The broker (Azure Function inmelodic-software/azure-iac,
ADR 0058, github-iac ADR 0015) refuses a mint for a rungithub-actions[bot]re-ran; it trusts
agithub-actions[bot]run only as a recordedpr-unblock-dispatch.ymldispatch
(dispatcher_run_id). - Feedback dispatch (
.github/workflows/pr-feedback-dispatch.yml) writes the
pr-feedback-dispatched-*records thatcheck-trusted-trigger's dispatcher check reads; that
check accepts a dispatcher run only at attempt 1 or re-run by a listed human, so a
github-actions[bot]re-run of it dispatches nothing the lanes trust.
Proposed change
Extend the pr-verify rule (reconciler record artifact reconcile-rerun-<run id>-2-<head sha>,
judged by the run GitHub attributes it to, never its contents) to:
- ci-workflows: add a
reconciler-workflowinput topr-review.ymlandpr-review-security.yml
and pass it to their trigger check; bump the pin in both caller workflows here. - Broker (
melodic-software/azure-iac): accept agithub-actions[bot]re-run at attempt 2 when
the same reconciler record exists, with the same run, branch, event, actor, tag and ancestry
checks asverifyReconcilerRerunin
.github/actions/check-trusted-trigger/check-trusted-trigger.mjs; send the needed field from
.github/actions/request-lane-token/action.yml. check-trusted-triggerdispatcher check: accept a dispatcher run at attempt 2 started by
github-actions[bot]with a reconciler record, or keep feedback dispatch person-only and say
so.- Add each workflow to
RECORDED_LANESinreconcile.mjs, and give each a "work already done"
test like pr-verify's verdict-upload-step check, so a cancelled run that already acted (posted
a review, pushed a refine commit, dispatched) is never re-run.
Acceptance
- A lost review, refine or feedback-dispatch run is re-run once by the reconciler and its gate
passes; agithub-actions[bot]re-run without a record still stops. - A run that already reached its side effect is left to a person.
- Broker and ci-workflows changes land and are pinned before this repo turns them on.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
priority: mediumReal value, no hard deadline; normal backlog flow.Real value, no hard deadline; normal backlog flow.status: readyTriaged, unblocked, and fully specified; eligible to pick up.Triaged, unblocked, and fully specified; eligible to pick up.work-class: structuralRefactors, migrations, contract changes; cross-cutting and hard to reverse.Refactors, migrations, contract changes; cross-cutting and hard to reverse.
Type
Fields
No fields configured for Feature.
Activity
github-actions commentedon Oct 20, 2026
Proposed work class: work-class: structural
Basis: automated intake triage; the lanes App applied this class.
Size: Medium
Complexity: Medium
Uncertainty: Known
Flags: none
Blocked by: #9073, #10142
#8524 (comment)
The decision settles the open questions. I read the work as: add lost conclusions to the babysit merge gate's re-run triggers for review runs, and add a sweep recovery for lost feedback-dispatch and refine runs. It also needs tests for each side-effect-already-done case and a README note. It depends on #9073 and #10142.
Automated triage, applied by the lanes App.
kyle-sexton commentedon Oct 25, 2026
Released for re-triage under the owner's 2026-10-24 rule: an issue stops for a person only for an intent or vision question, a security call, a host or attended run, or spend outside prepaid credit. The hold label came off and intake triage runs again now; it re-applies a hold, with a numbered question, when one of those applies.
To hold this issue again, add
needs-human(orstatus: needs-decision) and say why in a comment.melodic-automation-lanes commentedon Oct 25, 2026
The latest triage verdict holds nothing: the issue is at
status: ready.melodic-automation-lanes commentedon Oct 25, 2026
Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. On an issue a listed human filed, it counts as the issue's spec once posted, while unedited; it is still model text, so check it.
Intended outcome: Extend the pr-verify reconciler rule so lost review and refine runs are re-run once by the reconciler and pass their gates, with feedback dispatch likely kept person-only, landing ci-workflows, azure-iac and this repo as separate changes.
Dependencies
Related work
Open questions
RECOMMENDED: Keep feedback dispatch person-only, say so in the check-trusted-trigger README, and leave it out of RECORDED_LANES. Basis: judgment: intake triage recommendation; the records it writes are what other lanes trust, so this is the largest trust widening
RECOMMENDED: Yes, with the same run, branch, event, actor, tag and ancestry checks as verifyReconcilerRerun. Basis: judgment: security call about token minting that lives in melodic-software/azure-iac, outside this repo
RECOMMENDED: Yes: a ci-workflows issue, an azure-iac issue, and a this-repo issue that waits on both and then edits RECORDED_LANES (.github/scripts/maintenance-reconcile-lost-runs/reconcile.mjs:94), the pins and the work-already-done tests. Basis: .github/scripts/maintenance-reconcile-lost-runs/reconcile.mjs:94 lists only pr-verify.yml
Sources read
Automated refinement by the lanes App. Answer with a comment whose first line is
**Decision recorded.**.kyle-sexton commentedon Oct 26, 2026
Decision recorded. The owner answered in the Refinement and Issue Triage sessions (2026-10-25), deciding the open items below on the facts:
No owner ledger (
.work/*/interview-checklist.md,.work/gh-monitor-owner-decisions/) names this issue.Decided with no trust widening: no change to the ci-workflows trigger gate, the azure-iac broker, or
check-trusted-trigger's dispatcher check. The facts show each kind of lost run can be recovered without one, so the body's items 1 to 3 are not built.Questions from intake's refinement (#8524 (comment)), plus the review-lane premise:
reconciler-workflowinput, or a reconciler re-run?Decided: Neither. Both callers now use
pr-review-cli.yml(.github/workflows/pr-review-hosted.yml:172,.github/workflows/pr-review-security-hosted.yml:177), so the body's v0.48.0 premise is out of date. The reconciler stays out of review lanes: its own record says they are "listed in the summary for a person: their gates or the lanes token broker refuse a re-run github-actions[bot] starts" (.github/scripts/maintenance-reconcile-lost-runs/reconcile.mjs:43-45), and makinggithub-actions[bot]a re-run actor for them would widen a trust gate. A lost review run is recovered through the one path that already re-runs AI review checks: the babysit merge gate, which re-runs under the owner's token on the pinned live head, attempt 1 only, at most once per check per head (plugins/source-control/skills/babysit-prs/reference/safety.md:847-858). The lost conclusions (cancelled,timed_out,startup_failure, the reconciler'sLOSTset atreconcile.mjs:89) join that gate's re-run triggers, beside rate-limit, publish rate-limit and pr-review: the bubblewrap install step times out at 1 minute and fails review lanes with no outcome #9073'sinstall-failed, read from the one definition pr-review: the bubblewrap install step times out at 1 minute and fails review lanes with no outcome #9073 creates. This changes no actor, token or gate.github-actions[bot]attempt-2 dispatcher?Decided: No. It stays person-only:
check-trusted-triggeraccepts a dispatcher at attempt 1 or re-run by a listed human (.github/actions/check-trusted-trigger/check-trusted-trigger.mjs:732), unchanged. A lostpr-feedback-dispatch.ymlrun is recovered by the half-hourly sweep, a fresh attempt-1 run the gate already trusts (.github/scripts/pr-unblock/feedback-dispatch.mjs:52-62). The sweep looks for recovery only when its run listing shows a lostpr-feedback-dispatch.ymlrun, using the reconciler's definitions: aLOSTconclusion with at least one job started (reconcile.mjs:89,:123-126), within its lookback window (LOOKBACK_HOURS = 24,.github/scripts/maintenance-reconcile-lost-runs/infra-failure.mjs:36), so a run replaced in its concurrency group (.github/workflows/pr-feedback-dispatch.yml:63-66) does not count and one failed tick does not lose the recovery. In the common case that is one call per tick, which keeps within the owner's rate-limit constraint ("within our environmental, github, rate limit, etc. constraints", owner-quotes.md, Issue Triage line). It then dispatches a head thatdispatchFeedbackwould dispatch.dispatchFeedbackalready skips a head with an address-feedback check run or a queued or running address-feedback run (feedback-dispatch.mjs:29-34,:301-318), so a lost run that already dispatched is not dispatched twice. pr-pipeline: react to pull request comments without waiting for the next review run #8440 extends the same sweep for fix requests; both use one selection function.github-actions[bot]attempt-2 refine re-run mint a token?Decided: No. Refine is a
workflow_dispatchfrom the dispatcher (.github/workflows/pr-refine.yml, inputdispatcher-run-id), andfix-docs-dispatch.mjsruns in that dispatcher, so "the trigger gate and the broker check it as they check a resolve-conflicts dispatch" (.github/scripts/pr-unblock/fix-docs-dispatch.mjs:6-8). A lost refine run is recovered by one fresh dispatch on that path, which the broker already trusts. The rule is defined once, in pr-unblock: derive fix-ci-subscription's start and the restart cap from trusted values #8890's record: a fix-docs run lost to an outage with no check posted gets one more dispatch, and a completed run without a check otherwise makes the PR handled.Decided: Not needed. With no ci-workflows or azure-iac change, every edit is in this repository (question 0,
reconcile.mjs:89-98).Converted from the body's "Acceptance" list, wording unchanged except: "re-run once by the reconciler" becomes a re-run by the babysit merge gate for review runs, and refine and feedback-dispatch runs get one fresh dispatch on their trusted path (questions 1 and 2), and the "without a record" clause applies to those two only, since a review re-run passes its unchanged gate (question 0). The third bullet ("Broker and ci-workflows changes land and are pinned ...") is dropped, since neither repository changes.
Acceptance criteria:
cancelled,timed_outorstartup_failure) is re-run once by the babysit merge gate under its existing head, attempt and job-binding rules, read from the single re-run trigger definition pr-review: the bubblewrap install step times out at 1 minute and fails review lanes with no outcome #9073 creates; a lost refine or feedback-dispatch run gets one fresh dispatch on its trusted path; agithub-actions[bot]re-run of a refine or feedback-dispatch run still stops.failureoutside its listed classes; a test covers a plainfailurethat is not re-run.reconcile.mjsreview-lane exclusion unchanged).pr-feedback-dispatch.ymlrun in the lookback window, and a run with no started job does not count as lost. A test pins each.RECORDED_LANESandcheck-trusted-trigger.mjsare unchanged, and nothing changes inmelodic-software/ci-workflowsormelodic-software/azure-iac..github/actions/check-trusted-trigger/README.mdstates that feedback dispatch stays person-only and that the sweep recovers its lost runs.Depends on #9073's single re-run trigger definition, and on #10142's
rerun_bindsso attempt 2 binds.melodic-automation-lanes commentedon Oct 26, 2026
Not admitted to the issue lane yet: blocker #9073 is not closed by a listed human or a merge.