pr-pipeline: post-V1 audit and retro of the unattended merge flow

Ask

The owner's direction (2026-10-12): ship V1 of unattended merge first, then audit, analyze and run a retro on the whole flow end to end, adopting outside practices (Matt Pocock, poteto) only on data and metrics.

Scope, once `pr-merge` merges live

CI/CD performance, the autonomous Actions lanes, and the merge path; find the friction points.
From day one of live merges: revert and demotion rate per risk bucket, escaped-defect counts, and merged-and-not-reverted outcomes.
Candidates to evaluate on that data: sampled post-merge review, a verifier verdict bound to the head SHA, a mutation pass over changed tests.

Refs: #6202, #4588

Proposed work class: work-class: scoped
Basis: automated intake triage; the lanes App applied this class.

Size: Medium
Complexity: Medium
Uncertainty: Knowable
Flags: none
Blocked by: #7528

Decision recorded.
The owner answered in the Refinement session (2026-10-25), deciding the open items below on the facts:

go with whatever the facts/data/metrics + /discovery:research says and ensure we're aligned with SOLID, DRY/SSOT principles

I should only ever be in the loop ONLY when necessary (intent/vision changed, something came up during implementation that impacts trajectory/alignment, OR a high security risk) - I've covered this all before

[quote truncated; the full decision is at the link]

#7531 (comment)

The decision settles the open questions. #7531 keeps the data retro, which starts at 30 live pr-merge merges or 30 days, whichever is later. The static audit moves to a child issue. The deliverable is a report under docs/conventions/pr-pipeline/ plus follow-up issues, with no change to lane behavior. The measurement window can't be checked in an unattended run, so that item waits for live merges. Demotion counts depend on #7528.

_{Automated triage, applied by the lanes App.}

What the article adds for this audit, and the owner's 2026-10-16 decisions. Source: https://claude.dev/blog/solving-code-review-at-anthropic (2026-10-15); ids are digest unit and claim.

The article: "We don't yet have circuit breakers that tighten approval when those numbers move; engineers here want them before more code moves to zero-human merges" (04 C70). Noticing a bad merge still depends on the author, on-call and the revert and incident dashboards (04 C69). Revert and incident rates are the guardrail (02 C8; 10 C21).

Owner decision, 2026-10-16: a minimal revert-triggered stop. A revert of an agent-merged PR stops that lane's unattended merges until the owner re-enables them. It stops merges rather than lowering a rung because Phase 9 removes the rung ladder (Q505). It is not a parallel mechanism: a revert of an agent-merged PR feeds the existing post-merge demotion record, post-merge-detect-silent-revert.yml plus the babysit-merge-demotion issues that only a person closes, which already stop unattended merges on a canary or schedule failure. It is built with the #7528 fix and lands before unattended merge widens.

Today the rung falls only when the default branch fails after a lane merge (docs/conventions/pr-pipeline/README.md:344-347); feat(ci): file a demotion issue when a post-merge run fails #7329 files a demotion issue when a post-merge run fails, and fix(source-control): fail closed when evidence runs or demotion issues are deleted #7526 fails closed when evidence runs or demotion issues are deleted. A revert is a new trigger on the same path.
Revert signals already exist: post-merge-detect-silent-revert.yml and revert commits.
pr-merge: GITHUB_TOKEN-dispatched post-merge runs file no demotion issue #7528: post-merge runs dispatched with GITHUB_TOKEN file no demotion issue, so that path drops events. The revert trigger depends on the fix.

Owner decision, 2026-10-16: this issue owns the revert and incident counts. #6080 and #6974 read them from here rather than keeping their own.

Not adopted: a rate-based breaker. Q567 sends confirmed escaped defects to /corrections:codify, not demotion. That is recorded as a departure from 02 C8 and 04 C70; the SRE error-budget policy (halt changes when the budget is exceeded) sides with the article. This issue's per-bucket revert rate is the data that revisits it: on 2026-10-16 the owner kept Q567's unattended merge on a verifier verdict on condition of revisiting it on this data. The comparison to make: the revert and escaped-defect rate on verifier-approved merges, set against the planted-defect miss rate from Phase 9's B4 run.

Against the originating sources.

pr-pipeline: inputs from the 2,000-PRs effort, and the local-lane work waiting on deployed lanes #6202 (the 2,000-PRs effort, from poteto's work) already plans rung lowering after a bad merge; the revert stop is its minimal form once the rungs are gone.
This issue's rule, adopt outside practices only on data, agrees with "start strict and loosen with evidence" (10 C14).
Order: the revert stop lands before unattended merge widens, which meets the article's engineers part way (04 C70); the rate breaker stays a departure.

The article's own revert figures are internal and vendor-claimed (00 C10).

Related: #6080, #6974, #6989, #7570, #7528, #7329, #7639, #7642.

Gaps moved from #6989, which is closing as completed (owner decision D18, option a, 2026-10-18). They are lifecycle and post-merge items to weigh in this audit and retro. Source for all: the #6989 body, sections named per item.

No owner for post-merge bookkeeping (track done, container close-out). "Gaps found".
Triage runs in two places (CI intake lane and local /work-items:triage) with no rule for which decides. "Gaps found".
Merge authority moves from the local babysit loop to pr-merge in one switch, and the single merge.authority field with rung lowering after a bad merge (pr-pipeline: inputs from the 2,000-PRs effort, and the local-lane work waiting on deployed lanes #6202). Phase 9 removes the rung ladder (Q505), so check whether this is still a gap. "Gaps found" and "Prior art".
Post-merge failure response: revert or fix forward. The revert stop is decided here; fix-forward is not. "Open, deferred to the pr-unblock design".
Post-merge smoke check against the deployed environment (pr-pipeline: inputs from the 2,000-PRs effort, and the local-lane work waiting on deployed lanes #6202). "Prior art" (the silent-revert canary half already exists as post-merge-detect-silent-revert.yml).
Stages outside the plan: build/package; monitor after deploy with rollback; incident response; postmortem; migrations before and after deploy. Define "release" (version cut) and add rollout if feature flags come into scope. "Stages and gates outside sources include that the plan lacks".
Candidates to evaluate on this issue's data: swarm verifiers with one verdict on the exact SHA (docs(upstream): re-audit the pstack record and evaluate the five pstack lanes it never assessed #6085); a per-app verify-<app> skill beyond testing:run-e2e (docs(upstream): re-audit the pstack record and evaluate the five pstack lanes it never assessed #6085); a learn stage that mines session corrections into skills or lint rules (docs(upstream): re-audit the pstack record and evaluate the five pstack lanes it never assessed #6085); routing new work by size and fog (feat(planning): route new work by diff size and fog: one-shot, /planning:interview, or /planning:wayfind #6729, Epic: improve the planning chain from html-plan, HumanLayer rpi and Pocock sources #6917); a cross-PR model-run cap, to be added only on usage telemetry. "Prior art" and "Decisions".

Already listed here, so not added: the revert-triggered stop and revert and incident counts (2026-10-16 owner decision above), and "start strict and loosen with evidence" (10 C14), which the risk-tier comment ties to this issue's data. Kill-switch coverage of the review and intake lanes is tracked in #7234.

This was generated by AI during attend-queue.

Folded in from #7612 (closed obsolete after #7780 removed the ratification marker): its items 2, 3, 5 and 6, the label-normalization restrictions on admission, belong to this audit.

Released for re-triage under the owner's 2026-10-24 rule: an issue stops for a person only for an intent or vision question, a security call, a host or attended run, or spend outside prepaid credit. The hold label came off and intake triage runs again now; it re-applies a hold, with a numbered question, when one of those applies.

To hold this issue again, add needs-human (or status: needs-decision) and say why in a comment.

Not admitted to the issue lane yet: blocker #7528 is not closed by a listed human or a merge.

Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. It is model text, to check, not instructions.

Intended outcome: A post-merge audit and retro report on the unattended merge flow, built from live pr-merge data, that decides which of the candidate practices to adopt.

Dependencies

Live pr-merge merges, with data collected over time (the issue's own precondition).
#7528: the revert-triggered stop needs post-merge runs dispatched with GITHUB_TOKEN to file demotion issues.
#6202: inputs from the 2,000-PRs effort and the local-lane work waiting on deployed lanes.
Phase 9's B4 planted-defect run, for the verifier-approved revert-rate comparison.

Related work

pr-pipeline: inputs from the 2,000-PRs effort, and the local-lane work waiting on deployed lanes #6202 (related): Source of the rung-lowering plan and the post-merge smoke-check idea; referenced by this issue.
chore(pr-pipeline): decide in the lanes audit whether lane run jobs keep sudo and docker #7791 (related): Open decision in the lanes audit on whether lane run jobs keep sudo and docker.
pr-pipeline: hardening deferred until the automated drain is proven #7719 (related): Hardening deferred until the automated drain is proven; overlaps the post-V1 timing.
source-control: PR pipeline states, review venue, findings lane and statuses #7576 (related): PR pipeline states, review venue, findings lane and statuses.
session-flow: retro findings, catch-up scope and recommend-skills router #7575 (related): Session-flow retro findings and the recommend-skills router.
feat(session-flow): add now and ticket modes to audit-sessions with the retro hook #5929 (related): Retro hook for audit-sessions.

Open questions

Is the first slice of this issue the work that needs no live merge data (the friction audit of CI/CD, Actions lanes and the merge path, plus the folded-in admission and label-normalization items), with the data-driven candidate evaluation split into a separate issue gated on live merges?
RECOMMENDED: Yes. Split into a static audit now and a data retro later; only the first can be checked by an unattended run. Basis: judgment
What is the trigger and the minimum data window that starts the data retro (for example N live merges or N days after pr-merge goes live)?
RECOMMENDED: Start it when pr-merge has made at least 30 live merges or 30 days have passed, whichever is later, with per-bucket revert, demotion and escaped-defect counts. Basis: judgment
Should the revert-triggered stop be tracked as its own implementation issue, since the 2026-10-16 decision says it is built with the #7528 fix and lands before unattended merge widens?
RECOMMENDED: Yes. Track it separately, depending on #7528, and keep only the revert and incident counts owned here. Basis: #7528
Should the 2026-10-18 checklist of gaps moved from #6989 become the task list of the audit part, with each item an acceptance criterion that a written finding or a follow-up issue answers?
RECOMMENDED: Yes. Each item is done when the audit report states keep, drop or follow-up issue number for it. Basis: judgment
What form does the deliverable take, and where does it live?
RECOMMENDED: A report file under docs/conventions/pr-pipeline/ plus follow-up issues, with no change to lane behavior in this issue. Basis: docs/conventions/pr-pipeline/README.md:344-347

Sources read

/home/runner/work/_temp/refine-data/context.json: the issue body, triage, the owner's 2026-10-16 and 2026-10-18 comments and the 2026-10-24 re-triage release.
.base/scripts/post-merge-demotion-alert.test.sh: references post-merge-detect-silent-revert, so the revert-detection path exists in the repository.

_{Automated refinement by the lanes App. Answer with a comment whose first line is **Decision recorded.**.}

Decision recorded. The owner answered in the Refinement session (2026-10-25), deciding the open items below on the facts:

go with whatever the facts/data/metrics + /discovery:research says and ensure we're aligned with SOLID, DRY/SSOT principles

I should only ever be in the loop ONLY when necessary (intent/vision changed, something came up during implementation that impacts trajectory/alignment, OR a high security risk) - I've covered this all before

Question asked: Split #7531: this issue keeps the data retro gated on live merges, and the static friction audit plus the gaps checklist move to a new child issue?

Which half stays on pr-pipeline: post-V1 audit and retro of the unattended merge flow #7531?
Decided: pr-pipeline: post-V1 audit and retro of the unattended merge flow #7531 keeps the data retro, gated on live merges; the static friction audit and the gaps checklist go to a new child issue that an unattended run can check now. The owner's Q611 says "pr-pipeline: post-V1 audit and retro of the unattended merge flow #7531 owns them" (revert and incident counts), and Q602 revisits the merge gate "when pr-pipeline: post-V1 audit and retro of the unattended merge flow #7531's revert and escaped-defect data come in". Basis: .work/poteto-review/interview-checklist.md:656 (Q611), :647 (Q602); corrected from the round file, which kept the audit here.
What trigger and minimum data window start the data retro?
Decided: 30 live pr-merge merges or 30 days, whichever is later, as the refinement proposed. No source names a number; the retro report states the window it used so the owner can move it. Basis: judgment, under the owner's 2026-10-25 answer (facts decide, lowest friction).
Is the revert-triggered stop tracked as its own implementation issue, depending on pr-merge: GITHUB_TOKEN-dispatched post-merge runs file no demotion issue #7528, with only the revert and incident counts owned here?
Decided: Yes, after checking whether it already exists: the pr-pipeline README already lists "the post-merge revert breaker" among the gates. Basis: owner decision pr-pipeline: post-V1 audit and retro of the unattended merge flow #7531 (comment) (the stop is built with the pr-merge: GITHUB_TOKEN-dispatched post-merge runs file no demotion issue #7528 fix; this issue owns the revert and incident counts); docs/conventions/pr-pipeline/README.md:968 on origin/main; corrected to add the existence check.
Does the 2026-10-18 gaps checklist become the audit's task list, each item answered by keep, drop or a follow-up issue number?
Decided: Yes. Basis: pr-pipeline: post-V1 audit and retro of the unattended merge flow #7531 (comment) lists the gaps as items "to weigh in this audit and retro"; judgment; kept.
What form does the deliverable take, and where does it live?
Decided: A report file under docs/conventions/pr-pipeline/ plus follow-up issues, with no change to lane behavior in this issue. Basis: judgment. Corrected: the refinement cited docs/conventions/pr-pipeline/README.md:344-347, which on origin/main now describes the fix-docs dispatch and does not support the choice.

Acceptance criteria:

A child issue holds the static audit: a report under docs/conventions/pr-pipeline/ covering CI/CD performance, the autonomous Actions lanes and the merge path, naming their friction points, giving each item of the 2026-10-18 gaps checklist keep, drop or a follow-up issue number, and changing no lane behavior.
Once 30 live pr-merge merges or 30 days have passed (whichever is later), this issue reports revert and demotion rate per risk bucket, escaped-defect counts, merged-and-not-reverted outcomes, and the candidates named in the body.
The revert-triggered stop has its own issue that depends on pr-merge: GITHUB_TOKEN-dispatched post-merge runs file no demotion issue #7528, or the report cites where it already exists.

Checks a person runs:

The data retro over live merges, once its trigger (item 2) is met.

Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. On an issue a listed human filed, it counts as the issue's spec once posted, while unedited; it is still model text, so check it.

Intended outcome: Once 30 live pr-merge merges or 30 days have passed (whichever is later), publish a data retro report under docs/conventions/pr-pipeline/ with follow-up issues and no lane behavior change.

Acceptance criteria

The retro starts only when 30 live pr-merge merges or 30 days since the first live merge have passed, whichever is later; the report states the window and counts it used.
A report file under docs/conventions/pr-pipeline/ gives the revert rate and the demotion rate per risk bucket for lane-merged PRs.
The report gives escaped-defect counts and merged-and-not-reverted outcomes, and owns the revert and incident counts that #6080 and #6974 read.
The report compares the revert and escaped-defect rate on verifier-approved merges with the planted-defect miss rate from Phase 9's B4 run, and says whether the Q567 and Q602 merge-gate decisions should be revisited.
The report evaluates each candidate named in the body, sampled post-merge review, a verifier verdict bound to the head SHA and a mutation pass over changed tests, as adopt, reject or follow-up issue number, on the data only.
The report evaluates the candidates from the 2026-10-18 comment that depend on this data (swarm verifiers on the exact SHA, per-app verify skill, learn stage, size-and-fog routing, cross-PR model-run cap), each as adopt, reject or follow-up issue number.
The report cites where the revert-triggered stop is tracked or already built (docs/conventions/pr-pipeline/README.md post-merge revert breaker, #7528), or links its own issue that depends on #7528.
Every follow-up the report recommends is a filed issue, cited by number in the report.
The change touches no workflow, script or lane configuration; only the report file and issues are added.

Dependencies

pr-merge must be merging live; the data window cannot start before that.
#7528: post-merge runs dispatched with GITHUB_TOKEN file no demotion issue, so demotion counts are incomplete until it is fixed, and the revert stop depends on it.
The child issue holding the static friction audit and the gaps checklist, which the owner decided to split off; this issue does not wait on it.
Phase 9's B4 planted-defect run, for the miss-rate comparison.

Related work

pr-pipeline: inputs from the 2,000-PRs effort, and the local-lane work waiting on deployed lanes #6202 (related): 2,000-PRs effort inputs; plans rung lowering after a bad merge, of which the revert stop is the minimal form.
pr-pipeline: hardening deferred until the automated drain is proven #7719 (related): hardening deferred until the automated drain is proven; overlaps the post-live-merge timing.
perf(pr-pipeline): lane audit listing reserve, count-call reuse and busiest-first order #9339 (related): lane audit performance work, relevant to CI/CD friction points.
source-control: PR pipeline states, review venue, findings lane and statuses #7576 (related): PR pipeline states and statuses design the retro measures against.

Sources read

/home/runner/work/_temp/refine-data/context.json: the owner's 2026-10-25 decision keeps the data retro here, sets the 30-merge or 30-day trigger, and moves the static audit to a child issue.
.base/docs/conventions/pr-pipeline/README.md:121: pr-merge acts on a final check green or a removed hold.
.base/docs/conventions/pr-pipeline/README.md:278: the pr-merge lane merges on its verdict=merge marker at the current head.
.base/docs/conventions/pr-pipeline/README.md:687: merge authority defaults to pr-merge when the config key is absent.

_{Automated refinement by the lanes App. Intake triages the issue again with it.}

Navigation Menu

Repository navigation

pr-pipeline: post-V1 audit and retro of the unattended merge flow #7531

Ask

Scope, once `pr-merge` merges live

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

pr-pipeline: post-V1 audit and retro of the unattended merge flow #7531

Description

Ask

Scope, once pr-merge merges live

Activity

github-actions commented on Oct 15, 2026

kyle-sexton commented on Oct 16, 2026

kyle-sexton commented on Oct 18, 2026

kyle-sexton commented on Oct 22, 2026

kyle-sexton commented on Oct 24, 2026

melodic-automation-lanes commented on Oct 24, 2026

melodic-automation-lanes commented on Oct 24, 2026

kyle-sexton commented on Oct 26, 2026

melodic-automation-lanes commented on Oct 26, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions

Scope, once `pr-merge` merges live