diff --git a/.changes/pr-pipeline/fix-pr-unblock-route-not-park-5c1e7a20.md b/.changes/pr-pipeline/fix-pr-unblock-route-not-park-5c1e7a20.md new file mode 100644 index 0000000000..1898b1f3b9 --- /dev/null +++ b/.changes/pr-pipeline/fix-pr-unblock-route-not-park-5c1e7a20.md @@ -0,0 +1,22 @@ +--- +bump: minor +--- + +### Changed + +- A `pr-unblock` run that stops now ends on the route its reason has in [`escalation-routes.json`](https://github.com/melodic-software/claude-code-plugins/blob/main/.github/scripts/pr-unblock/escalation-routes.json), the one routing table that `report-escalation.sh`, `gate-source.mjs` and `clear-escalations.mjs` read. Only a reason a person must settle keeps the `needs-human` label: a path the lane token may not push or a supply-chain refusal, an intent, design or security question, a person's own thread or hold, a finding no adjudicator may judge (security-review or CRITICAL), or any `review-dependency` stop, since a failing status is per head and the head-bound merge marker comes from the lane's own re-run, so only the label holds it for a person. Every other reason (for example `ci-fix-escalated`, `feedback-not-landed`, the resolve-conflicts, address-feedback and fix-ci round and repeat caps) ends with no label, a failing commit status on the head the activity gated (`pr-unblock/conflict` for `resolve-conflicts`, `pr-unblock/` otherwise, posted by the `report-stop-status` job that replaces `report-conflict-status`), and one lanes App comment per activity that the next stop edits in place. A new head carries no status. ci-status, the one required check, fails while the newest `pr-unblock/*` status on the head by `github-actions[bot]` is `failure` or `error`, naming each such context, on pull request runs and in the merge queue (the queued pull request's head); a later `success` on the same head supersedes it. +- An `UNCERTAIN` or `VALID-defer` reply on a non-CRITICAL `claude-review` finding now goes to the adjudicator as an `INCORRECT` one does, instead of to a person; a security-review, CRITICAL or unknown-lane finding still holds for a person (`feedback-uncertain`). A `lane` verdict (the finding does not hold) resolves only an `INCORRECT` thread, where the lane disputed the finding with evidence: on an `UNCERTAIN` or `VALID-defer` thread it posts nothing, the thread stays open (the ruleset's thread resolution still holds the merge) and the round ends `feedback-unadjudicated`. A `reviewer` verdict hands the finding to the next round to fix, and a `human` verdict holds for a person. A pr-verify note left open ends `feedback-verifier-open`, and a finding that stood at adjudication and was left open again ends `feedback-stood-open`; both route to a failing status. A round's reason held for a person now outranks any reason routed to a failing status. +- `clear-escalations` takes off a `needs-human` an older run set for a reason now routed to a failing status: a `ci-fix-*` reason once a lane check of a `pr-require-checks.yml` run on the current head reads failed. Any other fail-routed reason keeps the label for a person: a new head carries no `pr-unblock/*` status that shows the cause gone or fails that head in the label's place. +- A pr-unblock stop routed to a failing status now skips `lane-stopped` (a new `skip-reason`) instead of `awaiting-human`. On a select job's own check (a cap, or a conflict no model can resolve) it ends the head, so the dispatchers stop re-running a deterministic stop. On the push, reply or verdict job's check, the dispatcher retries the same head under the cap it already keeps: + - `limit-stops-per-head` for resolve-conflicts + - `fix-ci-restarts` and the per-head cap for fix-ci + - `SWEEP_HEAD_CAP` for address-feedback + + The fix-ci dispatcher reads a fix-ci run and its in-run rung-up, which both post a `pr-unblock / fix-ci` check, as one attempt: a stop in either one's push job makes the attempt a failed one, and the pair counts once toward the caps. + + The fix-ci and address-feedback retries run one model rung up, read in the run from the head's own stop record. A retry that acts and does not stop posts success over the failing status; one that ends on a skip (a reply job with no model turn) leaves it. fix-ci's model escalations `protected-path` and `weaken-required` stay with a person, as `ci-fix-protected-path` and the new `ci-fix-weaken-required`. +- `fix-ci-context` and the fix-ci dispatcher read every check run on the head (`filter=all`) instead of the latest per name, so a contract-only run's later skip of a lane no longer hides that lane's failure, and a head with a hidden failure is not read as resolved. +- An address-feedback round that keeps no thread while a review thread is still unresolved (a reviewer the lane does not act on, such as Codex, a review of an older head, a thread it already answered) now ends `feedback-untaken-threads`, routed to a failing status, instead of a silent `not-applicable` skip. The dispatcher now dispatches an ungated head once for any unresolved thread, so that stop is reached; the select job's lane-stopped check ends the head, and the next tick does not dispatch it again. A reply job that fails with no escalation reason (a lane token it could not get) ends `feedback-publish-failed`, also routed to a failing status, instead of posting no stop. +- The monitoring lane gains signal 10, `stalled lane stop`. Each open pull request whose pr-unblock stop has had no new attempt on its head for over 2 hours is listed in one rolling issue. A skip is not an attempt, but a `not-applicable` skip after the stop ends it: for address-feedback on a non-draft pull request, and for fix-ci once a whole listing of the head's checks is settled with none failing. A stop includes a `needs-human` that `clear-escalations` took off for a reason now routed to a failing status. It reads the 40 most recently updated open pull requests and their newest comments in one GraphQL query, reading up to 200 older comments only for a pull request whose newest name no stop on its head; past 40 pull requests, or past those comments, it names what it left unread in the run's gaps. +- The merge path floor's `enforcement` class (`babysit_risk.REVIEW_FLOOR` and `merge-holds.mjs` `REVIEW_FLOOR`, from `FLOOR_CLASSES` and `HOLD_PATH_CLASSES`, and the review lanes' and ci-status's hold lists) also covers `.github/scripts/pr-unblock/escalation-routes.mjs`, which `clear-escalations.mjs` and `gate-source.mjs` now import, and `escalation-routes.json`, the routing table it reads, and the scripts that act on a route: `report-escalation.sh`, which labels a held reason, `adjudicate-feedback.mjs` and `publish-feedback.mjs`, which decide and resolve review threads, and `.github/scripts/apply-patch/apply-patch.mjs`, which holds fix-ci escalations, with every module they import (`request-budget.mjs`, `dependency-classify.mjs`, and `predicates.mjs`, `protected-paths.mjs` and `retry-fetch.mjs` in `apply-patch/`). So a change to any of them, such as one that moves a reason from `hold` to `fail`, stops a held reason from labeling or lets a model verdict resolve a thread a person must, does not merge until a clean security review binds to its head (`review-path:`); it does not hold for a person, which only the `trust` and `gate` classes (`HIGH_FLOOR`, `HOLD_PATH_FLOOR`) do. `test_merge_holds_sync.py` now also follows a module-relative file a floored module reads (`new URL("./…", import.meta.url)`), so the next such data file fails there until it is on the floor. +- A select job that finds a stop gone on its head clears that activity's failing status (`stop-resolved`): `address-feedback` once no hold and no unresolved review thread is left, `fix-ci` once a whole listing of the head's checks, every one completed, has none failing. fix-ci now reads the head's checks before its round cap, so a head re-run green after the cap clears too. A select that fails to read clears nothing. diff --git a/.changes/pr-pipeline/fix-verifier-note-roundcap-not-person-3b9d41e7.md b/.changes/pr-pipeline/fix-verifier-note-roundcap-not-person-3b9d41e7.md new file mode 100644 index 0000000000..c1ac3bd809 --- /dev/null +++ b/.changes/pr-pipeline/fix-verifier-note-roundcap-not-person-3b9d41e7.md @@ -0,0 +1,7 @@ +--- +bump: patch +--- + +### Fixed + +- A pr-verify note left open past the address-feedback round cap no longer hands the pull request to a person as `feedback-round-cap` with `needs-human`. It gets the fresh fixer's round as a code-review finding does, and once that round has run the head fails: `feedback-verifier-open` when only pr-verify notes are open, `feedback-not-converging` when a code-review finding is open beside them. A pr-verify note a trusted person directed, a security-review finding, a `claude[bot]` finding and a person's `needs-human` hold still escalate `feedback-round-cap`. diff --git a/.changes/source-control/fix-pr-unblock-route-not-park-f8205d45.md b/.changes/source-control/fix-pr-unblock-route-not-park-f8205d45.md new file mode 100644 index 0000000000..ab942c75d2 --- /dev/null +++ b/.changes/source-control/fix-pr-unblock-route-not-park-f8205d45.md @@ -0,0 +1,7 @@ +--- +bump: patch +--- + +### Changed + +- `babysit_risk.REVIEW_FLOOR` (the `enforcement` class of `FLOOR_CLASSES`, which holds a change until a clean security review binds to its head, not for a person) also covers `.github/scripts/pr-unblock/escalation-routes.mjs`, which `clear-escalations.mjs` and `gate-source.mjs` import, and `escalation-routes.json`, the routing table it reads, so the floor still holds every module its scripts load, and the scripts that act on a pr-unblock stop's route (`report-escalation.sh`, `adjudicate-feedback.mjs`, `publish-feedback.mjs`, `apply-patch.mjs`) with every module they import. diff --git a/.github/actions/collect-base-activity/collect-base-activity.test.mjs b/.github/actions/collect-base-activity/collect-base-activity.test.mjs index c49af0201b..bca30a08fb 100644 --- a/.github/actions/collect-base-activity/collect-base-activity.test.mjs +++ b/.github/actions/collect-base-activity/collect-base-activity.test.mjs @@ -106,7 +106,7 @@ before(() => { writeFileSync(resolved, JSON.stringify({ selected: { args: "ci" } })); // The issue lane's protected-path rules come from the base's apply-patch // and patch-rules.json, as committed. - for (const file of [".github/scripts/apply-patch", ".github/scripts/pr-merge/merge-holds.mjs", ".github/scripts/pr-unblock/dependency-classify.mjs", ".github/scripts/pr-unblock/thread-scope.mjs", ".github/actions/resolve-config/patch-rules.json"]) { + for (const file of [".github/scripts/apply-patch", ".github/scripts/pr-merge/merge-holds.mjs", ".github/scripts/pr-unblock/dependency-classify.mjs", ".github/scripts/pr-unblock/thread-scope.mjs", ".github/scripts/pr-unblock/escalation-routes.mjs", ".github/scripts/pr-unblock/escalation-routes.json", ".github/actions/resolve-config/patch-rules.json"]) { cpSync(path.join(REPO_ROOT, file), path.join(base, file), { recursive: true }); } }); diff --git a/.github/actions/resolve-config/callers.test.mjs b/.github/actions/resolve-config/callers.test.mjs index f9b7174a0d..dcfe0a3a86 100644 --- a/.github/actions/resolve-config/callers.test.mjs +++ b/.github/actions/resolve-config/callers.test.mjs @@ -23,6 +23,7 @@ import { parse } from "yaml"; import { loadRules, rowFor, select as selectPatch } from "../../scripts/apply-patch/apply-patch.mjs"; import { DEFAULTS as CI_FIX_DEFAULTS, skipReason } from "../../scripts/pr-unblock/fix-ci-dispatch.mjs"; import { VERDICTS } from "../../scripts/pr-unblock/review-dependency.mjs"; +import { stopSkip } from "../../scripts/pr-unblock/escalation-routes.mjs"; import { main as checkTrustedTrigger } from "../check-trusted-trigger/check-trusted-trigger.mjs"; import { generate as generateCarrierKey } from "../open-lane-output/keygen.mjs"; import { decide } from "../report-check-run/decide.mjs"; @@ -499,30 +500,116 @@ test("intake-triage.yml: the model may apply duplicate, which admission refuses, assert.match(text, /apply no `status: needs-decision`/); }); -test("pr-unblock.yml: report-conflict-status runs only for an unresolved-conflict escalation, from the select, push or scan-refusal job, with a gated head", () => { +test("pr-unblock.yml: report-stop-status runs when report-escalation does, or after an attempt that did not stop, with a gated head and never on a cancelled run", () => { const head = "c".repeat(40); const { jobs } = workflow("pr-unblock.yml"); - const job = jobs["report-conflict-status"]; - const context = (select, apply, headSha, refused = "") => ({ - "needs.resolve-conflicts-select.outputs.escalation-reason": select, - "needs.apply-conflict-patch.outputs.escalation-reason": apply, - "needs.conflict-scan-refusal.outputs.escalation-reason": refused, - "needs.resolve-conflicts-select.outputs.head-sha": headSha, + const job = jobs["report-stop-status"]; + const escalation = jobs["report-escalation"]; + assert.deepEqual(job.needs, escalation.needs); + const SOURCES = [ + "resolve-conflicts-select", + "apply-conflict-patch", + "conflict-scan-refusal", + "address-feedback-select", + "publish-feedback", + "review-dependency-select", + "apply-dependency-verdict", + "fix-ci-context", + "apply-patch", + ]; + const HEADS = ["resolve-conflicts-select", "address-feedback-select", "review-dependency-select", "fix-ci-context"]; + const EMITTERS = ["apply-conflict-patch", "publish-feedback", "apply-dependency-verdict", "apply-patch"]; + const RESOLVERS = ["address-feedback-select", "fix-ci-context"]; + const context = ({ from = "", reason = "", headFrom = "resolve-conflicts-select", sha = head, rungUp = "", rungUpResult = "skipped", rungUpReason = "", rungUpSkip = "", landed = "", skip = "", resolved = "", resolvedResult = "success" } = {}) => ({ + ...Object.fromEntries(SOURCES.map((id) => [`needs.${id}.outputs.escalation-reason`, id === from ? reason : ""])), + ...Object.fromEntries(RESOLVERS.map((id) => [`needs.${id}.result`, id === resolved ? resolvedResult : "skipped"])), + ...Object.fromEntries(RESOLVERS.map((id) => [`needs.${id}.outputs.stop-resolved`, id === resolved ? "true" : ""])), + ...Object.fromEntries(EMITTERS.map((id) => [`needs.${id}.result`, id === landed ? "success" : "skipped"])), + ...Object.fromEntries([...EMITTERS, "apply-patch-rung-up"].map((id) => [`needs.${id}.outputs.skip-reason`, id === landed ? skip : ""])), + ...Object.fromEntries(HEADS.map((id) => [`needs.${id}.outputs.head-sha`, id === headFrom ? sha : ""])), + "needs.apply-patch.outputs.rung-up": rungUp, + "needs.apply-patch-rung-up.result": rungUpResult, + "needs.apply-patch-rung-up.outputs.escalation-reason": rungUpReason, + "needs.apply-patch-rung-up.outputs.patch-skip-reason": rungUpSkip, }); - assert.deepEqual(job.needs, ["resolve-conflicts-select", "apply-conflict-patch", "conflict-scan-refusal"]); - assert.equal(evaluate(job.if, context("unresolved-conflict", "", head)), true, "before any model turn"); - assert.equal(evaluate(job.if, context("", "unresolved-conflict", head)), true, "a conflict the patch left"); - assert.equal(evaluate(job.if, context("", "", head, "unresolved-conflict")), true, "a scan that refused the model's output"); - for (const reason of ["repeat-cap", "protected-path-conflict", "workflow-staged", "conflict-patch-invalid", ""]) { - assert.equal(evaluate(job.if, context(reason, "", head)), false, reason); - assert.equal(evaluate(job.if, context("", reason, head)), false, reason); - } - assert.equal(evaluate(job.if, context("", "", "", "unresolved-conflict")), false, "no gated head to post on"); - assert.equal(evaluate(job.if, context("unresolved-conflict", "", "")), false, "no gated head to post on"); - // The head is the select job's, never a model call's. - const post = job.steps.find((step) => step.name === "Post the conflict status"); - assert.equal(post.env.HEAD_SHA, "${{ needs.resolve-conflicts-select.outputs.head-sha }}"); - assert.equal(post.env.ESCALATION_REASON, "unresolved-conflict"); + for (const from of SOURCES) { + assert.equal(evaluate(job.if, context({ from, reason: "x" })), true, from); + } + for (const headFrom of HEADS) { + assert.equal(evaluate(job.if, context({ from: "publish-feedback", reason: "x", headFrom })), true, headFrom); + } + assert.equal(evaluate(job.if, context()), false, "no reason"); + assert.equal(evaluate(job.if, context({ from: "apply-patch", reason: "x", sha: "" })), false, "no gated head to post on"); + assert.match(job.if, /^\$\{\{ !cancelled\(\) && /, "a cancelled run posts nothing"); + assert.doesNotMatch(job.if, /always\(\)/); + // The rung-up retry: a stop unless its patch applied cleanly. + assert.equal(evaluate(job.if, context({ headFrom: "fix-ci-context", rungUp: "true", rungUpResult: "failure" })), true); + assert.equal(evaluate(job.if, context({ headFrom: "fix-ci-context", rungUp: "true", rungUpResult: "success", rungUpReason: "ci-fix-invalid" })), true); + // An attempt that did not stop runs it too, for report-escalation.sh clear. + assert.equal(evaluate(job.if, context({ headFrom: "fix-ci-context", rungUp: "true", rungUpResult: "success" })), true, "the retry's patch applied: clear"); + for (const landed of EMITTERS) { + assert.equal(evaluate(job.if, context({ landed })), true, `${landed} ended with no stop: clear`); + // Review finding on #10252: a retry that succeeded on a skip (a reply + // job with no model turn) landed nothing, so it must not clear the + // failure an earlier stop left on the head. + for (const skip of ["no-model-turn", "superseded-sha", "rung-up"]) { + assert.equal(evaluate(job.if, context({ landed, skip })), false, `${landed} skipped ${skip}: no clear`); + } + } + assert.equal(evaluate(job.if, context({ landed: "apply-patch-rung-up", rungUpResult: "success", skip: "no-model-turn" })), false, "the rung-up push job's skip: no clear"); + // Review finding on #10252: a select job that finds its stop gone on the + // head (every thread resolved by hand, or no failing check) clears the + // status with no push, reply or verdict job; a failed select clears nothing. + const postStep = job.steps.find((step) => step.name === "Post the stop status"); + for (const resolved of RESOLVERS) { + const headFrom = resolved; + assert.equal(evaluate(job.if, context({ resolved, headFrom })), true, `${resolved} resolved: clear`); + assert.equal(evaluate(postStep.env.ESCALATION_REASON, context({ resolved, headFrom })), "", `${resolved} resolved: clear mode`); + assert.equal(evaluate(job.if, context({ resolved, headFrom, resolvedResult: "failure" })), false, `${resolved} failed: no clear`); + assert.equal(evaluate(job.if, context({ headFrom })), false, `${resolved} not resolved: no clear`); + } + assert.match(postStep.run, /echo status \|\| echo clear/); + // The write runner hands its skip up: the reply script's no-model-turn, or a gate stop. + const write = workflow("pr-run-activity-write.yml"); + assert.equal(write.on.workflow_call.outputs["skip-reason"].value, "${{ jobs.run.outputs.skip-reason }}"); + const steps = (extra = {}) => ({ + "steps.kill.outputs.proceed": "true", + "steps.kill.outputs.reason": "", + "steps.gate.outputs.proceed": "true", + "steps.gate.outputs.reason": "", + ...Object.fromEntries(["config", "head-recheck", "lane-token", "prepare", "promote", "act-script", "patch-select"].map((id) => [`steps.${id}.outputs.skip-reason`, ""])), + ...extra, + }); + const skipOf = (extra) => evaluate(write.jobs.run.outputs["skip-reason"], steps(extra)); + assert.equal(skipOf(), ""); + assert.equal(skipOf({ "steps.act-script.outputs.skip-reason": "no-model-turn" }), "no-model-turn"); + assert.equal(skipOf({ "steps.gate.outputs.proceed": "false", "steps.gate.outputs.reason": "untrusted-trigger" }), "untrusted-trigger"); + // #10092's 20:57 run: the reply job failed at its lane token with no + // escalation reason, so no stop was posted. The caller derives + // feedback-publish-failed from the failed result, and report-stop-status + // and report-escalation then run on it; a recorded reason still wins. + const failed = (extra = {}) => ({ ...context({ headFrom: "address-feedback-select", ...extra }), "needs.publish-feedback.result": "failure" }); + assert.equal(evaluate(job.if, failed()), true, "a failed reply job posts a stop"); + assert.equal(evaluate(job.steps.find((step) => step.name === "Post the stop status").env.ESCALATION_REASON, failed()), "feedback-publish-failed"); + assert.equal( + evaluate(job.steps.find((step) => step.name === "Post the stop status").env.ESCALATION_REASON, failed({ from: "publish-feedback", reason: "feedback-partial" })), + "feedback-partial", + ); + assert.equal(evaluate(escalation.if, failed()), true); + assert.equal(evaluate(escalation.with["escalation-reason"], failed()), "feedback-publish-failed"); + assert.equal(stopSkip("feedback-publish-failed"), "lane-stopped", "routed to a failing status"); + // The head is a select job's, never a model call's; the reason is the one + // report-escalation reads. + const post = job.steps.find((step) => step.name === "Post the stop status"); + assert.equal(post.env.HEAD_SHA.trim(), HEADS.map((id) => `needs.${id}.outputs.head-sha`).join(" || ").replace(/^/, "${{ ").replace(/$/, " }}")); + // report-escalation's reason, except that a rung-up retry whose patch + // applied is no stop. + const reason = (extra) => evaluate(post.env.ESCALATION_REASON, context(extra)); + assert.equal(reason({ from: "publish-feedback", reason: "feedback-not-landed" }), "feedback-not-landed"); + assert.equal(reason({ rungUp: "true", rungUpResult: "failure" }), "ci-fix-escalated"); + assert.equal(reason({ rungUp: "true", rungUpResult: "success" }), ""); + assert.equal(post.env.ACTIVITY, "${{ inputs.activity }}"); + assert.match(post.run, /report-escalation\.sh "\$\(\[ -n "\$ESCALATION_REASON" \] && echo status \|\| echo clear\)"$/); assert.deepEqual(job.permissions, { contents: "read", statuses: "write" }); assert.equal(job.uses, undefined, "a script job, never the write runner"); }); @@ -610,6 +697,7 @@ const rungUpContext = ({ rungUp = "true", result = "success", escalation = "", s "needs.apply-patch-rung-up.result": result, "needs.apply-patch-rung-up.outputs.escalation-reason": escalation, "needs.apply-patch-rung-up.outputs.patch-skip-reason": skip, + "needs.publish-feedback.result": "skipped", }); test("pr-unblock.yml: a rung-up stop runs fix-ci and apply-patch again at model-rung 1 in the same run", () => { @@ -624,9 +712,20 @@ test("pr-unblock.yml: a rung-up stop runs fix-ci and apply-patch again at model- assert.equal(up.with["model-rung"], "1"); assert.equal(upSubscription.with["model-rung"], "1"); assert.equal(apply.with["patch-model-rung"], "1"); - // The first pass runs at rung 0: no rung input, so a dispatch cannot skip it. - assert.equal(jobs["fix-ci"].with["model-rung"], undefined); - assert.equal(jobs["apply-patch"].with["patch-model-rung"], undefined); + // The first pass runs at the rung fix-ci-context read from the head's own + // trusted stop record (escalation-routes.mjs retryRung): no dispatch input + // sets it, so a dispatch cannot skip rung 0 or climb. + assert.equal(jobs["fix-ci"].with["model-rung"], "${{ needs.fix-ci-context.outputs.model-rung }}"); + assert.equal(jobs["fix-ci-subscription"].with["model-rung"], "${{ needs.fix-ci-context.outputs.model-rung }}"); + assert.equal(jobs["apply-patch"].with["patch-model-rung"], "${{ needs.fix-ci-context.outputs.model-rung }}"); + for (const id of ["address-feedback", "address-feedback-subscription"]) { + assert.equal(jobs[id].with["model-rung"], "${{ needs.address-feedback-select.outputs.model-rung }}", id); + } + // review-dependency's profile names no rung above it, and resolve-conflicts + // retries as a fresh session at its own rung. + for (const id of ["review-dependency", "review-dependency-subscription", "resolve-conflicts", "resolve-conflicts-subscription"]) { + assert.equal(jobs[id].with["model-rung"], undefined, id); + } assert.equal(workflow("pr-unblock.yml").on.workflow_dispatch.inputs["model-rung"], undefined); }); @@ -685,7 +784,7 @@ test("pr-unblock.yml: fix-ci is reported neutral when fix-ci-context skips, and assert.equal(evaluate(fixCi.if, context("true", "", head)), true); assert.equal(evaluate(notRun.if, context("true", "", head)), false); assert.equal(evaluate(notRun.if, context("false", "", head)), true); - assert.equal(evaluate(notRun.if, context("false", "ci-fix-round-cap", head)), false, "report-escalation owns it"); + assert.equal(evaluate(notRun.if, context("false", "ci-fix-round-cap", head)), true, "a stop posts its own skip, so the head's fix-ci runs record it"); assert.equal(evaluate(notRun.if, context("", "", "")), false, "a kill-switch stop names no head"); assert.equal(evaluate(notRun.if, context("", "", "", "resolve-conflicts")), false); @@ -698,6 +797,9 @@ test("pr-unblock.yml: fix-ci is reported neutral when fix-ci-context skips, and // A human-gated hold fix-ci-context read posts awaiting-human instead. const held = await postNotRunCheck(notRun, context("false", "", head, "fix-ci", "awaiting-human")); assert.equal(held[0].output.title, "Skipped: awaiting-human"); + // The round cap, routed to a failing status: lane-stopped ends the head. + const capped = await postNotRunCheck(notRun, context("false", "ci-fix-round-cap", head, "fix-ci", "lane-stopped")); + assert.equal(capped[0].output.title, "Skipped: lane-stopped"); // fix-ci-dispatch.mjs dispatches a head with no fix-ci check run; the // posted one, as the listing returns it, means the head was handled. @@ -2721,12 +2823,14 @@ test("leak scan: a dependency verdict is scanned and hashed, and only a known ve // address-feedback: whether the scanned replies hold an INCORRECT one, the // only reply adjudicate-feedback has work for. A line that is not JSON reads // as disputed, so the adjudicator's own checks decide it. -test("leak scan: replies-disputed is true only for scanned replies holding an INCORRECT reply or an unreadable line", async (t) => { +test("leak scan: replies-disputed is true only for scanned replies holding a reply that leaves its thread open or an unreadable line", async (t) => { const plantReplies = (text) => ({ temp }) => writeFileSync(join(temp, "skill-reply", "replies.jsonl"), text); const row = (disposition) => JSON.stringify({ thread: "PRRT_a", disposition, evidence: [], fix_sha: "", note: "n" }); for (const [text, want] of [ [`${row("VALID")}\n${row("INCORRECT")}\n`, "true"], - [`${row("VALID")}\n${row("UNCERTAIN")}\n`, ""], + [`${row("VALID")}\n${row("UNCERTAIN")}\n`, "true"], + [`${row("VALID-defer")}\n`, "true"], + [`${row("VALID")}\n`, ""], [`${row("VALID")}\n{not json\n`, "true"], [`${JSON.stringify({ disposition: "incorrect" })}\n`, ""], ]) { @@ -5364,7 +5468,9 @@ test("pr-unblock.yml: resolve-conflicts is reported neutral when the select job const head = "d".repeat(40); const { jobs } = workflow("pr-unblock.yml"); const notRun = jobs["resolve-conflicts-not-run"]; - const context = (proceed, headSha, { skip = "", state = "", activity = "resolve-conflicts" } = {}) => ({ + // The "Read the stop's skip reason" step runs escalation-routes.mjs skip. + const context = (proceed, headSha, { skip = "", state = "", activity = "resolve-conflicts", escalation = "" } = {}) => ({ + "steps.stop.outputs.skip": stopSkip(escalation), "inputs.activity": activity, "needs.resolve-conflicts-select.outputs.proceed": proceed, "needs.resolve-conflicts-select.outputs.head-sha": headSha, @@ -5381,7 +5487,9 @@ test("pr-unblock.yml: resolve-conflicts is reported neutral when the select job [{ skip: "not-applicable" }, "Skip reason: not-applicable"], [{ skip: "awaiting-human" }, "Skip reason: awaiting-human"], [{ state: "clean" }, "Skip reason: clean-merge"], - [{ state: "escalate" }, "Skip reason: awaiting-human"], + [{ state: "escalate", escalation: "protected-path-conflict" }, "Skip reason: awaiting-human"], + [{ state: "escalate", escalation: "repeat-cap" }, "Skip reason: lane-stopped"], + [{ state: "escalate", escalation: "unresolved-conflict" }, "Skip reason: lane-stopped"], ]) { const calls = await postNotRunChecks(notRun, context("false", head, given)); assert.deepEqual( @@ -6106,7 +6214,7 @@ test("pr-unblock.yml: a refused fix-ci collection runs apply-patch, whose select // select, with the empty patch-state a refused collection passes, escalates. const row = rowFor(loadRules(fileURLToPath(new URL("../../../", import.meta.url))), { prepare: "ci-context" }); const decision = selectPatch({ row, patchState: "", patchSha256: "", patchHeadSha: TRUSTED, headSha: TRUSTED, patchDir: "/nonexistent", reportDir: "/nonexistent" }); - assert.deepEqual(decision, { proceed: false, skipReason: "awaiting-human", escalationReason: "ci-fix-invalid" }); + assert.deepEqual(decision, { proceed: false, skipReason: "lane-stopped", escalationReason: "ci-fix-invalid" }); // report-escalation runs on that reason, and on nothing else here. const escalation = jobs["report-escalation"]; const quiet = Object.fromEntries([...escalation.if.matchAll(/needs\.[\w-]+\.(?:outputs\.[\w-]+|result)/g)].map(([ref]) => [ref, ""])); diff --git a/.github/actions/resolve-config/feedback-dispatch.test.mjs b/.github/actions/resolve-config/feedback-dispatch.test.mjs index ca075d3144..b3ff92b955 100644 --- a/.github/actions/resolve-config/feedback-dispatch.test.mjs +++ b/.github/actions/resolve-config/feedback-dispatch.test.mjs @@ -127,7 +127,8 @@ function fakes() { body: ``, })), listReviewThreads: async () => [ - { isResolved: false, comments: { nodes: [{ author: CLAUDE }], pageInfo: { hasNextPage: false } } }, + // A reviewer-bot finding on a review of the marked head. + { isResolved: false, comments: { nodes: [{ author: CLAUDE, pullRequestReview: { author: CLAUDE, commit: { oid: HEAD } } }], pageInfo: { hasNextPage: false } } }, ], listCheckRuns: async () => [], listWorkflowRuns: async () => [], diff --git a/.github/actions/resolve-config/fix-ci-context.test.mjs b/.github/actions/resolve-config/fix-ci-context.test.mjs index ec5112fbe4..307e4a07ff 100644 --- a/.github/actions/resolve-config/fix-ci-context.test.mjs +++ b/.github/actions/resolve-config/fix-ci-context.test.mjs @@ -21,6 +21,9 @@ const fixCommit = { const api = { getPull: async () => ({ state: "open", head: { sha: "a".repeat(40) } }), listPullCommits: async () => [fixCommit, fixCommit, fixCommit], + // A failing check: the round cap is read only when one fails. + listCommitCheckRuns: async () => + Object.assign([{ id: 11, name: "lint-shell", status: "completed", conclusion: "failure", app: { slug: "github-actions" } }], { complete: true }), }; test("with no FIX_CI_ROUNDS, main reads loop.fix-ci-rounds from the base config", async () => { diff --git a/.github/scripts/apply-patch/apply-patch.mjs b/.github/scripts/apply-patch/apply-patch.mjs index c7d78674af..9f87313545 100755 --- a/.github/scripts/apply-patch/apply-patch.mjs +++ b/.github/scripts/apply-patch/apply-patch.mjs @@ -68,6 +68,7 @@ import { pathToFileURL } from "node:url"; import { BODY_HOLD, LANES_APP_LOGIN } from "../pr-merge/merge-holds.mjs"; import { fixRefusal, parseConfig } from "../pr-unblock/dependency-classify.mjs"; +import { stopSkip } from "../pr-unblock/escalation-routes.mjs"; import { PREDICATES } from "./predicates.mjs"; import { PROTECTED_REFUSALS, segments } from "./protected-paths.mjs"; import { retryFetch } from "./retry-fetch.mjs"; @@ -158,6 +159,20 @@ export function rungUpEscalation(text) { return reasons.length > 0 && reasons.length === escalatedChecks(text).length && reasons.every((reason) => RUNG_UP_REASONS.has(reason)); } +// An escalation only a person settles, and the reason it stops with: a fix +// that needs a path the lane token may not push, or one that would weaken a +// required check (escalation-routes.json holds both). +const HELD_ESCALATIONS = [ + ["protected-path", "ci-fix-protected-path"], + ["weaken-required", "ci-fix-weaken-required"], +]; + +// The held reason a report's escalations name, or "". +export function heldEscalation(text) { + const reasons = reportLines(text).flatMap((line) => ESCALATED_REASON.exec(line)?.[1] ?? []); + return HELD_ESCALATIONS.find(([reason]) => reasons.includes(reason))?.[1] ?? ""; +} + const reportText = (reportDir) => { const file = reportDir ? join(reportDir, "reply.txt") : ""; return Boolean(file) && existsSync(file) && lstatSync(file).isFile() ? readFileSync(file, "utf8") : ""; @@ -169,6 +184,7 @@ const REPORTS = { "ci-escalations": { escalated: (reportDir) => escalatedChecks(reportText(reportDir)).length > 0, rungUp: (reportDir) => rungUpEscalation(reportText(reportDir)), + held: (reportDir) => heldEscalation(reportText(reportDir)), }, }; @@ -389,7 +405,12 @@ export function patchProblem(file, expected) { // the caller runs the producer once more at rung 1, instead of escalating. export function select({ row, patchState, patchSha256, patchHeadSha, headSha, patchDir, reportDir, rung = 0, mergeState = "" }) { checkRow(row); - const decide = (proceed, skipReason, escalationReason) => ({ proceed, skipReason, escalationReason }); + // A stop routed to a failing status skips lane-stopped (escalation-routes.mjs). + const decide = (proceed, skipReason, escalationReason) => ({ + proceed, + skipReason: skipReason === "awaiting-human" && escalationReason ? stopSkip(escalationReason) : skipReason, + escalationReason, + }); // A merge row: a clean merge commits with no patch; a conflicted one needs // the model's patch, and an empty one leaves the conflicts as git left them. if (isMerge(row)) { @@ -418,7 +439,8 @@ export function select({ row, patchState, patchSha256, patchHeadSha, headSha, pa if (escalated && rung === 0 && report.rungUp(reportDir)) { return decide(false, "rung-up", ""); } - return decide(false, "awaiting-human", reasonFor(row, escalated ? "escalated" : "empty")); + const held = escalated ? report.held(reportDir) : ""; + return decide(false, "awaiting-human", held || reasonFor(row, escalated ? "escalated" : "empty")); } let names = []; try { @@ -1110,7 +1132,8 @@ export async function main(argv, env, { api, cwd = process.cwd(), gitBin, log = appendFileSync(env.GITHUB_OUTPUT, `commit=${result.commit}\nbranch=${result.branch}\n`); } if (result.outcome !== "pushed") { - writeFileSync(env.SKIP_REASON_FILE, result.outcome === "superseded" ? "superseded-sha\n" : "awaiting-human\n"); + const skip = result.outcome === "superseded" ? "superseded-sha" : result.outcome === "escalate" ? stopSkip(result.escalationReason) : "awaiting-human"; + writeFileSync(env.SKIP_REASON_FILE, `${skip}\n`); } log(`apply-patch (${row.name}): ${result.outcome}${result.escalationReason ? ` (${result.escalationReason})` : ""}`); return 0; diff --git a/.github/scripts/apply-patch/apply-patch.test.mjs b/.github/scripts/apply-patch/apply-patch.test.mjs index bf5e25a1f8..cfe6caaa63 100644 --- a/.github/scripts/apply-patch/apply-patch.test.mjs +++ b/.github/scripts/apply-patch/apply-patch.test.mjs @@ -1107,7 +1107,7 @@ const patchDirOf = (t, text) => { }; const selectCi = (dir, sum, extra = {}) => select({ row: row("ci-context"), patchState: "written", patchSha256: sum, patchHeadSha: "a".repeat(40), headSha: "a".repeat(40), patchDir: dir, ...extra }); -const INVALID = { proceed: false, skipReason: "awaiting-human", escalationReason: "ci-fix-invalid" }; +const INVALID = { proceed: false, skipReason: "lane-stopped", escalationReason: "ci-fix-invalid" }; test("a draft set during the run drops the fix uncommitted", async (t) => { const fix = repoWithPatch(t, (repo) => writeFileSync(join(repo, "README.md"), "# s\n")); @@ -1247,21 +1247,25 @@ test("an empty patch-state whose report escalated a check names ci-fix-escalated writeFileSync(join(reportDir, "reply.txt"), "lint-shell: escalated not-reproducible needs a secret\n"); const head = "a".repeat(40); const base = { row: row("ci-context"), patchState: "empty", patchSha256: "", patchHeadSha: head, headSha: head, patchDir: scratch(t) }; - assert.deepEqual(select({ ...base, reportDir, rung: 1 }), { proceed: false, skipReason: "awaiting-human", escalationReason: "ci-fix-escalated" }); + assert.deepEqual(select({ ...base, reportDir, rung: 1 }), { proceed: false, skipReason: "lane-stopped", escalationReason: "ci-fix-escalated" }); assert.equal(select({ ...base, reportDir: scratch(t) }).escalationReason, "ci-fix-empty"); }); // A fix-ci run that changed nothing and escalated only // unfixed or not-reproducible checks retries once, one model rung up, instead -// of handing the PR to a person; protected-path, weaken-required and flaky -// stay with a person, and a rung-1 run never retries again. +// of stopping; flaky ends in a failing status (the dispatcher retries the +// head), protected-path and weaken-required stay with a person, and a rung-1 +// run never retries again in the same run. for (const [why, reply, rung, want] of [ ["unfixed at rung 0", "test-node: escalated unfixed three fixes failed\n", 0, { proceed: false, skipReason: "rung-up", escalationReason: "" }], ["unfixed and not-reproducible at rung 0", "a: escalated unfixed x\nb: escalated not-reproducible y\nc: fixed\n", 0, { proceed: false, skipReason: "rung-up", escalationReason: "" }], - ["unfixed at rung 1", "test-node: escalated unfixed three fixes failed\n", 1, { proceed: false, skipReason: "awaiting-human", escalationReason: "ci-fix-escalated" }], - ["protected-path at rung 0", "lint: escalated protected-path edits .github\n", 0, { proceed: false, skipReason: "awaiting-human", escalationReason: "ci-fix-escalated" }], - ["weaken-required beside unfixed at rung 0", "a: escalated unfixed x\nb: escalated weaken-required y\n", 0, { proceed: false, skipReason: "awaiting-human", escalationReason: "ci-fix-escalated" }], - ["flaky at rung 0", "a: escalated flaky x\n", 0, { proceed: false, skipReason: "awaiting-human", escalationReason: "ci-fix-escalated" }], + ["unfixed at rung 1", "test-node: escalated unfixed three fixes failed\n", 1, { proceed: false, skipReason: "lane-stopped", escalationReason: "ci-fix-escalated" }], + // A fix that needs a path the lane token may not push, or one that weakens + // a required check, stays with a person. + ["protected-path at rung 0", "lint: escalated protected-path edits .github\n", 0, { proceed: false, skipReason: "awaiting-human", escalationReason: "ci-fix-protected-path" }], + ["weaken-required beside unfixed at rung 0", "a: escalated unfixed x\nb: escalated weaken-required y\n", 0, { proceed: false, skipReason: "awaiting-human", escalationReason: "ci-fix-weaken-required" }], + ["protected-path beside weaken-required at rung 1", "a: escalated weaken-required x\nb: escalated protected-path y\n", 1, { proceed: false, skipReason: "awaiting-human", escalationReason: "ci-fix-protected-path" }], + ["flaky at rung 0", "a: escalated flaky x\n", 0, { proceed: false, skipReason: "lane-stopped", escalationReason: "ci-fix-escalated" }], ]) { test(`an empty fix-ci patch escalating ${why} selects ${want.skipReason}`, (t) => { const reportDir = scratch(t); @@ -1306,7 +1310,7 @@ test("a written fix escalates nothing, and a run that writes no patch escalates writeFileSync(join(nextReport, "reply.txt"), reply); const next = "b".repeat(40); const red = { row: row("ci-context"), patchState: "empty", patchSha256: "", patchHeadSha: next, headSha: next, patchDir: scratch(t), reportDir: nextReport }; - assert.deepEqual(select(red), { proceed: false, skipReason: "awaiting-human", escalationReason: "ci-fix-escalated" }); + assert.deepEqual(select(red), { proceed: false, skipReason: "lane-stopped", escalationReason: "ci-fix-escalated" }); }); test("select reads the rung from MODEL_RUNG and refuses any other value", async (t) => { @@ -1323,7 +1327,7 @@ test("select reads the rung from MODEL_RUNG and refuses any other value", async await main(["select"], env, { log: () => {} }); return readFileSync(out, "utf8"); }; - assert.match(await outputFor("1"), /^skip-reason=awaiting-human$/m); + assert.match(await outputFor("1"), /^skip-reason=lane-stopped$/m); assert.match(await outputFor("0"), /^skip-reason=rung-up$/m); assert.match(await outputFor(""), /^skip-reason=rung-up$/m); await assert.rejects(() => outputFor("2"), /MODEL_RUNG/); @@ -1900,12 +1904,12 @@ test("select under the merge-base row: a clean merge proceeds with no patch, an const at = (mergeState, patchState, extra = {}) => select({ row: row("merge-base"), mergeState, patchState, patchSha256: sum, patchHeadSha: head, headSha: head, patchDir: dir, ...extra }); assert.deepEqual(at("clean", ""), { proceed: true, skipReason: "", escalationReason: "" }); - assert.deepEqual(at("clean", "written"), { proceed: false, skipReason: "awaiting-human", escalationReason: "conflict-patch-invalid" }); + assert.deepEqual(at("clean", "written"), { proceed: false, skipReason: "lane-stopped", escalationReason: "conflict-patch-invalid" }); assert.deepEqual(at("conflicted", "written"), { proceed: true, skipReason: "", escalationReason: "" }); - assert.deepEqual(at("conflicted", "empty"), { proceed: false, skipReason: "awaiting-human", escalationReason: "unresolved-conflict" }); - assert.deepEqual(at("conflicted", ""), { proceed: false, skipReason: "awaiting-human", escalationReason: "conflict-patch-invalid" }); + assert.deepEqual(at("conflicted", "empty"), { proceed: false, skipReason: "lane-stopped", escalationReason: "unresolved-conflict" }); + assert.deepEqual(at("conflicted", ""), { proceed: false, skipReason: "lane-stopped", escalationReason: "conflict-patch-invalid" }); for (const state of ["", "escalate", "Clean"]) { - assert.deepEqual(at(state, ""), { proceed: false, skipReason: "awaiting-human", escalationReason: "conflict-patch-invalid" }, state); + assert.deepEqual(at(state, ""), { proceed: false, skipReason: "lane-stopped", escalationReason: "conflict-patch-invalid" }, state); } assert.deepEqual(at("clean", "", { patchHeadSha: "b".repeat(40) }), { proceed: false, skipReason: "superseded-sha", escalationReason: "" }); assert.deepEqual(at("conflicted", "written", { headSha: "b".repeat(40) }), { proceed: false, skipReason: "superseded-sha", escalationReason: "" }); diff --git a/.github/scripts/maintenance-monitor-signals/collect.mjs b/.github/scripts/maintenance-monitor-signals/collect.mjs index cede705a6c..1a7befe47d 100644 --- a/.github/scripts/maintenance-monitor-signals/collect.mjs +++ b/.github/scripts/maintenance-monitor-signals/collect.mjs @@ -3,7 +3,9 @@ // that fails to read is recorded in `gaps` and the others still run. Every // list is bounded, and `get` spends the run's call budget. import { parseOutcomeMarker } from "../work-resolve-issue/report-outcome.mjs"; -import { parseBudget } from "./signals.mjs"; +import { escalationActivity, escalationRoute } from "../pr-unblock/escalation-routes.mjs"; +import { failingChecks } from "../pr-unblock/gate-source.mjs"; +import { parseBudget, STALLED_HOURS } from "./signals.mjs"; export const LIMITS = { prMergeRuns: 12, @@ -15,7 +17,65 @@ export const LIMITS = { mergeGroupRuns: 10, sweepArtifacts: 5, commentPages: 5, + // Open pull requests the stalled-stop read covers, most recently updated + // first: one GraphQL query lists the first stalledPulls with each one's + // newest 100 comments and the open count; at most stalledCommentRounds more + // queries read 100 older comments for every PR whose newest name no stop on + // its head; at most stalledChecks check-run reads in all (a fix-ci + // resolution check's listing pages count among them). So the source + // spends at most 1 + 2 + 20 of CALL_BUDGET. stalledPulls covers every open + // pull request at today's count; past it, a gap leaves signal 10 unread + // rather than missing a stop. + stalledPulls: 40, + stalledCommentRounds: 2, + stalledChecks: 20, }; +// report-escalation.sh's comment for a stop routed to a failing status, one +// per activity, edited at each stop (gate-source.mjs LANE_STOPPED). +const LANE_STOPPED = /^$/; +const STOP_FIELD = (name) => new RegExp(`^- ${name}: \`([^\`]+)\``, "m"); +// clear-escalations.mjs's comment when it took off a needs-human an older run +// set (gate-source.mjs CLEARED): a reason now routed to a failing status left +// that head with no stop comment of its own. +const CLEARED = /^$/; +// One GraphQL query reading one field per item under its own alias, so a +// source reads every pull request it covers in one call. The repository is +// the query's $owner and $name. +export const batchQuery = (fields) => + `query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { ${fields.map((field, i) => `a${i}: ${field}`).join(" ")} } }`; +async function queryRepo(graphql, repo, query) { + const [owner, name] = repo.split("/"); + const result = await graphql(query, { owner, name }); + if (result?.errors?.length) throw new Error(`GraphQL: ${result.errors.map((e) => e.message).join("; ")}`); + return result?.data?.repository ?? {}; +} +// A comment connection's page, oldest first; GraphQL names a Bot without +// [bot], so its login is put back in the REST form agentIdentity uses. +const COMMENT_PAGE = "pageInfo { hasPreviousPage startCursor } nodes { author { __typename login } body createdAt updatedAt url }"; +const restComment = (node) => ({ + user: { login: node.author?.__typename === "Bot" ? `${node.author.login}[bot]` : node.author?.login }, + body: node.body, + created_at: node.createdAt, + updated_at: node.updatedAt, + html_url: node.url, +}); +// An attempt of the activity on the head: any of its check runs but a skip, +// which a select job posts without running the activity (a stop's own +// lane-stopped, a held awaiting-human, or not-applicable). +const attempted = (run) => !(run.conclusion === "neutral" && String(run.output?.title ?? "").startsWith("Skipped:")); +// A select job that finds the stop gone on its head (stop-resolved) skips +// not-applicable and report-stop-status posts success over the failing +// status, which this job cannot read. On a non-draft pull request (the slot's +// only not-applicable miss for both activities is `draft: [false]`, +// pr-pipeline.yaml) the address-feedback select skips not-applicable only +// when it is resolved (address-feedback.mjs select). fix-ci's also skips it +// with no fixable failure, so it counts only when the head's checks read +// whole, every one completed, none failing (fix-ci-context.mjs select's own +// rule) now; one that skipped while checks were still running and went green +// later reads as resolved too, though no success was posted. +const NOT_APPLICABLE = "Skipped: not-applicable"; +const RESOLVES = new Set(["address-feedback", "fix-ci"]); +const CHECK_PAGES = 3; export const AUDIT_WORKFLOWS = ["maintenance-audit-test-selection", "maintenance-check-upstream-drift"]; // Pull-request and merge-group runs hold no lane token, and a merge-group // failure is counted as an ejection instead. @@ -34,7 +94,7 @@ const RUN_ID = /^[0-9]+$/; const RUN_LOOKBACK_HOURS = 6; const iso = (date) => date.toISOString().replace(/\.\d{3}Z$/, "Z"); -export async function collect({ get, getStatus, readArtifact, repo, now, windowStart, agentIdentity, ownRate = null }) { +export async function collect({ get, graphql, getStatus, readArtifact, repo, now, windowStart, agentIdentity, ownRate = null }) { const gaps = []; const jobsMemo = new Map(); const annotationsMemo = new Map(); @@ -254,6 +314,123 @@ export async function collect({ get, getStatus, readArtifact, repo, now, windowS if (unread) gaps.push(`lane stops: over ${LIMITS.commentPages * 100} comments in the window; the oldest were not read`); }); + // Open pull requests whose head carries a stop the pr-unblock lane routed to + // a failing status (its lane-stopped comment, by the lanes App, names the + // PR's current head) with no attempt of that activity started on the head + // since the comment's last edit, and no stop-resolved skip ending it + // (NOT_APPLICABLE above): its per-head retries are spent. Any PR, not only + // the lanes App's: fix-ci adopts owners'. The status itself needs + // statuses: read, which this job's token lacks, so the comment and the + // activity's check runs stand for it. + await source("stalled stops", async () => { + data.stalledStops = []; + const cutoff = now.getTime() - STALLED_HOURS * 3600_000; + let checks = 0; + const listed = await queryRepo( + graphql, + repo, + `query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { pullRequests(states: OPEN, baseRefName: "main", first: ${LIMITS.stalledPulls}, orderBy: { field: UPDATED_AT, direction: DESC }) { totalCount nodes { number isDraft headRefOid comments(last: 100) { ${COMMENT_PAGE} } } } } }`, + ); + if (!listed.pullRequests) throw new Error("GraphQL: no open pull request list"); + const total = listed.pullRequests.totalCount ?? 0; + if (total > LIMITS.stalledPulls) { + gaps.push(`stalled stops: ${total} open pull requests, only the ${LIMITS.stalledPulls} most recently updated read`); + } + const pulls = (listed.pullRequests?.nodes ?? []).map((node) => ({ + number: node.number, + draft: node.isDraft === true, + head: { sha: node.headRefOid }, + comments: (node.comments?.nodes ?? []).map(restComment), + older: node.comments?.pageInfo?.hasPreviousPage ? node.comments.pageInfo.startCursor : null, + })); + // A PR's own markers on its current head: a lane-stopped comment naming it, + // or a cleared label for it. An older head's marker says nothing of this + // head's stop, which can sit in an older comment edited in place. + const marks = (pull) => + pull.comments.some((c) => { + if (c.user?.login !== agentIdentity) return false; + const body = String(c.body ?? "").replaceAll("\r\n", "\n"); + const first = body.split("\n")[0]; + return (LANE_STOPPED.test(first) && STOP_FIELD("Head").exec(body)?.[1] === pull.head.sha) || CLEARED.exec(first)?.[2] === pull.head.sha; + }); + for (let round = 0; round < LIMITS.stalledCommentRounds; round += 1) { + const deeper = pulls.filter((pull) => pull.older !== null && !marks(pull)); + if (deeper.length === 0) break; + const older = await queryRepo( + graphql, + repo, + batchQuery(deeper.map((pull) => `pullRequest(number: ${pull.number}) { comments(last: 100, before: ${JSON.stringify(pull.older)}) { ${COMMENT_PAGE} } }`)), + ); + for (const [i, pull] of deeper.entries()) { + const page = older[`a${i}`]?.comments; + pull.comments = [...(page?.nodes ?? []).map(restComment), ...pull.comments]; + pull.older = page?.pageInfo?.hasPreviousPage ? page.pageInfo.startCursor : null; + } + } + for (const pull of pulls) { + const head = pull.head?.sha; + const comments = pull.comments; + // Each stop on this head: the newest lane-stopped comment per activity, + // or a cleared label whose reason is now routed to a failing status. + const stops = new Map(); + for (const c of comments) { + if (c.user?.login !== agentIdentity) continue; + const lines = String(c.body ?? "").split("\n").map((line) => line.replace(/\r$/, "")); + const body = lines.join("\n"); + const activity = LANE_STOPPED.exec(lines[0])?.[1]; + if (activity) { + if (STOP_FIELD("Head").exec(body)?.[1] === head) { + stops.set(activity, { activity, reason: STOP_FIELD("Reason").exec(body)?.[1] ?? "unknown", stoppedAt: c.updated_at ?? c.created_at, url: c.html_url }); + } + continue; + } + const cleared = CLEARED.exec(lines[0]); + if (cleared && cleared[2] === head) { + for (const reason of cleared[1].split(",")) { + const of = escalationActivity(reason); + if (escalationRoute(reason) === "fail" && !stops.has(of)) { + stops.set(of, { activity: of, reason, stoppedAt: c.created_at, url: c.html_url }); + } + } + } + } + if (pull.older !== null && !marks(pull)) { + gaps.push(`stalled stops: #${pull.number} has over ${comments.length} comments and none names a stop on its head; older ones not read`); + } + for (const stop of stops.values()) { + if (!(Date.parse(stop.stoppedAt) < cutoff)) continue; + if (checks >= LIMITS.stalledChecks) { + gaps.push(`stalled stops: over ${LIMITS.stalledChecks} candidate stops; #${pull.number} and later not judged`); + return; + } + checks += 1; + const runs = (await get(`/repos/${repo}/commits/${head}/check-runs?check_name=${encodeURIComponent(`pr-unblock / ${stop.activity}`)}&filter=all&per_page=100`)).check_runs ?? []; + const since = runs.filter((run) => Date.parse(run.started_at) > Date.parse(stop.stoppedAt)); + if (since.some(attempted)) continue; + if (RESOLVES.has(stop.activity) && !pull.draft && since.some((run) => run.conclusion === "neutral" && run.output?.title === NOT_APPLICABLE)) { + if (stop.activity === "address-feedback") continue; + const listed = []; + let whole = false; + for (let page = 1; page <= CHECK_PAGES; page += 1) { + if (checks >= LIMITS.stalledChecks) { + gaps.push(`stalled stops: over ${LIMITS.stalledChecks} check-run reads; #${pull.number} and later not judged`); + return; + } + checks += 1; + const batch = (await get(`/repos/${repo}/commits/${head}/check-runs?filter=all&per_page=100&page=${page}`)).check_runs ?? []; + listed.push(...batch); + if (batch.length < 100) { + whole = true; + break; + } + } + if (whole && listed.every((run) => run.status === "completed") && failingChecks(listed).length === 0) continue; + } + data.stalledStops.push({ pr: pull.number, head, ...stop }); + } + } + }); + await source("audits", async () => { data.audits = []; for (const workflow of AUDIT_WORKFLOWS) { diff --git a/.github/scripts/maintenance-monitor-signals/collect.test.mjs b/.github/scripts/maintenance-monitor-signals/collect.test.mjs index c26327190c..8086b8da75 100644 --- a/.github/scripts/maintenance-monitor-signals/collect.test.mjs +++ b/.github/scripts/maintenance-monitor-signals/collect.test.mjs @@ -29,19 +29,58 @@ const artifact = (id, created_at, runId, extra = {}) => ({ ...extra, }); +// GraphQL answers come from the same REST routes: the open pull request +// list (newest update first) and each PR's comments, oldest first, as +// GraphQL shapes them: a Bot's login without [bot], pages of 100 from the +// newest back, a cursor being the index the page starts at. +const GRAPHQL = "POST /graphql"; +const node = (c) => ({ + author: c.user && { __typename: /\[bot\]$/.test(c.user.login) ? "Bot" : "User", login: c.user.login.replace(/\[bot\]$/, "") }, + body: c.body, + createdAt: c.created_at, + updatedAt: c.updated_at ?? c.created_at, + url: c.html_url, +}); +const commentPage = (all, before = all.length) => { + const start = Math.max(0, before - 100); + return { pageInfo: { hasPreviousPage: start > 0, startCursor: String(start) }, nodes: all.slice(start, before).map(node) }; +}; + function fakeApi(routes) { const calls = []; - const get = async (path) => { - calls.push(path); + const queries = []; + const resolve = (path) => { for (const [pattern, value] of routes) { - if (pattern.test(path)) { + if (pattern instanceof RegExp && pattern.test(path)) { if (value instanceof Error) throw value; return typeof value === "function" ? value(path) : value; } } throw new Error(`no route for ${path}`); }; - return { get, calls }; + const get = async (path) => { + calls.push(path); + return resolve(path); + }; + const comments = (number) => resolve(`/repos/${repo}/issues/${number}/comments`); + const graphql = async (query, variables) => { + calls.push(GRAPHQL); + queries.push({ query, variables }); + const value = routes.find(([pattern]) => pattern === GRAPHQL)?.[1]; + if (value instanceof Error) throw value; + const first = /pullRequests\(states: OPEN, baseRefName: "main", first: (\d+)/.exec(query); + if (first) { + const pulls = resolve(`/repos/${repo}/pulls?state=open&base=main&sort=updated&direction=desc&per_page=10000&page=1`); + const nodes = pulls.slice(0, Number(first[1])).map((pull) => ({ number: pull.number, isDraft: pull.draft === true, headRefOid: pull.head.sha, comments: commentPage(comments(pull.number)) })); + return { data: { repository: { pullRequests: { totalCount: pulls.length, nodes } } } }; + } + const repository = {}; + for (const [, alias, number, before] of query.matchAll(/(a\d+): pullRequest\(number: (\d+)\) \{ comments\(last: 100, before: "(\d+)"\)/g)) { + repository[alias] = { comments: commentPage(comments(number), Number(before)) }; + } + return { data: { repository } }; + }; + return { get, graphql, calls, queries }; } const status = { @@ -73,7 +112,35 @@ const laneComment = (id, body, extra = {}) => ({ ...extra, }); +// report-escalation.sh's lane-stopped comment for `activity` at `head`, last +// edited at `at`. +const sha = (c) => c.repeat(40); +const stopComment = (pr, head, at, { activity = "fix-ci", reason = "ci-fix-invalid", login = "melodic-automation-lanes[bot]" } = {}) => ({ + id: pr * 10, + body: `\nThe pr-unblock lane stopped.\n\n- Reason: \`${reason}\`\n- Head: \`${head}\`\n`, + user: { login }, + created_at: "2026-10-18T09:00:00Z", + updated_at: at, + html_url: `https://github.com/o/r/pull/${pr}#issuecomment-${pr * 10}`, +}); +// Open PRs: #8301 stalled (stopped at 12:00, no attempt since); #8302 stopped +// 20 minutes ago; #8303 stopped long ago but retried since; #8304 +// Dependabot's, stalled on resolve-conflicts; #8305's comment names an +// older head; #8306's marker is a person's. +const stallRoutes = () => [ + [/\/pulls\?state=open/, [8301, 8302, 8303, 8304, 8305, 8306].map((number) => ({ number, head: { sha: sha(String(number % 10)) } }))], + [/\/issues\/8301\/comments/, [stopComment(8301, sha("1"), "2026-10-19T12:00:00Z")]], + [/\/issues\/8302\/comments/, [stopComment(8302, sha("2"), "2026-10-19T16:20:00Z")]], + [/\/issues\/8303\/comments/, [stopComment(8303, sha("3"), "2026-10-19T12:00:00Z")]], + [/\/issues\/8304\/comments/, [stopComment(8304, sha("4"), "2026-10-19T11:30:00Z", { activity: "resolve-conflicts", reason: "conflict-patch-invalid" })]], + [/\/issues\/8305\/comments/, [stopComment(8305, sha("9"), "2026-10-19T10:00:00Z")]], + [/\/issues\/8306\/comments/, [stopComment(8306, sha("6"), "2026-10-19T10:00:00Z", { login: "someone" })]], + [/\/commits\/3{40}\/check-runs/, { check_runs: [{ started_at: "2026-10-19T12:30:00Z" }] }], + [/\/commits\/[0-9a-f]{40}\/check-runs/, { check_runs: [{ started_at: "2026-10-19T11:00:00Z" }] }], +]; + const routes = () => [ + ...stallRoutes(), [/workflows\/pr-merge\.yml\/runs/, { workflow_runs: [run(10)] }], [/runs\/10\/jobs/, { jobs: [{ id: 100, name: "merge", conclusion: "success" }, { id: 101, name: "post-merge-dispatch" }] }], [ @@ -133,6 +200,7 @@ const routes = () => [ const input = (api, extra = {}) => ({ get: api.get, + graphql: api.graphql, getStatus: async (url) => status[url], readArtifact: async (id) => ({ run_id: 1000 + id, event: "schedule", live: true, started_at: "2026-10-19T16:30:00Z", candidates: 10, refusal_counts: {}, results: [] }), repo, @@ -176,6 +244,14 @@ test("collect reads every source into the signals' data shape", async () => { data.status.incidents.map((i) => [i.id, i.components]), [["8tzbpcnl75jp", ["Actions", "Webhooks"]]], ); + // A stop with no attempt since, past STALLED_HOURS, on any author's PR. + assert.deepEqual( + data.stalledStops.map((s) => [s.pr, s.activity, s.reason]), + [ + [8301, "fix-ci", "ci-fix-invalid"], + [8304, "resolve-conflicts", "conflict-patch-invalid"], + ], + ); // Only the lanes App's markers created in the window, at the comment's start. assert.deepEqual( data.laneStops.map((s) => [s.id, s.outcome, s.issue]), @@ -426,3 +502,154 @@ test("listFiled lists only github-actions[bot]'s issues", async () => { await listFiled({ get: api.get, repo }); assert.deepEqual(api.calls.map((path) => new URL(path, "https://x").searchParams.get("creator")), ["github-actions[bot]"]); }); + +// GitHub's open pull request list, honoring sort direction, page and per_page. +const openPulls = (pulls) => (path) => { + const q = new URL(path, "https://x").searchParams; + const sorted = [...pulls].sort((a, b) => a.updated_at.localeCompare(b.updated_at) * (q.get("direction") === "desc" ? -1 : 1)); + const per = Number(q.get("per_page") ?? 30); + const page = Number(q.get("page") ?? 1); + return sorted.slice((page - 1) * per, page * per); +}; +const idle = (number, day) => ({ number, head: { sha: sha("e") }, updated_at: `2026-10-${String(day).padStart(2, "0")}T00:00:00Z` }); + +test("stalled stops: more open PRs than the read covers is a gap naming the true count, and the most recently updated are read", async () => { + const many = Array.from({ length: 145 }, (_, i) => idle(9000 + i, 1 + (i % 17))); + const api = fakeApi([[/\/pulls\?state=open/, openPulls(many)], [/\/issues\/9\d{3}\/comments/, []], ...routes()]); + const data = await collect(input(api)); + assert.ok(data.gaps.some((g) => g === "stalled stops: 145 open pull requests, only the 40 most recently updated read"), data.gaps.join("\n")); + // One query lists them, newest update first, with their comments. + assert.equal(api.calls.filter((p) => p === GRAPHQL).length, 1); + assert.match(api.queries[0].query, /pullRequests\(states: OPEN, baseRefName: "main", first: 40, orderBy: \{ field: UPDATED_AT, direction: DESC \}\)/); + assert.match(api.queries[0].query, /comments\(last: 100\)/); + assert.ok(!api.calls.some((p) => /\/pulls\?state=open|\/issues\/\d+\/comments/.test(p))); +}); + +// Review finding on #10252: the read spent up to 3 + 40 + 20 calls of the +// run's CALL_BUDGET. Worst case now: 145 open PRs, every one stopped long ago +// with no attempt, each with 350 comments so every comment round runs. +test("stalled stops: the worst case spends at most 1 + 2 + 20 calls", async () => { + const many = Array.from({ length: 145 }, (_, i) => idle(9000 + i, 1 + (i % 17))); + const chatter = Array.from({ length: 350 }, (_, i) => ({ id: i, body: "chatter", user: { login: "someone" }, created_at: "2026-10-17T00:00:00Z" })); + const api = fakeApi([ + [/\/pulls\?state=open/, openPulls(many)], + [/\/issues\/9\d{3}\/comments/, (path) => [...chatter, stopComment(Number(/issues\/(\d+)/.exec(path)[1]), sha("e"), "2026-10-19T10:00:00Z")]], + [/\/commits\/e{40}\/check-runs/, { check_runs: [] }], + ...routes(), + ]); + const data = await collect(input(api)); + const spent = api.calls.filter((p) => p === GRAPHQL || /\/pulls\?state=open|\/issues\/\d+\/comments|check-runs\?check_name/.test(p)).length; + assert.ok(spent <= 1 + 2 + 20, `${spent} calls`); + assert.equal(data.stalledStops.length, 20); +}); + +// Review finding on #10252: an older head's lane-stopped comment among the +// newest 100 hid the current head's stop in an older comment (edited in place +// at the new stop) and recorded no gap. Only a current-head marker ends the +// read; past the comment rounds, the PR is a gap. +test("stalled stops: an older head's marker does not stop the read before the current head's stop", async () => { + const chatter = (n) => Array.from({ length: n }, (_, i) => ({ id: i, body: "chatter", user: { login: "someone" }, created_at: "2026-10-18T10:00:00Z" })); + const current = stopComment(8501, sha("1"), "2026-10-19T12:00:00Z"); + const stale = { ...stopComment(8501, sha("9"), "2026-10-18T12:00:00Z", { activity: "address-feedback", reason: "feedback-partial" }), id: 2 }; + const api = fakeApi([ + [/\/pulls\?state=open/, [{ number: 8501, head: { sha: sha("1") } }, { number: 8502, head: { sha: sha("2") } }]], + [/\/issues\/8501\/comments/, [current, ...chatter(150), stale, ...chatter(20)]], + [/\/issues\/8502\/comments/, [stopComment(8502, sha("2"), "2026-10-19T12:00:00Z"), ...chatter(320), { ...stopComment(8502, sha("8"), "2026-10-18T12:00:00Z"), id: 3 }]], + [/\/commits\/[12]{40}\/check-runs/, { check_runs: [] }], + ...routes(), + ]); + const data = await collect(input(api)); + assert.deepEqual(data.stalledStops.map((s) => [s.pr, s.activity]), [[8501, "fix-ci"]]); + assert.ok(data.gaps.includes("stalled stops: #8502 has over 300 comments and none names a stop on its head; older ones not read"), data.gaps.join("\n")); +}); + +// A pull request stopped three hours ago sorts ahead of older idle ones, so +// more than 20 idle pull requests no longer hide it. +test("stalled stops: a freshly stopped PR behind more than 20 idle ones is read and listed", async () => { + const idlePulls = Array.from({ length: 25 }, (_, i) => idle(9100 + i, 1 + i)); + const fresh = { number: 9200, head: { sha: sha("f") }, updated_at: "2026-10-19T13:30:00Z" }; + const api = fakeApi([ + [/\/pulls\?state=open/, openPulls([...idlePulls, fresh])], + [/\/issues\/9200\/comments/, [stopComment(9200, sha("f"), "2026-10-19T13:30:00Z")]], + [/\/issues\/91\d{2}\/comments/, []], + [/\/commits\/f+\/check-runs/, { check_runs: [] }], + ...routes(), + ]); + const data = await collect(input(api)); + assert.deepEqual(data.stalledStops.map((s) => s.pr), [9200]); + assert.ok(!data.gaps.some((g) => g.startsWith("stalled stops:")), data.gaps.join("\n")); +}); + +// Shapes a skip-blind attempt rule missed: a select job's lane-stopped check +// posted after the comment's edit, and a draft's feedback skip (not-applicable +// from the slot's draft predicate). Neither ran the activity, so both stay +// stalled. A label clear-escalations took off for a reason now routed to a +// failing status (#10136's shape: no stop comment on that head) is a stop too. +test("stalled stops: a skip is no attempt, and a cleared fail-routed label is a stop", async () => { + const skip = (title, at) => ({ conclusion: "neutral", output: { title }, started_at: at }); + const cleared = (pr, head, reason) => ({ + id: pr, + body: `\nThe pr-unblock lane escalated this pull request.`, + user: { login: "melodic-automation-lanes[bot]" }, + created_at: "2026-10-19T10:00:00Z", + html_url: `https://github.com/o/r/pull/${pr}#c`, + }); + const api = fakeApi([ + [/\/pulls\?state=open/, [8401, 8402, 8403, 8404, 8405].map((number) => ({ number, draft: number === 8402, head: { sha: sha(String(number % 10)) } }))], + [/\/issues\/8401\/comments/, [stopComment(8401, sha("1"), "2026-10-19T12:00:00Z", { activity: "resolve-conflicts", reason: "repeat-cap" })]], + [/\/issues\/8402\/comments/, [stopComment(8402, sha("2"), "2026-10-19T12:00:00Z", { activity: "address-feedback", reason: "feedback-partial" })]], + [/\/issues\/8403\/comments/, [cleared(8403, sha("3"), "ci-fix-escalated")]], + [/\/issues\/8404\/comments/, [cleared(8404, sha("4"), "feedback-uncertain")]], + [/\/issues\/8405\/comments/, Array.from({ length: 350 }, (_, i) => ({ id: i, body: "chatter", user: { login: "someone" } }))], + [/\/commits\/1{40}\/check-runs/, { check_runs: [skip("Skipped: lane-stopped", "2026-10-19T12:00:05Z")] }], + [/\/commits\/2{40}\/check-runs/, { check_runs: [skip("Skipped: not-applicable", "2026-10-19T12:40:00Z")] }], + [/\/commits\/3{40}\/check-runs/, { check_runs: [{ conclusion: "success", output: { title: "Fixed" }, started_at: "2026-10-19T09:00:00Z" }] }], + ...routes(), + ]); + const data = await collect(input(api)); + assert.deepEqual( + data.stalledStops.map((s) => [s.pr, s.activity, s.reason]), + [ + [8401, "resolve-conflicts", "repeat-cap"], + [8402, "address-feedback", "feedback-partial"], + [8403, "fix-ci", "ci-fix-escalated"], + ], + "a held reason's clear is no stop", + ); + assert.ok(data.gaps.includes("stalled stops: #8405 has over 300 comments and none names a stop on its head; older ones not read"), data.gaps.join("\n")); +}); + +// Review finding on #10252: a select job that found the stop gone +// (stop-resolved) skips not-applicable and posts success over the failing +// status, which the monitor cannot read. address-feedback's select skips +// not-applicable only then, so on a non-draft PR that skip ends the stop. +// fix-ci's skips it also with no fixable failure, so it ends the stop only +// when the head's whole listing is settled with none failing. +test("stalled stops: a stop-resolved not-applicable skip ends an address-feedback or settled-green fix-ci stop", async () => { + const skip = (at) => ({ conclusion: "neutral", output: { title: "Skipped: not-applicable" }, started_at: at }); + const done = (id, conclusion) => ({ id, name: `check ${id}`, status: "completed", conclusion }); + const heads = { 8601: [done(1, "success")], 8603: [done(1, "success"), done(2, "skipped")], 8604: [done(1, "failure")], 8605: [done(1, "success"), { id: 2, name: "check 2", status: "in_progress", conclusion: null }] }; + const pulls = [8601, 8602, 8603, 8604, 8605]; + const api = fakeApi([ + [/\/pulls\?state=open/, pulls.map((number) => ({ number, draft: false, head: { sha: sha(String(number % 10)) } }))], + [/\/issues\/8601\/comments/, [stopComment(8601, sha("1"), "2026-10-19T12:00:00Z", { activity: "address-feedback", reason: "feedback-partial" })]], + [/\/issues\/8602\/comments/, [stopComment(8602, sha("2"), "2026-10-19T12:00:00Z", { activity: "resolve-conflicts", reason: "repeat-cap" })]], + ...[8603, 8604, 8605].map((pr) => [new RegExp(`/issues/${pr}/comments`), [stopComment(pr, sha(String(pr % 10)), "2026-10-19T12:00:00Z", { activity: "fix-ci", reason: "ci-fix-escalated" })]]), + [/\/commits\/([0-9a-f])\1{39}\/check-runs/, (path) => { + if (path.includes("check_name=")) return { check_runs: [skip("2026-10-19T12:30:00Z")] }; + const pr = pulls.find((number) => path.includes(sha(String(number % 10)))); + return { check_runs: heads[pr] ?? assert.fail(`no listing expected for #${pr}`) }; + }], + ...routes(), + ]); + const data = await collect(input(api)); + assert.deepEqual( + data.stalledStops.map((s) => [s.pr, s.activity]), + [ + [8602, "resolve-conflicts"], + [8604, "fix-ci"], + [8605, "fix-ci"], + ], + ); + assert.ok(!data.gaps.some((g) => g.startsWith("stalled stops:")), data.gaps.join("\n")); +}); diff --git a/.github/scripts/maintenance-monitor-signals/issues.mjs b/.github/scripts/maintenance-monitor-signals/issues.mjs index 04ff58b75a..975a6b51a7 100644 --- a/.github/scripts/maintenance-monitor-signals/issues.mjs +++ b/.github/scripts/maintenance-monitor-signals/issues.mjs @@ -15,6 +15,7 @@ export const SIGNALS = [ "6 report-only audit", "8 throughput", "9 lane stop", + "10 stalled lane stop", ]; export const LOG_ONLY = new Set(["7 platform incident"]); const MUTED = new Set(["not_planned", "duplicate"]); @@ -118,6 +119,7 @@ const SOURCES = { "pr-merge sweep": ["5 held PRs", "8 throughput"], "lane merges": ["8 throughput"], "lane stops": ["9 lane stop"], + "stalled stops": ["10 stalled lane stop"], audits: ["6 report-only audit"], "githubstatus.com": [], }; diff --git a/.github/scripts/maintenance-monitor-signals/issues.test.mjs b/.github/scripts/maintenance-monitor-signals/issues.test.mjs index 58774d7056..85e2652dd6 100644 --- a/.github/scripts/maintenance-monitor-signals/issues.test.mjs +++ b/.github/scripts/maintenance-monitor-signals/issues.test.mjs @@ -200,6 +200,7 @@ test("every detector's signal is a rolling family or logged only, and an unknown audits: [{ workflow: "w", run: { id: 1, url: "u", conclusion: "failure", createdAt: "t" } }], status: { components: [{ name: "Actions", status: "major_outage" }], incidents: [] }, laneStops: [{ id: 1, outcome: "funding", issue: 3, createdAt: "t", url: "u" }], + stalledStops: [{ pr: 5, head: "a", activity: "fix-ci", reason: "ci-fix-invalid", stoppedAt: "t", url: "u" }], }); assert.deepEqual(new Set(all.map((f) => f.signal)), new Set([...SIGNALS, ...LOG_ONLY])); const plan = planIssues({ findings: [{ ...findingFor([1]), signal: "10 renamed" }], existing: [] }); diff --git a/.github/scripts/maintenance-monitor-signals/monitor.mjs b/.github/scripts/maintenance-monitor-signals/monitor.mjs index 50306be51e..13f8843d8c 100644 --- a/.github/scripts/maintenance-monitor-signals/monitor.mjs +++ b/.github/scripts/maintenance-monitor-signals/monitor.mjs @@ -125,6 +125,7 @@ async function main(env = process.env) { const windowStart = new Date(now.getTime() - Number(env.WINDOW_MINUTES ?? 65) * 60_000); const { call, used } = client(env); const get = (path) => call("GET", path); + const graphql = (query, variables) => call("POST", "/graphql", { query, variables }); const getStatus = async (url) => { const response = await fetch(url, { signal: AbortSignal.timeout(20_000) }); if (!response.ok) throw new Error(`${url}: HTTP ${response.status}`); @@ -144,6 +145,7 @@ async function main(env = process.env) { const data = await collect({ get, + graphql, getStatus, readArtifact, repo, diff --git a/.github/scripts/maintenance-monitor-signals/signals.mjs b/.github/scripts/maintenance-monitor-signals/signals.mjs index 9e6ae03c9d..78b430dde7 100644 --- a/.github/scripts/maintenance-monitor-signals/signals.mjs +++ b/.github/scripts/maintenance-monitor-signals/signals.mjs @@ -13,6 +13,11 @@ export const ACTIONS_JOB_LIMIT = 60; export const EJECTIONS_PER_DAY = 2; export const HELD_HOURS = 24; export const STALL_HOURS = 3; +// pr-unblock-dispatch.yml runs every 30 minutes (cron 9,39), and the feedback +// sweep runs off each of those ticks, so a stop the lane will retry gets a new +// attempt within one tick; four ticks with none mean the per-head retries are +// spent (or a dispatcher refused). +export const STALLED_HOURS = 2; const SWEEP_MAX_AGE_MINUTES = 70; const MAX_FACTS = 25; @@ -315,6 +320,24 @@ export function laneStopFindings({ laneStops = [] }) { ); } +// Pull requests a pr-unblock stop left red with no retry left on their head +// (collect.mjs, "stalled stops"): no person was asked, so this tells one. +export function stalledFindings({ stalledStops = [] }) { + if (stalledStops.length === 0) return []; + const stops = [...stalledStops].sort((a, b) => a.pr - b.pr || a.activity.localeCompare(b.activity)); + return [ + finding({ + key: "stalled-lane-stop", + signal: "10 stalled lane stop", + title: `Stalled: ${stops.length} pull request stops the PR fix lane left red for over ${STALLED_HOURS} hours`, + area: "PR fix lane (pr-unblock)", + summary: `${stops.length} pull requests carry a failing pr-unblock status the lane can no longer retry on their head; each needs a new commit or a look at its stop comment.`, + facts: stops.map((s) => `#${s.pr} ${s.activity} ${s.reason} since ${s.stoppedAt}: ${s.url}`), + evidence: ["stalled-lane-stop", stops.map((s) => [s.pr, s.head, s.activity, s.reason])], + }), + ]; +} + const DETECTORS = [ mainRedFindings, ceilingFindings, @@ -325,6 +348,7 @@ const DETECTORS = [ platformFindings, throughputFindings, laneStopFindings, + stalledFindings, ]; export function detect(data) { diff --git a/.github/scripts/maintenance-monitor-signals/signals.test.mjs b/.github/scripts/maintenance-monitor-signals/signals.test.mjs index e750d78848..1762214fb8 100644 --- a/.github/scripts/maintenance-monitor-signals/signals.test.mjs +++ b/.github/scripts/maintenance-monitor-signals/signals.test.mjs @@ -14,6 +14,7 @@ import { mainRedFindings, parseBudget, platformFindings, + stalledFindings, throttleFindings, throughputFindings, } from "./signals.mjs"; @@ -429,3 +430,23 @@ test("every finding has a marker-safe key, a hex fingerprint, an area and inert for (const line of f.facts) assert.doesNotMatch(line, /`|\n/); } }); + +// collect.mjs hands on only stops past STALLED_HOURS with no attempt since; +// one finding names each, and its fingerprint moves only with the set. +test("stalled stops are one finding, a fact per PR, whose fingerprint moves with the stopped heads only", () => { + const stop = (pr, extra = {}) => ({ pr, head: "a".repeat(40), activity: "fix-ci", reason: "ci-fix-invalid", stoppedAt: "2026-10-19T12:00:00Z", url: `https://github.com/o/r/pull/${pr}#c`, ...extra }); + assert.deepEqual(stalledFindings({ stalledStops: [] }), []); + const [f] = stalledFindings({ stalledStops: [stop(9), stop(4, { activity: "resolve-conflicts", reason: "conflict-patch-invalid" })] }); + assert.equal(f.key, "stalled-lane-stop"); + assert.equal(f.signal, "10 stalled lane stop"); + assert.match(f.title, /^Stalled: 2 pull request stops/); + assert.deepEqual(f.facts, [ + "#4 resolve-conflicts conflict-patch-invalid since 2026-10-19T12:00:00Z: https://github.com/o/r/pull/4#c", + "#9 fix-ci ci-fix-invalid since 2026-10-19T12:00:00Z: https://github.com/o/r/pull/9#c", + ]); + const [again] = stalledFindings({ stalledStops: [stop(4, { activity: "resolve-conflicts", reason: "conflict-patch-invalid", stoppedAt: "later" }), stop(9)] }); + assert.equal(again.fingerprint, f.fingerprint, "the stop's time is not evidence"); + const [moved] = stalledFindings({ stalledStops: [stop(9, { head: "b".repeat(40) })] }); + assert.notEqual(moved.fingerprint, f.fingerprint); + assert.ok(detect({ ...base, stalledStops: [stop(9)] }).some((x) => x.key === "stalled-lane-stop")); +}); diff --git a/.github/scripts/pr-merge/clear-escalations.mjs b/.github/scripts/pr-merge/clear-escalations.mjs index 9e342c25cf..8341089cf2 100755 --- a/.github/scripts/pr-merge/clear-escalations.mjs +++ b/.github/scripts/pr-merge/clear-escalations.mjs @@ -32,6 +32,15 @@ // unresolved-conflict, GitHub reports the PR mergeable: no conflict // protected-path-conflict is left to resolve // feedback-protected-path the PR changes nothing under .github/workflows/ +// a fix-ci reason escalation-routes.json routes `fail` (a lane run before +// that route set the label): once a lane check of a pr-require-checks run +// on the current head reads failed (fix-ci-context's failingChecks; +// ci-status itself fails on the label, so it proves nothing, and an +// advisory lane's failure leaves ci-status green), so the head stays red +// without the label. Another fail-routed reason is a person's (below): a +// lane posts a pr-unblock/* status on a head only when it stops there or +// clears its own failure, so a moved head carries nothing that shows the +// cause gone or fails that head in the label's place // A listing that may be cut short, or a mergeable state not yet computed, // counts as the reason still holding. // Any other reason is decided by a person and is never cleared here. @@ -75,6 +84,9 @@ // // Environment: // GH_TOKEN the lane App token (pull-requests and issues write) +// CHECKS_TOKEN the job's own token (checks: read, actions: read), for +// the head's pr-require-checks runs and check runs, +// which the lane token cannot read // GITHUB_REPOSITORY owner/repo // KILL_SWITCH `proceed` from the kill-switch step; anything else stops // TRACKER_CONFIG the base .work-item-tracker.json, for the human-gated @@ -97,14 +109,19 @@ import { CLEARED, ESCALATION, escalationsSince, + escalationActivity, + escalationRoute, + failingChecks, foreignMarker, isApp, labelHistory, + LANE_STOPPED, laneHold, lines, openReviewHold, PAIRED_MS, RESUME_KEY, + ROUTES, same, spoken, } from "../pr-unblock/gate-source.mjs"; @@ -169,6 +186,45 @@ function laneCause(kind, reason) { return codes.every((code) => LANE_CAUSES[kind].some((entry) => (entry.endsWith(":") ? code.startsWith(entry) : code === entry))); } +// The check-run pages read per head, as fix-ci-context reads them. +const CHECK_PAGES = 3; + +// The workflow whose lanes ci-status aggregates. An advisory lane's failure +// (pr-test-windows) leaves ci-status green once the label is off, so only +// this workflow's checks count. +const REQUIRED_WORKFLOW = "pr-require-checks.yml"; + +// Whether a lane check of a pr-require-checks run on the head (not ci-status) +// failed. Every such run counts: a contract-only run posts its lanes skipped, +// and failingChecks prefers a lane's real result over that skip. One read per +// PR; a listing that fails, or may be cut short, reads as not failing. +async function headFailing({ checks, repo, pull }) { + try { + const required = (await checks("GET", `/repos/${repo}/actions/workflows/${REQUIRED_WORKFLOW}/runs?head_sha=${pull.head.sha}&per_page=100`))?.workflow_runs; + if (!Array.isArray(required) || required.length >= 100) { + return false; + } + const suites = new Set(required.map((run) => run.check_suite_id)); + const runs = []; + for (let page = 1; ; page += 1) { + const listed = (await checks("GET", `/repos/${repo}/commits/${pull.head.sha}/check-runs?filter=all&per_page=100&page=${page}`))?.check_runs; + if (!Array.isArray(listed)) { + return false; + } + runs.push(...listed); + if (listed.length < 100) { + break; + } + if (page === CHECK_PAGES) { + return false; + } + } + return failingChecks(runs.filter((run) => run.app?.slug === "github-actions" && suites.has(run.check_suite?.id))).length > 0; + } catch { + return false; + } +} + const escapeCommand = (text) => String(text).replaceAll(/[\r\n%]/g, " "); const reportError = (number, error) => console.log(`::error title=clear-escalations::#${number}: ${escapeCommand(error?.message ?? error)}`); @@ -193,6 +249,21 @@ const CHECKS = Object.freeze({ ? null : "the pull request changes nothing under .github/workflows/"; }, + // A fix-ci reason escalation-routes.json now routes to a failing status + // rather than a person, on a label an older lane run set: once the head's + // checks read failed, so it stays red without the label (green, missing or + // unreadable keeps it). Every other fail-routed reason is a person's here: + // a lane posts a pr-unblock/* status on a head only when it stops there or + // clears its own failure (report-escalation.sh), so nothing on a new head + // proves its cause gone or fails that head in the label's place. + ...Object.fromEntries( + Object.keys(ROUTES) + .filter((reason) => escalationRoute(reason) === "fail" && escalationActivity(reason) === "fix-ci") + .map((reason) => [ + reason, + async ({ failing }) => ((await failing()) ? "the lane now ends it in a failing status, not a person, and the head's own checks read failed" : null), + ]), + ), }); function mergeable(pull) { @@ -266,7 +337,7 @@ function pairedWithPrMerge(appComments, since) { let ours = false; let escalated = false; for (const item of appComments) { - const markers = lines(item.body).filter((line) => line.includes("\nStopped.", "2026-10-18T09:00:00Z", "2026-10-18T10:30:00Z"), + ], + }); + await run(w); + assert.equal(w.writes().at(-1)?.method, "DELETE"); +}); + test("a conflict reason clears only once GitHub reports the PR mergeable", async () => { const timeline = [labeled(BOT, "2026-10-18T10:00:00Z"), escalated("unresolved-conflict")]; const unknown = world({ timeline, mergeable: null }); diff --git a/.github/scripts/pr-merge/merge-holds.mjs b/.github/scripts/pr-merge/merge-holds.mjs index 16a1d73231..1793399779 100755 --- a/.github/scripts/pr-merge/merge-holds.mjs +++ b/.github/scripts/pr-merge/merge-holds.mjs @@ -244,7 +244,10 @@ export const matchesHoldPath = (path, glob) => posix.matchesGlob(undot(path), un // the config the token broker loads (resolve-config, effect-grants.json // included, and the pipeline config); `enforcement`, the other modules the // floored scripts import and the Python that decides the holds -// (test_merge_holds_sync.py derives both closures and requires them here). +// (test_merge_holds_sync.py derives both closures and requires them here), +// and the scripts that act on a pr-unblock stop's route (report-escalation.sh, +// adjudicate-feedback.mjs, publish-feedback.mjs, apply-patch.mjs) with every +// module they import and the routing table they read. // babysit_risk.FLOOR_CLASSES, which test_merge_holds_sync.py checks this // equals. A constant of the module ci-status and enable-auto-merge run from // the base commit, so a pull request cannot rewrite it. @@ -294,12 +297,23 @@ export const HOLD_PATH_CLASSES = Object.freeze([ [".github/workflows/pr-merge.yml", "gate"], ["docs/conventions/pr-pipeline.yaml", "gate"], [".github/scripts/pr-unblock/address-feedback.mjs", "enforcement"], + [".github/scripts/pr-unblock/escalation-routes.mjs", "enforcement"], + [".github/scripts/pr-unblock/escalation-routes.json", "enforcement"], [".github/scripts/pr-unblock/gate-source.mjs", "enforcement"], [".github/scripts/pr-unblock/thread-scope.mjs", "enforcement"], [".github/scripts/pr-unblock/retry-fetch.mjs", "enforcement"], [".github/scripts/work-resolve-issue/report-outcome.mjs", "enforcement"], [".github/scripts/work-resolve-issue/limits.mjs", "enforcement"], [".github/scripts/work-resolve-issue/retry-fetch.mjs", "enforcement"], + [".github/scripts/pr-unblock/report-escalation.sh", "enforcement"], + [".github/scripts/pr-unblock/adjudicate-feedback.mjs", "enforcement"], + [".github/scripts/pr-unblock/publish-feedback.mjs", "enforcement"], + [".github/scripts/pr-unblock/request-budget.mjs", "enforcement"], + [".github/scripts/pr-unblock/dependency-classify.mjs", "enforcement"], + [".github/scripts/apply-patch/apply-patch.mjs", "enforcement"], + [".github/scripts/apply-patch/predicates.mjs", "enforcement"], + [".github/scripts/apply-patch/protected-paths.mjs", "enforcement"], + [".github/scripts/apply-patch/retry-fetch.mjs", "enforcement"], ]); // The `trust` and `gate` globs (babysit_risk.HIGH_FLOOR), which hold for a // person. An `enforcement` glob (REVIEW_FLOOR) holds nothing here: the merge diff --git a/.github/scripts/pr-merge/merge-holds.test.mjs b/.github/scripts/pr-merge/merge-holds.test.mjs index 3e08fc72c0..28c57f136e 100644 --- a/.github/scripts/pr-merge/merge-holds.test.mjs +++ b/.github/scripts/pr-merge/merge-holds.test.mjs @@ -342,12 +342,23 @@ const ENFORCEMENT_FILES = [ "plugins/source-control/skills/babysit-prs/scripts/babysit_verify_evidence.py", "plugins/source-control/skills/babysit-prs/scripts/babysit_gh.py", ".github/scripts/pr-unblock/address-feedback.mjs", + ".github/scripts/pr-unblock/escalation-routes.mjs", + ".github/scripts/pr-unblock/escalation-routes.json", ".github/scripts/pr-unblock/gate-source.mjs", ".github/scripts/pr-unblock/thread-scope.mjs", ".github/scripts/pr-unblock/retry-fetch.mjs", ".github/scripts/work-resolve-issue/report-outcome.mjs", ".github/scripts/work-resolve-issue/limits.mjs", ".github/scripts/work-resolve-issue/retry-fetch.mjs", + ".github/scripts/pr-unblock/report-escalation.sh", + ".github/scripts/pr-unblock/adjudicate-feedback.mjs", + ".github/scripts/pr-unblock/publish-feedback.mjs", + ".github/scripts/pr-unblock/request-budget.mjs", + ".github/scripts/pr-unblock/dependency-classify.mjs", + ".github/scripts/apply-patch/apply-patch.mjs", + ".github/scripts/apply-patch/predicates.mjs", + ".github/scripts/apply-patch/protected-paths.mjs", + ".github/scripts/apply-patch/retry-fetch.mjs", ]; test("the floor's enforcement globs are the merge gate's, every other one holds here", () => { assert.deepEqual(REVIEW_FLOOR, HOLD_PATH_CLASSES.filter(([, kind]) => kind === "enforcement").map(([glob]) => glob)); diff --git a/.github/scripts/pr-unblock/address-feedback.mjs b/.github/scripts/pr-unblock/address-feedback.mjs index 706030acd1..33a2dd73ea 100644 --- a/.github/scripts/pr-unblock/address-feedback.mjs +++ b/.github/scripts/pr-unblock/address-feedback.mjs @@ -12,8 +12,8 @@ // skip (no such thread), escalate (the PR changes // .github/workflows/**; or the configured number of rounds already // ran, no write-access fix request came after the newest one, and -// a person gated the PR or a kept thread is a security or intent -// finding, needsPerson, or +// a person gated the PR or a kept thread is one only a person +// decides, needsPerson, or // the fresh fixer's round past the cap already ran) or proceed, // writing the thread file // the skill's review-thread tool reads, with the same-plugin files @@ -38,6 +38,7 @@ import { pathToFileURL } from "node:url"; import { commentItems, gateSource, listLabelEvents, ROUND_MARKER } from "./gate-source.mjs"; import { retryFetch } from "./retry-fetch.mjs"; import { isProtectedPath, namedInScope, prPathsOf } from "./thread-scope.mjs"; +import { retryRung, stopSkip } from "./escalation-routes.mjs"; // The round cap is the config's `loop.review-rounds`, passed in as // REVIEW_ROUNDS; the resolver's default is the fallback. @@ -97,16 +98,22 @@ const lastLine = (body) => export const laneOf = (body) => LANE_FOOTER.exec(lastLine(body))?.[1] ?? null; export const isLaneComment = (comment) => comment.authorId === MARKER_AUTHOR_ID && laneOf(comment.body) !== null; +// A thread whose first comment is a pr-verify verifier note (threadAuthors). +const isVerifierNote = (thread) => thread.comments[0]?.author === "verifier"; + // Past the round cap, the threads only a person may still decide: a security -// lane finding, a verifier note (pr-verify judges the PR against its linked -// issue, so a FAIL is an intent mismatch), and a finding whose lane no footer -// pr-review-cli appended names: only on a github-actions[bot] root -// (`rootAuthorId`) is the footer the reusable's, never model text. Only the -// code lane's finding goes to a fresh fixer. +// lane finding, a verifier note a trusted person directed, and a finding +// whose lane no footer pr-review-cli appended names: only on a +// github-actions[bot] root (`rootAuthorId`) is the footer the reusable's, +// never model text. The code lane's finding and an undirected verifier note +// go to a fresh fixer: a round cap is a lane bound, and pr-verify's FAIL +// already holds the merge and judges the next head afresh. export const needsPerson = (thread, rootAuthorId) => - rootAuthorId !== MARKER_AUTHOR_ID || - thread.comments[0]?.author !== "reviewer" || - laneOf(thread.comments[0].body) !== "claude-review"; + isVerifierNote(thread) + ? thread.comments.some((comment) => comment.direction === true) + : rootAuthorId !== MARKER_AUTHOR_ID || + thread.comments[0]?.author !== "reviewer" || + laneOf(thread.comments[0].body) !== "claude-review"; // Round markers the lanes App posted; any other author's copy is ignored. export const isRoundMarker = (comment, appId) => comment.user?.id === appId && firstLine(comment.body) === ROUND_MARKER; @@ -121,6 +128,8 @@ export function countRounds(issueComments, appId) { // or broker refusal after the push moved the head); a forged one only stops // the lane sooner. export const MODEL_CHECK = "pr-unblock / address-feedback"; +// The reply job's check: a lane stop there runs the head's retry a rung up. +export const REPLY_CHECK = "pr-unblock / publish-feedback"; const isModelRun = (run, conclusions) => run.name === MODEL_CHECK && run.app?.slug === "github-actions" && conclusions.includes(run.conclusion); export function countCheckRounds(checkRunsByCommit) { @@ -155,9 +164,10 @@ export function reviewedHeads(issueComments) { } // The first line of the reply publish-feedback.mjs posts on a thread it -// leaves open for a person: every disposition but VALID. +// leaves open: every disposition but VALID; older threads still carry the +// `for a person` forms. const ESCALATION_REPLY = - /^\*\*pr-unblock \/ address-feedback: (INCORRECT, left open|UNCERTAIN, left open for a person|VALID \(defer\), left open for a person)\*\*$/; + /^\*\*pr-unblock \/ address-feedback: (INCORRECT, left open|UNCERTAIN, left open(?: for a person)?|VALID \(defer\), left open(?: for a person)?)\*\*$/; // The first line of the reply publish-feedback.mjs posts when an adjudicator // finds a disputed finding stands (adjudicate-feedback.mjs): the next round // works that thread again and accepts no second INCORRECT on it. @@ -402,6 +412,19 @@ export function threadAuthors(comments, { reviewerId, appId, trustedHumans = new return authors; } +// Whether a thread threadAuthors accepted (`authors`) sits on a review its +// finder submitted on a marker-confirmed head: the reviewer bot's, or +// github-actions[bot]'s for a verifier note or a lane comment. A trusted +// person's direction (a fix request, or a reply to the lane's escalation) +// asks for the thread, so its review may be on an earlier head. `review` is +// { user: { id }, commit_id }. keepThreads and the dispatcher +// (feedback-dispatch.mjs hasWorkableThread) both judge a thread with this. +export function onReviewedHead({ authors, firstAuthorId, review, heads, reviewerId }) { + const directed = authors.some((a) => a.direction === true); + const reviewAuthor = authors[0].author === "verifier" || firstAuthorId === MARKER_AUTHOR_ID ? MARKER_AUTHOR_ID : reviewerId; + return review?.user?.id === reviewAuthor && (directed || heads.has(review.commit_id)); +} + // A thread is kept when it is unresolved, its first comment belongs to a // review its author (the reviewer bot, or github-actions[bot] for a verifier // note) submitted on a marker-confirmed head (any head for a thread carrying @@ -433,13 +456,8 @@ export function keepThreads({ threads, reviewComments, reviews, heads, reviewerI if (authors === null) { continue; } - // A trusted person's direction (a fix request, or a reply to the lane's - // escalation) asks for the thread, so its review may be on an earlier head. - const directed = authors.some((a) => a.direction === true); const review = reviewsById.get(rows[0].pull_request_review_id); - // A verifier note's or a lane comment's review is github-actions[bot]'s. - const reviewAuthor = authors[0].author === "verifier" || rows[0].user?.id === MARKER_AUTHOR_ID ? MARKER_AUTHOR_ID : reviewerId; - if (review?.user?.id !== reviewAuthor || (!directed && !heads.has(review.commit_id))) { + if (!onReviewedHead({ authors, firstAuthorId: rows[0].user?.id, review, heads, reviewerId })) { continue; } kept.push({ @@ -505,8 +523,18 @@ export async function select({ requesters: gated ? fixers : null, fixers, }); + // An unresolved thread the lane does not take (a reviewer it does not + // trust, a review on an older head, a reply it already gave) would block + // the merge with no stop on record, so it ends in a failing status. + // With no hold and no unresolved thread left, every thread stop is gone + // on this head: `resolved` lets report-stop-status clear its status. if (kept.length === 0) { - return { decision: "skip", skipReason: gated ? "awaiting-human" : "not-applicable", threads: kept }; + if (gated) { + return { decision: "skip", skipReason: "awaiting-human", threads: kept }; + } + return threads.some((thread) => thread.isResolved === false) + ? { decision: "escalate", escalationReason: "feedback-untaken-threads", threads: kept } + : { decision: "skip", skipReason: "not-applicable", resolved: true, threads: kept }; } // The files listing stops at FILES_CAP, so a longer diff could hide a // .github/workflows/ path and is treated as touching one. @@ -540,18 +568,20 @@ export async function select({ }), ); if (worked.length === 0) { - // Otherwise a security or intent thread still goes to a person; every - // other finding goes to a fresh fixer: one more round, a new session - // like every round, reading only the thread file and the PR head. Once - // that round has run, no automatic round ever runs again: the review - // is not converging, which is an intent question for a person. A PR a - // person gated also goes to a person: only a fresh fix request lifts - // their hold. + // Otherwise a thread only a person decides (needsPerson) still goes to + // one; every other finding goes to a fresh fixer: one more round, a + // new session like every round, reading only the thread file and the + // PR head. Once that round has run, no automatic round ever runs + // again, and the head fails: feedback-verifier-open when only verifier + // notes are left, feedback-not-converging otherwise. A PR a person + // gated also goes to a person: only a fresh fix request lifts their + // hold. if (gated || kept.some((thread) => needsPerson(thread, rows.get(thread.root_comment_id)?.user?.id))) { return { decision: "escalate", escalationReason: "feedback-round-cap", rounds, maxRounds: bound, threads: kept }; } if (rounds >= bound) { - return { decision: "escalate", escalationReason: "feedback-not-converging", rounds, maxRounds: bound, threads: kept }; + const escalationReason = kept.every(isVerifierNote) ? "feedback-verifier-open" : "feedback-not-converging"; + return { decision: "escalate", escalationReason, rounds, maxRounds: bound, threads: kept }; } worked = kept; } @@ -564,6 +594,8 @@ export async function select({ } return { decision: "proceed", + // A retry after the reply job stopped on this head runs one rung up. + modelRung: retryRung(await api.listCheckRuns(headSha, REPLY_CHECK)), rounds, maxRounds: bound, // True drops the human-gated label from the write runner's holds: every @@ -720,17 +752,22 @@ export function createGitHubApi({ token, repository, apiUrl, graphqlUrl, fetch: const url = `${repo}/commits/${sha}/check-runs?check_name=${encodeURIComponent(name)}&filter=all&per_page=100`; return (await request(url)).data.check_runs ?? []; }, - // Every check run on a commit, up to 300 (fix-ci-context.mjs). + // Every check run on a commit, re-runs and a contract-only run's skips + // included (failingChecks prefers a real result over a skip), up to 300 + // (fix-ci-context.mjs). async listCommitCheckRuns(sha) { const runs = []; + let total = Infinity; for (let page = 1; page <= 3; page += 1) { - const { data } = await request(`${repo}/commits/${sha}/check-runs?filter=latest&per_page=100&page=${page}`); + const { data } = await request(`${repo}/commits/${sha}/check-runs?filter=all&per_page=100&page=${page}`); runs.push(...(data.check_runs ?? [])); + total = data.total_count ?? Infinity; if ((data.check_runs ?? []).length < 100) { break; } } - return runs; + // complete: the listing holds every check run, not the first 300. + return Object.assign(runs, { complete: runs.length >= total }); }, listAnnotations: async (id) => (await request(`${repo}/check-runs/${id}/annotations?per_page=50`)).data, getJob: async (id) => (await request(`${repo}/actions/jobs/${id}`)).data, @@ -964,10 +1001,12 @@ export async function main(argv, env, { api, log = (line) => console.log(line) } // when this is true: every kept thread carries a write-access fix request, // or a lane escalation put the label on. output("fix-requested", String(result.fixRequested === true)); - output("skip-reason", result.decision === "skip" ? result.skipReason : result.decision === "escalate" ? "awaiting-human" : ""); + output("model-rung", result.modelRung ?? ""); + output("skip-reason", result.decision === "skip" ? result.skipReason : result.decision === "escalate" ? stopSkip(result.escalationReason) : ""); output("escalation-reason", result.escalationReason ?? ""); // The neutral checks' summary line when the PR stopped at the round cap. output("not-run-detail", result.decision === "escalate" && result.maxRounds ? roundLine(result.rounds, result.maxRounds) : ""); + output("stop-resolved", String(result.resolved === true)); log(`${result.decision}: ${result.threads.length} kept thread(s), ${result.rounds ?? 0} earlier round(s)${result.escalationReason ? `, ${result.escalationReason}` : ""}`); return 0; } diff --git a/.github/scripts/pr-unblock/address-feedback.test.mjs b/.github/scripts/pr-unblock/address-feedback.test.mjs index 7c8cbfc40d..ca8ff8bf5c 100644 --- a/.github/scripts/pr-unblock/address-feedback.test.mjs +++ b/.github/scripts/pr-unblock/address-feedback.test.mjs @@ -5,6 +5,8 @@ import { join } from "node:path"; import { test } from "node:test"; import { createGitHubApi, currentThreads, main, maxRoundsFrom, runRecord, select, trustedHumansFrom, VERIFY_NOTE_MARKER } from "./address-feedback.mjs"; +import { failingChecks } from "./gate-source.mjs"; +import { escalationRoute } from "./escalation-routes.mjs"; // Ids from .github/standards/trusted-actors/trusted-actors.json and the // GitHub users API (github-actions[bot] is 41898282). @@ -135,8 +137,8 @@ test("a thread with a human reply is never kept", async () => { const result = await run( fakeApi({ reviewComments: [comment(100, REVIEWER), comment(101, HUMAN)], threads: [thread("PRRT_1", [100, 101])] }), ); - assert.equal(result.decision, "skip"); - assert.equal(result.skipReason, "not-applicable"); + assert.equal(result.decision, "escalate"); + assert.equal(result.escalationReason, "feedback-untaken-threads"); }); // The thread on #7794: the reviewer bot's finding, the lanes App's UNCERTAIN @@ -186,14 +188,14 @@ test("a reply the lane did not leave open, or one not from the lanes App, is no [101, HUMAN, ESCALATION], [101, REVIEWER, ESCALATION], ]) { - assert.equal((await run(fakeApi(escalated({ escalation })))).decision, "skip", escalation[2]); + assert.equal((await run(fakeApi(escalated({ escalation })))).decision, "escalate", escalation[2]); } }); test("an escalation with no reply after it, or with the lane's reply last, is not kept", async () => { - assert.equal((await run(fakeApi(escalated({ replies: [] })))).decision, "skip"); + assert.equal((await run(fakeApi(escalated({ replies: [] })))).decision, "escalate"); const answered = escalated({ replies: [[102, HUMAN, DECISION], [103, APP, ESCALATION]] }); - assert.equal((await run(fakeApi(answered))).decision, "skip"); + assert.equal((await run(fakeApi(answered))).decision, "escalate"); // A second decision after the lane's second escalation is direction; the first stays context. const again = escalated({ replies: [[102, HUMAN, DECISION], [103, APP, ESCALATION], [104, HUMAN, "Do it anyway."]] }); const comments = (await run(fakeApi(again))).file.threads[0].comments; @@ -218,7 +220,7 @@ test("a thread whose finding stood at adjudication goes back to the lane with no // A trusted person's reply after it is direction; anyone else's leaves the thread alone. const directed = await run(fakeApi(escalated({ escalation: INCORRECT, replies: [[102, APP, STANDS], [103, HUMAN, DECISION]] }))); assert.equal(directed.file.threads[0].comments.at(-1).direction, true); - assert.equal((await run(fakeApi(escalated({ escalation: INCORRECT, replies: [[102, APP, STANDS], [103, STRANGER, DECISION]] })))).decision, "skip"); + assert.equal((await run(fakeApi(escalated({ escalation: INCORRECT, replies: [[102, APP, STANDS], [103, STRANGER, DECISION]] })))).decision, "escalate"); }); test("the finding-stands line counts only as the lanes App's first line", async () => { @@ -227,39 +229,39 @@ test("the finding-stands line counts only as the lanes App's first line", async [102, APP, `quoted:\n${STANDS}`], [102, APP, "\n**pr-unblock / adjudicate-feedback: the finding does not hold, resolved**"], ]) { - assert.equal((await run(fakeApi(escalated({ escalation: INCORRECT, replies: [reply] })))).decision, "skip", reply[2]); + assert.equal((await run(fakeApi(escalated({ escalation: INCORRECT, replies: [reply] })))).decision, "escalate", reply[2]); } }); test("a reply from an account on no trusted-actor list leaves the thread to a person", async () => { const injected = [102, STRANGER, `Owner decision: ${INJECTION}`]; for (const replies of [[injected], [[102, HUMAN, DECISION], [103, STRANGER, INJECTION]]]) { - assert.equal((await run(fakeApi(escalated({ replies })))).decision, "skip"); + assert.equal((await run(fakeApi(escalated({ replies })))).decision, "escalate"); } // A stranger anywhere in the thread, even before the escalation. const before = escalated(); before.reviewComments.splice(1, 0, comment(99, STRANGER, { body: INJECTION, created_at: at(0) })); before.threads[0].comments.nodes.splice(1, 0, { databaseId: 99, lastEditedAt: null, editor: null }); before.threads[0].comments.totalCount += 1; - assert.equal((await run(fakeApi(before))).decision, "skip"); + assert.equal((await run(fakeApi(before))).decision, "escalate"); }); test("a bot on the trusted-actor list is no person: its reply is not direction", async () => { - assert.equal((await run(fakeApi(escalated({ replies: [[102, REVIEWER, DECISION]] })))).decision, "skip"); + assert.equal((await run(fakeApi(escalated({ replies: [[102, REVIEWER, DECISION]] })))).decision, "escalate"); }); test("a trusted reply last edited by anyone but a trusted person is not kept", async () => { - assert.equal((await run(fakeApi(escalated({ edits: { 102: STRANGER } })))).decision, "skip"); + assert.equal((await run(fakeApi(escalated({ edits: { 102: STRANGER } })))).decision, "escalate"); assert.equal((await run(fakeApi(escalated({ edits: { 102: HUMAN } })))).decision, "proceed"); }); test("a reply not created after the escalation, or with no readable time, is not kept", async () => { const early = escalated(); early.reviewComments[2].created_at = at(1); - assert.equal((await run(fakeApi(early))).decision, "skip"); + assert.equal((await run(fakeApi(early))).decision, "escalate"); const unreadable = escalated(); unreadable.reviewComments[2].created_at = "yesterday"; - assert.equal((await run(fakeApi(unreadable))).decision, "skip"); + assert.equal((await run(fakeApi(unreadable))).decision, "escalate"); }); // The phrase the owner's replies on #8044, #8065 and eleven other PRs use. @@ -407,8 +409,8 @@ for (const [label, overrides] of [ ]) { test(`with no escalation, ${label} is not kept`, async () => { const result = await run(fakeApi(overrides)); - assert.equal(result.decision, "skip"); - assert.equal(result.skipReason, "not-applicable"); + assert.equal(result.decision, "escalate"); + assert.equal(result.escalationReason, "feedback-untaken-threads"); }); } @@ -456,8 +458,8 @@ for (const [label, overrides] of [ ]) { test(`on an earlier head, ${label} is not kept`, async () => { const result = await run(fakeApi({ ...OLD_REVIEW, ...overrides })); - assert.equal(result.decision, "skip"); - assert.equal(result.skipReason, "not-applicable"); + assert.equal(result.decision, "escalate"); + assert.equal(result.escalationReason, "feedback-untaken-threads"); }); } @@ -474,7 +476,7 @@ test("trustedHumansFrom reads only the people on the list", (t) => { test("a thread a human opened is never kept", async () => { const result = await run(fakeApi({ reviewComments: [comment(100, HUMAN)] })); - assert.equal(result.decision, "skip"); + assert.equal(result.decision, "escalate"); }); test("resolved threads are dropped", async () => { @@ -482,9 +484,45 @@ test("resolved threads are dropped", async () => { assert.equal(result.decision, "skip"); }); +// Review finding on #10252: once a person resolves every thread, a failing +// pr-unblock/address-feedback status (feedback-untaken-threads and the other +// thread stops) must come off the same head; nothing else resolves it. +test("a head with no unresolved thread resolves the address-feedback stop", async () => { + const handResolved = fakeApi({ + reviewComments: [comment(100, REVIEWER), comment(101, HUMAN)], + threads: [thread("PRRT_1", [100, 101], { isResolved: true })], + }); + for (const api of [handResolved, fakeApi({ threads: [] })]) { + const result = await run(api); + assert.deepEqual([result.decision, result.skipReason, result.resolved], ["skip", "not-applicable", true]); + } +}); + +test("an unresolved thread, taken or not, or a person's hold resolves nothing", async () => { + const untaken = await run(fakeApi({ reviewComments: [comment(100, REVIEWER), comment(101, HUMAN)], threads: [thread("PRRT_1", [100, 101])] })); + assert.deepEqual([untaken.escalationReason, untaken.resolved], ["feedback-untaken-threads", undefined]); + const taken = await run(fakeApi()); + assert.deepEqual([taken.decision, taken.resolved], ["proceed", undefined]); + const held = await run(fakeApi({ labels: ["needs-human"], threads: [] })); + assert.deepEqual([held.skipReason, held.resolved], ["awaiting-human", undefined]); +}); + +test("a thread read that fails throws, so main writes no outputs and resolves nothing", async (t) => { + const dir = mkdtempSync(join(tmpdir(), "address-feedback-")); + t.after(() => rmSync(dir, { recursive: true, force: true })); + const actors = join(dir, "actors.json"); + writeFileSync(actors, JSON.stringify({ actors: [{ id: REVIEWER, login: "claude[bot]", kind: "bot" }] })); + const env = { GITHUB_TOKEN: "t", GITHUB_REPOSITORY: "o/r", GITHUB_OUTPUT: join(dir, "out"), PR_NUMBER: "42", HEAD_SHA: HEAD, APP_SENDER_ID: String(APP), THREAD_FILE: join(dir, "t.json"), RUN_FILE: join(dir, "r.json"), TRUSTED_ACTORS: actors }; + const broken = { ...fakeApi(), listThreads: async () => Promise.reject(new Error("GraphQL: timeout")) }; + await assert.rejects(main(["select"], env, { api: broken, log: () => {} }), /timeout/); + assert.throws(() => readFileSync(env.GITHUB_OUTPUT, "utf8")); + await main(["select"], { ...env, GITHUB_OUTPUT: join(dir, "resolved") }, { api: fakeApi({ threads: [] }), log: () => {} }); + assert.match(readFileSync(join(dir, "resolved"), "utf8"), /^stop-resolved=true$/m); +}); + test("a review on a head no marker confirms is dropped", async () => { const result = await run(fakeApi({ reviews: [{ id: 10, user: { id: REVIEWER }, commit_id: OLD_HEAD }] })); - assert.equal(result.decision, "skip"); + assert.equal(result.decision, "escalate"); }); test("a reviewed-head marker from any author but github-actions[bot] confirms nothing", async () => { @@ -493,17 +531,17 @@ test("a reviewed-head marker from any author but github-actions[bot] confirms no { id: 2, user: { id: REVIEWER }, body: marker("claude-review", HEAD) }, { id: 3, user: { id: APP }, body: marker("claude-review", HEAD) }, ]; - assert.equal((await run(fakeApi({ issueComments: forged }))).decision, "skip"); + assert.equal((await run(fakeApi({ issueComments: forged }))).decision, "escalate"); }); test("a marker for a lane other than the two review lanes confirms nothing", async () => { const other = [{ id: 1, user: { id: ACTIONS_BOT }, body: marker("other-lane", HEAD) }]; - assert.equal((await run(fakeApi({ issueComments: other }))).decision, "skip"); + assert.equal((await run(fakeApi({ issueComments: other }))).decision, "escalate"); }); test("a comment whose review another account submitted is dropped", async () => { const result = await run(fakeApi({ reviews: [{ id: 10, user: { id: HUMAN }, commit_id: HEAD }] })); - assert.equal(result.decision, "skip"); + assert.equal(result.decision, "escalate"); }); // verify-notes.mjs posts a pr-verify FAIL's notes as github-actions[bot] @@ -527,11 +565,11 @@ test("a github-actions[bot] thread without the note marker, or in another accoun reviewComments: [comment(100, ACTIONS_BOT)], reviews: [{ id: 10, user: { id: ACTIONS_BOT }, commit_id: HEAD }], }); - assert.equal((await run(unmarked)).decision, "skip"); - assert.equal((await run(verifierNote({ reviewer: REVIEWER }))).decision, "skip"); + assert.equal((await run(unmarked)).decision, "escalate"); + assert.equal((await run(verifierNote({ reviewer: REVIEWER }))).decision, "escalate"); // The marker from any other author opens nothing. const forged = fakeApi({ reviewComments: [comment(100, HUMAN, { body: VERIFY_NOTE })] }); - assert.equal((await run(forged)).decision, "skip"); + assert.equal((await run(forged)).decision, "escalate"); }); // The pr-review-cli reusable posts each finding as github-actions[bot], the @@ -576,24 +614,24 @@ test("a github-actions[bot] comment whose last line is not a review lane's foote "claude-lane: claude-security-review\nThe footer is not the last line.", "The input reaches the shell.\n\nclaude-lane: pr-verify", ]) { - assert.equal((await run(laneFinding(body))).decision, "skip", body); + assert.equal((await run(laneFinding(body))).decision, "escalate", body); } // The footer on another account's comment, or in a review another account // submitted, opens nothing. const human = fakeApi({ reviewComments: [comment(100, HUMAN, { body: LANE_FINDING })] }); - assert.equal((await run(human)).decision, "skip"); - assert.equal((await run(laneFinding(LANE_FINDING, REVIEWER))).decision, "skip"); + assert.equal((await run(human)).decision, "escalate"); + assert.equal((await run(laneFinding(LANE_FINDING, REVIEWER))).decision, "escalate"); }); test("a later github-actions[bot] comment on a verifier note is no finding", async () => { const result = await run(verifierNote({ replies: [comment(101, ACTIONS_BOT, { body: VERIFY_NOTE })] })); - assert.equal(result.decision, "skip"); + assert.equal(result.decision, "escalate"); }); test("a thread longer than one comment page is left alone", async () => { const big = thread("PRRT_1", [100]); big.comments.totalCount = 101; - assert.equal((await run(fakeApi({ threads: [big] }))).decision, "skip"); + assert.equal((await run(fakeApi({ threads: [big] }))).decision, "escalate"); }); test("a PR that changes .github/workflows/** escalates feedback-protected-path before any model turn", async () => { @@ -769,7 +807,7 @@ test("at the cap, the code lane's findings go to a fresh fixer: every thread run for (const [label, roots] of [ ["a security lane finding", [LANE_FINDING]], ["a security lane finding beside a code lane one", [CODE_FINDING, LANE_FINDING]], - ["a verifier note, an intent mismatch", [VERIFY_NOTE]], + ["a security lane finding beside a verifier note", [VERIFY_NOTE, LANE_FINDING]], ]) { for (const rounds of [3, 4, 9]) { test(`after ${rounds} rounds at a cap of 3, ${label} still escalates feedback-round-cap`, async () => { @@ -809,6 +847,36 @@ test("once the fresh fixer's round ran, the code lane's findings escalate feedba assert.equal(fresh.maxRounds, 4); }); +test("at the cap, a verifier note goes to the fresh fixer as a code lane finding does, and past it fails the head instead of holding for a person", async () => { + for (const roots of [[VERIFY_NOTE], [VERIFY_NOTE, CODE_FINDING]]) { + const fresh = await capped({ rounds: 3, roots }); + assert.equal(fresh.decision, "proceed"); + assert.equal(fresh.escalationReason, undefined); + assert.deepEqual(fresh.file.threads.map((row) => row.id), roots.map((_, i) => `PRRT_${i + 1}`)); + } + for (const rounds of [4, 9]) { + const notes = await capped({ rounds, roots: [VERIFY_NOTE, `${VERIFY_NOTE}\nAnother note.`] }); + assert.equal(notes.decision, "escalate"); + assert.equal(notes.escalationReason, "feedback-verifier-open"); + assert.equal(escalationRoute(notes.escalationReason), "fail"); + assert.equal(notes.maxRounds, 4); + assert.equal(notes.file, undefined); + const mixed = await capped({ rounds, roots: [VERIFY_NOTE, CODE_FINDING] }); + assert.equal(mixed.escalationReason, "feedback-not-converging"); + assert.equal(escalationRoute(mixed.escalationReason), "fail"); + } +}); + +test("at the cap, a verifier note a trusted person directed still holds for a person", async () => { + for (const rounds of [3, 4]) { + const result = await capped({ rounds, roots: [VERIFY_NOTE], fixRequest: true, requestAt: at(5) }); + assert.equal(result.decision, "escalate"); + assert.equal(result.escalationReason, "feedback-round-cap"); + assert.equal(escalationRoute(result.escalationReason), "hold"); + assert.equal(result.file, undefined); + } +}); + test("past the fresh fixer's round, a write-access fix request still runs one more round on its thread", async () => { const result = await capped({ rounds: 4, roots: [CODE_FINDING, CODE_FINDING], fixRequest: true }); assert.equal(result.decision, "proceed"); @@ -866,7 +934,7 @@ test("main select writes outputs and the thread file; escalation writes no threa TRUSTED_ACTORS: actors, }); await main(["select"], env("ok"), { api: fakeApi(), log: () => {} }); - assert.equal(readFileSync(join(dir, "ok.out"), "utf8"), "proceed=true\nfix-requested=false\nskip-reason=\nescalation-reason=\nnot-run-detail=\n"); + assert.equal(readFileSync(join(dir, "ok.out"), "utf8"), "proceed=true\nfix-requested=false\nmodel-rung=\nskip-reason=\nescalation-reason=\nnot-run-detail=\nstop-resolved=false\n"); const okFile = JSON.parse(readFileSync(join(dir, "ok.json"), "utf8")); assert.equal(okFile.threads[0].id, "PRRT_1"); // No lane branch: the model job commits nothing. @@ -881,7 +949,7 @@ test("main select writes outputs and the thread file; escalation writes no threa // The people on the list come from TRUSTED_ACTORS. await main(["select"], env("decided"), { api: fakeApi(escalated()), log: () => {} }); - assert.equal(readFileSync(join(dir, "decided.out"), "utf8"), "proceed=true\nfix-requested=false\nskip-reason=\nescalation-reason=\nnot-run-detail=\n"); + assert.equal(readFileSync(join(dir, "decided.out"), "utf8"), "proceed=true\nfix-requested=false\nmodel-rung=\nskip-reason=\nescalation-reason=\nnot-run-detail=\nstop-resolved=false\n"); assert.equal(JSON.parse(readFileSync(join(dir, "decided.json"), "utf8")).threads[0].comments[2].direction, true); // The human-gated label comes from TRACKER_CONFIG; a fix request sets fix-requested. @@ -889,15 +957,15 @@ test("main select writes outputs and the thread file; escalation writes no threa writeFileSync(tracker, JSON.stringify({ config: { role_labels: { "human-gated": "status: blocked-human" } } })); const gatedEnv = { ...env("gated"), TRACKER_CONFIG: tracker }; await main(["select"], gatedEnv, { api: fakeApi({ ...escalated({ replies: [[102, HUMAN, REQUEST]] }), labels: ["status: blocked-human"] }), log: () => {} }); - assert.equal(readFileSync(join(dir, "gated.out"), "utf8"), "proceed=true\nfix-requested=true\nskip-reason=\nescalation-reason=\nnot-run-detail=\n"); + assert.equal(readFileSync(join(dir, "gated.out"), "utf8"), "proceed=true\nfix-requested=true\nmodel-rung=\nskip-reason=\nescalation-reason=\nnot-run-detail=\nstop-resolved=false\n"); const heldEnv = { ...env("held"), TRACKER_CONFIG: tracker }; await main(["select"], heldEnv, { api: fakeApi({ ...escalated(), labels: ["status: blocked-human"] }), log: () => {} }); - assert.equal(readFileSync(join(dir, "held.out"), "utf8"), "proceed=false\nfix-requested=false\nskip-reason=awaiting-human\nescalation-reason=\nnot-run-detail=\n"); + assert.equal(readFileSync(join(dir, "held.out"), "utf8"), "proceed=false\nfix-requested=false\nmodel-rung=\nskip-reason=awaiting-human\nescalation-reason=\nnot-run-detail=\nstop-resolved=false\n"); await main(["select"], env("esc"), { api: fakeApi({ files: [{ filename: ".github/workflows/x.yml" }] }), log: () => {} }); assert.equal( readFileSync(join(dir, "esc.out"), "utf8"), - "proceed=false\nfix-requested=false\nskip-reason=awaiting-human\nescalation-reason=feedback-protected-path\nnot-run-detail=\n", + "proceed=false\nfix-requested=false\nmodel-rung=\nskip-reason=awaiting-human\nescalation-reason=feedback-protected-path\nnot-run-detail=\nstop-resolved=false\n", ); assert.throws(() => readFileSync(join(dir, "esc.json"))); assert.throws(() => readFileSync(join(dir, "esc.run.json")), "an escalation writes no run.json"); @@ -915,7 +983,7 @@ test("main select writes outputs and the thread file; escalation writes no threa await main(["select"], { ...env("bound"), REVIEW_ROUNDS: "1" }, { api: bound, log: () => {} }); assert.equal( readFileSync(join(dir, "bound.out"), "utf8"), - "proceed=false\nfix-requested=false\nskip-reason=awaiting-human\nescalation-reason=feedback-not-converging\nnot-run-detail=round 2 of max 2\n", + "proceed=false\nfix-requested=false\nmodel-rung=\nskip-reason=lane-stopped\nescalation-reason=feedback-not-converging\nnot-run-detail=round 2 of max 2\nstop-resolved=false\n", ); }); @@ -982,6 +1050,40 @@ test("createGitHubApi follows rel=next pages and sends the token", async () => { ); }); +// A contract-only pr-require-checks run posts a lane skipped after the real +// run failed it. A latest-only listing returns only the skip, which hid the +// failure from failingChecks and let stop-resolved clear pr-unblock/fix-ci. +const SKIP_OVER_FAILURE = (url) => { + const failed = { id: 1, name: "test-node", status: "completed", conclusion: "failure" }; + const skipped = { id: 2, name: "test-node", status: "completed", conclusion: "skipped" }; + const filter = new URL(url).searchParams.get("filter"); + return filter === "all" ? [failed, skipped] : [skipped]; +}; + +test("listCommitCheckRuns lists every run, so a later skip does not hide a lane's failure", async () => { + const fetch = async (url) => { + const check_runs = SKIP_OVER_FAILURE(url); + return { ok: true, status: 200, json: async () => ({ total_count: check_runs.length, check_runs }), headers: { get: () => "" } }; + }; + const runs = await createGitHubApi({ token: "t", repository: "o/r", apiUrl: "https://api", graphqlUrl: "https://gql", fetch }).listCommitCheckRuns(HEAD); + assert.deepEqual(failingChecks(runs).map((run) => run.id), [1]); +}); + +test("listCommitCheckRuns marks a listing past its three pages, or with no total, as not complete", async () => { + const listing = (total, sizes) => { + const fetch = async (url) => { + const page = Number(new URL(url).searchParams.get("page")); + const check_runs = Array.from({ length: sizes[page - 1] ?? 0 }, (_, i) => ({ id: page * 1000 + i })); + return { ok: true, status: 200, json: async () => ({ ...(total === null ? {} : { total_count: total }), check_runs }), headers: { get: () => "" } }; + }; + return createGitHubApi({ token: "t", repository: "o/r", apiUrl: "https://api", graphqlUrl: "https://gql", fetch }).listCommitCheckRuns(HEAD); + }; + assert.equal((await listing(3, [3])).complete, true); + assert.equal((await listing(150, [100, 50])).complete, true); + assert.equal((await listing(301, [100, 100, 100])).complete, false); + assert.equal((await listing(null, [3])).complete, false); +}); + // A read, a GraphQL query and a write that repeats to the same state survive // one dropped socket; an issue and a mutation are sent once. test("createGitHubApi sends reads, GraphQL queries and repeatable writes again after a dropped socket, and an issue or a mutation once", async () => { @@ -1094,7 +1196,7 @@ test("a name from a trusted person without write access is not added", async () test("a name from an untrusted account leaves the thread alone, so nothing is added", async () => { const { api } = named([[101, STRANGER, OWNER_8116]], { permissions: { stranger: "admin" } }); - assert.equal((await run(api)).decision, "skip"); + assert.equal((await run(api)).decision, "escalate"); }); test("a name that is no regular file at the head is not added: missing, a symbolic link, a gitlink, or another letter case", async () => { @@ -1153,6 +1255,49 @@ test("a leading or trailing _ that belongs to the name is kept; only a matched _ assert.deepEqual([...result.file.threads[0].named_paths].sort(), [`${T}/__init__.py`, `${T}/_utils.js`, `${T}/run-transcript.test.js`]); }); +// A retry after the reply job stopped on this head (a lane-stopped +// publish-feedback check from the GitHub Actions app) runs one model rung up; +// a first round, or a stop record from another app, runs at the profile's own. +test("select asks for model rung 1 only after a lane stop on the head's reply check", async () => { + const stop = { status: "completed", conclusion: "neutral", output: { title: "Skipped: lane-stopped" }, app: { id: 15368 } }; + const withReply = (replyRuns) => { + const api = fakeApi(); + const base = api.listCheckRuns; + return { ...api, listCheckRuns: async (sha, name) => (name === "pr-unblock / publish-feedback" ? replyRuns : base(sha, name)) }; + }; + const rungOf = async (replyRuns) => (await select({ api: withReply(replyRuns), pr: 42, headSha: HEAD, repository: "o/r", appId: APP, reviewerId: REVIEWER, maxRounds: 3 })).modelRung; + assert.equal(await rungOf([stop]), "1"); + assert.equal(await rungOf([]), ""); + assert.equal(await rungOf([{ ...stop, app: { id: 1 } }]), ""); + assert.equal(await rungOf([{ ...stop, output: { title: "Skipped: no-model-turn" } }]), ""); +}); + +// #10092's silent stall: an unresolved thread the lane does not take left +// the head with a neutral not-applicable skip, no stop and no signal. The +// select job now ends it in feedback-untaken-threads (routed to a failing +// status); only a PR with no unresolved thread still skips. +const CODEX = 199175422; +test("an unresolved thread the lane does not take escalates feedback-untaken-threads", async () => { + const codex = await run( + fakeApi({ + reviewComments: [comment(100, CODEX)], + reviews: [{ id: 10, user: { id: CODEX }, commit_id: HEAD }], + }), + ); + assert.deepEqual([codex.decision, codex.escalationReason], ["escalate", "feedback-untaken-threads"], "a Codex-root thread"); + const stale = await run( + fakeApi({ + reviewComments: [comment(100, ACTIONS_BOT, { body: "The loop skips the last item.\n\nclaude-lane: claude-review" })], + reviews: [{ id: 10, user: { id: ACTIONS_BOT }, commit_id: OLD_HEAD }], + }), + ); + assert.deepEqual([stale.decision, stale.escalationReason], ["escalate", "feedback-untaken-threads"], "a lane thread on an older head"); + const none = await run(fakeApi({ threads: [thread("PRRT_1", [100], { isResolved: true })] })); + assert.deepEqual([none.decision, none.skipReason], ["skip", "not-applicable"], "no unresolved thread"); + const held = await run(fakeApi({ labels: ["needs-human"], reviewComments: [comment(100, CODEX)], reviews: [{ id: 10, user: { id: CODEX }, commit_id: HEAD }] })); + assert.deepEqual([held.decision, held.skipReason], ["skip", "awaiting-human"], "a person holds the PR"); +}); + // pr-feedback-dispatch.yml sends verify-pass-head after a pr-verify pass at // the PR's head; the select job then lists only the stale fail-note threads. test("main select with a pass head lists the stale fail-note threads for the reply job and keeps none for a model", async (t) => { diff --git a/.github/scripts/pr-unblock/adjudicate-feedback.mjs b/.github/scripts/pr-unblock/adjudicate-feedback.mjs index 87778fae8f..86ab094f3f 100755 --- a/.github/scripts/pr-unblock/adjudicate-feedback.mjs +++ b/.github/scripts/pr-unblock/adjudicate-feedback.mjs @@ -1,6 +1,7 @@ #!/usr/bin/env node // adjudicate-feedback.mjs: the pr-unblock lane's adjudication of a disputed -// review finding, one an address-feedback round answered INCORRECT. A +// review finding, one an address-feedback round answered INCORRECT, UNCERTAIN +// or VALID-defer (DISPUTED): any reply that leaves the thread open. A // fresh-context model sees the whole review comment, the locations both the // reviewer and the lane cited and those files at the head, never the lane's // reasoning, and returns a verdict as structured output; this file stages its @@ -10,7 +11,7 @@ // with the job's read-only token, before any model turn: reads the // select job's threads.json and the scan job's replies.jsonl, each // checked against the SHA-256 its writer recorded, and for each -// INCORRECT reply on an adjudicable thread no adjudication has +// DISPUTED reply on an adjudicable thread no adjudication has // stood before and no trusted person directed (up to MAX_THREADS) // writes // //: finding.json (path, line, head, and the @@ -24,7 +25,7 @@ // entry per indexed thread; an absent or malformed verdict is // null. Outputs `sha256`. // -// settleDisputes is publish-feedback.mjs's INCORRECT branch: after the lane's +// settleDisputes is publish-feedback.mjs's DISPUTED branch: after the lane's // replies are posted it applies each verdict (adjudicationReplies, fixed text // only) and returns the round's escalation reason. @@ -39,6 +40,13 @@ import { createGitHubApi, FINDING_STANDS, need, SHA } from "./address-feedback.m // model step per slot. A thread past it is not adjudicated, and the round // escalates feedback-unadjudicated. export const MAX_THREADS = 3; +// The dispositions that leave a thread open, each one an adjudicator judges +// on an adjudicable thread instead of a person. Only an INCORRECT reply (the +// lane asserted, with evidence, that the finding does not hold) can be +// resolved by a lane verdict; on UNCERTAIN or VALID-defer only a reviewer or +// human verdict settles it, so a model never closes a thread the lane did not +// dispute (settleDisputes). +export const DISPUTED = Object.freeze(["INCORRECT", "UNCERTAIN", "VALID-defer"]); export const VERDICTS = ["lane", "reviewer", "human"]; export const QUESTIONS = ["intent", "design", "security"]; // The first line of each adjudication reply the lanes App posts: this, or @@ -122,7 +130,7 @@ function readChecked(path, expected, what) { return text; } -// The slots to adjudicate: each INCORRECT reply whose thread the select job +// The slots to adjudicate: each DISPUTED reply whose thread the select job // kept, is adjudicable, no adjudication has stood and no trusted person // directed, in reply order, one slot per thread. Returns [{thread, finding, // evidence, cited, paths}] and the threads past MAX_THREADS. @@ -134,7 +142,7 @@ export function disputes(threadFile, repliesText, repository = "") { for (const line of repliesText.split("\n").filter((row) => row.trim() !== "")) { const reply = JSON.parse(line); const thread = kept.get(reply?.thread); - if (reply?.disposition !== "INCORRECT" || !thread || !THREAD_ID.test(thread.id) || seen.has(thread.id) || !adjudicable(thread.comments) || stood(thread.comments) || directed(thread.comments)) { + if (!DISPUTED.includes(reply?.disposition) || !thread || !THREAD_ID.test(thread.id) || seen.has(thread.id) || !adjudicable(thread.comments) || stood(thread.comments) || directed(thread.comments)) { continue; } seen.add(thread.id); @@ -266,45 +274,59 @@ export function adjudicationReply(verdict, evidence, headSha) { FINDING_STANDS, "**pr-unblock / adjudicate-feedback: the finding stands**", "", - `${seen} and found that the finding holds. The next address-feedback round fixes it; another INCORRECT reply on this thread is not accepted and hands the pull request to a person.`, + `${seen} and found that the finding holds. The next address-feedback round fixes it; a round that leaves this thread open again ends the lane in a failing status.`, "", evidenceLine(evidence), ].join("\n"); } // The round's escalation reasons, most urgent first; one round records one. +// A reason escalation-routes.json holds for a person outranks every reason it +// routes to a failing status, so a question for a person is never lost. const ORDER = [ - "feedback-partial", - "feedback-unadjudicated", "feedback-question-security", "feedback-question-intent", "feedback-question-design", "feedback-disputed", "feedback-uncertain", + "feedback-partial", + "feedback-unadjudicated", + "feedback-stood-open", + "feedback-verifier-open", "", ]; +// Whether a thread is a pr-verify verifier note (threadAuthors tags it). +const verifierNote = (comments) => comments?.[0]?.author === "verifier"; + // The reply job's skip reason for a round that left a finding standing with // the PR head where the select job read it: no review run follows to start // the round that fixes it, so feedback-dispatch.mjs's sweep dispatches that // head again. export const FINDING_STANDS_SKIP = "finding-stands"; -// publish-feedback.mjs's INCORRECT branch, after the lane's replies posted: +// publish-feedback.mjs's DISPUTED branch, after the lane's replies posted: // - a thread an adjudication stood before (stood) that this round left open -// is the cap: feedback-disputed, and no second adjudication; -// - an INCORRECT thread a trusted person directed (directed) is theirs: +// is the cap: feedback-stood-open, and no second adjudication; +// - a DISPUTED thread a trusted person directed (directed) is theirs: // feedback-disputed, and no adjudication; -// - an INCORRECT thread that is not adjudicable (a CRITICAL or -// security-review finding, a verifier note, or one of unknown lane) goes to -// a person: feedback-disputed, and any verdict for it is ignored; -// - an INCORRECT thread with no verdict escalates feedback-unadjudicated; +// - a DISPUTED pr-verify verifier note: feedback-verifier-open (pr-verify's +// own FAIL holds the merge, and it judges the next head afresh); +// - any other DISPUTED thread that is not adjudicable (a CRITICAL or +// security-review finding, or one of unknown lane) goes to a person: +// feedback-disputed when the reply is INCORRECT, else feedback-uncertain, +// and any verdict for it is ignored; +// - a DISPUTED thread with no verdict escalates feedback-unadjudicated; // - human: no reply, feedback-question-; a question // only the owner answers stays the owner's; -// - lane: the fixed reply, then the thread is resolved; -// - reviewer: the fixed FINDING_STANDS reply; when the head did not move -// this round (`pushed` false) and nothing escalates, the round skips -// FINDING_STANDS_SKIP. +// - lane: on an INCORRECT reply, the fixed reply, then the thread is +// resolved; on an UNCERTAIN or VALID-defer reply (the lane could not tell, +// or agreed the finding holds) no reply, the thread stays open (the +// ruleset's thread resolution still holds the merge), and +// feedback-unadjudicated; +// - reviewer: the fixed FINDING_STANDS reply, which the next round reads as +// a finding to fix; when the head did not move this round (`pushed` +// false) and nothing escalates, the round skips FINDING_STANDS_SKIP. // A thread that changed, a moved head or a failed write stops here as // feedback-partial. Returns {escalationReason, skipReason}. export async function settleDisputes({ api, pr, adjudicatedSha, posted, replies, shown, now, verdicts, pushed, headMoved, resolveAfterReply, escalationReason, log }) { @@ -317,24 +339,30 @@ export async function settleDisputes({ api, pr, adjudicatedSha, posted, replies, }; const byThread = new Map(replies.map((reply) => [reply.thread, reply])); for (const { thread, disposition } of posted) { - if (stood(now.get(thread)?.comments)) { + const comments = now.get(thread)?.comments; + if (stood(comments)) { if (disposition !== "VALID") { log(`${thread}: the finding stood at adjudication and this round left it open (${disposition})`); - raise("feedback-disputed"); + raise("feedback-stood-open"); } continue; } - if (disposition !== "INCORRECT") { + if (!DISPUTED.includes(disposition)) { continue; } - if (directed(now.get(thread)?.comments)) { + if (directed(comments)) { log(`${thread}: a trusted person directed this thread; it goes back to them, not to an adjudicator`); raise("feedback-disputed"); continue; } - if (!adjudicable(now.get(thread)?.comments)) { + if (verifierNote(comments)) { + log(`${thread}: a pr-verify note left open (${disposition}); pr-verify judges the next head`); + raise("feedback-verifier-open"); + continue; + } + if (!adjudicable(comments)) { log(`${thread}: not a non-CRITICAL code-review finding; it goes to a person, not to an adjudicator`); - raise("feedback-disputed"); + raise(disposition === "INCORRECT" ? "feedback-disputed" : "feedback-uncertain"); continue; } const verdict = verdicts.get(thread); @@ -348,6 +376,14 @@ export async function settleDisputes({ api, pr, adjudicatedSha, posted, replies, raise(`feedback-question-${verdict.question}`); continue; } + // Only an INCORRECT reply is the lane disputing the finding with + // evidence. On UNCERTAIN or VALID-defer a lane verdict alone would let one + // model turn close a finding no one disputed, so the thread stays open. + if (verdict.verdict === "lane" && disposition !== "INCORRECT") { + log(`${thread}: a lane verdict settles only an INCORRECT reply, not ${disposition}; left open`); + raise("feedback-unadjudicated"); + continue; + } try { const ids = shown.get(thread).comment_ids; const before = await api.readThread(thread); diff --git a/.github/scripts/pr-unblock/adjudicate-feedback.prompt.md b/.github/scripts/pr-unblock/adjudicate-feedback.prompt.md index c7e8166e81..e70e726f6f 100644 --- a/.github/scripts/pr-unblock/adjudicate-feedback.prompt.md +++ b/.github/scripts/pr-unblock/adjudicate-feedback.prompt.md @@ -1,8 +1,8 @@ # Adjudicate one disputed review finding -A review lane posted a finding on a pull request. The lane that fixes findings answered that the -finding is incorrect and cited locations in the code as its evidence. You decide which side the -code supports. You see the whole review comment, the locations each side cited and the files at +A review lane posted a finding on a pull request. The lane that fixes findings left it open: it +answered that the finding is incorrect, that it could not tell, or that the fix belongs elsewhere, +and may have cited locations in the code as its evidence. You decide which side the code supports. You see the whole review comment, the locations each side cited and the files at the pull request head. You do not see the fixing lane's reasoning. No one can answer a question, and you change nothing: your structured output is the only result, and a script acts on it. diff --git a/.github/scripts/pr-unblock/adjudicate-feedback.test.mjs b/.github/scripts/pr-unblock/adjudicate-feedback.test.mjs index bebfa753d2..15a0092b38 100644 --- a/.github/scripts/pr-unblock/adjudicate-feedback.test.mjs +++ b/.github/scripts/pr-unblock/adjudicate-feedback.test.mjs @@ -5,7 +5,8 @@ import { tmpdir } from "node:os"; import { join } from "node:path"; import { test } from "node:test"; -import { adjudicationReply, collect, disputes, MAX_THREADS, main, parseVerdicts, prepare, reviewerEvidence, settleDisputes, verdictOf } from "./adjudicate-feedback.mjs"; +import { adjudicationReply, collect, DISPUTED, disputes, MAX_THREADS, main, parseVerdicts, prepare, reviewerEvidence, settleDisputes, verdictOf } from "./adjudicate-feedback.mjs"; +import { escalationRoute } from "./escalation-routes.mjs"; import { noteComments } from "./verify-notes.mjs"; const HEAD = "a".repeat(40); @@ -116,26 +117,68 @@ test("disputes never adjudicates a CRITICAL code-review finding or a verifier no assert.deepEqual(disputes(fileOf(important), incorrect("PRRT_i")).slots.map((slot) => slot.thread), ["PRRT_i"]); }); -test("settleDisputes sends an INCORRECT reply on a CRITICAL code-review finding or a verifier note to a person, whatever verdict arrived", async () => { - const api = new Proxy({}, { get: (_, name) => () => assert.fail(`no API call expected, got ${String(name)}`) }); - for (const now of [new Map([["PRRT_a", criticalThread("PRRT_a")]]), new Map([["PRRT_a", verifierThread("PRRT_a")]])]) { - const result = await settleDisputes({ - api, - pr: 42, - adjudicatedSha: HEAD, - posted: [{ thread: "PRRT_a", disposition: "INCORRECT" }], - replies: [], - shown: new Map([["PRRT_a", { comment_ids: [1, 2] }]]), - now, - verdicts: new Map([["PRRT_a", { verdict: "lane", question: "none" }]]), - pushed: false, - headMoved: async () => false, - resolveAfterReply: async () => assert.fail("not resolved"), - escalationReason: "", - log: () => {}, - }); - assert.deepEqual(result, { escalationReason: "feedback-disputed", skipReason: "" }); +// settleDisputes with no API call allowed, on one posted thread. +const settleOffline = (now, disposition, verdicts = new Map([["PRRT_a", { verdict: "lane", question: "none" }]]), escalationReason = "") => + settleDisputes({ + api: new Proxy({}, { get: (_, name) => () => assert.fail(`no API call expected, got ${String(name)}`) }), + pr: 42, + adjudicatedSha: HEAD, + posted: [{ thread: "PRRT_a", disposition }], + replies: [], + shown: new Map([["PRRT_a", { comment_ids: [1, 2] }]]), + now, + verdicts, + pushed: false, + headMoved: async () => false, + resolveAfterReply: async () => assert.fail("not resolved"), + escalationReason, + log: () => {}, + }); + +test("settleDisputes holds a CRITICAL code-review finding left open for a person, whatever verdict arrived: disputed when INCORRECT, else uncertain", async () => { + const now = new Map([["PRRT_a", criticalThread("PRRT_a")]]); + assert.deepEqual(await settleOffline(now, "INCORRECT"), { escalationReason: "feedback-disputed", skipReason: "" }); + for (const disposition of ["UNCERTAIN", "VALID-defer"]) { + assert.deepEqual(await settleOffline(now, disposition), { escalationReason: "feedback-uncertain", skipReason: "" }, disposition); + } +}); + +test("settleDisputes ends a verifier note left open in feedback-verifier-open, a failing status and no person, whatever verdict arrived", async () => { + const now = new Map([["PRRT_a", verifierThread("PRRT_a")]]); + for (const disposition of DISPUTED) { + assert.deepEqual(await settleOffline(now, disposition), { escalationReason: "feedback-verifier-open", skipReason: "" }, disposition); + } +}); + +test("a reason held for a person outranks every fail-routed reason in one round", async () => { + const now = new Map([["PRRT_a", criticalThread("PRRT_a")]]); + assert.deepEqual(await settleOffline(now, "UNCERTAIN", undefined, "feedback-partial"), { escalationReason: "feedback-uncertain", skipReason: "" }); + const verifier = new Map([["PRRT_a", verifierThread("PRRT_a")]]); + assert.deepEqual(await settleOffline(verifier, "UNCERTAIN", undefined, "feedback-partial"), { escalationReason: "feedback-partial", skipReason: "" }); +}); + +// Only an INCORRECT reply is the lane disputing the finding with evidence. +// On UNCERTAIN (the lane could not tell) or VALID-defer (it agreed the finding +// holds), a lane verdict would let one model turn close a finding no one +// disputed (#10252 security review), so neither settles it: no reply, the +// thread stays open (the ruleset's thread resolution still holds the merge) +// and the round ends in a failing status. settleOffline fails on any API call +// or resolve. +test("settleDisputes resolves only an INCORRECT finding on a lane verdict: UNCERTAIN and VALID-defer stay open as feedback-unadjudicated", async () => { + const now = new Map([["PRRT_a", thread("PRRT_a")]]); + for (const disposition of ["UNCERTAIN", "VALID-defer"]) { + assert.deepEqual(await settleOffline(now, disposition), { escalationReason: "feedback-unadjudicated", skipReason: "" }, disposition); + // A person's reason in the same round still outranks it. + assert.deepEqual(await settleOffline(now, disposition, undefined, "feedback-uncertain"), { escalationReason: "feedback-uncertain", skipReason: "" }, disposition); + } +}); + +test("disputes stages an UNCERTAIN or VALID-defer reply on an adjudicable thread as it does an INCORRECT one", () => { + for (const disposition of ["UNCERTAIN", "VALID-defer"]) { + const { slots } = disputes(fileOf(thread("PRRT_a"), criticalThread("PRRT_c")), [incorrect("PRRT_a", { disposition }), incorrect("PRRT_c", { disposition })].join("\n")); + assert.deepEqual(slots.map((slot) => slot.thread), ["PRRT_a"], disposition); } + assert.deepEqual(disputes(fileOf(thread("PRRT_a")), incorrect("PRRT_a", { disposition: "VALID" })).slots, []); }); test("the adjudication reply says what the adjudicator read: the review comment, not the lane's reasoning", () => { @@ -248,6 +291,25 @@ test("settleDisputes sends an INCORRECT reply on a security-review or unknown-la } }); +// The floor for a reply the lane could not settle: on a security-review, +// unknown-lane or claude[bot] finding an UNCERTAIN or VALID-defer reply holds +// for a person whatever verdict arrived, and that reason routes `hold` (the +// needs-human label), not a failing status a new head would clear. +test("settleDisputes holds an UNCERTAIN or VALID-defer security-review or unknown-lane finding for a person, whatever verdict arrived", async () => { + for (const [label, make] of [["security-review", securityThread], ["unknown lane", unnamedThread], ["claude[bot]", claudeBotThread]]) { + for (const disposition of ["UNCERTAIN", "VALID-defer"]) { + for (const verdict of ["lane", "reviewer"]) { + const now = new Map([["PRRT_a", make("PRRT_a")]]); + const verdicts = new Map([["PRRT_a", { verdict, question: "none" }]]); + assert.deepEqual(await settleOffline(now, disposition, verdicts), { escalationReason: "feedback-uncertain", skipReason: "" }, `${label} ${disposition} ${verdict}`); + } + } + assert.deepEqual(disputes(fileOf(make("PRRT_s")), ["UNCERTAIN", "VALID-defer"].map((disposition) => incorrect("PRRT_s", { disposition })).join("\n")).slots, [], `${label}: no slot`); + } + assert.equal(escalationRoute("feedback-uncertain"), "hold"); + assert.equal(escalationRoute("feedback-disputed"), "hold"); +}); + test("prepare writes one slot per dispute with the files at the head, and refuses another head's thread file", async (t) => { const dir = mkdtempSync(join(tmpdir(), "adjudicate-")); t.after(() => rmSync(dir, { recursive: true, force: true })); diff --git a/.github/scripts/pr-unblock/dispatch.mjs b/.github/scripts/pr-unblock/dispatch.mjs index c55b888e9c..c8092d6fd1 100644 --- a/.github/scripts/pr-unblock/dispatch.mjs +++ b/.github/scripts/pr-unblock/dispatch.mjs @@ -38,6 +38,7 @@ import { pathToFileURL } from "node:url"; import { loadTrustedList } from "../../actions/check-trusted-trigger/check-trusted-trigger.mjs"; import { testedRun } from "../maintenance-reconcile-lost-runs/infra-failure.mjs"; +import { LIMIT_DEFAULTS } from "../work-resolve-issue/limits.mjs"; import { DEPENDABOT_ID, POLICY_PATH } from "./dependency-classify.mjs"; import { dispatchDependencyReviews } from "./dependency-dispatch.mjs"; import { emptyRecord, isHandled, markHandled, pruneRecord, readRecord, writeRecord } from "./dispatch-record.mjs"; @@ -45,7 +46,7 @@ import { prHold } from "./gate-source.mjs"; import { runTitle, shareWorkflowRuns } from "./feedback-dispatch.mjs"; import { DEFAULTS as CI_FIX_DEFAULTS, dispatchCiFixes, humanGatedLabel, LIST_QUERY as CI_FIX_QUERY } from "./fix-ci-dispatch.mjs"; import { DEFAULTS as DOCS_FIX_DEFAULTS, dispatchDocsFixes, HISTORY_QUERY as DOCS_FIX_HISTORY_QUERY, LIST_QUERY as DOCS_FIX_QUERY } from "./fix-docs-dispatch.mjs"; -import { anyHandles, cappedAt, limitStop, pendingStop, permanentlyHandles } from "./limit-stop.mjs"; +import { anyHandles, cappedAt, limitStop, pendingStop, permanentlyHandles, releasedBy, stopRelease } from "./limit-stop.mjs"; import { budgetedFetch, serverFailure } from "./request-budget.mjs"; import { readConfig, readPolicy } from "./review-dependency.mjs"; @@ -54,6 +55,9 @@ export const DEFAULTS = Object.freeze({ workflow: "pr-unblock.yml", activity: "resolve-conflicts", checkName: "pr-unblock / resolve-conflicts", + // The push job's check: its lane-stopped skip releases the head for a + // same-head retry (limit-stop.mjs stopRelease). + stopCheckName: "pr-unblock / apply-conflict-patch", // The GitHub Actions app: only a check run it created counts, so another // app cannot suppress a dispatch by posting a run under the same name. appId: 15368, @@ -120,6 +124,7 @@ export async function dispatchConflicting({ workflow = DEFAULTS.workflow, activity = DEFAULTS.activity, checkName = DEFAULTS.checkName, + stopCheckName = DEFAULTS.stopCheckName, appId = DEFAULTS.appId, maxDispatch = DEFAULTS.maxDispatch, unknownRetries = DEFAULTS.unknownRetries, @@ -206,8 +211,20 @@ export async function dispatchConflicting({ log(`skip #${node.number}: ${why}`); continue; } - const existing = await api.listCheckRuns({ sha: node.headRefOid, checkName, appId }); + let existing = await api.listCheckRuns({ sha: node.headRefOid, checkName, appId }); const options = { limits, failuresRetry: true, heldSkipsRelease: true }; + // A push job that stopped on a reason routed to a failing status + // releases the head's earlier checks: the next run retries the same + // head, a fresh model session, until limit-stops-per-head such stops + // stand on it. + if (anyHandles(existing, { now, ...options })) { + const cap = (limits ?? LIMIT_DEFAULTS)["limit-stops-per-head"]; + const { cut, stops } = stopRelease(await api.listCheckRuns({ sha: node.headRefOid, checkName: stopCheckName, appId }), cap); + existing = releasedBy(existing, cut, limits); + if (cut === null && stops >= cap) { + log(`::warning::#${node.number}: head ${node.headRefOid}'s ${stopCheckName} stopped ${stops} time(s), the most one head gets; no ${activity} retry until a new push`); + } + } handled = anyHandles(existing, { now, ...options }); // Handled only because a limit stop has not reset: the log names it. waiting = handled && !anyHandles(existing, { now: Infinity, ...options }) ? pendingStop(existing, { now, limits }) : null; @@ -382,12 +399,12 @@ export function createGitHubApi({ }, // A held PR's issue timeline (gate-source.mjs reads its REST shape). listTimeline: (number) => paginate(`issues/${Number(number)}/timeline`), - // Every check run on a commit, up to 300, as fix-ci-context reads them + // Every check run on a commit, skips included, up to 300, as fix-ci-context reads them // (address-feedback.mjs listCommitCheckRuns). async listCommitCheckRuns(sha) { const runs = []; for (let page = 1; page <= 3; page += 1) { - const data = await request(`${apiUrl}/repos/${owner}/${name}/commits/${sha}/check-runs?filter=latest&per_page=100&page=${page}`); + const data = await request(`${apiUrl}/repos/${owner}/${name}/commits/${sha}/check-runs?filter=all&per_page=100&page=${page}`); runs.push(...(data.check_runs ?? [])); if ((data.check_runs ?? []).length < 100) { break; diff --git a/.github/scripts/pr-unblock/dispatch.test.mjs b/.github/scripts/pr-unblock/dispatch.test.mjs index 009c660557..234f31ede3 100644 --- a/.github/scripts/pr-unblock/dispatch.test.mjs +++ b/.github/scripts/pr-unblock/dispatch.test.mjs @@ -64,8 +64,10 @@ function pr(number, overrides = {}) { // or a Map from head SHA to the check runs GitHub returns for it. `runIds` // maps a PR number to the workflow_run_id its dispatch returns, undefined // included; any other PR gets 9000 plus its number. `workflowRuns` is the -// pr-unblock.yml run listing. +// pr-unblock.yml run listing. `stops` maps a head SHA to its +// pr-unblock / apply-conflict-patch check runs, read into calls.stops. function fakeApi({ + stops = new Map(), pages = [[]], requeries = [], checked = new Set(), @@ -78,7 +80,7 @@ function fakeApi({ comments = new Map(), labelEvents = new Map(), } = {}) { - const calls = { list: [], get: [], checks: [], dispatch: [], runs: [], timeline: [] }; + const calls = { list: [], get: [], checks: [], stops: [], dispatch: [], runs: [], timeline: [] }; let round = 0; return { calls, @@ -115,6 +117,10 @@ function fakeApi({ return answer.filter((node) => numbers.includes(node.number)); }, async listCheckRuns({ sha, checkName, appId }) { + if (checkName === "pr-unblock / apply-conflict-patch") { + calls.stops.push({ sha, appId }); + return stops.get(sha) ?? []; + } calls.checks.push({ sha, checkName, appId }); if (failChecks.has(sha)) { throw new Error("check-runs read failed"); @@ -1372,3 +1378,48 @@ test("main records a fix-ci head at the per-head cap in DISPATCH_RECORD, warns o rmSync(dir, { recursive: true, force: true }); } }); + +// A same-head retry after a stop routed to a failing status: the push job's +// lane-stopped skip (escalation-routes.mjs stopSkip) releases the head's +// earlier resolve-conflicts checks, up to limit-stops-per-head stops. +const at = (time) => `2026-10-26T${time}:00Z`; +const modelRun = (time) => ({ ...SUCCESS_RUN, started_at: at(time), completed_at: at(time) }); +const laneStopRun = (time) => ({ status: "completed", conclusion: "neutral", output: { title: "Skipped: lane-stopped" }, started_at: at(time) }); + +test("a push job's lane stop releases the head for a same-head retry, up to limit-stops-per-head stops", async () => { + const head = pr(1).headRefOid; + const once = harness({ pages: [[pr(1)]], checked: new Map([[head, [modelRun("10:00")]]]), stops: new Map([[head, [laneStopRun("10:05")]]]) }); + await once.run(); + assert.deepEqual(dispatchedNumbers(once.api), ["1"], "one stop: retried"); + + const record = emptyRecord(); + const three = harness( + { + pages: [[pr(1)]], + checked: new Map([[head, [modelRun("10:00"), modelRun("10:30"), modelRun("11:00")]]]), + stops: new Map([[head, [laneStopRun("10:05"), laneStopRun("10:35"), laneStopRun("11:05")]]]), + }, + { record }, + ); + await three.run(); + assert.deepEqual(dispatchedNumbers(three.api), [], "three stops: the cap"); + assert.ok(three.lines.some((line) => line.startsWith("::warning::#1:") && line.includes("stopped 3 time(s)"))); + assert.ok(record.handled.get(CHECK_NAME).has(head), "the capped head is recorded"); +}); + +test("a retry that did not stop, or a select job's own lane stop, handles the head", async () => { + const head = pr(1).headRefOid; + // The newest push job check is not a stop: the retry landed. + const landed = harness({ + pages: [[pr(1)]], + checked: new Map([[head, [modelRun("10:00"), modelRun("10:30")]]]), + stops: new Map([[head, [laneStopRun("10:05"), { ...SUCCESS_RUN, started_at: at("10:35") }]]]), + }); + await landed.run(); + assert.deepEqual(dispatchedNumbers(landed.api), []); + // A select job's stop (the repeat cap) is deterministic: no retry. + const terminal = harness({ pages: [[pr(1)]], checked: new Map([[head, [laneStopRun("10:00")]]]) }); + await terminal.run(); + assert.deepEqual(dispatchedNumbers(terminal.api), []); + assert.deepEqual(terminal.api.calls.stops.map((call) => call.sha), [head], "the push job's checks are read, and hold no stop"); +}); diff --git a/.github/scripts/pr-unblock/escalation-routes.json b/.github/scripts/pr-unblock/escalation-routes.json new file mode 100644 index 0000000000..657b2f5f39 --- /dev/null +++ b/.github/scripts/pr-unblock/escalation-routes.json @@ -0,0 +1,44 @@ +{ + "protected-path-conflict": { "activity": "resolve-conflicts", "route": "hold", "why": "a conflicted path is a workflow, submodule config or a settings file, which the lane does not resolve" }, + "workflow-staged": { "activity": "resolve-conflicts", "route": "hold", "why": "the merge stages a change under .github/workflows/, which the lane does not push" }, + "unexpected-path-staged": { "activity": "resolve-conflicts", "route": "fail", "why": "the conflict resolution changes a path outside the conflicted ones, which the lane does not push" }, + "repeat-cap": { "activity": "resolve-conflicts", "route": "fail", "why": "the lane has resolved conflicts on this branch as many times in a row as its cap allows" }, + "unresolved-conflict": { "activity": "resolve-conflicts", "route": "fail", "why": "the lane could not resolve every conflict" }, + "conflict-patch-invalid": { "activity": "resolve-conflicts", "route": "fail", "why": "the conflict resolution the lane wrote did not pass validation, so the merge was not pushed" }, + "conflict-patch-not-applied": { "activity": "resolve-conflicts", "route": "fail", "why": "the conflict resolution did not apply to the merge it was written for, so the merge was not pushed" }, + "conflict-patch-protected-path": { "activity": "resolve-conflicts", "route": "hold", "why": "the conflict resolution changes a path the lane does not push: .github/workflows/, CODEOWNERS, .gitmodules or a settings file" }, + "feedback-protected-path": { "activity": "address-feedback", "route": "hold", "why": "the pull request changes .github/workflows/, which the lane token cannot push, so the lane does not address its review threads" }, + "feedback-round-cap": { "activity": "address-feedback", "route": "hold", "why": "the lane has addressed review threads on this pull request as many rounds as its cap allows, and an open thread is a security or intent finding or a person holds the pull request, so a person decides" }, + "feedback-not-converging": { "activity": "address-feedback", "route": "fail", "why": "review rounds are not converging: after the round cap and one fresh fixer round the code review still has open findings, so the fixer and the reviewer disagree on what the change should be" }, + "feedback-uncertain": { "activity": "address-feedback", "route": "hold", "why": "the lane could not settle a security-review or CRITICAL finding, or one no known review lane wrote, which no adjudicator judges, so a person decides it" }, + "feedback-verifier-open": { "activity": "address-feedback", "route": "fail", "why": "the lane left a pr-verify note open (uncertain, deferred or answered incorrect, or still open after the round cap and one fresh fixer round); pr-verify judges the next head afresh" }, + "feedback-not-landed": { "activity": "address-feedback", "route": "fail", "why": "a fix the lane recorded did not reach the branch, so none of its review replies was posted" }, + "feedback-invalid": { "activity": "address-feedback", "route": "fail", "why": "the review replies the lane recorded did not pass validation, so none was posted" }, + "feedback-partial": { "activity": "address-feedback", "route": "fail", "why": "posting the review replies stopped partway, or a fixed thread could not be resolved" }, + "feedback-untaken-threads": { "activity": "address-feedback", "route": "fail", "why": "the pull request has an unresolved review thread the lane does not take (a reviewer it does not act on, a review of an older head, or a thread it already answered), so nothing will resolve it" }, + "feedback-publish-failed": { "activity": "address-feedback", "route": "fail", "why": "the reply job failed before it decided anything (a lane token it could not get, or a script error), so no reply was posted" }, + "feedback-incomplete": { "activity": "address-feedback", "route": "fail", "why": "the lane did not record a disposition for every review thread it kept, so none was posted" }, + "feedback-disputed": { "activity": "address-feedback", "route": "hold", "why": "a disputed review finding is one only a person decides (a security-review or CRITICAL finding, one no known review lane wrote, or a thread a person directed)" }, + "feedback-stood-open": { "activity": "address-feedback", "route": "fail", "why": "an adjudicator found a review finding stands, and the next round left it open again" }, + "feedback-unadjudicated": { "activity": "address-feedback", "route": "fail", "why": "the lane left a review finding open and no adjudicator verdict settled it: none was recorded, or it found the finding does not hold where the lane had not disputed it (an UNCERTAIN or VALID-defer reply), which only an INCORRECT reply lets a verdict resolve" }, + "feedback-question-intent": { "activity": "address-feedback", "route": "hold", "why": "an adjudicator found that a disputed review finding turns on what the change is meant to do, which a person decides" }, + "feedback-question-design": { "activity": "address-feedback", "route": "hold", "why": "an adjudicator found that a disputed review finding turns on a design choice, which a person decides" }, + "feedback-question-security": { "activity": "address-feedback", "route": "hold", "why": "an adjudicator found that a disputed review finding turns on a security judgment, which a person decides" }, + "dependency-round-cap": { "activity": "review-dependency", "route": "hold", "why": "the lane has reviewed this dependency update as many rounds as its cap allows" }, + "dependency-invalid": { "activity": "review-dependency", "route": "hold", "why": "the dependency review verdict did not pass validation, so the lane acted on none of it" }, + "dependency-fix-not-landed": { "activity": "review-dependency", "route": "hold", "why": "the dependency review chose a fix, but no fix commit reached the branch (the lane lands a fix only when it changes nothing but the versions in npm or pip manifest and lock files outside .github/)" }, + "dependency-security-close": { "activity": "review-dependency", "route": "hold", "why": "the dependency review would close a security update, which the lane never closes" }, + "dependency-escalate": { "activity": "review-dependency", "route": "hold", "why": "the dependency review found a risk the lane does not decide alone" }, + "dependency-config-unreadable": { "activity": "review-dependency", "route": "hold", "why": "the lane could not read the base .github/dependabot.yml, so it does not mark this dependency update for merge" }, + "dependency-files-truncated": { "activity": "review-dependency", "route": "hold", "why": "the dependency review chose merge, but the files listing it read was cut short, so the lane does not mark this dependency update for merge" }, + "dependency-fix-refused": { "activity": "review-dependency", "route": "hold", "why": "a commit on the dependency update other than Dependabot's changes more than a version of an npm or pip manifest or lock file, as a regular file outside .github/, so the lane does not act on the review" }, + "dependency-control-uncited": { "activity": "review-dependency", "route": "hold", "why": "the dependency review chose merge for a review-only dependency without quoting its control, so the lane does not mark it for merge" }, + "dependency-workflow-unpinned": { "activity": "review-dependency", "route": "hold", "why": "the dependency review chose merge, but the update changes a workflow or action file by more than swapping one full-SHA action pin for another, so the lane does not mark it for merge" }, + "ci-fix-round-cap": { "activity": "fix-ci", "route": "fail", "why": "the lane has pushed as many CI fixes to this pull request as its cap allows, and CI still fails" }, + "ci-fix-empty": { "activity": "fix-ci", "route": "fail", "why": "the lane changed nothing that fixes the failing checks" }, + "ci-fix-invalid": { "activity": "fix-ci", "route": "fail", "why": "the CI fix the lane wrote did not pass validation, so it was not pushed" }, + "ci-fix-protected-path": { "activity": "fix-ci", "route": "hold", "why": "the CI fix changes a path the lane does not push: .github/workflows/, CODEOWNERS, .gitmodules or a settings file" }, + "ci-fix-weaken-required": { "activity": "fix-ci", "route": "hold", "why": "the lane could fix a failing check only by weakening a required check, which a person decides" }, + "ci-fix-not-applied": { "activity": "fix-ci", "route": "fail", "why": "the CI fix did not apply to the head it was written for, so it was not pushed" }, + "ci-fix-escalated": { "activity": "fix-ci", "route": "fail", "why": "the lane could not fix every failing check, after one retry a model rung up; its run log names each one and why" } +} diff --git a/.github/scripts/pr-unblock/escalation-routes.mjs b/.github/scripts/pr-unblock/escalation-routes.mjs new file mode 100644 index 0000000000..49cb784fa8 --- /dev/null +++ b/.github/scripts/pr-unblock/escalation-routes.mjs @@ -0,0 +1,45 @@ +// escalation-routes.json, the one routing table: each stop reason's activity +// and route (`hold` hands the PR to a person with the label; `fail` ends in a +// failing status and no label), as report-escalation.sh reads it. No +// network; imported by gate-source.mjs, the select and emitter scripts, and +// apply-patch.mjs (which copies this file and the table beside its own copy). +import { readFileSync } from "node:fs"; +import { pathToFileURL } from "node:url"; + +export const ROUTES = Object.freeze(JSON.parse(readFileSync(new URL("./escalation-routes.json", import.meta.url), "utf8"))); + +// The activity a reason resumes, or null for a reason the table does not +// list, which holds every activity. +export const escalationActivity = (reason) => (Object.hasOwn(ROUTES, reason) ? ROUTES[reason].activity : null); + +// A reason's route, or null for a reason the table does not list. +export const escalationRoute = (reason) => (Object.hasOwn(ROUTES, reason) ? ROUTES[reason].route : null); + +// The skip reason a job that stops on `reason` posts on its check: a stop +// routed to a failing status is `lane-stopped`, which limit-stop.mjs reads; +// one held for a person, or one the table does not list, `awaiting-human`. +export const stopSkip = (reason) => (escalationRoute(reason) === "fail" ? "lane-stopped" : "awaiting-human"); + +// A stop routed to a failing status, posted after a model turn: the emitter +// job's neutral check carrying stopSkip's lane-stopped (limit-stop.mjs +// releases the head on it). The same skip on the activity's own check is a +// select job's stop, which handles the head for good like any other check. +export const LANE_STOP = "Skipped: lane-stopped"; +export const laneStop = (run) => run?.status === "completed" && run.conclusion === "neutral" && run.output?.title === LANE_STOP; + +// The GitHub Actions app: only its check runs are a trusted stop record. +export const ACTIONS_APP_ID = 15368; + +// The model rung a same-head retry runs at: "1" (the profile's escalate-to, +// resolve.mjs) once the head carries a lane stop from the emitter job +// (`stopRuns`, REST shape, by the GitHub Actions app), else "" (its own). +export const retryRung = (stopRuns) => (stopRuns.some((run) => laneStop(run) && run.app?.id === ACTIONS_APP_ID) ? "1" : ""); + +// `node escalation-routes.mjs skip` prints stopSkip of the REASON variable, +// for a workflow job that posts a select job's stop. +if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) { + if (process.argv[2] !== "skip") { + throw new Error("usage: escalation-routes.mjs skip"); + } + process.stdout.write(stopSkip(process.env.REASON ?? "")); +} diff --git a/.github/scripts/pr-unblock/feedback-dispatch.mjs b/.github/scripts/pr-unblock/feedback-dispatch.mjs index d8161d7f8a..0b34135711 100644 --- a/.github/scripts/pr-unblock/feedback-dispatch.mjs +++ b/.github/scripts/pr-unblock/feedback-dispatch.mjs @@ -83,10 +83,10 @@ import { appendFileSync, readFileSync } from "node:fs"; import { pathToFileURL } from "node:url"; import { commitPulls } from "../../actions/check-trusted-trigger/check-trusted-trigger.mjs"; -import { fixRequesters, humanGatedLabelFrom, threadAuthors, trustedHumansFrom } from "./address-feedback.mjs"; +import { fixRequesters, humanGatedLabelFrom, onReviewedHead, reviewedHeads, threadAuthors, trustedHumansFrom } from "./address-feedback.mjs"; import { commentItems, gateSource, listLabelEvents } from "./gate-source.mjs"; import { emptyRecord, isHandled, markHandled, readRecord, writeRecord } from "./dispatch-record.mjs"; -import { anyHandles, limitResetsAt, permanentlyHandles } from "./limit-stop.mjs"; +import { anyHandles, LANE_STOP, laneStop, limitResetsAt, permanentlyHandles } from "./limit-stop.mjs"; import { budgetedFetch } from "./request-budget.mjs"; export const DEFAULTS = Object.freeze({ @@ -130,8 +130,10 @@ export const SWEEP_WORKFLOW = ".github/workflows/pr-unblock-dispatch.yml"; export const SWEEP_HEAD_CAP = 3; const startOf = (run) => Date.parse(run.started_at ?? ""); -// The reply check skips that leave the head for another round. -const RELEASES = new Set(["Skipped: no-model-turn", "Skipped: finding-stands"]); +// The reply check skips that leave the head for another round: a lane stop +// (a stop escalation-routes.json routes to a failing status) is retried on +// the same head, a fresh model session, under SWEEP_HEAD_CAP with the rest. +const RELEASES = new Set(["Skipped: no-model-turn", "Skipped: finding-stands", LANE_STOP]); const releases = (run) => run?.status === "completed" && run.conclusion === "neutral" && RELEASES.has(run.output?.title); // A pass-only round's reply check (publish-feedback.mjs PASS_ONLY_SKIP) // answers no thread, so it neither releases a head nor ends a release. @@ -147,23 +149,37 @@ export function markedHead(comments, lane, markerAuthorId = DEFAULTS.markerAutho return SHA.test(sha) ? sha : ""; } -// An unresolved thread, read in full, that threadAuthors accepts. -export function hasWorkableThread(threads, { reviewerId, appId, trustedHumans, requesters = null, fixers = requesters ?? new Set() }) { - return threads.some( - (thread) => - thread.isResolved === false && - thread.comments.pageInfo?.hasNextPage !== true && - threadAuthors( - thread.comments.nodes.map((node) => ({ - authorId: node.author?.databaseId, - createdAt: node.createdAt, - body: node.body, - lastEditedAt: node.lastEditedAt, - editorId: node.editor?.databaseId, - })), - { reviewerId, appId, trustedHumans, requesters, fixers }, - ) !== null, - ); +// An unresolved thread, read in full, that the select job keeps: threadAuthors +// accepts it and its review is on a marker-confirmed head (onReviewedHead). +// `heads` is reviewedHeads of the PR's issue comments. +export function hasWorkableThread(threads, { reviewerId, appId, trustedHumans, heads, requesters = null, fixers = requesters ?? new Set() }) { + return threads.some((thread) => { + if (thread.isResolved !== false || thread.comments.pageInfo?.hasNextPage === true) { + return false; + } + const nodes = thread.comments.nodes; + const authors = threadAuthors( + nodes.map((node) => ({ + authorId: node.author?.databaseId, + createdAt: node.createdAt, + body: node.body, + lastEditedAt: node.lastEditedAt, + editorId: node.editor?.databaseId, + })), + { reviewerId, appId, trustedHumans, requesters, fixers }, + ); + const review = nodes[0]?.pullRequestReview; + return ( + authors !== null && + onReviewedHead({ + authors, + firstAuthorId: nodes[0].author?.databaseId, + review: review && { user: { id: review.author?.databaseId }, commit_id: review.commit?.oid }, + heads, + reviewerId, + }) + ); + }); } export const VERIFY_WORKFLOW = ".github/workflows/pr-verify.yml"; @@ -314,12 +330,16 @@ export async function dispatchFeedback({ permission: api.getPermission, }); const requesters = gated ? fixers : null; - if (!hasWorkableThread(threads, { reviewerId: settings.reviewerId, appId: appSenderId, trustedHumans, requesters, fixers })) { - return skip( - gated - ? `#${number} is labeled ${humanGatedLabel} (${gate.source}: ${gate.reason}) and has no thread a trusted person with write access asked the lane to fix` - : `#${number} has no unresolved thread left by the reviewer bot alone, asked to be fixed, or answered by a trusted person`, - ); + // On an ungated head, any unresolved thread dispatches, even one the select + // job will not keep (a reviewer the lane does not act on, a review of an + // older head): select then escalates feedback-untaken-threads, and its + // lane-stopped check handles the head, so the stall gets a stop once. + const workable = hasWorkableThread(threads, { reviewerId: settings.reviewerId, appId: appSenderId, trustedHumans, heads: reviewedHeads(comments), requesters, fixers }); + if (gated && !workable) { + return skip(`#${number} is labeled ${humanGatedLabel} (${gate.source}: ${gate.reason}) and has no thread a trusted person with write access asked the lane to fix`); + } + if (!workable && !threads.some((thread) => thread.isResolved === false)) { + return skip(`#${number} has no unresolved review thread`); } } // Both review lanes finish on one head, and their dispatchers queue in one @@ -362,10 +382,13 @@ export async function dispatchFeedback({ // limit stop is never released: limit-stop.mjs still holds the head until // its reset and counts it toward limits.limit-stops-per-head. let replies = []; + // A select job's own stop (lane-stopped on this check) ends the head; only + // a reply job's skip releases it. + const selectStopped = existing.some(laneStop); if (anyHandles(existing, { now, limits: resolved.limits })) { replies = roundReplies(await api.listCheckRuns({ sha: head, checkName: settings.replyCheckName, appId: settings.appId })).filter(sinceNotes); const newest = replies.reduce((a, b) => (startOf(b) > startOf(a) ? b : a), replies[0]); - if (releases(newest)) { + if (releases(newest) && !selectStopped) { const cut = startOf(newest); existing = existing.filter((run) => limitResetsAt(run, resolved.limits) !== null || !(startOf(run) < cut)); } @@ -457,6 +480,7 @@ const THREADS_QUERY = `query($owner: String!, $name: String!, $number: Int!, $af author { login ... on Bot { databaseId } ... on User { databaseId } } createdAt body lastEditedAt editor { ... on Bot { databaseId } ... on User { databaseId } } + pullRequestReview { author { ... on Bot { databaseId } ... on User { databaseId } } commit { oid } } } pageInfo { hasNextPage } } diff --git a/.github/scripts/pr-unblock/feedback-dispatch.test.mjs b/.github/scripts/pr-unblock/feedback-dispatch.test.mjs index 336763d185..d684fba8f9 100644 --- a/.github/scripts/pr-unblock/feedback-dispatch.test.mjs +++ b/.github/scripts/pr-unblock/feedback-dispatch.test.mjs @@ -46,9 +46,12 @@ const failedRun = (line, completed = "2026-10-17T10:00:00Z") => ({ completed_at: completed, output: { title: "Failed", summary: line === null ? "The activity step failed." : `${line}\n\nThe activity step failed.` }, }); -const thread = (authors, isResolved = false) => ({ +// The first comment's review, as THREADS_QUERY reads it: its author's +// review of `reviewed`. +const onReview = (author, reviewed = HEAD) => ({ pullRequestReview: { author, commit: { oid: reviewed } } }); +const thread = (authors, isResolved = false, reviewed = HEAD) => ({ isResolved, - comments: { nodes: authors.map((author) => ({ author })), pageInfo: { hasNextPage: false } }, + comments: { nodes: authors.map((author, i) => ({ author, ...(i === 0 ? onReview(author, reviewed) : {}) })), pageInfo: { hasNextPage: false } }, }); // The #7794 shape: the reviewer bot's finding, the lanes App's UNCERTAIN reply // (publish-feedback.mjs renderReply), then `reply`, each a minute apart. @@ -58,7 +61,7 @@ const decidedThread = (reply = { author: KYLE }) => ({ isResolved: false, comments: { nodes: [ - { author: CLAUDE, body: "finding" }, + { author: CLAUDE, body: "finding", ...onReview(CLAUDE) }, { author: LANE, body: "**pr-unblock / address-feedback: UNCERTAIN, left open for a person**\n\nnote" }, { body: "Owner decision: drop the reference.", ...reply }, ].map((node, i) => ({ createdAt: `2026-10-17T20:0${i}:00Z`, lastEditedAt: null, editor: null, ...node })), @@ -240,7 +243,7 @@ test("the security review workflow is an upstream too", async () => { // footer (ci-workflows publish-review publish.cjs commentBody at v0.63.0). const laneThread = (body) => ({ isResolved: false, - comments: { nodes: [{ author: ACTIONS_BOT_NODE, body }], pageInfo: { hasNextPage: false } }, + comments: { nodes: [{ author: ACTIONS_BOT_NODE, body, ...onReview(ACTIONS_BOT_NODE) }], pageInfo: { hasNextPage: false } }, }); test("a github-actions[bot] thread ending in the lane footer dispatches like the reviewer bot's", async () => { @@ -253,13 +256,23 @@ test("the code lane's github-actions[bot] thread dispatches like the reviewer bo assert.equal((await decide(api)).result.number, 42); }); -test("a github-actions[bot] thread without a review lane's footer last dispatches nothing", async () => { - for (const body of ["Unquoted input.", "claude-lane: claude-security-review\nUnquoted input."]) { - const api = fakeApi({ threads: [laneThread(body)] }); - assert.equal((await decide(api)).result, null, body); - assert.deepEqual(api.calls.dispatch, []); - } -}); +// An unresolved thread the select job will not keep still dispatches once on +// an ungated head: select escalates feedback-untaken-threads, so the stall +// ends in a stop instead of silence (#10092). +for (const [label, threads] of [ + ["a github-actions[bot] thread without a review lane's footer last", [laneThread("Unquoted input.")]], + ["a github-actions[bot] thread with the footer first", [laneThread("claude-lane: claude-security-review\nUnquoted input.")]], + ["the only open thread has a human reply", [thread([CLAUDE, KYLE])]], + ["the only open thread is a human's", [thread([KYLE])]], + ["the reply after the lane's escalation is a stranger's", [decidedThread({ author: STRANGER })]], + ["the trusted reply was last edited by a stranger", [decidedThread({ author: KYLE, lastEditedAt: "2026-10-17T20:30:00Z", editor: STRANGER })]], +]) { + test(`${label} dispatches once, for select to escalate feedback-untaken-threads`, async () => { + const api = fakeApi({ threads }); + assert.equal((await decide(api)).result.number, 42); + assert.equal(api.calls.dispatch.length, 1); + }); +} for (const [label, options] of [ ["the upstream is another workflow", { run: { path: ".github/workflows/ci.yml" } }], @@ -287,12 +300,6 @@ for (const [label, options] of [ comments: [marker("claude-review", HEAD, { id: 9999001 }), marker("claude-security-review")], }], ["every thread is resolved", { threads: [thread([CLAUDE], true)] }], - ["the only open thread has a human reply", { threads: [thread([CLAUDE, KYLE])] }], - ["the only open thread is a human's", { threads: [thread([KYLE])] }], - ["the reply after the lane's escalation is a stranger's", { threads: [decidedThread({ author: STRANGER })] }], - ["the trusted reply was last edited by a stranger", { - threads: [decidedThread({ author: KYLE, lastEditedAt: "2026-10-17T20:30:00Z", editor: STRANGER })], - }], ["the head already has an address-feedback check run", { checkRuns: [{ status: "completed", conclusion: "success", output: { title: "Done" } }], }], @@ -517,7 +524,7 @@ const VERIFY_RUN = upstreamRun({ id: 99, path: ".github/workflows/pr-verify.yml" // run at the head. const VERIFY_CREATED = "2026-10-17T12:00:00Z"; function verifyApi(options = {}) { - const note = { isResolved: false, comments: { nodes: [{ author: ACTIONS_BOT_NODE, body: "\nnotes" }], pageInfo: { hasNextPage: false } } }; + const note = { isResolved: false, comments: { nodes: [{ author: ACTIONS_BOT_NODE, body: "\nnotes", ...onReview(ACTIONS_BOT_NODE) }], pageInfo: { hasNextPage: false } } }; const api = fakeApi({ threads: [note], ...options }); api.getWorkflowRun = async (id) => (id === VERIFY_RUN.id ? { ...VERIFY_RUN, created_at: VERIFY_CREATED } : null); api.listCompletedRuns = async () => [upstreamRun({ id: 13, created_at: "2026-10-17T10:30:00Z" })]; @@ -913,7 +920,7 @@ test("without needs-human a trusted reply dispatches with no permission read", a const askedThread = (reply) => ({ isResolved: false, comments: { - nodes: [{ author: CLAUDE, body: "finding" }, reply].map((node, i) => ({ + nodes: [{ author: CLAUDE, body: "finding", ...onReview(CLAUDE, OLD_HEAD) }, reply].map((node, i) => ({ createdAt: `2026-10-17T20:0${i}:00Z`, lastEditedAt: null, editor: null, @@ -938,10 +945,10 @@ for (const [label, options] of [ }], ["a trusted reply that asks no fix", { threads: [askedThread({ author: KYLE_LOGIN, body: "Owner decision: drop it." })] }], ]) { - test(`a thread the lane never answered dispatches nothing for ${label}`, async () => { + test(`a thread the lane never answered dispatches once for select to escalate, for ${label}`, async () => { const api = fakeApi(options); - assert.equal((await decide(api)).result, null); - assert.deepEqual(api.calls.dispatch, []); + assert.equal((await decide(api)).result.number, 42); + assert.equal(api.calls.dispatch.length, 1); }); } @@ -991,6 +998,58 @@ test("the API client reads review threads across pages through GraphQL", async ( ); }); +// #10092: a PR whose only unresolved threads the select job will not keep (a +// Codex-root thread, a review of an older head) stalled with no stop. On an +// ungated head the dispatcher fires once; select escalates +// feedback-untaken-threads, and the not-run job's lane-stopped checks (on +// address-feedback and publish-feedback) handle the head, so the next tick +// does not fire again. +const CODEX = { databaseId: 199175422 }; +const selectStopped = [ + roundRun("address-feedback", "neutral", "Skipped: lane-stopped", "2026-10-17T10:00:00Z"), + roundRun("publish-feedback", "neutral", "Skipped: lane-stopped", "2026-10-17T10:00:01Z"), +]; +for (const [label, author, reviewed] of [ + ["a Codex-only PR", CODEX, HEAD], + ["a stale-head-only PR", CLAUDE, OLD_HEAD], +]) { + test(`${label} dispatches once, then not again on the same head`, async () => { + const open = { isResolved: false, comments: { nodes: [{ author, body: "finding", ...onReview(author, reviewed) }], pageInfo: { hasNextPage: false } } }; + const first = fakeApi({ threads: [open] }); + assert.equal((await decide(first)).result.number, 42); + assert.equal(first.calls.dispatch.length, 1); + const second = fakeApi({ threads: [open], checkRuns: selectStopped }); + assert.equal((await decide(second)).result, null); + assert.deepEqual(second.calls.dispatch, []); + }); +} + +test("a PR with no unresolved thread does not dispatch", async () => { + const api = fakeApi({ threads: [thread([CODEX], true), thread([CLAUDE], true)] }); + const { result, lines } = await decide(api); + assert.equal(result, null); + assert.deepEqual(api.calls.dispatch, []); + assert.match(lines.at(-1), /has no unresolved review thread/); +}); + +test("on a gated head only a workable thread dispatches; an untaken one does not", async () => { + const gated = { pulls: pull({ labels: [{ name: "needs-human" }] }) }; + const api = fakeApi({ ...gated, threads: [thread([CODEX])] }); + assert.equal((await decide(api)).result, null); + assert.deepEqual(api.calls.dispatch, []); +}); + +test("the API client reads each thread's review and its commit", async () => { + const bodies = []; + const fetch = async (url, init) => { + bodies.push(JSON.parse(init.body)); + return new Response(JSON.stringify({ data: { repository: { pullRequest: { reviewThreads: { nodes: [], pageInfo: { hasNextPage: false } } } } } })); + }; + const api = createGitHubApi({ token: "t", repository: REPOSITORY, apiUrl: "https://api.example", graphqlUrl: "https://api.example/graphql", fetch }); + await api.listReviewThreads(42); + assert.match(bodies[0].query, /pullRequestReview \{ author \{ \.\.\. on Bot \{ databaseId \} \.\.\. on User \{ databaseId \} \} commit \{ oid \} \}/); +}); + // The record one run leaves for the next at the same head. `counting` wraps // a fake so every API call is counted by method. function counting(api) { @@ -1152,6 +1211,7 @@ function sweepApi({ openPulls = pull(), checksAt = { [HEAD]: [failedRound, noMod // Both review lanes marked each PR's head reviewed. const headOf = (number) => openPulls.find((p) => p.number === number).head.sha; api.listIssueComments = async (number) => [marker("claude-review", headOf(number)), marker("claude-security-review", headOf(number))]; + api.listReviewThreads = async (number) => [thread([CLAUDE], false, headOf(number))]; api.listCheckRuns = async ({ sha, checkName }) => (checksAt[sha] ?? []).filter((run) => (run.name ?? "pr-unblock / address-feedback") === checkName); api.listCompletedRuns = async (path, head) => (path.endsWith("pr-review-hosted.yml") ? (reviewRuns[head] ?? []) : []); return api; @@ -1289,6 +1349,29 @@ test("a PR the sweep cannot read is logged and the next PR is still swept", asyn assert.ok(lines.some((line) => line.startsWith("::warning::#42:") && line.includes("502")), lines.join("\n")); }); +// A round whose reply job stopped on a reason routed to a failing status +// (feedback-not-landed, -invalid, -partial, -incomplete, -unadjudicated and +// the rest): its lane-stopped skip releases the head like no-model-turn, so +// the tick retries it on the same head, a fresh model session, under +// SWEEP_HEAD_CAP with the other releasing skips. +const laneStopped = (started) => roundRun("publish-feedback", "neutral", "Skipped: lane-stopped", started); + +test("a scheduled tick retries a head whose reply job stopped, until SWEEP_HEAD_CAP stops stand", async (t) => { + const api = sweepApi({ checksAt: { [HEAD]: [answeredRound, laneStopped("2026-10-17T10:05:00Z")] } }); + await sweepMain(t, api); + assert.deepEqual(api.calls.dispatch.map((call) => call.inputs["pr-number"]), ["42"]); + const capped = sweepApi({ + checksAt: { [HEAD]: [answeredRound, laneStopped("2026-10-17T10:05:00Z"), laneStopped("2026-10-17T11:05:00Z"), laneStopped("2026-10-17T12:05:00Z")] }, + }); + assert.equal((await sweepMain(t, capped)).output, ""); + assert.deepEqual(capped.calls.dispatch, []); + // A select job's own stop (the round cap) posts on the address-feedback + // check, after the reply check: it holds the head. + const terminal = sweepApi({ checksAt: { [HEAD]: [answeredRound, laneStopped("2026-10-17T10:05:00Z"), roundRun("address-feedback", "neutral", "Skipped: lane-stopped", "2026-10-17T11:00:00Z")] } }); + await sweepMain(t, terminal); + assert.deepEqual(terminal.calls.dispatch, []); +}); + // A pr-verify pass whose post-verify-notes job found an open fail-note thread // from an earlier head names its head in VERIFY_PASS_HEAD_SHA: the dispatch // sends a pass-only round, which needs no workable thread, no review marker diff --git a/.github/scripts/pr-unblock/fix-ci-context.mjs b/.github/scripts/pr-unblock/fix-ci-context.mjs index aaac9fd41a..60e93764a2 100644 --- a/.github/scripts/pr-unblock/fix-ci-context.mjs +++ b/.github/scripts/pr-unblock/fix-ci-context.mjs @@ -4,10 +4,10 @@ // // It skips a pull request the human-gated label holds (awaiting-human) unless // another lane's escalation put the label on (gate-source.mjs prHold), -// escalates `ci-fix-round-cap` once the lanes App has pushed -// `loop.fix-ci-rounds` fix-ci commits to the pull request, skips a closed pull -// request or a head with no failing check (one cancelled past its time limit -// counts), and otherwise writes the context +// skips a closed pull request or a head with no failing check (one cancelled +// past its time limit counts), escalates `ci-fix-round-cap` once the lanes App +// has pushed `loop.fix-ci-rounds` fix-ci commits to the pull request, and +// otherwise writes the context // file the `source-control:pull-request ci --ci` skill reads: each failing // check's name, the workflow file that defines its job, its annotations and // the end of its log, and the review findings address-feedback would work @@ -23,14 +23,16 @@ import { pathToFileURL } from "node:url"; import { timeLimitNoted } from "../maintenance-reconcile-lost-runs/infra-failure.mjs"; import { createGitHubApi, humanGatedLabelFrom, keepThreads, need, positive, reviewedHeads, reviewerIdFrom, SHA } from "./address-feedback.mjs"; -import { prHold } from "./gate-source.mjs"; +import { failingChecks, prHold } from "./gate-source.mjs"; + +// clear-escalations reads the head the same way, so the reader lives on the +// merge path floor in gate-source.mjs. +export { failingChecks }; +import { retryRung, stopSkip } from "./escalation-routes.mjs"; // The trailer apply-patch.mjs writes on every fix commit, from the ci-context // row of patch-rules.json; a round is one. export const FIX_TRAILER = "Lane-Activity: pr-unblock/fix-ci"; -const FAILING = new Set(["failure", "timed_out", "startup_failure"]); -// The aggregate of the others: fixing them fixes it. -const AGGREGATE = "ci-status"; const WORKFLOW_PATH = /^\.github\/workflows\/[A-Za-z0-9._-]+\.ya?ml$/; const MAX_CHECKS = 10; const MAX_ANNOTATIONS = 20; @@ -65,27 +67,6 @@ export function logExcerpt(text) { return excerpt; } -// A contract-only run skips every lane, and ci-status then carries the last -// full run's recorded verdict forward, so a skip says nothing about a check. -const SILENT = new Set(["skipped", "neutral"]); -const silent = (run) => run.status === "completed" && SILENT.has(run.conclusion); - -// The latest run of each check name that is not a skip, when it failed or was -// cancelled; select keeps a cancelled one only past its time limit. The fix-ci -// dispatcher reads a cancelled head through it too, so the two agree. -export function failingChecks(checkRuns) { - const latest = new Map(); - for (const run of checkRuns) { - const kept = latest.get(run.name); - if (!kept || (silent(kept) && !silent(run)) || (silent(kept) === silent(run) && run.id > kept.id)) { - latest.set(run.name, run); - } - } - return [...latest.values()] - .filter((run) => run.name !== AGGREGATE && run.status === "completed" && (FAILING.has(run.conclusion) || run.conclusion === "cancelled")) - .sort((a, b) => a.id - b.id); -} - const annotationOf = (row) => ({ path: String(row.path ?? ""), line: Number.isSafeInteger(row.start_line) ? row.start_line : null, @@ -157,12 +138,22 @@ export async function select({ api, pr, appId, reviewerId, maxRounds, humanGated if (held) { return { decision: "skip", skipReason: "awaiting-human", headSha }; } + // A whole listing of settled checks with none failing: every fix-ci stop + // is gone on this head, round cap included, and `resolved` lets + // report-stop-status clear its status. A cut-short or unsettled listing + // decides nothing. + const runs = await api.listCommitCheckRuns(headSha); + const failing = failingChecks(runs); + if (failing.length === 0) { + const settled = runs.complete === true && runs.every((run) => run.status === "completed"); + return { decision: "skip", skipReason: "not-applicable", ...(settled ? { resolved: true } : {}), headSha, laneEscalated }; + } const rounds = countFixRounds(await api.listPullCommits(pr), appId); if (rounds >= maxRounds) { return { decision: "escalate", escalationReason: "ci-fix-round-cap", rounds, headSha, laneEscalated }; } const checks = []; - for (const run of failingChecks(await api.listCommitCheckRuns(headSha))) { + for (const run of failing) { if (checks.length === MAX_CHECKS) { break; } @@ -190,7 +181,9 @@ export async function select({ api, pr, appId, reviewerId, maxRounds, humanGated } catch (error) { log(`::warning::the review findings could not be read (${error.message}); the context holds none`); } - return { decision: "proceed", rounds, headSha, laneEscalated, file: { pr, head_sha: headSha, checks, review_findings } }; + // A retry after the push job stopped on this head runs one rung up. + const modelRung = retryRung(await api.listCheckRuns(headSha, "pr-unblock / apply-patch")); + return { decision: "proceed", rounds, headSha, laneEscalated, modelRung, file: { pr, head_sha: headSha, checks, review_findings } }; } // The base config's resolved loop caps, through the base checkout's @@ -231,10 +224,13 @@ export async function main(env, { api, log = (line) => console.log(line) } = {}) output("head-sha", result.headSha); output("escalation-reason", result.escalationReason ?? ""); output("context-sha256", sha256); - // fix-ci-not-run's check, and whether apply-patch may push past the - // human-gated label a lane escalation put on. - output("skip-reason", result.decision === "skip" ? result.skipReason : ""); + // fix-ci-not-run's check (a stop's own skip, so the head's fix-ci runs + // record it), and whether apply-patch may push past the human-gated label + // a lane escalation put on. + output("skip-reason", result.decision === "skip" ? result.skipReason : result.decision === "escalate" ? stopSkip(result.escalationReason) : ""); output("lane-escalated", String(result.laneEscalated === true)); + output("model-rung", result.modelRung ?? ""); + output("stop-resolved", String(result.resolved === true)); log( `${result.decision}: ${result.file?.checks.length ?? 0} failing check(s), ${result.file?.review_findings.length ?? 0} review finding(s), ${result.rounds ?? 0} earlier round(s)${result.escalationReason ? `, ${result.escalationReason}` : ""}`, ); diff --git a/.github/scripts/pr-unblock/fix-ci-context.test.mjs b/.github/scripts/pr-unblock/fix-ci-context.test.mjs index 3353f41192..0520b303a3 100644 --- a/.github/scripts/pr-unblock/fix-ci-context.test.mjs +++ b/.github/scripts/pr-unblock/fix-ci-context.test.mjs @@ -42,7 +42,14 @@ function fakeApi(overrides = {}) { return { getPull: async () => overrides.pull ?? { state: "open", head: { sha: HEAD } }, listPullCommits: async () => overrides.commits ?? [], - listCommitCheckRuns: async () => overrides.checkRuns ?? [run(11, "lint-shell", "failure"), run(13, "ci-status", "failure")], + listCommitCheckRuns: async () => { + if (overrides.checkRunsError) { + throw overrides.checkRunsError; + } + return Object.assign([...(overrides.checkRuns ?? [run(11, "lint-shell", "failure"), run(13, "ci-status", "failure")])], { complete: overrides.complete ?? true }); + }, + // The head's pr-unblock / apply-patch checks: a lane stop runs the retry a rung up. + listCheckRuns: async () => overrides.stopRuns ?? [], listAnnotations: async (id) => overrides.annotations?.[id] ?? [ { path: "scripts/x.sh", start_line: 12, annotation_level: "failure", message: "SC2034: foo appears unused" }, @@ -294,6 +301,49 @@ test("a commit apply-patch writes for fix-ci counts as a round", () => { assert.equal(countFixRounds([{ sha: "e".repeat(40), author: { id: APP }, commit: { message } }], APP), 1); }); +// Review finding on #10252: a failing pr-unblock/fix-ci status must come off +// a head whose CI went green with no new commit (fixed by hand, or a flaky +// lane re-run green), including after the round cap; only a whole, settled +// check listing with nothing failing says so. +test("a head with no failing check resolves the fix-ci stop, at the round cap too", async () => { + const green = [run(11, "lint-shell", "failure"), run(14, "lint-shell", "success"), run(13, "ci-status", "failure")]; + for (const commits of [[], [fixCommit(), fixCommit(), fixCommit()]]) { + const result = await selectWith(fakeApi({ checkRuns: green, commits })); + assert.deepEqual([result.decision, result.skipReason, result.resolved, result.headSha], ["skip", "not-applicable", true, HEAD]); + } +}); + +test("a failing, cancelled or unsettled check, a cut-short listing, a hold or a closed PR resolves nothing", async () => { + const cases = { + failing: fakeApi(), + "failing at the round cap": fakeApi({ commits: [fixCommit(), fixCommit(), fixCommit()] }), + "cancelled with no time-limit note": fakeApi({ checkRuns: [run(12, "test-bash (1)", "cancelled")], annotations: { 12: CANCEL_ONLY } }), + "a re-run still in progress": fakeApi({ checkRuns: [run(11, "lint-shell", null, { status: "in_progress" })] }), + "a cut-short listing": fakeApi({ checkRuns: [run(14, "lint-shell", "success")], complete: false }), + closed: fakeApi({ pull: { state: "closed", head: { sha: HEAD } }, checkRuns: [] }), + held: gatedApi({ login: "kyle-sexton", id: HUMAN, type: "User" }, "feedback-disputed"), + }; + for (const [name, api] of Object.entries(cases)) { + const result = await selectWith(api); + assert.notEqual(result.resolved, true, name); + } +}); + +test("a check listing that fails to read throws, so main writes no outputs and resolves nothing", async () => { + const dir = mkdtempSync(join(tmpdir(), "fix-ci-context-")); + try { + const env = { GITHUB_TOKEN: "t", GITHUB_REPOSITORY: "o/r", PR_NUMBER: "42", APP_SENDER_ID: String(APP), CONTEXT_FILE: join(dir, "context.json"), FIX_CI_ROUNDS: "3", TRUSTED_ACTORS, GITHUB_OUTPUT: join(dir, "out") }; + await assert.rejects(main(env, { api: fakeApi({ checkRunsError: new Error("GET check-runs returned 502") }), log: () => {} }), /502/); + assert.throws(() => readFileSync(env.GITHUB_OUTPUT, "utf8")); + await main({ ...env, GITHUB_OUTPUT: join(dir, "green") }, { api: fakeApi({ checkRuns: [run(14, "lint-shell", "success")] }), log: () => {} }); + assert.match(readFileSync(join(dir, "green"), "utf8"), /^stop-resolved=true$/m); + await main({ ...env, GITHUB_OUTPUT: join(dir, "red") }, { api: fakeApi(), log: () => {} }); + assert.match(readFileSync(join(dir, "red"), "utf8"), /^stop-resolved=false$/m); + } finally { + rmSync(dir, { recursive: true, force: true }); + } +}); + test("a closed pull request is skipped", async () => { const result = await selectWith(fakeApi({ pull: { state: "closed", head: { sha: HEAD } } })); assert.equal(result.decision, "skip"); @@ -428,3 +478,12 @@ test("main refuses a malformed round cap", async () => { const env = { GITHUB_TOKEN: "t", GITHUB_REPOSITORY: "o/r", GITHUB_OUTPUT: "/dev/null", PR_NUMBER: "42", APP_SENDER_ID: String(APP), CONTEXT_FILE: "/dev/null", TRUSTED_ACTORS }; await assert.rejects(main({ ...env, FIX_CI_ROUNDS: "0" }, { api: fakeApi() }), /FIX_CI_ROUNDS/); }); + +// A retry after the push job stopped on this head runs one model rung up, so +// apply-patch takes no second in-run rung-up. +test("a head whose push job stopped proceeds at model rung 1; any other at its own", async () => { + const stop = { status: "completed", conclusion: "neutral", output: { title: "Skipped: lane-stopped" }, app: { id: 15368 } }; + assert.equal((await selectWith(fakeApi({ stopRuns: [stop] }))).modelRung, "1"); + assert.equal((await selectWith(fakeApi())).modelRung, ""); + assert.equal((await selectWith(fakeApi({ stopRuns: [{ ...stop, app: { id: 2 } }] }))).modelRung, ""); +}); diff --git a/.github/scripts/pr-unblock/fix-ci-dispatch.mjs b/.github/scripts/pr-unblock/fix-ci-dispatch.mjs index 8d5c54ad4b..f1b051fb67 100644 --- a/.github/scripts/pr-unblock/fix-ci-dispatch.mjs +++ b/.github/scripts/pr-unblock/fix-ci-dispatch.mjs @@ -55,6 +55,7 @@ import { emptyRecord, isHandled, markHandled } from "./dispatch-record.mjs"; import { runTitle } from "./feedback-dispatch.mjs"; import { failingChecks } from "./fix-ci-context.mjs"; import { prHold } from "./gate-source.mjs"; +import { laneStop } from "./limit-stop.mjs"; import { serverFailure } from "./request-budget.mjs"; export const DEFAULTS = Object.freeze({ @@ -62,6 +63,10 @@ export const DEFAULTS = Object.freeze({ workflow: "pr-unblock.yml", activity: "fix-ci", checkName: "pr-unblock / fix-ci", + // The push job's check: a lane-stopped skip there (a stop routed to a + // failing status after the model turn) counts the fix-ci run before it as + // a failed one, which the per-head cap and the restarts retry. + stopCheckName: "pr-unblock / apply-patch", // The run whose failure counts: pr-require-checks when its lanes failed, // else the advisory pr-test-windows, which ci-status does not aggregate but // the merge gate holds on. @@ -154,6 +159,44 @@ const fromRest = (run) => ({ completedAt: run.completed_at ?? null, }); +// The head's fix-ci runs (REST) in the rollup's shape, one per workflow run: +// fix-ci and the in-run fix-ci-rung-up both post a fix-ci check, and the pair +// is one attempt, read from its last check. An attempt a lane-stopped push job +// check (`stopRuns`, REST) followed reads as a failure: the run stopped, so +// the head may be retried under the same caps a failed run is. A check's +// workflow run is the id in its details_url (report-check-run sets it to the +// posting run's page); not its check suite, which GitHub picks for a check +// created over the API. A stop marks the attempt of its own run; a check with +// no run id falls back to timing, a stop after it and before the next fix-ci +// check started. +const RUN_PAGE = /\/actions\/runs\/([1-9][0-9]{0,19})$/; +export function withStops(runs, stopRuns) { + const startOf = (run) => Date.parse(run.started_at ?? ""); + const runOf = (run) => RUN_PAGE.exec(String(run.details_url ?? ""))?.[1] ?? null; + const stops = stopRuns.filter(laneStop); + const starts = runs.map(startOf).sort((a, b) => a - b); + const stoppedAt = (run) => { + const from = startOf(run); + const until = starts.find((start) => start > from) ?? Infinity; + return stops.some((stop) => + runOf(stop) !== null && runOf(run) !== null + ? runOf(stop) === runOf(run) + : startOf(stop) > from && startOf(stop) < until, + ); + }; + const attempts = new Map(); + for (const run of runs) { + const key = runOf(run) ?? Symbol(); + const attempt = attempts.get(key) ?? { last: run, stopped: false }; + if (startOf(run) >= startOf(attempt.last)) { + attempt.last = run; + } + attempt.stopped ||= stoppedAt(run); + attempts.set(key, attempt); + } + return [...attempts.values()].map(({ last, stopped }) => (stopped ? { ...fromRest(last), conclusion: "FAILURE" } : fromRest(last))); +} + // The fix-ci runs that count: an untrusted-trigger skip ran nothing. const counted = (runs) => runs.filter((run) => !(run.conclusion === "NEUTRAL" && String(run.title ?? "").startsWith("Skipped: untrusted-trigger"))); @@ -367,6 +410,7 @@ export async function dispatchCiFixes({ workflow = DEFAULTS.workflow, activity = DEFAULTS.activity, checkName = DEFAULTS.checkName, + stopCheckName = DEFAULTS.stopCheckName, actionsAppId = DEFAULTS.actionsAppId, holdLabels = DEFAULTS.holdLabels, gateLabel = null, @@ -470,7 +514,10 @@ export async function dispatchCiFixes({ // read is not dispatched. let fixCiRuns; try { - fixCiRuns = (await api.listCheckRuns({ sha: node.headRefOid, checkName, appId: actionsAppId })).map(fromRest); + fixCiRuns = withStops( + await api.listCheckRuns({ sha: node.headRefOid, checkName, appId: actionsAppId }), + await api.listCheckRuns({ sha: node.headRefOid, checkName: stopCheckName, appId: actionsAppId }), + ); } catch (error) { if (error.rateLimited) { log(`::warning::#${node.number}: its ${checkName} run read is rate limited (${error.message}); no fix-ci dispatched, a later run retries`); diff --git a/.github/scripts/pr-unblock/fix-ci-dispatch.test.mjs b/.github/scripts/pr-unblock/fix-ci-dispatch.test.mjs index 349651a19f..2d4a91a07e 100644 --- a/.github/scripts/pr-unblock/fix-ci-dispatch.test.mjs +++ b/.github/scripts/pr-unblock/fix-ci-dispatch.test.mjs @@ -3,6 +3,7 @@ import { readFileSync } from "node:fs"; import { test } from "node:test"; import { createGitHubApi } from "./dispatch.mjs"; +import { failingChecks } from "./gate-source.mjs"; import { emptyRecord, isHandled } from "./dispatch-record.mjs"; import { RERUN_DUE_MINUTES } from "../maintenance-reconcile-lost-runs/infra-failure.mjs"; import { dispatchCiFixes, humanGatedLabel, LIST_QUERY, skipReason } from "./fix-ci-dispatch.mjs"; @@ -85,10 +86,11 @@ function fakeApi({ comments = new Map(), labelEvents = new Map(), history = new Map(), + stops = new Map(), annotations = new Map(), commitRuns = new Map(), } = {}) { - const calls = { list: 0, runs: 0, dispatch: [], jobs: [], paths: [], timeline: [], history: [], annotations: [], commit: [] }; + const calls = { list: 0, runs: 0, dispatch: [], jobs: [], paths: [], timeline: [], history: [], stops: [], annotations: [], commit: [] }; return { calls, // One check run's annotations, by its id; none unless the test names it. @@ -119,6 +121,11 @@ function fakeApi({ // The head's check runs of one name and app over REST, every one of them: // the test's history for the head, else the rollup's runs in REST's shape. async listCheckRuns({ sha, checkName, appId }) { + // The push job's checks: `stops` for the head. + if (checkName === "pr-unblock / apply-patch") { + calls.stops.push(sha); + return stops.get(sha) ?? []; + } calls.history.push(sha); const rows = history.get(sha); if (rows instanceof Error) { @@ -808,7 +815,7 @@ test("the adapter sends one query naming the PR, its last commit's check rollup } }); -test("the adapter reads a head's check runs as fix-ci-context does: latest, 100 a page, at most 3 pages", async () => { +test("the adapter reads a head's check runs as fix-ci-context does: every run, 100 a page, at most 3 pages", async () => { const urls = []; const api = createGitHubApi({ token: "t", @@ -824,10 +831,29 @@ test("the adapter reads a head's check runs as fix-ci-context does: latest, 100 assert.equal(runs.length, 300); assert.deepEqual( urls, - [1, 2, 3].map((page) => `https://api.example/repos/octo/repo/commits/${"a".repeat(40)}/check-runs?filter=latest&per_page=100&page=${page}`), + [1, 2, 3].map((page) => `https://api.example/repos/octo/repo/commits/${"a".repeat(40)}/check-runs?filter=all&per_page=100&page=${page}`), ); }); +// A contract-only pr-require-checks run posts a lane skipped after the real +// run failed it; a latest-only listing returned only the skip, and the +// dispatcher read the head as not failing where fix-ci-context must agree. +test("the adapter's listing keeps a lane's failure under a later skip of it", async () => { + const api = createGitHubApi({ + token: "t", + repository: "octo/repo", + apiUrl: "https://api.example", + graphqlUrl: "https://api.example/graphql", + fetch: async (url) => { + const skipped = { id: 2, name: "test-node", status: "completed", conclusion: "skipped" }; + const all = new URL(url).searchParams.get("filter") === "all"; + const check_runs = all ? [{ id: 1, name: "test-node", status: "completed", conclusion: "failure" }, skipped] : [skipped]; + return new Response(JSON.stringify({ check_runs }), { status: 200 }); + }, + }); + assert.deepEqual(failingChecks(await api.listCommitCheckRuns("a".repeat(40))).map((run) => run.id), [1]); +}); + test("a head whose ci-status failed on infra only gets no fix-ci run, and takes no place under the cap", async () => { const head = (number) => String(number).padStart(40, "a"); const { api, lines, result } = await run({ pages: [[pr(5), pr(6), pr(7)]], requiredRuns: new Map([[head(5), { run: tested(1), jobs: INFRA_JOBS }]]) }); @@ -1141,3 +1167,53 @@ test("a failed job whose steps never listed is logged as such past the deadline" ); assert.ok(lines.some((line) => line.startsWith("dispatched fix-ci #5 ") && line.endsWith(`(cause: steps unlisted past ${RERUN_DUE_MINUTES} min in lint-repo)`)), lines.join("\n")); }); + +// A fix-ci run whose push job stopped on a reason routed to a failing status +// (a lane-stopped skip on pr-unblock / apply-patch) counts as a failed run: +// the head is retried under fix-ci-restarts and the per-head cap, as a run +// that failed is. +test("a fix-ci run its push job stopped is retried on the same head once, then left at fix-ci-restarts", async () => { + const head = "7".padStart(40, "a"); + const at = (time) => `2026-10-22T${time}:00Z`; + const fixCi = (time) => ({ name: "pr-unblock / fix-ci", status: "completed", conclusion: "success", output: { title: "Fixed" }, started_at: at(time), completed_at: at(time) }); + const stop = (time) => ({ name: "pr-unblock / apply-patch", status: "completed", conclusion: "neutral", output: { title: "Skipped: lane-stopped" }, started_at: at(time) }); + const once = await run({ pages: [[pr(7)]], history: new Map([[head, [fixCi("10:00")]]]), stops: new Map([[head, [stop("10:05")]]]) }); + assert.ok(once.lines.some((line) => line.startsWith("dispatched fix-ci #7 ")), once.lines.join("\n")); + const twice = await run({ + pages: [[pr(7)]], + history: new Map([[head, [fixCi("10:00"), fixCi("10:30")]]]), + stops: new Map([[head, [stop("10:05"), stop("10:35")]]]), + }); + assert.ok(twice.lines.some((line) => line.includes("failed 2 time(s) on this base, past 1 restart(s)")), twice.lines.join("\n")); + // A push job that did not stop leaves the run handling the head. + const pushed = await run({ pages: [[pr(7)]], history: new Map([[head, [fixCi("10:00")]]]), stops: new Map([[head, [{ ...stop("10:05"), output: { title: "Pushed" }, conclusion: "success" }]]]) }); + assert.ok(pushed.lines.some((line) => line.includes("head already has a pr-unblock / fix-ci check run")), pushed.lines.join("\n")); + // A select job's own stop (the round cap) on the fix-ci check is final. + const capped = await run({ pages: [[pr(7)]], history: new Map([[head, [{ ...stop("10:00"), name: "pr-unblock / fix-ci" }]]]) }); + assert.ok(capped.lines.some((line) => line.includes("head already has a pr-unblock / fix-ci check run")), capped.lines.join("\n")); +}); + +// fix-ci and the in-run fix-ci-rung-up each post a fix-ci check in one +// workflow run (the run id in each check's details_url). When the rung-up's +// push job stops, the pair +// is one stopped attempt: the first check's "Passed: no changes" does not +// leave the head handled, and the pair spends one restart, not two. +test("a fix-ci run whose rung-up push job stopped is one failed attempt, retried on the same head", async () => { + const head = "7".padStart(40, "a"); + const at = (time) => `2026-10-22T${time}:00Z`; + const fixCi = (time, suite, title) => ({ name: "pr-unblock / fix-ci", status: "completed", conclusion: "success", output: { title }, started_at: at(time), completed_at: at(time), details_url: `https://github.com/o/r/actions/runs/${suite}` }); + const stop = (time, suite) => ({ name: "pr-unblock / apply-patch", status: "completed", conclusion: "neutral", output: { title: "Skipped: lane-stopped" }, started_at: at(time), details_url: `https://github.com/o/r/actions/runs/${suite}` }); + const pair = [fixCi("10:00", 1, "Passed: no changes"), fixCi("10:10", 1, "Fixed")]; + const once = await run({ pages: [[pr(7)]], history: new Map([[head, pair]]), stops: new Map([[head, [stop("10:15", 1)]]]) }); + assert.ok(once.lines.some((line) => line.startsWith("dispatched fix-ci #7 ")), once.lines.join("\n")); + // A stop in another workflow run marks only that run. + const other = await run({ pages: [[pr(7)]], history: new Map([[head, pair]]), stops: new Map([[head, [stop("10:15", 2)]]]) }); + assert.ok(other.lines.some((line) => line.includes("head already has a pr-unblock / fix-ci check run")), other.lines.join("\n")); + // A second stopped attempt is past the one restart. + const twice = await run({ + pages: [[pr(7)]], + history: new Map([[head, [...pair, fixCi("11:00", 3, "Passed: no changes"), fixCi("11:10", 3, "Fixed")]]]), + stops: new Map([[head, [stop("10:15", 1), stop("11:15", 3)]]]), + }); + assert.ok(twice.lines.some((line) => line.includes("failed 2 time(s) on this base, past 1 restart(s)")), twice.lines.join("\n")); +}); diff --git a/.github/scripts/pr-unblock/gate-source.mjs b/.github/scripts/pr-unblock/gate-source.mjs index fe380fb4ad..e15a805d56 100644 --- a/.github/scripts/pr-unblock/gate-source.mjs +++ b/.github/scripts/pr-unblock/gate-source.mjs @@ -17,6 +17,8 @@ // `label`, `created_at`, `updated_at`. Text written by anyone but those // authors is never read. +import { escalationActivity } from "./escalation-routes.mjs"; + export const ESCALATION = ""; export const RESUME_KEY = /^$/; // Any escalation marker, of any lane: one that is not a whole ESCALATION @@ -27,6 +29,9 @@ const ANY_ESCALATION = ""; const MODEL_RUN = /^$/; export const CLEARED = /^$/; +// report-escalation.sh's comment for a fail-routed stop, one per activity, +// edited in place; it adds no label. +export const LANE_STOPPED = /^$/; // A review lane's hold comment opens a line with this marker. const REVIEW_HOLD = "$/; @@ -129,7 +134,7 @@ export function gateSource({ timeline, label, head, isLanesApp }) { // A whole escalation's two marker lines and address-feedback's own round // record are known; any other marker, another lane's escalation or an // escalation line out of place included, is not. - const known = (line) => line === ESCALATION || RESUME_KEY.test(line) || line === ROUND_MARKER || MODEL_RUN.test(line); + const known = (line) => line === ESCALATION || RESUME_KEY.test(line) || line === ROUND_MARKER || MODEL_RUN.test(line) || LANE_STOPPED.test(line); const malformed = appComments .filter((item) => writtenSince(item, since)) .map((item) => lines(item.body)) @@ -144,21 +149,34 @@ export function gateSource({ timeline, label, head, isLanesApp }) { // The Reason line report-escalation.sh writes into its escalation comment. const REASON_LINE = /^- Reason: `([a-z][a-z-]*)`$/; -// resolve-conflicts's reasons, which report-escalation.sh maps by default. -const CONFLICT_REASONS = new Set(["protected-path-conflict", "workflow-staged", "unexpected-path-staged", "repeat-cap", "unresolved-conflict"]); +export { escalationActivity, escalationRoute, ROUTES } from "./escalation-routes.mjs"; + +const FAILING = new Set(["failure", "timed_out", "startup_failure"]); +// The aggregate of the others: fixing them fixes it. It also fails on a hold +// label, so it never stands for the head's CI. +const AGGREGATE = "ci-status"; +// A contract-only run skips every lane, and ci-status then carries the last +// full run's recorded verdict forward, so a skip says nothing about a check. +const SILENT = new Set(["skipped", "neutral"]); +const silent = (run) => run.status === "completed" && SILENT.has(run.conclusion); + +// The latest run of each check name that is not a skip, when it failed or was +// cancelled; select keeps a cancelled one only past its time limit. The fix-ci +// dispatcher reads a cancelled head through it too, so the two agree, and +// clear-escalations reads a fix-ci stop's head through it. +export function failingChecks(checkRuns) { + const latest = new Map(); + for (const run of checkRuns) { + const kept = latest.get(run.name); + if (!kept || (silent(kept) && !silent(run)) || (silent(kept) === silent(run) && run.id > kept.id)) { + latest.set(run.name, run); + } + } + return [...latest.values()] + .filter((run) => run.name !== AGGREGATE && run.status === "completed" && (FAILING.has(run.conclusion) || run.conclusion === "cancelled")) + .sort((a, b) => a.id - b.id); +} -// The activity a reason resumes, as report-escalation.sh maps it, or null for -// a reason it does not list, which holds every activity. -export const escalationActivity = (reason) => - reason.startsWith("feedback-") - ? "address-feedback" - : reason.startsWith("dependency-") - ? "review-dependency" - : reason.startsWith("ci-fix-") - ? "fix-ci" - : CONFLICT_REASONS.has(reason) || reason.startsWith("conflict-patch-") - ? "resolve-conflicts" - : null; // When a timeline item was written, and by whom, for a comment or a review // (clear-escalations.mjs reads people speaking the same way). diff --git a/.github/scripts/pr-unblock/gate-source.test.mjs b/.github/scripts/pr-unblock/gate-source.test.mjs index 209b224303..cdcc013c11 100644 --- a/.github/scripts/pr-unblock/gate-source.test.mjs +++ b/.github/scripts/pr-unblock/gate-source.test.mjs @@ -2,9 +2,10 @@ // review lanes' hold marker (docs/conventions/pr-pipeline/README.md, "Review-lane // holds"), and from GitHub's GraphQL LabeledEvent and UnlabeledEvent shapes. import assert from "node:assert/strict"; +import { readFileSync } from "node:fs"; import { test } from "node:test"; -import { gateSource, laneEscalationAllows, listLabelEvents, prHold } from "./gate-source.mjs"; +import { escalationActivity, escalationRoute, gateSource, laneEscalationAllows, listLabelEvents, prHold, ROUTES } from "./gate-source.mjs"; const APP = { login: "melodic-automation-lanes[bot]", id: 337595936, type: "Bot" }; const ACTIONS = { login: "github-actions[bot]", id: 41898282, type: "Bot" }; @@ -51,6 +52,37 @@ test("another lane's marker on a lanes App comment since the label is unknown", assert.equal(source(timeline), "unknown"); }); +test("report-escalation.sh's lane-stopped comment, posted or edited since the label, leaves a lane escalation workable", () => { + const timeline = [ + label("labeled", APP, "2026-10-18T10:00:00Z"), + escalation(), + { ...commented(APP, "\nStopped.", "2026-10-18T09:00:00Z"), updated_at: "2026-10-18T10:30:00Z" }, + ]; + assert.equal(source(timeline), "lane-escalation"); + // Only that marker's exact shape is known. + const odd = [label("labeled", APP, "2026-10-18T10:00:00Z"), escalation(), commented(APP, "", "2026-10-18T10:30:00Z")]; + assert.equal(source(odd), "unknown"); +}); + +// The write runner's escalation enum (pr-run-activity-write.yml, its first +// step) is the list of reasons a run may hand report-escalation.sh; the +// routing table must route each one, and route nothing else. +test("escalation-routes.json routes every reason the write runner accepts, each to one activity and one route", () => { + const workflow = readFileSync(new URL("../../workflows/pr-run-activity-write.yml", import.meta.url), "utf8"); + const block = workflow.slice(workflow.indexOf("case $ESCALATION_REASON in"), workflow.indexOf("patch-empty | patch-invalid")); + const accepted = [...block.matchAll(/(?<=^|[\s|])([a-z]+(?:-[a-z]+)+)(?=\s*(?:\||\)))/gm)].map((match) => match[1]); + assert.ok(accepted.length > 30, `read ${accepted.length} reasons from the enum`); + assert.deepEqual([...new Set(accepted)].sort(), Object.keys(ROUTES).sort()); + for (const [reason, entry] of Object.entries(ROUTES)) { + assert.ok(["resolve-conflicts", "address-feedback", "review-dependency", "fix-ci"].includes(entry.activity), reason); + assert.ok(["hold", "fail"].includes(entry.route), reason); + assert.equal(escalationActivity(reason), entry.activity); + assert.equal(escalationRoute(reason), entry.route); + } + assert.equal(escalationActivity("bogus"), null); + assert.equal(escalationRoute("constructor"), null); +}); + test("address-feedback's own round records since the escalation leave it workable", () => { const timeline = [ label("labeled", APP, "2026-10-18T10:00:00Z"), diff --git a/.github/scripts/pr-unblock/limit-stop.mjs b/.github/scripts/pr-unblock/limit-stop.mjs index 4142e8d8bc..0eed6a21f7 100644 --- a/.github/scripts/pr-unblock/limit-stop.mjs +++ b/.github/scripts/pr-unblock/limit-stop.mjs @@ -21,6 +21,9 @@ // `limit-stops-per-head` stops the dispatchers stop re-dispatching the head; // the next push makes a new one. import { LIMIT_DEFAULTS, clampReset } from "../work-resolve-issue/limits.mjs"; +import { laneStop } from "./escalation-routes.mjs"; + +export { ACTIONS_APP_ID, LANE_STOP, laneStop, retryRung } from "./escalation-routes.mjs"; const ISO = String.raw`\d{4}-\d{2}-\d{2}T\d{2}:\d{2}(?::\d{2}(?:\.\d+)?)?Z`; const NAME = "[a-z][a-z0-9]*(?:-[a-z0-9]+)*"; @@ -147,3 +150,21 @@ export function cappedAt(runs, { limits = LIMIT_DEFAULTS, failuresRetry = false, const { done, retryable } = completedRuns(runs, { limits, failuresRetry, heldSkipsRelease }); return done.length >= limits["limit-stops-per-head"] && done.length > 0 && done.every(retryable) ? done.length : 0; } + +const startOf = (run) => Date.parse(run?.started_at ?? ""); + +// How a head's emitter checks (`stopRuns`, REST shape) release its activity +// checks for a same-head retry: `cut` is the newest emitter check's start +// when that check is a lane stop and fewer than `cap` lane stops stand on the +// head (the dispatcher's own per-head cap), so activity checks that started +// before it stop counting; null otherwise. `stops` counts the lane stops. +export function stopRelease(stopRuns, cap) { + const stops = stopRuns.filter(laneStop).length; + const newest = stopRuns.reduce((a, b) => (startOf(b) > startOf(a) ? b : a), stopRuns[0]); + return { cut: laneStop(newest) && stops < cap ? startOf(newest) : null, stops }; +} + +// `runs` without those started before `cut`, a limit stop kept. +export const releasedBy = (runs, cut, limits = LIMIT_DEFAULTS) => + cut === null ? runs : runs.filter((run) => limitResetsAt(run, limits) !== null || !(startOf(run) < cut)); + diff --git a/.github/scripts/pr-unblock/publish-feedback.mjs b/.github/scripts/pr-unblock/publish-feedback.mjs index bab830675f..d5cb75ed70 100755 --- a/.github/scripts/pr-unblock/publish-feedback.mjs +++ b/.github/scripts/pr-unblock/publish-feedback.mjs @@ -38,11 +38,12 @@ // refusal or a named upload feedback-invalid. Otherwise escalates // feedback-invalid, feedback-incomplete (a kept thread has no // disposition), feedback-not-landed, feedback-partial (posting stopped -// after a reply), feedback-uncertain (an UNCERTAIN or VALID-defer reply), -// or a reason adjudicate-feedback.mjs settleDisputes gives for an -// INCORRECT reply: feedback-unadjudicated, feedback-question-intent, -// feedback-question-design, feedback-question-security, or -// feedback-disputed (the finding stood at an earlier adjudication and +// after a reply), or a reason adjudicate-feedback.mjs settleDisputes +// gives for an INCORRECT, UNCERTAIN or VALID-defer reply: +// feedback-unadjudicated, feedback-question-intent, +// feedback-question-design, feedback-question-security, +// feedback-disputed, feedback-uncertain, feedback-verifier-open, or +// feedback-stood-open (the finding stood at an earlier adjudication and // this round left it open). A finding that stands with the head unmoved // skips finding-stands instead, which feedback-dispatch.mjs's sweep // dispatches again. @@ -101,8 +102,9 @@ import { staleVerifyNotes, trustedHumansFrom, } from "./address-feedback.mjs"; -import { parseVerdicts, settleDisputes } from "./adjudicate-feedback.mjs"; +import { DISPUTED, parseVerdicts, settleDisputes } from "./adjudicate-feedback.mjs"; import { inThreadScope } from "./thread-scope.mjs"; +import { stopSkip } from "./escalation-routes.mjs"; // bytes: replies.jsonl's cap, the read runner's carry, seal and scan cap too; // runBytes: run.json's, the 1 MiB the read runner accepts for it. @@ -128,8 +130,8 @@ const KEYS = ["disposition", "evidence", "fix_sha", "note", "thread"]; const LABELS = { VALID: "VALID, fixed", INCORRECT: "INCORRECT, left open", - UNCERTAIN: "UNCERTAIN, left open for a person", - "VALID-defer": "VALID (defer), left open for a person", + UNCERTAIN: "UNCERTAIN, left open", + "VALID-defer": "VALID (defer), left open", }; class Invalid extends Error {} @@ -633,9 +635,6 @@ async function postReplies({ api, pr, headSha, headRef, appId, patchState = "", return { escalationReason, posted: [] }; } replies = replies.map((reply) => (reply.disposition === "VALID" ? { ...reply, fixSha: landed.get(reply.thread) } : reply)); - if (replies.some((reply) => reply.disposition === "UNCERTAIN" || reply.disposition === "VALID-defer")) { - escalationReason = "feedback-uncertain"; - } const posted = []; for (const reply of replies) { const ids = shown.get(reply.thread).comment_ids; @@ -679,11 +678,12 @@ async function postReplies({ api, pr, headSha, headRef, appId, patchState = "", } log(`replied on ${reply.thread} (${reply.disposition})${reply.disposition === "VALID" ? "; resolved" : ""}`); } - // An INCORRECT thread goes to the adjudicator's verdict, not straight to - // a person (owner decision): it is resolved, handed back to the next round, - // or escalated with the question the verdict names; a thread whose finding - // stood at an earlier adjudication and is still open escalates - // feedback-disputed (adjudicate-feedback.mjs settleDisputes). The + // A thread left open (INCORRECT, UNCERTAIN or VALID-defer) goes to the + // adjudicator's verdict, not straight to a person (owner decision): it is + // resolved, handed back to the next round, or escalated with the question + // the verdict names; a thread whose finding stood at an earlier + // adjudication and is still open ends feedback-stood-open + // (adjudicate-feedback.mjs settleDisputes). The // adjudicator read the select job's head, which this round's fix push // may have moved past headSha. // A head that moved this round gets a review run, which starts the next @@ -696,7 +696,7 @@ async function postReplies({ api, pr, headSha, headRef, appId, patchState = "", replies, shown, now, - verdicts: posted.some((entry) => entry.disposition === "INCORRECT") ? parseVerdicts(readAdjudication(), run.head_sha) : new Map(), + verdicts: posted.some((entry) => DISPUTED.includes(entry.disposition)) ? parseVerdicts(readAdjudication(), run.head_sha) : new Map(), pushed: run.head_sha !== headSha, headMoved, resolveAfterReply, @@ -817,7 +817,7 @@ export async function main(env, { api, log = (line) => console.log(line) } = {}) }); if (escalationReason) { writeFileSync(env.ESCALATION_REASON_FILE, `${escalationReason}\n`); - writeFileSync(env.SKIP_REASON_FILE, "awaiting-human\n"); + writeFileSync(env.SKIP_REASON_FILE, `${stopSkip(escalationReason)}\n`); } else if (skipReason) { writeFileSync(env.SKIP_REASON_FILE, `${skipReason}\n`); } diff --git a/.github/scripts/pr-unblock/publish-feedback.test.mjs b/.github/scripts/pr-unblock/publish-feedback.test.mjs index 42cdbd7eff..08e736d96f 100644 --- a/.github/scripts/pr-unblock/publish-feedback.test.mjs +++ b/.github/scripts/pr-unblock/publish-feedback.test.mjs @@ -244,7 +244,7 @@ test("every reply the reply job leaves open reads as an escalation; a VALID one assert.equal(isEscalationReply({ user: { id: APP }, body: body("VALID") }, APP), false); }); -test("INCORRECT is posted and left open; UNCERTAIN and VALID-defer are posted, left open, and escalate", async () => { +test("INCORRECT, UNCERTAIN and VALID-defer are posted, left open, and go to the adjudicator, not a person", async () => { const api = fakeApi(); const result = await run( api, @@ -253,20 +253,34 @@ test("INCORRECT is posted and left open; UNCERTAIN and VALID-defer are posted, l reply({ thread: "PRRT_b", disposition: "UNCERTAIN", evidence: [], note: "Needs the vendor docs." }), ), undefined, - // The finding stands, so INCORRECT leaves the thread open; UNCERTAIN - // still escalates. + // The finding stands, so INCORRECT leaves the thread open; the UNCERTAIN + // thread has no verdict, so the round ends unadjudicated: a failing + // status, not a person. verdicts(judged("PRRT_a", "reviewer")), ); - assert.equal(result.escalationReason, "feedback-uncertain"); + assert.equal(result.escalationReason, "feedback-unadjudicated"); assert.deepEqual(api.resolved(), []); assert.match(api.posted()[0][3], /^\*\*pr-unblock \/ address-feedback: INCORRECT, left open\*\*\n\nEvidence: `src\/a.js:3`, `docs\/b.md:1`\n/); - assert.match(api.posted()[1][3], /UNCERTAIN, left open for a person\*\*\n\nEvidence: none in the checkout\n/); + assert.match(api.posted()[1][3], /UNCERTAIN, left open\*\*\n\nEvidence: none in the checkout\n/); assert.deepEqual(api.calls.find((call) => call[0] === "compare"), ["compare", GATED, NOW], "a moved branch is checked even with no VALID reply"); + // A lane verdict never resolves a deferred finding the lane agreed holds: + // the thread stays open with only the lane's reply, and the round ends + // unadjudicated. A stands verdict hands it to the next round. One that + // finds an UNCERTAIN finding stands does the same. const defer = fakeApi(); - assert.equal((await run(defer, lines(reply({ disposition: "VALID-defer" })))).escalationReason, "feedback-uncertain"); - assert.match(defer.posted()[0][3], /VALID \(defer\), left open for a person/); + assert.equal((await run(defer, lines(reply({ disposition: "VALID-defer" })), undefined, verdicts(judged("PRRT_a", "lane")))).escalationReason, "feedback-unadjudicated"); + assert.match(defer.posted()[0][3], /VALID \(defer\), left open\*\*/); + assert.equal(defer.posted().length, 1); assert.deepEqual(defer.resolved(), []); + const deferStands = fakeApi(); + assert.equal((await run(deferStands, lines(reply({ disposition: "VALID-defer" })), undefined, verdicts(judged("PRRT_a", "reviewer")))).escalationReason, ""); + assert.match(deferStands.posted()[1][3], /finding-stands/); + assert.deepEqual(deferStands.resolved(), []); + const unsure = fakeApi(); + assert.equal((await run(unsure, lines(reply({ disposition: "UNCERTAIN" })), undefined, verdicts(judged("PRRT_a", "reviewer")))).escalationReason, ""); + assert.match(unsure.posted()[1][3], /finding-stands/); + assert.deepEqual(unsure.resolved(), []); }); test("the round marker is posted once, as the first line, numbered after the App's earlier rounds", async () => { @@ -533,7 +547,7 @@ test("a reviewer verdict posts the finding-stands reply and leaves the thread op const stands = api.posted()[2][3].split("\n"); assert.equal(stands[0], ""); assert.equal(stands[1], "**pr-unblock / adjudicate-feedback: the finding stands**"); - assert.match(stands[3], /another INCORRECT reply on this thread is not accepted/); + assert.match(stands[3], /a round that leaves this thread open again ends the lane in a failing status/); assert.equal(stands[5], "Evidence: `docs/b.md:1`"); }); @@ -552,7 +566,8 @@ test("a finding that stands with the head unmoved skips finding-stands and escal assert.equal(api.posted().length, 2, "the lane's reply and the finding-stands reply"); assert.deepEqual(api.resolved(), []); - // An UNCERTAIN reply in the same round still goes to a person. + // An UNCERTAIN reply in the same round with no verdict of its own ends the + // round unadjudicated. const uncertain = fakeApi({ compare: unmoved }); const both = await run( uncertain, @@ -560,7 +575,7 @@ test("a finding that stands with the head unmoved skips finding-stands and escal atNow(), verdictsAt(NOW, judged("PRRT_a", "reviewer")), ); - assert.deepEqual([both.escalationReason, both.skipReason], ["feedback-uncertain", ""]); + assert.deepEqual([both.escalationReason, both.skipReason], ["feedback-unadjudicated", ""]); }); // #9560's fallback lands a VALID reply on an earlier lane commit: nothing is @@ -629,20 +644,20 @@ const stoodApi = (overrides = {}) => }); const stoodRun = JSON.stringify({ ...RUN, threads: [{ id: "PRRT_a", path: "src/a.js", comment_ids: [100, 101, 102] }] }); -test("no second dispute: an INCORRECT reply on a thread whose finding stood escalates feedback-disputed and is not adjudicated", async () => { +test("no second dispute: an INCORRECT reply on a thread whose finding stood ends feedback-stood-open and is not adjudicated", async () => { const api = stoodApi({ compare: { status: "identical", commits: [], files: [] } }); const result = await run(api, lines(reply()), stoodRun, verdicts(judged("PRRT_a", "lane"))); - assert.equal(result.escalationReason, "feedback-disputed"); + assert.equal(result.escalationReason, "feedback-stood-open"); assert.equal(api.posted().length, 1, "the lane's reply, and no adjudication reply"); assert.match(api.posted()[0][3], /INCORRECT, left open/); assert.deepEqual(api.resolved(), []); }); -test("the cap: a stood thread the next round leaves open escalates feedback-disputed; one it fixes resolves", async () => { +test("the cap: a stood thread the next round leaves open ends feedback-stood-open; one it fixes resolves", async () => { for (const disposition of ["UNCERTAIN", "VALID-defer"]) { const api = stoodApi({ compare: { status: "identical", commits: [], files: [] } }); const result = await run(api, lines(reply({ disposition, evidence: [] })), stoodRun); - assert.equal(result.escalationReason, "feedback-disputed", disposition); + assert.equal(result.escalationReason, "feedback-stood-open", disposition); assert.deepEqual(api.resolved(), []); } const fixed = stoodApi(); @@ -1131,7 +1146,7 @@ test("main writes the escalation and skip reason files and refuses a symlinked a }; const api = fakeApi(); assert.equal(await main(env, { api, log: () => {} }), 0); - assert.equal(readFileSync(env.ESCALATION_REASON_FILE, "utf8"), "feedback-uncertain\n"); + assert.equal(readFileSync(env.ESCALATION_REASON_FILE, "utf8"), "feedback-unadjudicated\n"); // replies.jsonl must be the bytes the scan job uploaded, whose SHA-256 it // passed out. @@ -1142,7 +1157,7 @@ test("main writes the escalation and skip reason files and refuses a symlinked a assert.deepEqual(forged.posted(), []); assert.equal(forged.markers().length, 1); } - assert.equal(readFileSync(env.SKIP_REASON_FILE, "utf8"), "awaiting-human\n"); + assert.equal(readFileSync(env.SKIP_REASON_FILE, "utf8"), "lane-stopped\n"); assert.equal(api.posted().length, 1); assert.equal(api.markers()[0][2].split("\n")[1], MODEL_RUN); // No model attempt names no run, so even a marker already on the PR from diff --git a/.github/scripts/pr-unblock/report-escalation.sh b/.github/scripts/pr-unblock/report-escalation.sh index c7f509f3d1..ef8dc3a11f 100755 --- a/.github/scripts/pr-unblock/report-escalation.sh +++ b/.github/scripts/pr-unblock/report-escalation.sh @@ -1,38 +1,27 @@ #!/usr/bin/env bash -# report-escalation.sh: the pr-unblock lane's report-escalation leaf. Hands the -# PR to a person: one machine-marked comment with the reason, the resume key -# and how to resume, plus the human-gated role label. Run from the base copy; -# it reads only the trusted base checkout and the GitHub API, never a PR head +# report-escalation.sh: the pr-unblock lane's report-escalation leaf. Ends a +# lane run that stopped, on the route escalation-routes.json (beside this +# script, the one routing table) gives its reason. Run from the base copy; it +# reads only the trusted base checkout and the GitHub API, never a PR head # file, and never writes to the branch. # +# - route `hold` (SAFETY, or a question for a person: intent, design, HIGH or +# CRITICAL security, a person's own hold): hands the PR to a person with +# one machine-marked comment holding the reason, the resume key and how to +# resume, plus the human-gated role label. +# - route `fail` (a defect the lane's own bounded retries did not mend): no +# label. One lanes App comment per activity, marked LANE_STOPPED, is posted +# once and edited in place after, naming the reason and the head; status +# mode (below) puts a failing commit status on that head. A new head carries +# no status, so the lane's next run there starts clean. +# # Environment: # GH_TOKEN token with pull-requests: write and issues: write # (statuses: write in status mode) # GITHUB_REPOSITORY owner/repo # PR_NUMBER the pull request # HEAD_SHA the gated head; it is the resume key -# ESCALATION_REASON protected-path-conflict | workflow-staged -# | unexpected-path-staged | repeat-cap -# | unresolved-conflict | feedback-protected-path -# | feedback-round-cap | feedback-uncertain -# | feedback-not-landed | feedback-invalid -# | feedback-partial | feedback-incomplete -# | feedback-disputed | feedback-unadjudicated -# | feedback-question-intent | feedback-question-design -# | feedback-question-security | feedback-not-converging -# | dependency-round-cap -# | dependency-invalid | dependency-fix-not-landed -# | dependency-security-close | dependency-escalate -# | dependency-config-unreadable -# | dependency-files-truncated -# | dependency-control-uncited -# | dependency-workflow-unpinned -# | dependency-fix-refused | ci-fix-round-cap -# | ci-fix-empty | ci-fix-invalid -# | ci-fix-protected-path | ci-fix-not-applied -# | ci-fix-escalated | conflict-patch-invalid -# | conflict-patch-not-applied -# | conflict-patch-protected-path +# ESCALATION_REASON a key of escalation-routes.json # TRACKER_CONFIG base .work-item-tracker.json # (default .base/.work-item-tracker.json) # LANES_BOT_LOGIN the lanes App bot login, the only author whose marker @@ -40,20 +29,24 @@ # # The role label is config.role_labels["human-gated"]. A missing file or entry # means needs-human; a present entry that is not a non-empty string fails. -# Idempotent: with the label on the PR and a LANES_BOT_LOGIN marker for this -# resume key and reason, it writes nothing; another reason on the same head -# gets its own comment, so gate-source.mjs reads every reason the label holds -# for. Without the label (a person removed it to resume), it +# Hold is idempotent: with the label on the PR and a LANES_BOT_LOGIN marker +# for this resume key and reason, it writes nothing; another reason on the +# same head gets its own comment, so gate-source.mjs reads every reason the +# label holds for. Without the label (a person removed it to resume), it # escalates again. The label is written before the comment, so a retry after a # failure between the two finds the label and posts at most one comment. The # comment write is tried three times, so a transient failure does not leave the -# label without its explanation. +# label without its explanation. Fail writes nothing when its comment already +# reads as it would. +# +# With the argument `clear` (below), it reads ACTIVITY in place of the reason. # -# With the argument `status`, it instead posts one failing `pr-unblock/conflict` -# commit status on HEAD_SHA when the reason is unresolved-conflict, and nothing -# for any other reason. GitHub starts no pull_request workflow on a conflicted -# PR, so this status is what shows the head why its CI is missing. That mode -# needs GH_TOKEN with statuses: write and reads TARGET_URL (optional, the run). +# With the argument `status`, it instead posts one failing commit status on +# HEAD_SHA for a fail-routed reason, and nothing for a held one: context +# `pr-unblock/conflict` for resolve-conflicts (GitHub starts no pull_request +# workflow on a conflicted PR, so this status is also what shows the head why +# its CI is missing), else `pr-unblock/`. That mode needs GH_TOKEN +# with statuses: write and reads TARGET_URL (optional, the run). # # Exit: 0 when escalated or already escalated; 1 on bad input or API failure. set -euo pipefail @@ -63,52 +56,35 @@ die() { exit 1 } -case ${ESCALATION_REASON:-} in - protected-path-conflict) why='a conflicted path is a workflow, submodule config or a settings file, which the lane does not resolve' ;; - workflow-staged) why='the merge stages a change under .github/workflows/, which the lane does not push' ;; - unexpected-path-staged) why='the conflict resolution changes a path outside the conflicted ones, which the lane does not push' ;; - repeat-cap) why='the lane has resolved conflicts on this branch as many times in a row as its cap allows' ;; - unresolved-conflict) why='the lane could not resolve every conflict' ;; - feedback-protected-path) why='the pull request changes .github/workflows/, which the lane token cannot push, so the lane does not address its review threads' ;; - feedback-round-cap) why='the lane has addressed review threads on this pull request as many rounds as its cap allows, and an open thread is a security or intent finding or a person holds the pull request, so a person decides' ;; - feedback-not-converging) why='review rounds are not converging: after the round cap and one fresh fixer round the code review still has open findings, so the fixer and the reviewer disagree on what the change should be' ;; - feedback-uncertain) why='a review finding is uncertain or deferred, so a person decides it' ;; - feedback-not-landed) why='a fix the lane recorded did not reach the branch, so none of its review replies was posted' ;; - feedback-invalid) why='the review replies the lane recorded did not pass validation, so none was posted' ;; - feedback-partial) why='posting the review replies stopped partway, or a fixed thread could not be resolved, so a person finishes them' ;; - feedback-incomplete) why='the lane did not record a disposition for every review thread it kept, so none was posted' ;; - feedback-disputed) why='a disputed review finding is one only a person decides (a security-review finding, a thread a person directed, or one an adjudicator found stands and the next round left open again)' ;; - feedback-unadjudicated) why='the lane replied with evidence that a review finding is incorrect, and no adjudicator verdict on that thread was recorded, so a person decides it' ;; - feedback-question-intent) why='an adjudicator found that a disputed review finding turns on what the change is meant to do, which a person decides' ;; - feedback-question-design) why='an adjudicator found that a disputed review finding turns on a design choice, which a person decides' ;; - feedback-question-security) why='an adjudicator found that a disputed review finding turns on a security judgment, which a person decides' ;; - dependency-round-cap) why='the lane has reviewed this dependency update as many rounds as its cap allows' ;; - dependency-invalid) why='the dependency review verdict did not pass validation, so the lane acted on none of it' ;; - dependency-fix-not-landed) why='the dependency review chose a fix, but no fix commit reached the branch (the lane lands a fix only when it changes nothing but the versions in npm or pip manifest and lock files outside .github/)' ;; - dependency-security-close) why='the dependency review would close a security update, which the lane never closes' ;; - dependency-escalate) why='the dependency review found a risk the lane does not decide alone' ;; - dependency-config-unreadable) why='the lane could not read the base .github/dependabot.yml, so it does not mark this dependency update for merge' ;; - dependency-files-truncated) why='the dependency review chose merge, but the files listing it read was cut short, so the lane does not mark this dependency update for merge' ;; - dependency-fix-refused) why='a commit on the dependency update other than Dependabot'"'"'s changes more than a version of an npm or pip manifest or lock file, as a regular file outside .github/, so the lane does not act on the review' ;; - dependency-control-uncited) why='the dependency review chose merge for a review-only dependency without quoting its control, so the lane does not mark it for merge' ;; - dependency-workflow-unpinned) why='the dependency review chose merge, but the update changes a workflow or action file by more than swapping one full-SHA action pin for another, so the lane does not mark it for merge' ;; - ci-fix-round-cap) why='the lane has pushed as many CI fixes to this pull request as its cap allows, and CI still fails' ;; - ci-fix-empty) why='the lane changed nothing that fixes the failing checks' ;; - ci-fix-invalid) why='the CI fix the lane wrote did not pass validation, so it was not pushed' ;; - ci-fix-protected-path) why='the CI fix changes a path the lane does not push: .github/workflows/, CODEOWNERS, .gitmodules or a settings file' ;; - ci-fix-not-applied) why='the CI fix did not apply to the head it was written for, so it was not pushed' ;; - ci-fix-escalated) why='the lane could not fix every failing check; its run log names each one and why' ;; - conflict-patch-invalid) why='the conflict resolution the lane wrote did not pass validation, so the merge was not pushed' ;; - conflict-patch-not-applied) why='the conflict resolution did not apply to the merge it was written for, so the merge was not pushed' ;; - conflict-patch-protected-path) why='the conflict resolution changes a path the lane does not push: .github/workflows/, CODEOWNERS, .gitmodules or a settings file' ;; - *) die "unknown escalation reason [${ESCALATION_REASON:-}]" ;; -esac -case $ESCALATION_REASON in - feedback-*) activity=address-feedback ;; - dependency-*) activity=review-dependency ;; - ci-fix-*) activity=fix-ci ;; - *) activity=resolve-conflicts ;; -esac +routes=$(dirname "${BASH_SOURCE[0]}")/escalation-routes.json + +# `clear`: an attempt of ACTIVITY on HEAD_SHA ended without a stop (a same-head +# retry landed). When the newest status of the activity's context on the +# head, by github-actions[bot], is a failure, it posts success over it, so a +# stop the retry mended no longer fails the head. Needs statuses: write. +if [[ ${1:-} == clear ]]; then + [[ ${HEAD_SHA:-} =~ ^([0-9a-f]{40}|[0-9a-f]{64})$ ]] || die "HEAD_SHA is not a commit id" + [[ ${GITHUB_REPOSITORY:-} =~ ^[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+$ ]] || die "GITHUB_REPOSITORY is not owner/repo" + jq -e --arg a "${ACTIVITY:-}" 'any(.[]; .activity == $a)' "$routes" >/dev/null || die "ACTIVITY is not an activity the table routes" + if [[ $ACTIVITY == resolve-conflicts ]]; then context=pr-unblock/conflict; else context=pr-unblock/$ACTIVITY; fi + state=$(gh api "repos/$GITHUB_REPOSITORY/commits/$HEAD_SHA/statuses?per_page=100" | jq -r --arg c "$context" \ + '[.[] | select(.context == $c and .creator.login == "github-actions[bot]")][0].state // ""') + if [[ $state != failure ]]; then + echo "no failing $context on $HEAD_SHA to clear" + exit 0 + fi + args=(-f state=success -f "context=$context" -f "description=The lane ran again on this head and did not stop") + [[ -n ${TARGET_URL:-} ]] && args+=(-f "target_url=$TARGET_URL") + gh api -X POST "repos/$GITHUB_REPOSITORY/statuses/$HEAD_SHA" "${args[@]}" >/dev/null + echo "cleared $context on $HEAD_SHA" + exit 0 +fi +[[ ${ESCALATION_REASON:-} =~ ^[a-z]+(-[a-z]+)*$ ]] || die "unknown escalation reason [${ESCALATION_REASON:-}]" +entry=$(jq -ce --arg r "$ESCALATION_REASON" 'if has($r) then .[$r] else null end' "$routes") || + die "unknown escalation reason [$ESCALATION_REASON]" +why=$(jq -er '.why | strings' <<<"$entry") || die "no why for $ESCALATION_REASON in $routes" +activity=$(jq -er '.activity | strings' <<<"$entry") || die "no activity for $ESCALATION_REASON in $routes" +route=$(jq -er '.route | select(. == "hold" or . == "fail")' <<<"$entry") || die "no route for $ESCALATION_REASON in $routes" [[ ${PR_NUMBER:-} =~ ^[1-9][0-9]*$ ]] || die "PR_NUMBER is not a pull request number" [[ ${HEAD_SHA:-} =~ ^([0-9a-f]{40}|[0-9a-f]{64})$ ]] || die "HEAD_SHA is not a commit id" [[ ${GITHUB_REPOSITORY:-} =~ ^[A-Za-z0-9_.-]+/[A-Za-z0-9_.-]+$ ]] || die "GITHUB_REPOSITORY is not owner/repo" @@ -116,43 +92,64 @@ esac case ${1:-} in '') ;; status) - if [[ $ESCALATION_REASON != unresolved-conflict ]]; then - echo "no conflict status for $ESCALATION_REASON" + if [[ $route != fail ]]; then + echo "no status for $ESCALATION_REASON: it holds for a person" exit 0 fi - args=(-f state=failure -f context=pr-unblock/conflict - -f "description=Conflicts with the base branch; CI does not run until they are resolved") + if [[ $activity == resolve-conflicts ]]; then + context=pr-unblock/conflict + else + context=pr-unblock/$activity + fi + if [[ $ESCALATION_REASON == unresolved-conflict ]]; then + description='Conflicts with the base branch; CI does not run until they are resolved' + else + description="Lane stopped: $ESCALATION_REASON; checked again on a new head" + fi + args=(-f state=failure -f "context=$context" -f "description=$description") [[ -n ${TARGET_URL:-} ]] && args+=(-f "target_url=$TARGET_URL") gh api -X POST "repos/$GITHUB_REPOSITORY/statuses/$HEAD_SHA" "${args[@]}" >/dev/null - echo "posted pr-unblock/conflict failure on $HEAD_SHA" + echo "posted $context failure on $HEAD_SHA" exit 0 ;; *) die "unknown mode [$1]" ;; esac -tracker=${TRACKER_CONFIG:-.base/.work-item-tracker.json} -label=needs-human -if [[ -f $tracker ]]; then - label=$(jq -er '.config.role_labels // {} - | if has("human-gated") then .["human-gated"] | strings | select(length > 0) else "needs-human" end' "$tracker") || - die "cannot read the human-gated role label from $tracker" -fi - bot=${LANES_BOT_LOGIN:-melodic-automation-lanes[bot]} issue="repos/$GITHUB_REPOSITORY/issues/$PR_NUMBER" -marker='' -key_line="" - -labeled=$(gh api "$issue" | jq --arg l "$label" '[.labels[].name] | index($l) != null') -if [[ $labeled == true ]]; then - marked=$(gh api --paginate "$issue/comments?per_page=100" | jq -s --arg bot "$bot" --arg m "$marker" --arg k "$key_line" \ - --arg r "- Reason: \`$ESCALATION_REASON\`" \ - '[add[] | select(.user.login == $bot) | (.body // "") | split("\n") | map(rtrimstr("\r")) - | select(.[0] == $m and (.[1] // "") == $k and any(.[]; . == $r))] | length > 0') - if [[ $marked == true ]]; then - echo "already escalated $ESCALATION_REASON for resume key $HEAD_SHA" - exit 0 +dispatch="gh workflow run pr-unblock.yml -R $GITHUB_REPOSITORY -f pr-number=$PR_NUMBER -f activity=$activity" + +if [[ $route == hold ]]; then + tracker=${TRACKER_CONFIG:-.base/.work-item-tracker.json} + label=needs-human + if [[ -f $tracker ]]; then + label=$(jq -er '.config.role_labels // {} + | if has("human-gated") then .["human-gated"] | strings | select(length > 0) else "needs-human" end' "$tracker") || + die "cannot read the human-gated role label from $tracker" + fi + marker='' + key_line="" + labeled=$(gh api "$issue" | jq --arg l "$label" '[.labels[].name] | index($l) != null') + if [[ $labeled == true ]]; then + marked=$(gh api --paginate "$issue/comments?per_page=100" | jq -s --arg bot "$bot" --arg m "$marker" --arg k "$key_line" \ + --arg r "- Reason: \`$ESCALATION_REASON\`" \ + '[add[] | select(.user.login == $bot) | (.body // "") | split("\n") | map(rtrimstr("\r")) + | select(.[0] == $m and (.[1] // "") == $k and any(.[]; . == $r))] | length > 0') + if [[ $marked == true ]]; then + echo "already escalated $ESCALATION_REASON for resume key $HEAD_SHA" + exit 0 + fi fi +else + # gate-source.mjs LANE_STOPPED: one per activity, edited in place. + marker="" + if [[ $activity == resolve-conflicts ]]; then + context=pr-unblock/conflict + else + context=pr-unblock/$activity + fi + existing=$(gh api --paginate "$issue/comments?per_page=100" | jq -sc --arg bot "$bot" --arg m "$marker" \ + '[add[] | select(.user.login == $bot and ((.body // "") | split("\n")[0] | rtrimstr("\r")) == $m)] | last // {} | {id, body}') fi # For a review that is not converging: each open review thread. The count @@ -161,7 +158,6 @@ fi # path is data: backticks, line breaks and angle brackets are dropped so it # stays one code span and cannot open an HTML comment (a marker line). details='' -resume="To resume, deal with the reason above, remove the \`$label\` label (and \`do-not-merge\`, if set), then dispatch the \`pr-unblock\` workflow with \`pr-number=$PR_NUMBER\` and \`activity=$activity\`:" if [[ $ESCALATION_REASON == feedback-not-converging ]]; then # Page by page, as address-feedback.mjs listThreads reads them: an open # thread past the first 100 can be the one that stopped the lane. Anyone @@ -198,16 +194,61 @@ if [[ $ESCALATION_REASON == feedback-not-converging ]]; then threads='- the open review threads could not be read; see the pull request'"'"'s Files changed tab' fi details=$'\n'"The review rounds are not converging after the round cap and one fresh fixer round. Open review threads:"$'\n\n'"$threads"$'\n' - # The rounds already run keep counting, so removing the label alone would - # stop again on the same head; only the threads move it. - resume="Removing the \`$label\` label alone does not resume: the rounds already run still count, so the lane stops again on this head. Do one of these instead: - -1. Resolve each open review thread yourself (fix it, or decide it stays as it is), then remove the \`$label\` label (and \`do-not-merge\`, if set). -2. From an account with write access, reply on a thread with a line that starts \`PR-fix lane: please fix this finding\`, then dispatch the \`pr-unblock\` workflow with \`pr-number=$PR_NUMBER\` and \`activity=$activity\`. The lane runs one round on the threads you asked it to fix:" fi body=$(mktemp) trap 'rm -f "$body"' EXIT + +if [[ $route == fail ]]; then + # The rounds already run keep counting for a capped reason, so a new head + # meets the cap again; a write-access fix request, or a person resolving + # the threads, moves a review that is not converging. + # The run that stopped: each stop names its own, so the comment's last edit + # is the newest stop's time (the monitoring lane's stalled-PR signal reads + # it). + run_line='' + if [[ ${GITHUB_RUN_ID:-} =~ ^[1-9][0-9]*$ ]]; then + run_line="- Run: ${GITHUB_SERVER_URL:-https://github.com}/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID"$'\n' + fi + more='' + if [[ $ESCALATION_REASON == feedback-not-converging ]]; then + more=" A reply on a thread, from an account with write access, starting \`PR-fix lane: please fix this finding\` grants one more round on that thread." + fi + cat >"$body" </dev/null + echo "stopped: $ESCALATION_REASON on $HEAD_SHA, comment $id edited" + exit 0 + fi + for attempt in 1 2 3; do + gh api -X POST "$issue/comments" -F "body=@$body" >/dev/null && break + ((attempt < 3)) || die "cannot post the lane-stopped comment after $attempt attempts" + sleep $((attempt * 5)) + done + echo "stopped: $ESCALATION_REASON on $HEAD_SHA, no label" + exit 0 +fi + cat >"$body" </dev/null || echo '[]' + ;; "POST repos/o/r/statuses/"*) [[ -f $STUB/fail_status ]] && { echo 'stub gh: status POST failed' >&2; exit 1; } echo '{}' @@ -107,10 +113,49 @@ run() { RC=$? set -e } -writes() { grep -c '^POST' "$STUB/requests" || true; } -comment() { printf '[{"body":%s,"user":{"login":"%s","type":"%s"}}]' "$(jq -Rs . <<<"$1")" "$2" "$3"; } +writes() { grep -cE '^(POST|PATCH)' "$STUB/requests" || true; } +# comment [id] +comment() { printf '[{"id":%s,"body":%s,"user":{"login":"%s","type":"%s"}}]' "${4:-1}" "$(jq -Rs . <<<"$1")" "$2" "$3"; } # bot_marker [reason]: the lanes App's escalation comment for that head and reason. -bot_marker() { comment "$MARKER"$'\n'""$'\nbody\n\n- Reason: `'"${2:-unresolved-conflict}"'`' 'melodic-automation-lanes[bot]' Bot; } +bot_marker() { comment "$MARKER"$'\n'""$'\nbody\n\n- Reason: `'"${2:-protected-path-conflict}"'`' 'melodic-automation-lanes[bot]' Bot; } +activity_of() { + case $1 in + feedback-*) echo address-feedback ;; + dependency-*) echo review-dependency ;; + ci-fix-*) echo fix-ci ;; + *) echo resolve-conflicts ;; + esac +} +context_of() { + local activity + activity=$(activity_of "$1") + if [[ $activity == resolve-conflicts ]]; then echo pr-unblock/conflict; else echo "pr-unblock/$activity"; fi +} + +# The routes: a person only for SAFETY (a path the lane token may not push, +# supply-chain trust) or a question only a person answers (intent, design, +# security, a person's own thread or hold), and every review-dependency stop, +# whose failing status would be per head while the merge marker comes from the +# lane's own re-run; every other stop is a failing status and no label. +HOLD_REASONS=(protected-path-conflict workflow-staged conflict-patch-protected-path + feedback-protected-path feedback-round-cap feedback-uncertain feedback-disputed + feedback-question-intent feedback-question-design feedback-question-security + dependency-round-cap dependency-invalid dependency-fix-not-landed dependency-config-unreadable + dependency-files-truncated dependency-control-uncited + dependency-security-close dependency-escalate dependency-fix-refused dependency-workflow-unpinned + ci-fix-protected-path ci-fix-weaken-required) +FAIL_REASONS=(unexpected-path-staged repeat-cap unresolved-conflict conflict-patch-invalid conflict-patch-not-applied + feedback-not-converging feedback-verifier-open feedback-not-landed feedback-invalid feedback-partial + feedback-incomplete feedback-stood-open feedback-unadjudicated feedback-untaken-threads feedback-publish-failed + ci-fix-round-cap ci-fix-empty ci-fix-invalid ci-fix-not-applied ci-fix-escalated) +ROUTES="$SCRIPT_DIR/escalation-routes.json" +check "the routing table holds exactly these reasons" \ + "$(printf '%s\n' "${HOLD_REASONS[@]}" "${FAIL_REASONS[@]}" | sort)" "$(jq -r 'keys[]' "$ROUTES" | sort)" +check "the routing table holds these for a person" \ + "$(printf '%s\n' "${HOLD_REASONS[@]}" | sort)" "$(jq -r 'to_entries[] | select(.value.route == "hold") | .key' "$ROUTES" | sort)" +for reason in "${HOLD_REASONS[@]}" "${FAIL_REASONS[@]}"; do + check "$reason: the table's activity" "$(activity_of "$reason")" "$(jq -r --arg r "$reason" '.[$r].activity' "$ROUTES")" +done # Rejected reasons: workflow-files and dependency-workflow-held are retired names, the rest are not in the enum. for reason in workflow-files dependency-workflow-held bogus '' 'repeat-cap ' REPEAT-CAP; do @@ -120,15 +165,8 @@ for reason in workflow-files dependency-workflow-held bogus '' 'repeat-cap ' REP check "rejects reason [$reason]: no gh call" '' "$(cat "$STUB/calls")" done -# Accepted reasons: one comment, one label, both on the PR. -for reason in protected-path-conflict workflow-staged unexpected-path-staged repeat-cap unresolved-conflict \ - feedback-protected-path feedback-round-cap feedback-uncertain feedback-not-landed feedback-invalid \ - feedback-partial feedback-incomplete feedback-disputed feedback-not-converging \ - feedback-unadjudicated feedback-question-intent feedback-question-design feedback-question-security \ - dependency-round-cap dependency-invalid dependency-fix-not-landed dependency-security-close dependency-escalate \ - dependency-config-unreadable dependency-files-truncated dependency-control-uncited dependency-workflow-unpinned dependency-fix-refused \ - ci-fix-round-cap ci-fix-empty ci-fix-invalid ci-fix-protected-path ci-fix-not-applied ci-fix-escalated \ - conflict-patch-invalid conflict-patch-not-applied conflict-patch-protected-path; do +# Held reasons: one comment, one label, both on the PR. +for reason in "${HOLD_REASONS[@]}"; do setup '[]' '[]' run "$reason" check "$reason: exit 0" 0 "$RC" @@ -141,27 +179,64 @@ for reason in protected-path-conflict workflow-staged unexpected-path-staged rep has "$reason: comment says to remove the label" 'remove the `needs-human` label' "$STUB/bodies" has "$reason: comment says to dispatch pr-unblock" 'pr-unblock' "$STUB/bodies" has "$reason: comment names the PR to dispatch" 'pr-number=42' "$STUB/bodies" - case $reason in - feedback-*) want=address-feedback ;; - dependency-*) want=review-dependency ;; - ci-fix-*) want=fix-ci ;; - *) want=resolve-conflicts ;; - esac + want=$(activity_of "$reason") has "$reason: comment names the activity to dispatch" "-f pr-number=42 -f activity=$want" "$STUB/bodies" has "$reason: label defaults to needs-human" 'labels[]=needs-human' "$STUB/calls" check "$reason: the comment run posts no commit status" 0 "$(grep -c statuses "$STUB/requests" || true)" done -ALL_REASONS=(protected-path-conflict workflow-staged unexpected-path-staged repeat-cap unresolved-conflict - feedback-protected-path feedback-round-cap feedback-uncertain feedback-not-landed feedback-invalid - feedback-partial feedback-incomplete feedback-disputed - feedback-unadjudicated feedback-question-intent feedback-question-design feedback-question-security - dependency-round-cap dependency-invalid dependency-fix-not-landed dependency-security-close dependency-escalate - dependency-config-unreadable dependency-files-truncated dependency-control-uncited dependency-workflow-unpinned dependency-fix-refused - ci-fix-round-cap ci-fix-empty ci-fix-invalid ci-fix-protected-path ci-fix-not-applied ci-fix-escalated - conflict-patch-invalid conflict-patch-not-applied conflict-patch-protected-path) +# Fail-routed reasons: one lanes-stopped comment, no label, no status from +# the comment run. +for reason in "${FAIL_REASONS[@]}"; do + setup '[]' '[]' + run "$reason" + want=$(activity_of "$reason") + check "$reason, fail: exit 0" 0 "$RC" + check "$reason, fail: one comment, no label" 'POST repos/o/r/issues/42/comments' "$(grep -E '^(POST|PATCH)' "$STUB/requests")" + check "$reason, fail: marker is the first line" "" "$(head -n 1 "$STUB/bodies")" + check "$reason, fail: no escalation marker" 0 "$(grep -c 'work-items:escalation' "$STUB/bodies" || true)" + has "$reason, fail: names the reason" "- Reason: \`$reason\`" "$STUB/bodies" + has "$reason, fail: names the head" "- Head: \`$HEAD_A\`" "$STUB/bodies" + has "$reason, fail: names the failing status" "failing \`$(context_of "$reason")\` status" "$STUB/bodies" + has "$reason, fail: says a retry or a new head runs again" 'A new commit to the branch starts clean' "$STUB/bodies" + has "$reason, fail: gives the dispatch" "-f pr-number=42 -f activity=$want" "$STUB/bodies" + check "$reason, fail: names no label" 0 "$(grep -c 'needs-human' "$STUB/bodies" "$STUB/calls" | awk -F: '{s+=$2} END {print s}')" + check "$reason, fail: reads no issue labels" 0 "$(grep -c '^GET repos/o/r/issues/42$' "$STUB/requests" || true)" + check "$reason, fail: no commit status" 0 "$(grep -c statuses "$STUB/requests" || true)" +done -# Status mode: a failing pr-unblock/conflict status on the head, only for unresolved-conflict. +# The fail comment is one per activity, edited in place: unchanged, nothing +# is written; on a new head it is edited, never posted again. +STOP_MARKER='' +setup '[]' '[]' +run ci-fix-escalated +fail_body=$(sed '/^=====$/d' "$STUB/bodies") +setup '[]' "$(comment "$fail_body" 'melodic-automation-lanes[bot]' Bot 77)" +run ci-fix-escalated +check "fail, same head, comment unchanged: exit 0" 0 "$RC" +check "fail, same head, comment unchanged: no write" 0 "$(writes)" + +setup '[]' "$(comment "$STOP_MARKER"$'\nolder text' 'melodic-automation-lanes[bot]' Bot 77)" +run ci-fix-escalated "$HEAD_B" +check "fail, new head: edits the comment in place" 'PATCH repos/o/r/issues/comments/77' "$(grep -E '^(POST|PATCH)' "$STUB/requests")" +has "fail, new head: the edit names the new head" "- Head: \`$HEAD_B\`" "$STUB/bodies" + +setup '[]' "$(comment "$STOP_MARKER"$'\nolder text' someone User 77)" +run ci-fix-escalated +check "fail: a marker a user wrote is not the lane's comment" 'POST repos/o/r/issues/42/comments' "$(grep -E '^(POST|PATCH)' "$STUB/requests")" + +setup '[]' "$(comment ''$'\nolder' 'melodic-automation-lanes[bot]' Bot 77)" +run ci-fix-escalated +check "fail: another activity's comment is left alone" 'POST repos/o/r/issues/42/comments' "$(grep -E '^(POST|PATCH)' "$STUB/requests")" + +# A held reason on a PR that already shows a fail comment still holds. +setup '[]' "$(comment "$STOP_MARKER"$'\nolder text' 'melodic-automation-lanes[bot]' Bot 77)" +run ci-fix-protected-path +check "hold beside a fail comment: label then a new comment" \ + $'POST repos/o/r/issues/42/labels\nPOST repos/o/r/issues/42/comments' "$(grep -E '^(POST|PATCH)' "$STUB/requests")" + +# Status mode: a failing commit status on the head for a fail-routed reason, +# pr-unblock/conflict for resolve-conflicts, pr-unblock/ otherwise. setup '[]' '[]' MODE=status TARGET_URL=https://example.test/run/1 run unresolved-conflict check "status, unresolved-conflict: exit 0" 0 "$RC" @@ -173,17 +248,29 @@ has "status, unresolved-conflict: description says CI does not run" \ 'description=Conflicts with the base branch; CI does not run until they are resolved' "$STUB/calls" has "status, unresolved-conflict: links the run" 'target_url=https://example.test/run/1' "$STUB/calls" +# Re-armed on a new head: the status is the head's, so a later head starts +# with none and gets its own only if the lane stops there too. setup '[]' '[]' MODE=status run unresolved-conflict "$HEAD_B" check "status on another head: posted on that head" "POST repos/o/r/statuses/$HEAD_B" "$(cat "$STUB/requests")" check "status without TARGET_URL: no target_url" 0 "$(grep -c target_url "$STUB/calls" || true)" -for reason in "${ALL_REASONS[@]}"; do +for reason in "${FAIL_REASONS[@]}"; do [[ $reason == unresolved-conflict ]] && continue setup '[]' '[]' MODE=status run "$reason" check "status, $reason: exit 0" 0 "$RC" - check "status, $reason: no gh call" '' "$(cat "$STUB/calls")" + check "status, $reason: one status POST on the head" "POST repos/o/r/statuses/$HEAD_A" "$(cat "$STUB/requests")" + has "status, $reason: state failure" 'state=failure' "$STUB/calls" + has "status, $reason: context" "context=$(context_of "$reason")" "$STUB/calls" + has "status, $reason: description names the reason" "description=Lane stopped: $reason; checked again on a new head" "$STUB/calls" +done + +for reason in "${HOLD_REASONS[@]}"; do + setup '[]' '[]' + MODE=status run "$reason" + check "status, held $reason: exit 0" 0 "$RC" + check "status, held $reason: no gh call" '' "$(cat "$STUB/calls")" done setup '[]' '[]' @@ -204,36 +291,36 @@ check "unknown mode: no gh call" '' "$(cat "$STUB/calls")" # The role label resolves from the base tracker config, not from the workspace root. setup '[]' '[]' '{"config":{"role_labels":{"human-gated":"status: blocked-human"}}}' printf '{"config":{"role_labels":{"human-gated":"from-head"}}}' >"$STUB/ws/.work-item-tracker.json" -run repeat-cap +run workflow-staged check "role label: exit 0" 0 "$RC" has "role label: label from config" 'labels[]=status: blocked-human' "$STUB/calls" has "role label: comment names it" 'remove the `status: blocked-human` label' "$STUB/bodies" check "role label: workspace-root config ignored" 0 "$(grep -c from-head "$STUB/calls" || true)" setup '[]' '[]' '{"config":{"role_labels":{"autonomous-eligible":"go"}}}' -run repeat-cap +run workflow-staged has "role label: absent role defaults to needs-human" 'labels[]=needs-human' "$STUB/calls" setup '[]' '[]' '{not json' -run repeat-cap +run workflow-staged check "role label: unparsable config fails" 1 "$([[ $RC -ne 0 ]] && echo 1 || echo 0)" check "role label: unparsable config writes nothing" 0 "$(writes)" for entry in false null 7 '""' '["needs-human"]'; do setup '[]' '[]' "{\"config\":{\"role_labels\":{\"human-gated\":$entry}}}" - run repeat-cap + run workflow-staged check "role label: present entry $entry fails" 1 "$([[ $RC -ne 0 ]] && echo 1 || echo 0)" check "role label: present entry $entry makes no gh call" '' "$(cat "$STUB/calls")" done # Idempotency. setup '[{"name":"needs-human"}]' "$(bot_marker "$HEAD_A")" -run unresolved-conflict +run protected-path-conflict check "re-run on the same head: exit 0" 0 "$RC" check "re-run on the same head: no write" 0 "$(writes)" setup '[{"name":"needs-human"}]' "$(bot_marker "$HEAD_B")" -run unresolved-conflict +run protected-path-conflict check "labeled, marker for another head: comment only" 'POST repos/o/r/issues/42/comments' "$(grep '^POST' "$STUB/requests")" # Another activity's escalation on the same head is recorded too, so @@ -244,7 +331,7 @@ check "labeled, same head, another reason: comment only" 'POST repos/o/r/issues/ has "labeled, same head, another reason: comment names the new reason" 'ci-fix-round-cap' "$STUB/bodies" setup '[]' "$(bot_marker "$HEAD_A")" -run unresolved-conflict +run protected-path-conflict check "label removed by a human, same head: escalates again" \ $'POST repos/o/r/issues/42/labels\nPOST repos/o/r/issues/42/comments' "$(grep '^POST' "$STUB/requests")" @@ -253,13 +340,13 @@ check "label removed by a human, same head: escalates again" \ LABEL_THEN_3_COMMENTS=$'POST repos/o/r/issues/42/labels\nPOST repos/o/r/issues/42/comments\nPOST repos/o/r/issues/42/comments\nPOST repos/o/r/issues/42/comments' setup '[]' '[]' touch "$STUB/fail_labels" -run unresolved-conflict +run protected-path-conflict check "label write fails: exit 1" 1 "$([[ $RC -ne 0 ]] && echo 1 || echo 0)" check "label write fails: no comment posted" 0 "$(grep -c '^POST repos/o/r/issues/42/comments$' "$STUB/requests" || true)" setup '[]' '[]' echo 1 >"$STUB/fail_comments" -run unresolved-conflict +run protected-path-conflict check "comment write fails once: exit 0" 0 "$RC" check "comment write fails once: label, then the comment twice" \ $'POST repos/o/r/issues/42/labels\nPOST repos/o/r/issues/42/comments\nPOST repos/o/r/issues/42/comments' \ @@ -272,7 +359,7 @@ check "comment write fails once: backs off once" 'sleep 5' "$(cat "$STUB/sleeps" exhausted() { setup '[]' '[]' echo 99 >"$STUB/fail_comments" - run unresolved-conflict + run protected-path-conflict check "$1: comment write fails every attempt: exit 1" 1 "$([[ $RC -ne 0 ]] && echo 1 || echo 0)" check "$1: comment write fails every attempt: label on, three comment attempts" \ "$LABEL_THEN_3_COMMENTS" "$(grep '^POST' "$STUB/requests")" @@ -283,33 +370,33 @@ exhausted() { } exhausted "retry after a failed comment" -run unresolved-conflict +run protected-path-conflict check "retry after a failed comment: exit 0" 0 "$RC" check "retry after a failed comment: one comment, no label" 'POST repos/o/r/issues/42/comments' "$(grep '^POST' "$STUB/requests")" # A label left by a failed comment write is explained by the next run, on a later head too. exhausted "label left, later head" -run unresolved-conflict "$HEAD_B" +run protected-path-conflict "$HEAD_B" check "label left, later head: exit 0" 0 "$RC" check "label left, later head: explains the label" 'POST repos/o/r/issues/42/comments' "$(grep '^POST' "$STUB/requests")" has "label left, later head: comment carries the later resume key" "resume-key=$HEAD_B" "$STUB/bodies" setup '[{"name":"needs-human"}]' "$(comment "$MARKER"$'\n'"" someone User)" -run unresolved-conflict +run protected-path-conflict check "marker written by a user does not count" 'POST repos/o/r/issues/42/comments' "$(grep '^POST' "$STUB/requests")" setup '[{"name":"needs-human"}]' "$(comment "$MARKER"$'\n'"" 'other-app[bot]' Bot)" -run unresolved-conflict +run protected-path-conflict check "marker written by another bot does not count" 'POST repos/o/r/issues/42/comments' "$(grep '^POST' "$STUB/requests")" -setup '[{"name":"needs-human"}]' "$(comment "$MARKER"$'\n'""$'\n- Reason: `unresolved-conflict`' 'other-app[bot]' Bot)" -LBL='other-app[bot]' run unresolved-conflict +setup '[{"name":"needs-human"}]' "$(comment "$MARKER"$'\n'""$'\n- Reason: `protected-path-conflict`' 'other-app[bot]' Bot)" +LBL='other-app[bot]' run protected-path-conflict check "LANES_BOT_LOGIN names the lanes identity: no write" 0 "$(writes)" -# feedback-not-converging: the comment names each open review thread, gives -# no count of its own (select's count, which also reads model check runs, -# is on the neutral checks), and resumes through the threads, never through -# removing the label alone, which would stop again on the same head. +# feedback-not-converging, a cap: no label; the comment names each open +# review thread, gives no count of its own (select's count, which also reads +# model check runs, is on the neutral checks), and names the write-access fix +# request that grants one more round. ROUND='' rounds_json=$(jq -n --arg r "$ROUND" '[range(4) | {body: ($r + "\npr-unblock / address-feedback finished round \(. + 1) of max 4 on x."), user: {login: "melodic-automation-lanes[bot]", type: "Bot"}}]') setup '[]' "$rounds_json" @@ -330,10 +417,9 @@ has "not-converging: a path is shown without backticks or line breaks" '- `docs/ check "not-converging: a path cannot open an HTML comment" 0 "$(grep -c ''$'\nold' 'melodic-automation-lanes[bot]' Bot 9)" + run "$reason" + check "$reason: never runs git" 0 "$(grep -c '^git' "$STUB/calls" || true)" + check "$reason: never calls the contents API" 0 "$(grep -c 'contents/' "$STUB/calls" || true)" + check "$reason: writes only the issue, its comments and labels" 0 \ + "$(grep -vE '^(GET|POST) repos/o/r/issues/42(/comments(\?per_page=100)?|/labels)?$|^PATCH repos/o/r/issues/comments/9$' "$STUB/requests" | grep -c . || true)" +done exit "$FAILED" diff --git a/.github/scripts/pr-unblock/workflow-gates.test.mjs b/.github/scripts/pr-unblock/workflow-gates.test.mjs index ef31de86b7..20def222b5 100644 --- a/.github/scripts/pr-unblock/workflow-gates.test.mjs +++ b/.github/scripts/pr-unblock/workflow-gates.test.mjs @@ -493,7 +493,7 @@ const CONFLICT_JOBS = [ "resolve-conflicts-subscription-handoff", "resolve-conflicts-fallback", "apply-conflict-patch", - "report-conflict-status", + "report-stop-status", ]; // As for review-dependency: a job that needs a skipped subscription call and diff --git a/.github/workflows/pr-merge.yml b/.github/workflows/pr-merge.yml index 6b7c137e22..e7b72e4740 100644 --- a/.github/workflows/pr-merge.yml +++ b/.github/workflows/pr-merge.yml @@ -1449,6 +1449,9 @@ jobs: timeout-minutes: 15 env: GH_TOKEN: ${{ steps.lane-token.outputs.token }} + # The job's own token reads a fix-ci stop's head runs and checks + # (actions: read, checks: read); the lane token holds neither. + CHECKS_TOKEN: ${{ github.token }} BABYSIT_VERIFY_GH_TOKEN: ${{ github.token }} BABYSIT_VERIFY_LISTING: ${{ runner.temp }}/verify-listing.json BABYSIT_VERIFY_HEAD_REFS: ${{ runner.temp }}/head-refs-in.json diff --git a/.github/workflows/pr-require-checks.yml b/.github/workflows/pr-require-checks.yml index e4e9d0db8a..a630323714 100644 --- a/.github/workflows/pr-require-checks.yml +++ b/.github/workflows/pr-require-checks.yml @@ -3233,11 +3233,12 @@ jobs: # of own-gate retries, 7 s of status-write retries and the 90 s # `rerun-wait-seconds` wait, about four and a half minutes, plus up to # 15 s of retry sleeps for each of the hold-lift step's two reads, the - # review-lane step's six and the lanes-hold step's eight (in the merge + # review-lane step's six, the lanes-hold step's eight (in the merge # queue the verdict reader's reads more, about 73 with pr-merge's carried # reads and about 860 without them, measured 2026-10-25; its wall time is - # unmeasured, and a cold one can reach the cap, which fails red), about four - # minutes, plus the cost-hold step's reads: one for most pull requests, + # unmeasured, and a cold one can reach the cap, which fails red) and the + # pr-unblock status step's one (two in the merge queue), about four and a + # half minutes, plus the cost-hold step's reads: one for most pull requests, # which stop at the label events; for a lane-labeled one whose body cites a # decision, about eight (nine on a Refs issue), and two more per review # run since the window opened, plus the base-config checkout's fetch @@ -4463,7 +4464,7 @@ jobs: } if [[ $AUTHOR_ID =~ ^[0-9]+$ && " $exempt_author_ids " == *" $AUTHOR_ID "* ]]; then # The trust-gate paths, as bash patterns (`*` crosses `/`). - hold_patterns=('.github/standards/trusted-actors/*' '.github/actions/check-trusted-trigger/*' '.github/actions/resolve-config/*' '.github/actions/request-lane-token/*' '.github/actions/check-kill-switch/*' '.github/actions/log-api-budget/rate-guard.mjs' '.github/actions/log-api-budget/rate-limits.json' '.github/scripts/pr-merge/merge-holds.mjs' '.github/scripts/pr-merge/enable-auto-merge.mjs' '.github/scripts/pr-merge/superseded.mjs' '.github/scripts/pr-merge/clear-escalations.mjs' '.github/scripts/pr-merge/carry-run-cache.mjs' '.github/scripts/pr-merge/gate-check.mjs' '.github/scripts/pr-merge/head-ref-map.mjs' '.github/scripts/pr-merge/lane-records.mjs' '.github/scripts/pr-merge/rate-stop.bash' '.github/scripts/pr-merge/refire-refused.mjs' '.github/scripts/pr-merge/rotation.mjs' '.github/scripts/pr-merge/sweep-summary.mjs' '.github/scripts/pr-merge/who.mjs' '.github/scripts/pr-merge/post-unarmed-status.mjs' '.github/scripts/pr-merge/verify-answers.mjs' '.github/scripts/pr-unblock/address-feedback.mjs' '.github/scripts/pr-unblock/gate-source.mjs' '.github/scripts/pr-unblock/thread-scope.mjs' '.github/scripts/pr-unblock/retry-fetch.mjs' '.github/scripts/work-resolve-issue/report-outcome.mjs' '.github/scripts/work-resolve-issue/limits.mjs' '.github/scripts/work-resolve-issue/retry-fetch.mjs' '.github/workflows/pr-require-checks.yml' '.github/workflows/pr-merge.yml' 'plugins/source-control/skills/babysit-prs/scripts/babysit_risk.py' 'plugins/source-control/skills/babysit-prs/scripts/babysit_check_inputs.py' 'plugins/source-control/skills/babysit-prs/scripts/babysit_checks.py' 'plugins/source-control/skills/babysit-prs/scripts/babysit_classify.py' 'plugins/source-control/skills/babysit-prs/scripts/babysit_dependabot.py' 'plugins/source-control/skills/babysit-prs/scripts/babysit_feedback.py' 'plugins/source-control/skills/babysit-prs/scripts/babysit_gh.py' 'plugins/source-control/skills/babysit-prs/scripts/babysit_merge.py' 'plugins/source-control/skills/babysit-prs/scripts/babysit_merge_policy.py' 'plugins/source-control/skills/babysit-prs/scripts/babysit_postmerge.py' 'plugins/source-control/skills/babysit-prs/scripts/babysit_repo_config.py' 'plugins/source-control/skills/babysit-prs/scripts/babysit_review_evidence.py' 'plugins/source-control/skills/babysit-prs/scripts/babysit_review_trigger.py' 'plugins/source-control/skills/babysit-prs/scripts/babysit_run_cache.py' 'plugins/source-control/skills/babysit-prs/scripts/babysit_state.py' 'plugins/source-control/skills/babysit-prs/scripts/babysit_util.py' 'plugins/source-control/skills/babysit-prs/scripts/babysit_verify_evidence.py' 'plugins/source-control/skills/babysit-prs/scripts/babysit_whole_review.py' 'docs/conventions/pr-pipeline.yaml' 'CODEOWNERS' '*/CODEOWNERS' '.gitmodules' '*/.gitmodules') + hold_patterns=('.github/standards/trusted-actors/*' '.github/actions/check-trusted-trigger/*' '.github/actions/resolve-config/*' '.github/actions/request-lane-token/*' '.github/actions/check-kill-switch/*' '.github/actions/log-api-budget/rate-guard.mjs' '.github/actions/log-api-budget/rate-limits.json' '.github/scripts/pr-merge/merge-holds.mjs' '.github/scripts/pr-merge/enable-auto-merge.mjs' '.github/scripts/pr-merge/superseded.mjs' '.github/scripts/pr-merge/clear-escalations.mjs' '.github/scripts/pr-merge/carry-run-cache.mjs' '.github/scripts/pr-merge/gate-check.mjs' '.github/scripts/pr-merge/head-ref-map.mjs' '.github/scripts/pr-merge/lane-records.mjs' '.github/scripts/pr-merge/rate-stop.bash' '.github/scripts/pr-merge/refire-refused.mjs' '.github/scripts/pr-merge/rotation.mjs' '.github/scripts/pr-merge/sweep-summary.mjs' '.github/scripts/pr-merge/who.mjs' '.github/scripts/pr-merge/post-unarmed-status.mjs' '.github/scripts/pr-merge/verify-answers.mjs' '.github/scripts/pr-unblock/address-feedback.mjs' '.github/scripts/pr-unblock/escalation-routes.mjs' '.github/scripts/pr-unblock/escalation-routes.json' '.github/scripts/pr-unblock/gate-source.mjs' '.github/scripts/pr-unblock/thread-scope.mjs' '.github/scripts/pr-unblock/retry-fetch.mjs' '.github/scripts/work-resolve-issue/report-outcome.mjs' '.github/scripts/work-resolve-issue/limits.mjs' '.github/scripts/work-resolve-issue/retry-fetch.mjs' '.github/scripts/pr-unblock/report-escalation.sh' '.github/scripts/pr-unblock/adjudicate-feedback.mjs' '.github/scripts/pr-unblock/publish-feedback.mjs' '.github/scripts/pr-unblock/request-budget.mjs' '.github/scripts/pr-unblock/dependency-classify.mjs' '.github/scripts/apply-patch/apply-patch.mjs' '.github/scripts/apply-patch/predicates.mjs' '.github/scripts/apply-patch/protected-paths.mjs' '.github/scripts/apply-patch/retry-fetch.mjs' '.github/workflows/pr-require-checks.yml' '.github/workflows/pr-merge.yml' 'plugins/source-control/skills/babysit-prs/scripts/babysit_risk.py' 'plugins/source-control/skills/babysit-prs/scripts/babysit_check_inputs.py' 'plugins/source-control/skills/babysit-prs/scripts/babysit_checks.py' 'plugins/source-control/skills/babysit-prs/scripts/babysit_classify.py' 'plugins/source-control/skills/babysit-prs/scripts/babysit_dependabot.py' 'plugins/source-control/skills/babysit-prs/scripts/babysit_feedback.py' 'plugins/source-control/skills/babysit-prs/scripts/babysit_gh.py' 'plugins/source-control/skills/babysit-prs/scripts/babysit_merge.py' 'plugins/source-control/skills/babysit-prs/scripts/babysit_merge_policy.py' 'plugins/source-control/skills/babysit-prs/scripts/babysit_postmerge.py' 'plugins/source-control/skills/babysit-prs/scripts/babysit_repo_config.py' 'plugins/source-control/skills/babysit-prs/scripts/babysit_review_evidence.py' 'plugins/source-control/skills/babysit-prs/scripts/babysit_review_trigger.py' 'plugins/source-control/skills/babysit-prs/scripts/babysit_run_cache.py' 'plugins/source-control/skills/babysit-prs/scripts/babysit_state.py' 'plugins/source-control/skills/babysit-prs/scripts/babysit_util.py' 'plugins/source-control/skills/babysit-prs/scripts/babysit_verify_evidence.py' 'plugins/source-control/skills/babysit-prs/scripts/babysit_whole_review.py' 'docs/conventions/pr-pipeline.yaml' 'CODEOWNERS' '*/CODEOWNERS' '.gitmodules' '*/.gitmodules') if [[ ! $PR_NUMBER =~ ^[1-9][0-9]*$ ]]; then echo "::error::the pull request number '$PR_NUMBER' is not a number." exit 1 @@ -4891,6 +4892,65 @@ jobs: # reader holds it (verify:spec-changed). A record or link that does not # read holds. # Pinned by scripts/ci-lanes-pr-hold.test.sh, which runs this body. + # A HEAD A PR-UNBLOCK LANE STOPPED IS HELD. A lane stop that + # escalation-routes.json routes `fail` ends in a failing + # `pr-unblock/` commit status on the head (pr-unblock.yml + # report-stop-status), and only ci-status is required, so this step + # fails while the newest status of any `pr-unblock/*` context on the + # head is `failure` or `error`, naming each. Newest per context, by + # github-actions[bot] only, as report-escalation.sh `clear` reads it: + # its later `success` on the same head supersedes the failure, and + # another account's status neither holds nor clears. On a pull request + # event the head is the event's; in the merge queue it is the head of + # the pull request named in the group's ref, read live (the ref's SHA is + # the group's base). A status event re-runs nothing, so a stop posted + # after this step passed holds at the merge queue's run. An unparsable + # ref, a head that is not a commit SHA, or an unreadable pull request or + # status list fails closed. + # Pinned by scripts/ci-hold-pr-unblock-status.test.sh, which runs this body. + - name: Fail a head a pr-unblock lane stopped + id: pr_unblock_status + if: ${{ !cancelled() && (github.event_name == 'pull_request' || github.event_name == 'merge_group') }} + env: + GH_TOKEN: ${{ github.token }} + HEAD_SHA: ${{ github.event.pull_request.head.sha }} + run: | + . "$RUNNER_TEMP/gh-read.sh" + read3() { + local attempt + for attempt in 1 2 3; do + gh_read "${@:2}" && return 0 + [ "$attempt" -lt 3 ] || { echo "::error::could not read $1; re-run this run." >&2; return 1; } + sleep $((attempt * 5)) + done + } + head=$HEAD_SHA + if [ "$GITHUB_EVENT_NAME" = merge_group ]; then + pr=$(sed -nE 's#^(refs/heads/)?gh-readonly-queue/.+/pr-([0-9]+)-[0-9a-f]{40}$#\2#p' <<<"$GITHUB_REF") + if [ -z "$pr" ]; then + echo "::error::cannot read a pull request number from the merge-group ref '$GITHUB_REF'." + exit 1 + fi + pull=$(read3 "#$pr" "repos/$GITHUB_REPOSITORY/pulls/$pr") || exit 1 + head=$(jq -r '.head.sha // ""' <<<"$pull") || head= + fi + if [[ ! $head =~ ^[0-9a-f]{40}$ ]]; then + echo "::error::the head '$head' is not a commit SHA." + exit 1 + fi + # Newest first across pages, so the first row of a context is its newest. + rows=$(read3 "the statuses of $head" "repos/$GITHUB_REPOSITORY/commits/$head/statuses?per_page=100" --paginate \ + --jq '.[] | select((.context | startswith("pr-unblock/")) and .creator.login == "github-actions[bot]") + | [.context, .state, (.description // "" | gsub("[\t\r\n]"; " "))] | @tsv') || exit 1 + failing=$(awk -F'\t' '!seen[$1]++ && ($2 == "failure" || $2 == "error")' <<<"$rows") + if [ -z "$failing" ]; then + echo "No pr-unblock/* status fails on $head." + exit 0 + fi + while IFS=$'\t' read -r context state description; do + echo "::error::'$context' is $state on $head: ${description//%/%25}. The lane that stopped clears it by running again on this head, or a new head carries none." + done <<<"$failing" + exit 1 - name: Fail a lanes App pull request its body or a member holds if: ${{ !cancelled() && (github.event_name == 'pull_request' || github.event_name == 'merge_group') }} env: diff --git a/.github/workflows/pr-review-hosted.yml b/.github/workflows/pr-review-hosted.yml index 8674a1c918..88c4bd0f63 100644 --- a/.github/workflows/pr-review-hosted.yml +++ b/.github/workflows/pr-review-hosted.yml @@ -223,12 +223,23 @@ jobs: .github/scripts/pr-merge/post-unarmed-status.mjs .github/scripts/pr-merge/verify-answers.mjs .github/scripts/pr-unblock/address-feedback.mjs + .github/scripts/pr-unblock/escalation-routes.mjs + .github/scripts/pr-unblock/escalation-routes.json .github/scripts/pr-unblock/gate-source.mjs .github/scripts/pr-unblock/thread-scope.mjs .github/scripts/pr-unblock/retry-fetch.mjs .github/scripts/work-resolve-issue/report-outcome.mjs .github/scripts/work-resolve-issue/limits.mjs .github/scripts/work-resolve-issue/retry-fetch.mjs + .github/scripts/pr-unblock/report-escalation.sh + .github/scripts/pr-unblock/adjudicate-feedback.mjs + .github/scripts/pr-unblock/publish-feedback.mjs + .github/scripts/pr-unblock/request-budget.mjs + .github/scripts/pr-unblock/dependency-classify.mjs + .github/scripts/apply-patch/apply-patch.mjs + .github/scripts/apply-patch/predicates.mjs + .github/scripts/apply-patch/protected-paths.mjs + .github/scripts/apply-patch/retry-fetch.mjs .github/workflows/pr-require-checks.yml .github/workflows/pr-merge.yml plugins/source-control/skills/babysit-prs/scripts/babysit_risk.py diff --git a/.github/workflows/pr-review-security-hosted.yml b/.github/workflows/pr-review-security-hosted.yml index 47fb86da95..1b1570ba90 100644 --- a/.github/workflows/pr-review-security-hosted.yml +++ b/.github/workflows/pr-review-security-hosted.yml @@ -223,12 +223,23 @@ jobs: .github/scripts/pr-merge/post-unarmed-status.mjs .github/scripts/pr-merge/verify-answers.mjs .github/scripts/pr-unblock/address-feedback.mjs + .github/scripts/pr-unblock/escalation-routes.mjs + .github/scripts/pr-unblock/escalation-routes.json .github/scripts/pr-unblock/gate-source.mjs .github/scripts/pr-unblock/thread-scope.mjs .github/scripts/pr-unblock/retry-fetch.mjs .github/scripts/work-resolve-issue/report-outcome.mjs .github/scripts/work-resolve-issue/limits.mjs .github/scripts/work-resolve-issue/retry-fetch.mjs + .github/scripts/pr-unblock/report-escalation.sh + .github/scripts/pr-unblock/adjudicate-feedback.mjs + .github/scripts/pr-unblock/publish-feedback.mjs + .github/scripts/pr-unblock/request-budget.mjs + .github/scripts/pr-unblock/dependency-classify.mjs + .github/scripts/apply-patch/apply-patch.mjs + .github/scripts/apply-patch/predicates.mjs + .github/scripts/apply-patch/protected-paths.mjs + .github/scripts/apply-patch/retry-fetch.mjs .github/workflows/pr-require-checks.yml .github/workflows/pr-merge.yml plugins/source-control/skills/babysit-prs/scripts/babysit_risk.py diff --git a/.github/workflows/pr-run-activity-read.yml b/.github/workflows/pr-run-activity-read.yml index 89fa73a554..54138d9b96 100644 --- a/.github/workflows/pr-run-activity-read.yml +++ b/.github/workflows/pr-run-activity-read.yml @@ -2555,11 +2555,12 @@ jobs: if (patch) core.setOutput('patch-sha256', crypto.createHash('sha256').update(patch).digest('hex')); if (replies) { core.setOutput('replies-sha256', crypto.createHash('sha256').update(replies).digest('hex')); - // adjudicate-feedback's gate: an INCORRECT reply, or a line - // that is not JSON, which the adjudicator's own checks refuse. + // adjudicate-feedback's gate: a reply that leaves its thread + // open (adjudicate-feedback.mjs DISPUTED), or a line that is + // not JSON, which the adjudicator's own checks refuse. const disputed = replies.toString('utf8').split('\n').filter((line) => line.trim() !== '').some((line) => { try { - return JSON.parse(line)?.disposition === 'INCORRECT'; + return ['INCORRECT', 'UNCERTAIN', 'VALID-defer'].includes(JSON.parse(line)?.disposition); } catch { return true; } diff --git a/.github/workflows/pr-run-activity-write.yml b/.github/workflows/pr-run-activity-write.yml index cd5ad5d0db..7fce072a8c 100644 --- a/.github/workflows/pr-run-activity-write.yml +++ b/.github/workflows/pr-run-activity-write.yml @@ -400,17 +400,21 @@ on: outputs: escalation-reason: description: >- - Why the activity handed the PR to a person: for the + Why the activity stopped (report-escalation routes it to a person + or to a failing status, escalation-routes.json): for the review-replies script feedback-uncertain, feedback-not-landed or feedback-invalid, feedback-partial, feedback-incomplete, - feedback-disputed, feedback-unadjudicated, feedback-question-intent, + (feedback-publish-failed is the caller's, for a reply job that failed), + feedback-disputed, feedback-stood-open, feedback-verifier-open, + feedback-unadjudicated, feedback-question-intent, feedback-question-design or feedback-question-security; for the dependency-verdict script dependency-invalid, dependency-fix-not-landed, dependency-config-unreadable, dependency-files-truncated, dependency-control-uncited, dependency-workflow-unpinned, dependency-fix-refused, dependency-security-close or dependency-escalate; for apply-patch, before the mint, - ci-fix-empty, ci-fix-invalid or ci-fix-escalated, and from its + ci-fix-empty, ci-fix-invalid, ci-fix-escalated, ci-fix-protected-path + or ci-fix-weaken-required, and from its script ci-fix-invalid, ci-fix-protected-path or ci-fix-not-applied; for issue-mode apply-patch, patch-empty, patch-invalid, patch-not-applied or patch-protected-path; for apply-patch under @@ -430,6 +434,13 @@ on: apply-patch: the skip reason the patch's select gave, empty when it proceeded. value: ${{ jobs.run.outputs.patch-skip-reason }} + skip-reason: + description: >- + Why the run did no work: the kill switch's or trigger gate's reason, + or the skip reason its check run posts (a script's own, such as the + reply script's no-model-turn); empty when it acted. A caller that + reads a success result as a landed attempt checks this is empty. + value: ${{ jobs.run.outputs.skip-reason }} head-sha: description: The PR head the gate passed, before any push of this run. Set only by runner steps. value: ${{ jobs.run.outputs.head-sha }} @@ -481,6 +492,15 @@ jobs: ${{ steps.act-script.outputs.escalation-reason || steps.patch-select.outputs.escalation-reason }} rung-up: ${{ steps.patch-select.outputs.skip-reason == 'rung-up' }} patch-skip-reason: ${{ steps.patch-select.outputs.skip-reason }} + # The verdict step's skip reason, after the gate's reason. + skip-reason: >- + ${{ steps.kill.outputs.proceed != 'true' && steps.kill.outputs.reason + || steps.gate.outputs.proceed != 'true' && steps.gate.outputs.reason + || steps.config.outputs.skip-reason || steps.head-recheck.outputs.skip-reason + || steps.lane-token.outputs.skip-reason + || steps.prepare.outputs.skip-reason || steps.promote.outputs.skip-reason + || steps.act-script.outputs.skip-reason + || steps.patch-select.outputs.skip-reason }} # The base action's own failure text, so the report names why the run # job failed before its act step. run-failure: ${{ steps.lane-token.outputs.failure-reason }} @@ -584,14 +604,15 @@ jobs: "" | protected-path-conflict | workflow-staged | unexpected-path-staged | repeat-cap | unresolved-conflict) ;; feedback-protected-path | feedback-round-cap | feedback-not-converging | feedback-uncertain | feedback-not-landed | feedback-invalid) ;; feedback-partial | feedback-incomplete | feedback-disputed) ;; + feedback-untaken-threads | feedback-publish-failed) ;; feedback-unadjudicated | feedback-question-intent | feedback-question-design) ;; - feedback-question-security) ;; + feedback-question-security | feedback-stood-open | feedback-verifier-open) ;; dependency-round-cap | dependency-invalid | dependency-fix-not-landed) ;; dependency-security-close | dependency-escalate | dependency-config-unreadable) ;; dependency-files-truncated | dependency-control-uncited | dependency-workflow-unpinned) ;; dependency-fix-refused) ;; ci-fix-round-cap | ci-fix-empty | ci-fix-invalid | ci-fix-escalated) ;; - ci-fix-protected-path | ci-fix-not-applied) ;; + ci-fix-protected-path | ci-fix-not-applied | ci-fix-weaken-required) ;; conflict-patch-invalid | conflict-patch-not-applied | conflict-patch-protected-path) ;; patch-empty | patch-invalid | patch-not-applied | patch-protected-path | patch-apply-failed) if [ -z "$ISSUE_NUMBER" ]; then @@ -1203,10 +1224,10 @@ jobs: echo "required=$required" >>"$GITHUB_OUTPUT" # apply-patch: the head checkout replaces .base, so the script's sibling - # imports (thread-scope.mjs, dependency-classify.mjs and merge-holds.mjs, - # beside the script's base-script copy), the patch-rules.json it reads, - # and the dependabot.yml whose pairs the dependency-manifests rule reads - # are copied out of the base first. + # imports (thread-scope.mjs, dependency-classify.mjs, merge-holds.mjs and + # escalation-routes.mjs with its table, beside the script's base-script + # copy), the patch-rules.json it reads, and the dependabot.yml whose pairs + # the dependency-manifests rule reads are copied out of the base first. - name: Copy the apply-patch base files id: apply-base if: steps.lane-token.outputs.token != '' && steps.config.outputs.prepare == 'apply-patch' @@ -1217,6 +1238,8 @@ jobs: mkdir -p "$RUNNER_TEMP/pr-unblock" "$RUNNER_TEMP/pr-merge" "$DEST/$(dirname "$RULES")" cp .base/.github/scripts/pr-unblock/thread-scope.mjs "$RUNNER_TEMP/pr-unblock/" cp .base/.github/scripts/pr-unblock/dependency-classify.mjs "$RUNNER_TEMP/pr-unblock/" + cp .base/.github/scripts/pr-unblock/escalation-routes.mjs "$RUNNER_TEMP/pr-unblock/" + cp .base/.github/scripts/pr-unblock/escalation-routes.json "$RUNNER_TEMP/pr-unblock/" cp .base/.github/scripts/pr-merge/merge-holds.mjs "$RUNNER_TEMP/pr-merge/" cp ".base/$RULES" "$DEST/$RULES" # A base without one leaves the rule's pairs unreadable: a row that @@ -1933,7 +1956,7 @@ jobs: ?*:feedback-uncertain | ?*:feedback-not-landed | ?*:feedback-invalid) ;; ?*:feedback-partial | ?*:feedback-incomplete | ?*:feedback-disputed) ;; ?*:feedback-unadjudicated | ?*:feedback-question-intent | ?*:feedback-question-design) ;; - ?*:feedback-question-security) ;; + ?*:feedback-question-security | ?*:feedback-stood-open | ?*:feedback-verifier-open) ;; :ci-fix-invalid | :ci-fix-protected-path | :ci-fix-not-applied | :patch-invalid | :patch-protected-path | :patch-not-applied | :patch-empty) [ "$PREPARE" = apply-patch ] || { echo "::error::the escalation reason file holds no reason this script may give"; exit 1; } ;; diff --git a/.github/workflows/pr-unblock.yml b/.github/workflows/pr-unblock.yml index 2f0bf6074e..2fcc0f4dd4 100644 --- a/.github/workflows/pr-unblock.yml +++ b/.github/workflows/pr-unblock.yml @@ -261,8 +261,10 @@ jobs: # no model job, so no resolve-conflicts check would land on the head and # dispatch.mjs would dispatch it again on every run. This job posts it # neutral on the head the select job read: the slot's skip reason, else - # clean-merge (the push job commits it), else awaiting-human. A - # kill-switch stop reads no head, and nothing is posted. + # clean-merge (the push job commits it), else the stop's own skip + # (escalation-routes.mjs stopSkip: lane-stopped for a stop routed to a + # failing status, which ends the head; awaiting-human for one held for a + # person). A kill-switch stop reads no head, and nothing is posted. resolve-conflicts-not-run: needs: resolve-conflicts-select if: >- @@ -288,6 +290,12 @@ jobs: node-version: 24 package-manager-cache: false + - name: Read the stop's skip reason + id: stop + env: + REASON: ${{ needs.resolve-conflicts-select.outputs.escalation-reason }} + run: echo "skip=$(node .base/.github/scripts/pr-unblock/escalation-routes.mjs skip)" >>"$GITHUB_OUTPUT" + - name: Post the resolve-conflicts check run uses: ./.base/.github/actions/report-check-run # zizmor: ignore[self-repository] with: @@ -297,7 +305,7 @@ jobs: not-run-reason: >- ${{ needs.resolve-conflicts-select.outputs.skip-reason || needs.resolve-conflicts-select.outputs.merge-state == 'clean' && 'clean-merge' - || 'awaiting-human' }} + || steps.stop.outputs.skip }} head-sha: ${{ needs.resolve-conflicts-select.outputs.head-sha }} can-commit: "false" base-path: .base @@ -619,9 +627,15 @@ jobs: escalation-reason: ${{ steps.select.outputs.escalation-reason }} skip-reason: ${{ steps.slot.outputs.applies == 'false' && steps.slot.outputs.skip-reason || steps.select.outputs.skip-reason }} fix-requested: ${{ steps.select.outputs.fix-requested }} + # "1" for a same-head retry after the reply job stopped there + # (escalation-routes.mjs retryRung), else empty. + model-rung: ${{ steps.select.outputs.model-rung }} # `round of max ` when the round cap stopped the PR, for the # neutral checks' summary. not-run-detail: ${{ steps.select.outputs.not-run-detail }} + # true when no hold and no unresolved thread is left on the head, so + # report-stop-status clears a failing pr-unblock/address-feedback. + stop-resolved: ${{ steps.select.outputs.stop-resolved }} threads-sha256: ${{ steps.select.outputs.threads-sha256 }} run-sha256: ${{ steps.select.outputs.run-sha256 }} # The artifact name's attempt, so a re-run of a later job alone still @@ -818,6 +832,7 @@ jobs: pr-number: ${{ inputs.pr-number }} dispatcher-run-id: ${{ inputs.dispatcher-run-id }} trusted-base: default-tip + model-rung: ${{ needs.address-feedback-select.outputs.model-rung }} feedback-context-attempt: ${{ needs.address-feedback-select.outputs.context-attempt }} feedback-context-sha256: ${{ needs.address-feedback-select.outputs.threads-sha256 }} head-sha: ${{ needs.address-feedback-select.outputs.head-sha }} @@ -895,6 +910,7 @@ jobs: pr-number: ${{ inputs.pr-number }} dispatcher-run-id: ${{ inputs.dispatcher-run-id }} trusted-base: default-tip + model-rung: ${{ needs.address-feedback-select.outputs.model-rung }} feedback-context-attempt: ${{ needs.address-feedback-select.outputs.context-attempt }} feedback-context-sha256: ${{ needs.address-feedback-select.outputs.threads-sha256 }} head-sha: ${{ needs.address-feedback-select.outputs.head-sha }} @@ -960,6 +976,7 @@ jobs: pr-number: ${{ inputs.pr-number }} dispatcher-run-id: ${{ inputs.dispatcher-run-id }} trusted-base: default-tip + model-rung: ${{ needs.address-feedback-select.outputs.model-rung }} feedback-context-attempt: ${{ needs.address-feedback-select.outputs.context-attempt }} feedback-context-sha256: ${{ needs.address-feedback-select.outputs.threads-sha256 }} head-sha: ${{ needs.address-feedback-select.outputs.head-sha }} @@ -975,7 +992,7 @@ jobs: # without force. It runs only on a written patch: no change (every thread # INCORRECT or UNCERTAIN) needs no push. report-escalation does not need it: # a patch that does not land leaves no fix commit, and publish-feedback - # hands the PR to a person as feedback-not-landed. + # ends the round feedback-not-landed. apply-feedback-patch: # After adjudicate-feedback too, so the push lands right before the reply # job: a review run on the pushed head that finishes while this run still @@ -1023,9 +1040,10 @@ jobs: reply-attempt: ${{ needs.address-feedback-fallback.result != 'skipped' && needs.address-feedback-fallback.outputs.scan-reply-attempt || needs.address-feedback-fallback.result == 'skipped' && needs.address-feedback-subscription.result != 'skipped' && needs.address-feedback-subscription.outputs.scan-reply-attempt || needs.address-feedback-fallback.result == 'skipped' && needs.address-feedback-subscription.result == 'skipped' && needs.address-feedback.outputs.scan-reply-attempt || '' }} # The adjudicator, after the model job and before the reply job, only when - # the answering leg's scan job read an INCORRECT reply (feedback-disputed); - # skipping it leaves any such reply with no verdict, which escalates - # feedback-unadjudicated. For each INCORRECT reply on a non-CRITICAL + # the answering leg's scan job read a reply that leaves its thread open + # (INCORRECT, UNCERTAIN or VALID-defer: feedback-disputed); skipping it + # leaves any such reply with no verdict, which ends + # feedback-unadjudicated. For each such reply on a non-CRITICAL # code-review thread no adjudication stood before, up to three # (adjudicate-feedback.mjs MAX_THREADS), a fresh-context model reads the # whole review comment, the locations the reviewer and the lane cited and @@ -1885,11 +1903,17 @@ jobs: head-sha: ${{ steps.context.outputs.head-sha }} escalation-reason: ${{ steps.context.outputs.escalation-reason }} context-sha256: ${{ steps.context.outputs.context-sha256 }} - # awaiting-human when the human-gated label holds fix-ci, which - # fix-ci-not-run posts; lane-escalated when another lane's escalation + # awaiting-human when the human-gated label holds fix-ci, lane-stopped + # on the round cap, which fix-ci-not-run posts; lane-escalated when another lane's escalation # put the label on, so apply-patch pushes past it. skip-reason: ${{ steps.context.outputs.skip-reason }} lane-escalated: ${{ steps.context.outputs.lane-escalated }} + # true when a whole, settled listing of the head's checks has none + # failing, so report-stop-status clears a failing pr-unblock/fix-ci. + stop-resolved: ${{ steps.context.outputs.stop-resolved }} + # "1" for a same-head retry after the push job stopped there + # (escalation-routes.mjs retryRung), else empty. + model-rung: ${{ steps.context.outputs.model-rung }} # The artifact name's attempt, so a re-run of the model job alone still # finds it. context-attempt: ${{ steps.upload.outcome == 'success' && github.run_attempt || '' }} @@ -1958,7 +1982,7 @@ jobs: tokens: github-token # A fix-ci-context skip (a closed PR, a human-gated hold, or no failing - # check) runs no fix-ci job, + # check) or stop (the round cap, lane-stopped) runs no fix-ci job, # so no fix-ci check would land on the head and fix-ci-dispatch.mjs would # dispatch it again on every run. This job posts fix-ci neutral on that head, # as publish-feedback-not-run does; a kill-switch stop leaves the head empty. @@ -1967,7 +1991,6 @@ jobs: if: >- ${{ !cancelled() && inputs.activity == 'fix-ci' && needs.fix-ci-context.outputs.proceed == 'false' - && needs.fix-ci-context.outputs.escalation-reason == '' && needs.fix-ci-context.outputs.head-sha != '' }} runs-on: ubuntu-24.04 timeout-minutes: 10 @@ -2027,6 +2050,7 @@ jobs: pr-number: ${{ inputs.pr-number }} dispatcher-run-id: ${{ inputs.dispatcher-run-id }} trusted-base: default-tip + model-rung: ${{ needs.fix-ci-context.outputs.model-rung }} ci-context-attempt: ${{ needs.fix-ci-context.outputs.context-attempt }} ci-context-sha256: ${{ needs.fix-ci-context.outputs.context-sha256 }} head-sha: ${{ needs.fix-ci-context.outputs.head-sha }} @@ -2109,6 +2133,7 @@ jobs: pr-number: ${{ inputs.pr-number }} dispatcher-run-id: ${{ inputs.dispatcher-run-id }} trusted-base: default-tip + model-rung: ${{ needs.fix-ci-context.outputs.model-rung }} ci-context-attempt: ${{ needs.fix-ci-context.outputs.context-attempt }} ci-context-sha256: ${{ needs.fix-ci-context.outputs.context-sha256 }} head-sha: ${{ needs.fix-ci-context.outputs.head-sha }} @@ -2174,6 +2199,7 @@ jobs: pr-number: ${{ inputs.pr-number }} dispatcher-run-id: ${{ inputs.dispatcher-run-id }} trusted-base: default-tip + model-rung: ${{ needs.fix-ci-context.outputs.model-rung }} ci-context-attempt: ${{ needs.fix-ci-context.outputs.context-attempt }} ci-context-sha256: ${{ needs.fix-ci-context.outputs.context-sha256 }} head-sha: ${{ needs.fix-ci-context.outputs.head-sha }} @@ -2221,6 +2247,8 @@ jobs: patch-attempt: ${{ needs.fix-ci-fallback.result != 'skipped' && needs.fix-ci-fallback.outputs.scan-patch-attempt || needs.fix-ci-fallback.result == 'skipped' && needs.fix-ci-subscription.result != 'skipped' && needs.fix-ci-subscription.outputs.scan-patch-attempt || needs.fix-ci-fallback.result == 'skipped' && needs.fix-ci-subscription.result == 'skipped' && needs.fix-ci.outputs.scan-patch-attempt || '' }} patch-activity: fix-ci patch-prepare: ci-context + # A same-head retry's producer ran a rung up: no second in-run rung-up. + patch-model-rung: ${{ needs.fix-ci-context.outputs.model-rung }} # The head fix-ci-context gated, from a job that ran no head code; the # read runner's own head-sha output is the run job's, which head code # could forge. Both fix-ci calls refuse a context for another head, so a @@ -2498,20 +2526,64 @@ jobs: head-sha-fallback: ${{ needs.fix-ci-context.outputs.head-sha }} github-token: ${{ github.token }} - # GitHub starts no pull_request workflow on a conflicted PR, so a head the - # lane could not resolve gets no CI and no sign why. This script job, with - # no model, posts a failing pr-unblock/conflict commit status on that head, - # whether the select job escalated before any model turn, the push job - # found a conflict left, or the scan refused the model's output. It is the - # lane's only statuses: write, on its own GITHUB_TOKEN: the lanes App token - # the escalation comment uses carries no statuses permission. - report-conflict-status: - needs: [resolve-conflicts-select, apply-conflict-patch, conflict-scan-refusal] + # A run that stopped on a reason escalation-routes.json routes `fail` ends + # in a failing commit status on the head its activity gated, never a + # person: pr-unblock/conflict for resolve-conflicts (GitHub starts no + # pull_request workflow on a conflicted PR, so it also shows that head why + # its CI is missing), pr-unblock/ otherwise; a new head carries + # none. It runs when report-escalation does, on the same reason, except + # on a cancelled run, and report-escalation.sh status posts nothing for a + # reason held for a person. It also runs after an attempt whose push, reply + # or verdict job acted and ended with no stop: report-escalation.sh clear + # then posts success over a failure an earlier stop left on the same head, + # so a same-head retry that landed leaves nothing red. A job that succeeded + # on a skip (the reply script's no-model-turn, a gate or head-recheck skip) + # (a rung-up skip included) landed nothing and leaves the failure in place; + # the address-feedback sweep counts a no-model-turn skip toward + # SWEEP_HEAD_CAP (feedback-dispatch.mjs RELEASES). clear also runs when a + # select job that succeeded finds the stop gone on its head (stop-resolved: + # every thread resolved, or no check failing), since no push, reply or + # verdict job runs then. This script job, with no + # model, is the lane's only + # statuses: write, on its own GITHUB_TOKEN: the lanes App token the + # escalation comment uses carries no statuses permission. + report-stop-status: + needs: + - resolve-conflicts-select + - apply-conflict-patch + - conflict-scan-refusal + - address-feedback-select + - publish-feedback + - review-dependency-select + - apply-dependency-verdict + - fix-ci-context + - apply-patch + - apply-patch-rung-up if: >- - ${{ !cancelled() && needs.resolve-conflicts-select.outputs.head-sha != '' - && (needs.resolve-conflicts-select.outputs.escalation-reason == 'unresolved-conflict' - || needs.apply-conflict-patch.outputs.escalation-reason == 'unresolved-conflict' - || needs.conflict-scan-refusal.outputs.escalation-reason == 'unresolved-conflict') }} + ${{ !cancelled() + && (needs.resolve-conflicts-select.outputs.head-sha || needs.address-feedback-select.outputs.head-sha + || needs.review-dependency-select.outputs.head-sha || needs.fix-ci-context.outputs.head-sha || '') != '' + && (needs.resolve-conflicts-select.outputs.escalation-reason != '' + || needs.apply-conflict-patch.outputs.escalation-reason != '' + || needs.conflict-scan-refusal.outputs.escalation-reason != '' + || needs.address-feedback-select.outputs.escalation-reason != '' + || needs.publish-feedback.outputs.escalation-reason != '' + || needs.publish-feedback.result == 'failure' + || needs.review-dependency-select.outputs.escalation-reason != '' + || needs.apply-dependency-verdict.outputs.escalation-reason != '' + || needs.fix-ci-context.outputs.escalation-reason != '' + || needs.apply-patch.outputs.escalation-reason != '' + || needs.apply-patch.outputs.rung-up == 'true' + && !(needs.apply-patch-rung-up.result == 'success' + && needs.apply-patch-rung-up.outputs.escalation-reason == '' + && needs.apply-patch-rung-up.outputs.patch-skip-reason == '') + || needs.apply-conflict-patch.result == 'success' && needs.apply-conflict-patch.outputs.skip-reason == '' + || needs.publish-feedback.result == 'success' && needs.publish-feedback.outputs.skip-reason == '' + || needs.apply-dependency-verdict.result == 'success' && needs.apply-dependency-verdict.outputs.skip-reason == '' + || needs.apply-patch.result == 'success' && needs.apply-patch.outputs.skip-reason == '' + || needs.apply-patch-rung-up.result == 'success' && needs.apply-patch-rung-up.outputs.skip-reason == '' + || needs.address-feedback-select.result == 'success' && needs.address-feedback-select.outputs.stop-resolved == 'true' + || needs.fix-ci-context.result == 'success' && needs.fix-ci-context.outputs.stop-resolved == 'true') }} runs-on: ubuntu-24.04 timeout-minutes: 10 permissions: @@ -2525,14 +2597,35 @@ jobs: path: .base persist-credentials: false - - name: Post the conflict status + # One activity runs per dispatch, so one select job's head is set; a + # reason is a fixed enum value from a base script or this run's jobs, + # which report-escalation.sh rejects unless the routing table lists it. + # No reason: the attempt did not stop, and clear runs instead. + - name: Post the stop status env: GH_TOKEN: ${{ github.token }} PR_NUMBER: ${{ inputs.pr-number }} - HEAD_SHA: ${{ needs.resolve-conflicts-select.outputs.head-sha }} - ESCALATION_REASON: unresolved-conflict + HEAD_SHA: >- + ${{ needs.resolve-conflicts-select.outputs.head-sha || needs.address-feedback-select.outputs.head-sha + || needs.review-dependency-select.outputs.head-sha || needs.fix-ci-context.outputs.head-sha }} + ESCALATION_REASON: >- + ${{ needs.resolve-conflicts-select.outputs.escalation-reason + || needs.apply-conflict-patch.outputs.escalation-reason + || needs.conflict-scan-refusal.outputs.escalation-reason + || needs.address-feedback-select.outputs.escalation-reason + || needs.publish-feedback.outputs.escalation-reason + || needs.publish-feedback.result == 'failure' && 'feedback-publish-failed' + || needs.review-dependency-select.outputs.escalation-reason + || needs.apply-dependency-verdict.outputs.escalation-reason + || needs.fix-ci-context.outputs.escalation-reason + || needs.apply-patch.outputs.escalation-reason + || needs.apply-patch-rung-up.outputs.escalation-reason + || needs.apply-patch.outputs.rung-up == 'true' + && !(needs.apply-patch-rung-up.result == 'success' && needs.apply-patch-rung-up.outputs.patch-skip-reason == '') + && 'ci-fix-escalated' || '' }} + ACTIVITY: ${{ inputs.activity }} TARGET_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} - run: bash .base/.github/scripts/pr-unblock/report-escalation.sh status + run: bash .base/.github/scripts/pr-unblock/report-escalation.sh "$([ -n "$ESCALATION_REASON" ] && echo status || echo clear)" report-escalation: needs: @@ -2546,6 +2639,8 @@ jobs: - fix-ci-context - apply-patch - apply-patch-rung-up + # A reply job that failed with no reason (its lane token, a script error) + # stops as feedback-publish-failed, derived here from its result. # escalation-reason is set only by the write runner's own steps, the # base's reply, verdict and apply-patch scripts, and the base scripts of # resolve-conflicts-select, conflict-scan-refusal, address-feedback-select, @@ -2563,6 +2658,7 @@ jobs: || needs.conflict-scan-refusal.outputs.escalation-reason != '' || needs.address-feedback-select.outputs.escalation-reason != '' || needs.publish-feedback.outputs.escalation-reason != '' + || needs.publish-feedback.result == 'failure' || needs.review-dependency-select.outputs.escalation-reason != '' || needs.apply-dependency-verdict.outputs.escalation-reason != '' || needs.fix-ci-context.outputs.escalation-reason != '' @@ -2592,6 +2688,7 @@ jobs: || needs.conflict-scan-refusal.outputs.escalation-reason || needs.address-feedback-select.outputs.escalation-reason || needs.publish-feedback.outputs.escalation-reason + || needs.publish-feedback.result == 'failure' && 'feedback-publish-failed' || needs.review-dependency-select.outputs.escalation-reason || needs.apply-dependency-verdict.outputs.escalation-reason || needs.fix-ci-context.outputs.escalation-reason diff --git a/docs/conventions/pr-pipeline.yaml b/docs/conventions/pr-pipeline.yaml index 5c53feecf6..335a3152b0 100644 --- a/docs/conventions/pr-pipeline.yaml +++ b/docs/conventions/pr-pipeline.yaml @@ -90,11 +90,12 @@ activities: # opened on a reviewed head, left by it alone or answered by a trusted # person after the lane escalated them, hands the pull request to a person # when it changes .github/workflows/ or, once `loop.review-rounds` rounds - # ran, a person gated it or a kept thread is a security or intent - # finding (a write-access fix + # ran, a person gated it or a kept thread is a security finding or one + # a person directed (a write-access fix # request newer than the last round still runs one more round on its - # threads; other findings get one fresh fixer round, then go to a person - # as feedback-not-converging), and + # threads; other findings, pr-verify notes included, get one fresh fixer + # round, then fail the head as feedback-verifier-open or + # feedback-not-converging), and # writes the thread file; the read runner # checks that file and gives the skill the review-thread tool. prepare: review-threads @@ -383,7 +384,10 @@ activities: reads-untrusted: false inputs: [issue] report-escalation: - # Labels the PR needs-human and posts the reason with a resume key. + # Ends a stopped run on its reason's route (escalation-routes.json): + # hold labels the PR needs-human and posts the reason with a resume key; + # fail posts or edits one lane-stopped comment, and report-stop-status a + # failing status on the head. script: .github/scripts/pr-unblock/report-escalation.sh effect: publish-artifact gating: advisory diff --git a/docs/conventions/pr-pipeline/README.md b/docs/conventions/pr-pipeline/README.md index 8538c564cd..ae4843fd49 100644 --- a/docs/conventions/pr-pipeline/README.md +++ b/docs/conventions/pr-pipeline/README.md @@ -154,14 +154,15 @@ merge with the head and the pinned base as parents ([`pr-run-activity.md`](pr-run-activity.md#prepare-merge-base)). GitHub starts no `pull_request` workflow on a PR with a merge conflict, so a conflicted head gets -no CI. When `resolve-conflicts` escalates with `unresolved-conflict`, `pr-unblock.yml`'s -`report-conflict-status` job (a script job with no model, the lane's only `statuses: write`, on its -own `GITHUB_TOKEN`) posts a `failure` commit status with context `pr-unblock/conflict` on that -head, saying CI does not run until the conflicts are resolved. A push that resolves them is a new -head, which the status does not reach. A conflict that clears with the head unchanged (main moved -again) leaves the failing status on it until a push or a person replaces it. The other -`resolve-conflicts` escalations post no status; they hand the PR to a person through the comment -and label alone. +no CI. When `resolve-conflicts` stops on a reason routed to a failing status (below), +`pr-unblock.yml`'s `report-stop-status` job (a script job with no model, the lane's only +`statuses: write`, on its own `GITHUB_TOKEN`) posts a `failure` commit status with context +`pr-unblock/conflict` on that head; for `unresolved-conflict` it says CI does not run until the +conflicts are resolved. A push that resolves them is a new head, which the status does not reach. +A conflict that clears with the head unchanged (main moved again) leaves the failing status on it +until a push or a person replaces it. An `address-feedback` or `fix-ci` failing status stays too +when a person resolves the threads or gets the checks green on that head: a `pr-unblock` dispatch +of that activity then finds the cause gone and clears it, and no dispatcher starts that run. `pr-unblock`'s `address-feedback` dispatches come from `pr-feedback-dispatch.yml`, also scripted, on `workflow_run` `completed` of `pr-review-hosted` and `pr-review-security-hosted`. It takes only @@ -313,8 +314,11 @@ head's CI decides, and a red new head gets its own run. The one exception: a run `model-rung` 0 that changed nothing and escalated only `unfixed` or `not-reproducible` checks skips `apply-patch` with `rung-up`, and the same `pr-unblock` run runs `fix-ci` and `apply-patch` once more at `model-rung` 1 (`fix-ci-rung-up`, `fix-ci-rung-up-subscription`, `apply-patch-rung-up`; [Models](#models)). `report-escalation` -hands the PR to a person unless that retry's patch applied cleanly: the retry's own reason, else -`ci-fix-escalated` (a neutral or failed retry). A cancelled run applies no label: the write +ends the run unless that retry's patch applied cleanly: the retry's own reason, else +`ci-fix-escalated` (a neutral or failed retry), each routed to a failing `pr-unblock/fix-ci` +status but `ci-fix-protected-path` and `ci-fix-weaken-required` (a check the model could fix only +through a path the lane token may not push, or by weakening a required check), which go to a +person. A cancelled run applies no label: the write workflow's `run` job has no status function in its `if:`, so it skips on a cancel, nothing pushes and `ci-status` stays red. In a run that changed nothing, `protected-path`, `weaken-required` and `flaky` go to a person without a retry; beside a pushed patch, the new @@ -541,6 +545,37 @@ An activity is idempotent: a rerun on the same commit gives the same verdict or mutating activity with nothing to change makes no commit, and each commit it makes names the activity in a trailer. A verdict is bound to the commit it judged and is not reused for any other commit. +A `pr-unblock` run that stops ends on the route +[`escalation-routes.json`](../../../.github/scripts/pr-unblock/escalation-routes.json), the one +routing table, gives its reason; `report-escalation.sh` reads it. Route `hold` is for a reason only +a person settles: a path the lane token may not push or a supply-chain refusal (SAFETY), an intent, +design or security question, a person's own thread or hold, a finding no adjudicator may judge (a +security-review or CRITICAL one), or any `review-dependency` stop: its failing status would be per head, +and the merge gate's head-bound `verdict=merge` marker comes from the lane's own re-run, so only the label +holds it for a person. Every other reason is a defect the lane's bounded retries (the +`fix-ci` rung-up, the adjudicator, the round, repeat and per-head caps) did not mend, routed `fail`: +no label, a failing commit status on the head the activity gated (`pr-unblock/conflict` for +`resolve-conflicts`, `pr-unblock/` otherwise; `report-stop-status` posts it), and one +lanes App comment per activity (marked ``, +naming the run) that the next stop edits in place. The stopping job's check skips `lane-stopped`. +A select job's stop (a cap, or a conflict no model can resolve) is final for its head. A stop +after a model turn (the push, reply or verdict job's) is retried on the same head by the +activity's dispatcher, under the cap it already keeps: `limit-stops-per-head` for +`resolve-conflicts`, `fix-ci-restarts` and the per-head cap for `fix-ci`, +`SWEEP_HEAD_CAP` for `address-feedback`. A `fix-ci` or `address-feedback` retry runs one model +rung up (`escalate-to`), read in the run from the head's own stop record, never from a dispatch +input; the others retry as a fresh session. A retry that acts and does not stop posts success over the +failing status (`report-escalation.sh clear`); one that ends on a skip, such as a reply job with no +model turn, leaves it. A select job that finds the stop gone on its head also posts success +(`stop-resolved`): `address-feedback` once no hold and no unresolved review thread is left, +`fix-ci` once a whole listing of the head's checks, every one completed, has none failing (round +cap included). So a thread resolved by hand, or CI fixed by hand or re-run green, frees the same +head at the next dispatch of that activity (the one the lane-stopped comment gives); nothing +dispatches it on its own, and a `ci-status` that already failed stays failed until something +re-runs it (the next review lane's `rerun-ci-status`, or a person). Once the retries are spent, the monitoring lane's +`stalled lane stop` signal names the PR. A new head carries no status, so the lane's next run +there starts clean. + When an activity needs a human, it escalates with a resume key and exits instead of blocking. An activity whose `labels-absent` lists `needs-human` or `do-not-merge` skips with `awaiting-human` while that label is on the PR; a human removes the label, then dispatches the lane, to resume it @@ -562,6 +597,7 @@ head: | `workflow-staged` | main changes nothing under `.github/workflows/` since its merge base with the head | | `unresolved-conflict`, `protected-path-conflict` | GitHub reports the PR mergeable | | `feedback-protected-path` | the PR changes nothing under `.github/workflows/` | +| a reason `escalation-routes.json` routes `fail`, on a label set before it did | a `ci-fix-*` reason once a lane check of a `pr-require-checks.yml` run on the current head reads failed (`ci-status` itself fails on the label, so it is not read, and an advisory lane such as `pr-test-windows` does not count; green, missing or unreadable keeps the label). Any other fail-routed reason stays for a person: a lane posts a `pr-unblock/*` status on a head only when it stops there or clears its own failure, so a new head carries nothing that shows the cause gone or fails that head in the label's place | | `head-moved` (pr-merge's `babysit-merge-verify-escalation`, `babysit-merge-risk` and `babysit-merge-shared-copy-escalation`, with no pr-unblock escalation since the label, and the marked App comment written last before the label is pr-merge's and within a minute of it) | the newest App comment carrying each marker names, as ``at head `` ``, a 40-hex head other than the current one; pr-merge then judges the new head and escalates again if its refusal still holds | | `refusal-cleared` (under `merge.lane-prs: native`, `enable-auto-merge`'s own unedited person refusal comment on a lanes App pull request, naming the current head, whose recorded verifier or risk cause is one the lanes clear: every code on `LANE_CAUSES`) | the gate's reader at that head finds nothing refusing, or only causes the lanes clear, read at least as far as the recorded one. A recorded cause a person clears (an issue link, a security finding, a trust-gate widening, a cost hold, a high path, an untrusted spec, the repeated-fail cap, any code off that list) is never read or lifted at its head (owner decision Q-CLEAR B): a person removes the label, or the `head-moved` clear does once the head moves. At most three reader runs a run | @@ -704,8 +740,11 @@ round whose model never ran, so the scan uploaded no replies), `finding-stands` address-feedback round's reply job, when an adjudicator let a disputed finding stand and the head did not move, so no review run starts the round that fixes it), `clean-merge` (a resolve-conflicts round whose base merge left no conflict, so no model ran and the push job commits the merge), -`pass-only` (a pass-only address-feedback round's reply job, which ran no model and only resolved -fail-note threads a pr-verify pass superseded; the dispatcher's release reads skip it), or, for an issue run under +`lane-stopped` (a pr-unblock stop `escalation-routes.json` routes to a failing status: on a select +job's activity check it ends the head; on a push, reply or verdict job's check the dispatcher +retries the head under its per-head cap), `pass-only` (a pass-only address-feedback round's reply +job, which ran no model and only resolved fail-note threads a pr-verify pass superseded; the +dispatcher's release reads skip it), or, for an issue run under `models.issue-dimensions`, `research-first` or `probe-first` (its Uncertainty), or `split-required` (its Size is `split`). Issue mode posts no check, so work-resolve-issue.yml reports those three dimension skips on the issue instead: `needs-human`, no `agent-ready`, and a comment naming the next step @@ -843,8 +882,14 @@ the rules that only tighten in `babysit_review_evidence.py`, which compute the security review's clear condition. - `enforcement`, a clean security review clears it: every other module the floored scripts import (`.github/scripts/pr-unblock/address-feedback.mjs`, `gate-source.mjs` and - `thread-scope.mjs`, `.github/scripts/work-resolve-issue/report-outcome.mjs` and `limits.mjs`, - and the `retry-fetch.mjs` copy in each of those two directories) + `thread-scope.mjs`, `escalation-routes.mjs` and the routing table `escalation-routes.json` it + reads, `.github/scripts/work-resolve-issue/report-outcome.mjs` and `limits.mjs`, and the + `retry-fetch.mjs` copy in each of those two directories), the scripts that act on a pr-unblock + stop's route (`report-escalation.sh`, which labels a held reason, `adjudicate-feedback.mjs` and + `publish-feedback.mjs`, which decide and resolve threads, and + `.github/scripts/apply-patch/apply-patch.mjs`, which holds fix-ci escalations) with every module + they import (`request-budget.mjs`, `dependency-classify.mjs`, and `predicates.mjs`, + `protected-paths.mjs` and `retry-fetch.mjs` in `apply-patch/`), and the rest of the Python that decides the holds (`babysit_verify_evidence.py`, which `enable-auto-merge.mjs` runs, and the `babysit-prs/scripts/` modules it and `babysit_risk.py` import). It scores HIGH as `review-path:` until the `claude-security-review` evidence @@ -1965,6 +2010,11 @@ them on every `pull_request` run, contract-only runs included: author id, except that their PR fails when a changed file (or a renamed file's old path) is a trust-gate path (the hold-paths above less the security review's own files), since no lane reviews it; a person merges it. A draft is exempt. +- **A stopped lane holds its head.** ci-status fails while the newest `pr-unblock/*` commit status + on the head by `github-actions[bot]` is `failure` or `error`, naming each such context; a later + `success` on the same head (`report-escalation.sh clear`) supersedes it, and another account's + status is not read. The merge-group run checks the queued pull request's head too. A status post + re-runs nothing, so a stop posted after ci-status passed holds at the merge queue's run. A review usually finishes after ci-status has read it, so `pr-feedback-dispatch.yml`'s `rerun-ci-status` job re-runs the ci-status job of the newest `pr-require-checks` run on the head @@ -2148,6 +2198,7 @@ findings listed by key. Its scripts and tests are in | 7 platform incident | githubstatus.com has an incident naming Actions that is open or began in the window, or shows Actions not operational; listed in the run summary only, never filed, since nothing in this repository acts on it | `platform:`, `platform:actions` | | 8 throughput | no PR by `merge.agent-identity` merged for 3 hours while pr-merge's newest live sweep, at most 70 minutes old, counts candidates | `throughput-zero` | | 9 lane stop | a comment by `merge.agent-identity` created in the window starts with a `` marker, alone or after the attended queue's escalation marker line, one issue per outcome listing each stopped issue | `lane-stop:funding`, `lane-stop:limited`, `lane-stop:turn-cap` | +| 10 stalled lane stop | an open pull request into main, of any author, carries a pr-unblock lane-stopped comment by `merge.agent-identity` naming its current head, or a `needs-human-cleared` comment for that head whose reason now routes to a failing status, older than 2 hours (four `pr-unblock-dispatch.yml` ticks), with no `pr-unblock / ` check run but a skip started on that head since, and no `not-applicable` skip since that ends it (address-feedback on a non-draft pull request; fix-ci only once a whole listing of the head's checks is settled with none failing): its same-head retries are spent; one issue lists each such stop | `stalled-lane-stop` | A rolling issue, titled `Monitoring: `, has the first body line ``; the body names the signal and its state @@ -2297,13 +2348,14 @@ One counter per PR, in the findings store, is shared by every lane. After `loop. full review rounds, one more high-severity-only pass runs; if it still has findings, the PR escalates. Once `loop.review-rounds` of the address-feedback activity's rounds have run, a write-access trusted person's fix request on a thread after the newest round runs one more round, -on the requested threads only. With no such request, a security lane finding, a pr-verify note (an -intent mismatch), a `claude[bot]` finding, whose lane no footer proves, or a person's `needs-human` hold +on the requested threads only. With no such request, a security lane finding, a pr-verify note a +trusted person directed, a `claude[bot]` finding, whose lane no footer proves, or a person's `needs-human` hold hands the PR to a person; any other -finding goes to one fresh fixer round, a new session like every round. That is the last automatic -round: once it has run, no sweep, schedule or dispatch starts another, and the next select hands -the PR to a person as `feedback-not-converging`, a comment that names the open threads (a review -that does not converge is an intent question). Removing the label alone does not resume it, since +finding, an undirected pr-verify note included, goes to one fresh fixer round, a new session like every round. That is the last automatic +round: once it has run, no sweep, schedule or dispatch starts another, and the next select fails +the head as `feedback-verifier-open` when only pr-verify notes are open, else as +`feedback-not-converging`, a comment that names the open threads. A round cap is a lane bound, not +an intent question, and pr-verify's FAIL already holds the merge until it passes a later head. Removing the label alone does not resume it, since the rounds already run still count: a person resolves the open threads, or a write-access fix request still gets its own round, as the comment's resume steps say. The round marker and the neutral checks show `round of max `, where M is `loop.review-rounds` plus the fresh fixer's round. The default is 2; this diff --git a/docs/conventions/pr-pipeline/pr-pipeline.schema.json b/docs/conventions/pr-pipeline/pr-pipeline.schema.json index 3b5d1e2834..1cf57a3336 100644 --- a/docs/conventions/pr-pipeline/pr-pipeline.schema.json +++ b/docs/conventions/pr-pipeline/pr-pipeline.schema.json @@ -459,6 +459,7 @@ "no-model-turn", "finding-stands", "clean-merge", + "lane-stopped", "pass-only" ], "description": "The one reason a skipped activity's neutral check carries. Not used by the config file; the runner and check-run reporter validate reasons against it. Meaning: README.md, Outputs and skips." diff --git a/docs/conventions/pr-pipeline/pr-run-activity.md b/docs/conventions/pr-pipeline/pr-run-activity.md index 99d7e58484..faab161fc2 100644 --- a/docs/conventions/pr-pipeline/pr-run-activity.md +++ b/docs/conventions/pr-pipeline/pr-run-activity.md @@ -843,9 +843,9 @@ the repeat cap counts by), `loop.no-progress` as `LOOP_NO_PROGRESS` and a job output, and escalates `unresolved-conflict` when one says the scan refused or the download failed (`conflict-scan-refusal.mjs`, through `scanRefusal`); no marker escalates nothing. -6. `report-conflict-status` posts a failing `pr-unblock/conflict` status on the select job's head - when the select, push or `conflict-scan-refusal` job escalates `unresolved-conflict`; `report-escalation` - takes any of their reasons. +6. `report-stop-status` posts a failing `pr-unblock/conflict` status on the select job's head + when the select, push or `conflict-scan-refusal` job stops on a reason routed `fail`; + `report-escalation` takes any of their reasons. The three merges of one round must agree on the conflict tree; one that does not fails red and nothing is resolved or pushed. The kill switch (step 3) stops new runs only; a run already past it @@ -912,7 +912,13 @@ the base's `.github/scripts/pr-unblock/address-feedback.mjs`, `apply-patch.mjs` same way. On a PR carrying the base's human-gated role label it keeps a thread only when one of those replies is such a fix request by a write-access human, and reports `fix-requested=true`, which lifts only the human-gated hold of the push job. No kept - thread skips `not-applicable`, or `awaiting-human` on such a PR. A PR diff touching + thread skips `awaiting-human` on such a PR; elsewhere it escalates `feedback-untaken-threads` + (routed to a failing status) while any review thread is unresolved, and skips `not-applicable` + only when none is. The dispatcher (`feedback-dispatch.mjs`) dispatches an ungated head for any + unresolved thread, so such a head reaches this step once: its lane-stopped check then ends the + head, which a reply job's skip does not release. A failed reply job with no escalation reason + (its lane token, a script error) stops as `feedback-publish-failed`, which `pr-unblock.yml` + derives from the job's result. A PR diff touching `.github/workflows/**` escalates `feedback-protected-path`. Rounds count as the larger of the round markers the lanes App posted and the PR commits carrying a `github-actions` `pr-unblock / address-feedback` check run that ended `success` or `failure` (a round whose @@ -921,10 +927,12 @@ the base's `.github/scripts/pr-unblock/address-feedback.mjs`, `apply-patch.mjs` or the completion of a model check run that ended `success`, `failure`, `cancelled` or `timed_out`) still runs one more round, on those threads only; a round with no readable time grants none. With no such thread, a PR a person gated with the human-gated label, or a kept thread whose first comment is not a `github-actions[bot]` - finding ending in the `claude-lane: claude-review` footer (a security lane finding, a verifier - note, or a `claude[bot]` finding, whose footer would be model text) escalates `feedback-round-cap`; otherwise one more round runs on every kept + finding ending in the `claude-lane: claude-review` footer nor a pr-verify note (a security lane + finding, or a `claude[bot]` finding, whose footer would be model text), or a pr-verify note a + trusted person directed, escalates `feedback-round-cap`, which skips `awaiting-human`; otherwise one more round runs on every kept thread (the fresh fixer), and once it has run the select job escalates - `feedback-not-converging`. The escalations skip `awaiting-human`; no escalation or skip runs a + `feedback-verifier-open` when every kept thread is a pr-verify note, else + `feedback-not-converging`; both route to a failing status and skip `lane-stopped`. No escalation or skip runs a model job. A round-cap escalation also writes `not-run-detail` (`round of max `, M being the cap plus the fresh fixer's round), which both neutral checks add to their summary. Otherwise it writes `threads.json` and `run.json` (the gated head and the kept threads' ids, paths, comment ids and named paths, and `pr_paths`, from the thread file) and uploads both as @@ -1051,11 +1059,13 @@ the base's `.github/scripts/pr-unblock/address-feedback.mjs`, `apply-patch.mjs` second line names the workflow run and `review-replies-attempt`, and a re-run of the reply job alone finds a lanes App marker with that line and posts none, so one round counts once, while a re-run of the model job counts another. A round with no replies attempt names none, - and its marker is always posted. It escalates - `feedback-uncertain` for an `UNCERTAIN` or `VALID-defer` reply; - - after the replies, settles each posted `INCORRECT` reply by the adjudicator's verdict + and its marker is always posted; + - after the replies, settles each posted `INCORRECT`, `UNCERTAIN` or `VALID-defer` reply by + the adjudicator's verdict ([below](#the-adjudicator)): `lane` posts the fixed finding-answered reply and resolves the - thread; `reviewer` posts the fixed finding-stands reply and leaves the thread open for the + thread only on an `INCORRECT` reply, where the lane disputed the finding with evidence; on an + `UNCERTAIN` or `VALID-defer` reply the thread stays open, with no reply, and escalates + `feedback-unadjudicated`, so a model verdict never closes a finding the lane did not dispute; `reviewer` posts the fixed finding-stands reply and leaves the thread open for the next round; when the PR head did not move this round, no review run follows to start that round, so the reply job skips `finding-stands` (when nothing escalates), and the dispatcher's half-hourly sweep dispatches the head again, within the select job's round cap and three @@ -1063,8 +1073,14 @@ the base's `.github/scripts/pr-unblock/address-feedback.mjs`, `apply-patch.mjs` `feedback-question-intent`, `feedback-question-design` or `feedback-question-security`; no verdict escalates `feedback-unadjudicated`. A thread whose finding stood at an earlier adjudication is not adjudicated again: any reply but `VALID` on it escalates - `feedback-disputed`. A thread or head that changed after the lane's reply, or a failed write, - escalates `feedback-partial`. One round records one reason, the most urgent in that order. + `feedback-stood-open`. A thread no adjudicator may judge goes elsewhere: a pr-verify note + left open escalates `feedback-verifier-open`, a thread a trusted person directed, or an + `INCORRECT` reply on a security-review, CRITICAL or unknown-lane finding, escalates + `feedback-disputed`, and an `UNCERTAIN` or `VALID-defer` reply on one escalates + `feedback-uncertain`. A thread or head that changed after the lane's reply, or a failed + write, escalates `feedback-partial`. One round records one reason: one held for a person + (`feedback-question-*`, `feedback-disputed`, `feedback-uncertain`) before any routed to a + failing status. The script writes its reason to `ESCALATION_REASON_FILE`; the runner accepts only those reasons, and only with `review-replies`, as the run's `escalation-reason` output. @@ -1086,12 +1102,13 @@ model turn. #### The adjudicator pr-unblock.yml's `adjudicate-feedback` job runs after the model job, before the push and reply jobs, when -the leg the reply job reads uploaded replies holding an `INCORRECT` reply or a line that is not -JSON (the read runner's scan job sets `feedback-disputed` from the bytes it uploaded); otherwise it -is skipped, and an `INCORRECT` reply with no verdict still escalates `feedback-unadjudicated`. It +the leg the reply job reads uploaded replies holding a reply that leaves its thread open +(`INCORRECT`, `UNCERTAIN` or `VALID-defer`) or a line that is not JSON (the read runner's scan job +sets `feedback-disputed` from the bytes it uploaded); otherwise it is skipped, and such a reply +with no verdict still escalates `feedback-unadjudicated`. It runs only base code with a read-only token and no OIDC token. `adjudicate-feedback.mjs prepare` checks the select job's thread file and the scan job's replies against the SHA-256 each writer -recorded, then stages each `INCORRECT` reply on a thread no adjudication stood before and no +recorded, then stages each such reply on a thread no adjudication stood before and no trusted person directed, up to three: the finding's path and line, the whole review comment, the locations the comment cites (`path:line` tokens and line-numbered permalinks into this repository, at most ten), the locations the lane cited, at most ten (never its note), and those files at the @@ -1099,15 +1116,17 @@ select job's head, read through the contents API with each response's rate-limit the job's end (`.github/actions/log-api-budget`), each up to 200 KiB and staged with a `.txt` suffix so a cited `CLAUDE.md` or `.claude/**` file is never loaded as instructions. A thread a trusted person directed (a fix request, or a reply after an escalation) stays theirs: an -`INCORRECT` reply on it escalates `feedback-disputed` with no adjudication. Only a +reply that leaves it open escalates `feedback-disputed` with no adjudication. Only a github-actions[bot] review comment whose footer names the code-review lane (`claude-lane: claude-review`; the select job tags the thread file's comment with that `lane`, and a footer in a claude[bot] comment, whose body the model writes, names none) and that carries no -`severity: CRITICAL` line is adjudicated. A CRITICAL finding, a security-review finding, one whose -lane its root comment does not name, and a pr-verify verifier note always go to a person: an -`INCORRECT` reply on one escalates `feedback-disputed`, takes no slot, and any verdict for it is -ignored. A verifier note is never adjudicated because resolving its thread clears nothing: the -FAIL still holds the merge, and pr-verify judges each head once. Each thread gets one fresh +`severity: CRITICAL` line is adjudicated. A CRITICAL finding, a security-review finding and one +whose lane its root comment does not name go to a person: an `INCORRECT` reply on one escalates +`feedback-disputed`, an `UNCERTAIN` or `VALID-defer` one `feedback-uncertain`; it takes no slot, +and any verdict for it is ignored. A pr-verify verifier note is never adjudicated because +resolving its thread clears nothing: the FAIL still holds the merge, and pr-verify judges each +head once; a reply that leaves one open ends `feedback-verifier-open`, a failing status and no +person. Each thread gets one fresh `run-claude-cli` call, after `apply-claude-managed-settings` and runner isolation, on the pr-unblock API key (`PR_UNBLOCK_CLAUDE_API_KEY`), under `dontAsk` with Read alone, `Read(./adjudicate/**)` the one allow rule, Read denied on the base checkout and the runner's temp directory (which holds the lane's replies and @@ -1300,8 +1319,9 @@ its fallback call on the role's OAuth token after the key call hands off `.github/scripts/pr-unblock/fix-ci-context.mjs` with a token that reads pull requests, checks and Actions, and checks out no PR head. After the kill switch it hands the pull request to a person (`ci-fix-round-cap`) once the lanes App has pushed `loop.fix-ci-rounds` commits with - the trailer `Lane-Activity: pr-unblock/fix-ci`. Otherwise it reads the head's latest check run - per name and keeps those that ended `failure`, `timed_out` or `startup_failure`, except + the trailer `Lane-Activity: pr-unblock/fix-ci`. Otherwise it reads every check run on the head + (`filter=all`, as the fix-ci dispatcher does), takes per name the latest that is not a skip, so a + contract-only run's later skip does not hide a lane's failure, and keeps those that ended `failure`, `timed_out` or `startup_failure`, except `ci-status` and any GitHub Actions check run that is no workflow job (a lane's own check), at most 10. For each it writes the name, the workflow file that defines the job (or `null`), at most 20 annotations of at most 500 characters, and the last 80 lines, at most 3,000 diff --git a/plugins/source-control/skills/babysit-prs/scripts/babysit_risk.py b/plugins/source-control/skills/babysit-prs/scripts/babysit_risk.py index 985ef2e3b2..ddd95f3090 100755 --- a/plugins/source-control/skills/babysit-prs/scripts/babysit_risk.py +++ b/plugins/source-control/skills/babysit-prs/scripts/babysit_risk.py @@ -203,7 +203,10 @@ class RiskResult: # the pipeline config. # - `enforcement`: the rest of the modules the floored scripts import and # the Python that decides the holds (test_merge_holds_sync.py derives both -# closures and requires them here). +# closures and requires them here), and the scripts that act on a +# pr-unblock stop's route (report-escalation.sh, adjudicate-feedback.mjs, +# publish-feedback.mjs, apply-patch.mjs) with every module they import and +# the routing table they read, so a held reason cannot be dropped unreviewed. # A `trust` or `gate` path holds for a person (HIGH_FLOOR). An `enforcement` # path holds until the security lane's review binds to the head with no # open finding (REVIEW_FLOOR, `review-path:`), so no person is needed. @@ -254,12 +257,23 @@ class RiskResult: (".github/workflows/pr-merge.yml", "gate"), ("docs/conventions/pr-pipeline.yaml", "gate"), (".github/scripts/pr-unblock/address-feedback.mjs", "enforcement"), + (".github/scripts/pr-unblock/escalation-routes.mjs", "enforcement"), + (".github/scripts/pr-unblock/escalation-routes.json", "enforcement"), (".github/scripts/pr-unblock/gate-source.mjs", "enforcement"), (".github/scripts/pr-unblock/thread-scope.mjs", "enforcement"), (".github/scripts/pr-unblock/retry-fetch.mjs", "enforcement"), (".github/scripts/work-resolve-issue/report-outcome.mjs", "enforcement"), (".github/scripts/work-resolve-issue/limits.mjs", "enforcement"), (".github/scripts/work-resolve-issue/retry-fetch.mjs", "enforcement"), + (".github/scripts/pr-unblock/report-escalation.sh", "enforcement"), + (".github/scripts/pr-unblock/adjudicate-feedback.mjs", "enforcement"), + (".github/scripts/pr-unblock/publish-feedback.mjs", "enforcement"), + (".github/scripts/pr-unblock/request-budget.mjs", "enforcement"), + (".github/scripts/pr-unblock/dependency-classify.mjs", "enforcement"), + (".github/scripts/apply-patch/apply-patch.mjs", "enforcement"), + (".github/scripts/apply-patch/predicates.mjs", "enforcement"), + (".github/scripts/apply-patch/protected-paths.mjs", "enforcement"), + (".github/scripts/apply-patch/retry-fetch.mjs", "enforcement"), ) # fmt: skip PERSON_CLASSES = frozenset({"trust", "gate"}) HIGH_FLOOR: tuple[str, ...] = tuple(g for g, c in FLOOR_CLASSES if c in PERSON_CLASSES) diff --git a/plugins/source-control/skills/babysit-prs/scripts/tests/test_babysit_risk.py b/plugins/source-control/skills/babysit-prs/scripts/tests/test_babysit_risk.py index b08074a324..a0343d94a4 100644 --- a/plugins/source-control/skills/babysit-prs/scripts/tests/test_babysit_risk.py +++ b/plugins/source-control/skills/babysit-prs/scripts/tests/test_babysit_risk.py @@ -325,12 +325,23 @@ def test_floor_paths_are_high_whatever_the_switch(self): (".github/workflows/pr-merge.yml", "gate"), ("docs/conventions/pr-pipeline.yaml", "gate"), (".github/scripts/pr-unblock/address-feedback.mjs", "enforcement"), + (".github/scripts/pr-unblock/escalation-routes.mjs", "enforcement"), + (".github/scripts/pr-unblock/escalation-routes.json", "enforcement"), (".github/scripts/pr-unblock/gate-source.mjs", "enforcement"), (".github/scripts/pr-unblock/thread-scope.mjs", "enforcement"), (".github/scripts/pr-unblock/retry-fetch.mjs", "enforcement"), (".github/scripts/work-resolve-issue/report-outcome.mjs", "enforcement"), (".github/scripts/work-resolve-issue/limits.mjs", "enforcement"), (".github/scripts/work-resolve-issue/retry-fetch.mjs", "enforcement"), + (".github/scripts/pr-unblock/report-escalation.sh", "enforcement"), + (".github/scripts/pr-unblock/adjudicate-feedback.mjs", "enforcement"), + (".github/scripts/pr-unblock/publish-feedback.mjs", "enforcement"), + (".github/scripts/pr-unblock/request-budget.mjs", "enforcement"), + (".github/scripts/pr-unblock/dependency-classify.mjs", "enforcement"), + (".github/scripts/apply-patch/apply-patch.mjs", "enforcement"), + (".github/scripts/apply-patch/predicates.mjs", "enforcement"), + (".github/scripts/apply-patch/protected-paths.mjs", "enforcement"), + (".github/scripts/apply-patch/retry-fetch.mjs", "enforcement"), ), ) # fmt: skip self.assertEqual( @@ -364,12 +375,23 @@ def test_floor_paths_are_high_whatever_the_switch(self): ("docs/conventions/pr-pipeline.yaml", "docs/conventions/pr-pipeline.yaml"), ("Docs/Conventions/PR-Pipeline.yaml", "docs/conventions/pr-pipeline.yaml"), (".github/scripts/pr-unblock/address-feedback.mjs", ".github/scripts/pr-unblock/address-feedback.mjs"), + (".github/scripts/pr-unblock/escalation-routes.mjs", ".github/scripts/pr-unblock/escalation-routes.mjs"), + (".github/scripts/pr-unblock/escalation-routes.json", ".github/scripts/pr-unblock/escalation-routes.json"), (".github/scripts/pr-unblock/gate-source.mjs", ".github/scripts/pr-unblock/gate-source.mjs"), (".github/scripts/pr-unblock/thread-scope.mjs", ".github/scripts/pr-unblock/thread-scope.mjs"), (".github/scripts/pr-unblock/retry-fetch.mjs", ".github/scripts/pr-unblock/retry-fetch.mjs"), (".github/scripts/work-resolve-issue/report-outcome.mjs", ".github/scripts/work-resolve-issue/report-outcome.mjs"), (".github/scripts/work-resolve-issue/limits.mjs", ".github/scripts/work-resolve-issue/limits.mjs"), (".github/scripts/work-resolve-issue/retry-fetch.mjs", ".github/scripts/work-resolve-issue/retry-fetch.mjs"), + (".github/scripts/pr-unblock/report-escalation.sh", ".github/scripts/pr-unblock/report-escalation.sh"), + (".github/scripts/pr-unblock/adjudicate-feedback.mjs", ".github/scripts/pr-unblock/adjudicate-feedback.mjs"), + (".github/scripts/pr-unblock/publish-feedback.mjs", ".github/scripts/pr-unblock/publish-feedback.mjs"), + (".github/scripts/pr-unblock/request-budget.mjs", ".github/scripts/pr-unblock/request-budget.mjs"), + (".github/scripts/pr-unblock/dependency-classify.mjs", ".github/scripts/pr-unblock/dependency-classify.mjs"), + (".github/scripts/apply-patch/apply-patch.mjs", ".github/scripts/apply-patch/apply-patch.mjs"), + (".github/scripts/apply-patch/predicates.mjs", ".github/scripts/apply-patch/predicates.mjs"), + (".github/scripts/apply-patch/protected-paths.mjs", ".github/scripts/apply-patch/protected-paths.mjs"), + (".github/scripts/apply-patch/retry-fetch.mjs", ".github/scripts/apply-patch/retry-fetch.mjs"), ] # fmt: skip unreviewed = ( _lane("claude-review"), @@ -1233,12 +1255,23 @@ def test_the_floor_holds_with_the_switch_off(self): ".github/workflows/pr-merge.yml", "docs/conventions/pr-pipeline.yaml", ".github/scripts/pr-unblock/address-feedback.mjs", + ".github/scripts/pr-unblock/escalation-routes.mjs", + ".github/scripts/pr-unblock/escalation-routes.json", ".github/scripts/pr-unblock/gate-source.mjs", ".github/scripts/pr-unblock/thread-scope.mjs", ".github/scripts/pr-unblock/retry-fetch.mjs", ".github/scripts/work-resolve-issue/report-outcome.mjs", ".github/scripts/work-resolve-issue/limits.mjs", ".github/scripts/work-resolve-issue/retry-fetch.mjs", + ".github/scripts/pr-unblock/report-escalation.sh", + ".github/scripts/pr-unblock/adjudicate-feedback.mjs", + ".github/scripts/pr-unblock/publish-feedback.mjs", + ".github/scripts/pr-unblock/request-budget.mjs", + ".github/scripts/pr-unblock/dependency-classify.mjs", + ".github/scripts/apply-patch/apply-patch.mjs", + ".github/scripts/apply-patch/predicates.mjs", + ".github/scripts/apply-patch/protected-paths.mjs", + ".github/scripts/apply-patch/retry-fetch.mjs", ): with self.subTest(path): unreviewed = ( @@ -1800,7 +1833,7 @@ def _score(self, path, security): return score(_input(files=(_file(path),), lanes=lanes)) def test_a_clean_bound_security_review_clears_each_enforcement_path(self): - self.assertEqual(len(babysit_risk.REVIEW_FLOOR), 21) + self.assertEqual(len(babysit_risk.REVIEW_FLOOR), 32) for glob in babysit_risk.REVIEW_FLOOR: with self.subTest(glob): result = self._score(glob, (_lane("claude-security-review"),)) diff --git a/plugins/source-control/skills/babysit-prs/scripts/tests/test_merge_holds_sync.py b/plugins/source-control/skills/babysit-prs/scripts/tests/test_merge_holds_sync.py index 127cdc8f71..6636a62332 100644 --- a/plugins/source-control/skills/babysit-prs/scripts/tests/test_merge_holds_sync.py +++ b/plugins/source-control/skills/babysit-prs/scripts/tests/test_merge_holds_sync.py @@ -66,6 +66,17 @@ ".github/scripts/pr-merge/superseded.mjs", ".github/scripts/pr-merge/clear-escalations.mjs", ) +# The scripts that act on a pr-unblock stop's route: report-escalation.sh adds +# `needs-human` for a held reason, adjudicate-feedback.mjs decides which threads +# a model verdict may settle, publish-feedback.mjs resolves threads, and +# apply-patch.mjs decides which fix-ci escalations hold. Flooring the routing +# table alone would let an edit to one of them drop a hold. +ROUTE_ENTRIES = ( + ".github/scripts/pr-unblock/report-escalation.sh", + ".github/scripts/pr-unblock/adjudicate-feedback.mjs", + ".github/scripts/pr-unblock/publish-feedback.mjs", + ".github/scripts/apply-patch/apply-patch.mjs", +) # A relative import, or a module-relative file a module reads at load # (`new URL("./x.json", import.meta.url)`): data such as a routing table # decides as much as the code that reads it. @@ -226,8 +237,8 @@ def test_the_floor_is_the_same_on_both_sides(self) -> None: self.assertEqual(answer["classes"], [list(row) for row in risk.FLOOR_CLASSES]) self.assertEqual(answer["floor"], list(risk.HIGH_FLOOR)) self.assertEqual(answer["review"], list(risk.REVIEW_FLOOR)) - self.assertEqual(len(risk.FLOOR_CLASSES), 51) - self.assertEqual(len(risk.REVIEW_FLOOR), 21) + self.assertEqual(len(risk.FLOOR_CLASSES), 62) + self.assertEqual(len(risk.REVIEW_FLOOR), 32) self.assertEqual( {kind for _, kind in risk.FLOOR_CLASSES}, {"trust", "gate", "enforcement"} ) @@ -253,6 +264,20 @@ def test_the_floor_holds_what_its_actions_load(self) -> None: # The rate-limit defaults rate-guard.mjs reads decide when the lanes # stop short of GitHub's limits, so they are held like the module. self.assertIn(".github/actions/log-api-budget/rate-limits.json", files) + # The routing table escalation-routes.mjs reads decides hold or fail + # per stop reason, so it is held like the module. + self.assertIn(".github/scripts/pr-unblock/escalation-routes.json", files) + answer = module(files) + for path, held in zip(files, answer["held"], strict=True): + with self.subTest(path): + self.assertTrue(any(risk.glob_match(g, path) for g in risk.FLOOR)) + self.assertEqual( + held, any(risk.glob_match(g, path) for g in risk.HIGH_FLOOR) + ) + + def test_the_floor_holds_the_code_that_acts_on_a_stop_route(self) -> None: + files = import_closure(ROUTE_ENTRIES) + self.assertIn(".github/scripts/apply-patch/protected-paths.mjs", files) answer = module(files) for path, held in zip(files, answer["held"], strict=True): with self.subTest(path): diff --git a/scripts/ci-hold-pr-unblock-status.test.sh b/scripts/ci-hold-pr-unblock-status.test.sh new file mode 100755 index 0000000000..34a1085b77 --- /dev/null +++ b/scripts/ci-hold-pr-unblock-status.test.sh @@ -0,0 +1,171 @@ +#!/usr/bin/env bash +# Runs the `Fail a head a pr-unblock lane stopped` step body of +# .github/workflows/pr-require-checks.yml's ci-status under the shell Actions +# gives a step with no `shell:` (bash -e), against stub `gh` and `sleep`. +# +# ci-status fails while the newest `pr-unblock/*` status on the head, by +# github-actions[bot], is a failure or error, and names each such context. A +# later success on the same head supersedes a failure. In the merge queue the +# head is the queued pull request's, read live. It fails closed when it cannot +# tell. +# test-scope: .github/workflows/pr-require-checks.yml +set -uo pipefail + +ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +WORKFLOW="$ROOT/.github/workflows/pr-require-checks.yml" + +# shellcheck source=lib/test-harness.sh +. "$ROOT/scripts/lib/test-harness.sh" + +TMP_ROOT="$(mktemp -d)" +trap 'rm -rf "$TMP_ROOT"' EXIT + +# shellcheck source=lib/ci-gh-read.sh +. "$ROOT/scripts/lib/ci-gh-read.sh" +export RUNNER_TEMP="$TMP_ROOT" +ci_gh_read::install "$WORKFLOW" "$TMP_ROOT" budget || fail "the ci-status gh_read helper step is missing or fails" + +NAME='Fail a head a pr-unblock lane stopped' +step="$(awk -v name=" - name: $NAME" '$0 == name { on = 1; print; next } + on && /^ - / { exit } + on { print }' "$WORKFLOW")" +awk '/^ run: \|$/ { on = 1; next } + on && /^ / { print substr($0, 11); next } + on && /^[[:blank:]]*$/ { print ""; next } + on { exit }' <<<"$step" >"$TMP_ROOT/body.sh" + +if [[ ! -s "$TMP_ROOT/body.sh" ]]; then + fail "no '$NAME' step with a run: | body in $WORKFLOW" + test_harness::report + exit 1 +fi +if [[ "$step" == *"if: \${{ !cancelled() && (github.event_name == 'pull_request' || github.event_name == 'merge_group') }}"* ]]; then + ok "step runs on every pull_request and merge_group run that is not cancelled" +else + fail "step lost its pull_request and merge_group condition" +fi +if [[ "$step" == *"continue-on-error"* ]]; then + fail "step must not continue on error: its exit code is the hold" +else + ok "step fails the job on its exit code" +fi +order="$(grep -nE "^ ci-status:$|^ - name: ($NAME|Aggregate lane results)$" "$WORKFLOW" | cut -d: -f1 | paste -sd' ')" +read -r job_line hold_line agg_line <<<"$order" +if [[ -n "${agg_line:-}" && "$job_line" -lt "$hold_line" && "$hold_line" -lt "$agg_line" ]]; then + ok "step sits in ci-status before the aggregate" +else + fail "step is not inside ci-status before the aggregate (lines: $order)" +fi + +# The gh stub logs its arguments. A pulls read answers STUB_PULL, after the +# api-budget wrapper's header block when it added -i. A statuses read fails +# while STUB_STATUS_FAILS is set, else applies the step's --jq filter to the +# STUB_STATUSES array (newest first, as the API lists them), as gh does per +# page. +mkdir -p "$TMP_ROOT/bin" +cat >"$TMP_ROOT/bin/gh" <<'EOF' +#!/usr/bin/env bash +printf '%s\n' "$*" >>"$STUB_LOG" +case "$*" in +*/pulls/*) + [[ -z "${STUB_PULL_FAILS:-}" ]] || exit 1 + [[ "$2" != -i ]] || printf 'HTTP/2.0 200 OK\r\nX-Ratelimit-Remaining: 4000\r\nX-Ratelimit-Resource: core\r\n\r\n' + printf '%s\n' "$STUB_PULL" + ;; +*/statuses*) + [[ -z "${STUB_STATUS_FAILS:-}" ]] || exit 1 + while [[ $# -gt 0 && "$1" != --jq ]]; do shift; done + jq -r "$2" <<<"${STUB_STATUSES:-[]}" + ;; +*) exit 1 ;; +esac +EOF +cat >"$TMP_ROOT/bin/sleep" <<'EOF' +#!/usr/bin/env bash +printf '%s\n' "$*" >>"$STUB_SLEEPS" +EOF +chmod +x "$TMP_ROOT/bin/gh" "$TMP_ROOT/bin/sleep" + +HEAD=0123456789abcdef0123456789abcdef01234567 +QUEUED=fedcba9876543210fedcba9876543210fedcba98 +GROUP_BASE=89abcdef0123456789abcdef0123456789abcdef +LANE='{"login":"github-actions[bot]"}' +# status [creator] [description] +status() { jq -cn --arg c "$1" --arg s "$2" --argjson u "${3:-$LANE}" --arg d "${4:-The lane stopped}" \ + '{context: $c, state: $s, creator: $u, description: $d}'; } + +# expect