diff --git a/.changes/source-control/feat-4588-promotion-evidence-publisher-3634507d.md b/.changes/source-control/feat-4588-promotion-evidence-publisher-3634507d.md new file mode 100644 index 0000000000..8a6fdc8a15 --- /dev/null +++ b/.changes/source-control/feat-4588-promotion-evidence-publisher-3634507d.md @@ -0,0 +1,7 @@ +--- +bump: minor +--- + +### Added + +- `babysit-loop` ships `scripts/publish-promotion-evidence.mjs`, an operator-side publisher for the promotion-evidence events file (#4588). It reads GitHub only, through read-only `gh api` calls, and emits the `{cell, event, at}` contrary events the security-binding checker reads (`reverted-merge`, `gate-failure`, and `verification-divergence` for a missed-blocking finding a human confirmed) plus a C3 promotion-evidence report: each promotable cell's predicate terms with counts, window and pass or fail. Report mode, the default, writes nothing. `--out ` writes `evidence.json` and an `evidence.meta.json` sidecar (`generated_at`, `through`, `sha256`, `prev_sha256`) by temp file and rename, puts the previous events file back when the sidecar rename fails, and refuses a set that drops a previously published event or an existing file that does not match its sidecar. Work classes are read from the labels of the PR and of the issues it linked at merge, as of the merge. A merge counts as autonomous when no human enqueued it or a `--lane-login` identity merged it. An autonomous merge with no readable class is listed as unclassed and its contrary events count against every cell. Only the post-merge gate counts toward `gate-failure`. That gate is the full and canary workflow runs on any later commit containing the merge, read as `babysit_postmerge.py` reads them, plus the check runs and statuses named by the required `--gate-check` on the merge commit. A cancelled run counts unless a later run of that check succeeded, and a red full run covering several merges counts against each. As in `babysit_postmerge.py`, a merge with no lane dispatch run within 6 hours, or a lane-enqueued merge whose pin comment is missing, edited or names another head, is a gate failure. Where it runs and under which identity is the operator's deployment choice (an App lane is passed as `--lane-login name[bot]`), and the resolver does not read the sidecar; merge authority is unchanged. diff --git a/plugins/source-control/skills/babysit-loop/reference/promotion-evidence-bootstrap.md b/plugins/source-control/skills/babysit-loop/reference/promotion-evidence-bootstrap.md index cb3a01cb83..71406c6658 100644 --- a/plugins/source-control/skills/babysit-loop/reference/promotion-evidence-bootstrap.md +++ b/plugins/source-control/skills/babysit-loop/reference/promotion-evidence-bootstrap.md @@ -147,6 +147,49 @@ Agent-unwritable, because an agent that can write the file can delete a `reverte keep its cell promoted. Outside the target repository's blast radius, for the same reason as the binding. +#### The publisher + +[`scripts/publish-promotion-evidence.mjs`](../scripts/publish-promotion-evidence.mjs) builds the +events file from GitHub alone, through read-only `gh api` calls: merged pull requests classed by +the `work-class:` labels their linked issues and the pull request carried at merge time, which of +them merged autonomously (no human enqueued them, or a `--lane-login` identity merged them), +reverts of those merges, the post-merge gate (the full and canary workflow runs on any later +commit that contains the merge, the dispatch deadline and the pin read-back, read as +`babysit-prs/scripts/babysit_postmerge.py` reads them, and the checks named by the required +`--gate-check` on the merge commit), and the missed-blocking label a human applied. An autonomous +merge whose class cannot be read is listed as unclassed, and its contrary events count against +every promotable cell. Its header comment states what counts as each event, including the +judgment mappings and how to set `--lane-login` for merge-queue merges. + +- **Report mode** (no `--out`) prints the events array and the C3 promotion-evidence report to + stdout and writes nothing. The report gives each promotable cell's predicate terms from + [work-classes.md](https://raw.githubusercontent.com/melodic-software/claude-code-plugins/main/plugins/autonomy/reference/guardrails/work-classes.md) + "Suggested default predicates" with counts, window and pass or fail, and lists merged reverts + it could not match to a merge. +- **Deployment defaults:** `--missed-blocking-label ai-review: missed-blocking` (the script's + default), and `--ai-reviewer claude[bot] --ai-reviewer chatgpt-codex-connector[bot]`, the REST + spellings of the review bots on this repository's pull requests (read from #7361's reviews, + 2026-10-13). +- **Every revert counts** as `reverted-merge`, whoever merged or enqueued the revert pull request + (a human, a bot or the lane), and so does a revert commit pushed with no pull request. The + trade-off: a bot able to merge reverts can hold a cell down. That fails safe, because a revert + can only demote. +- **`--out `** writes `evidence.json` and the sidecar `evidence.meta.json` + (`generated_at`, `through`, `sha256` of the events file, `prev_sha256`), each by a temp file + and a rename. It refuses to publish when the previous events file does not match its sidecar's + `sha256`, or when the new events drop one the previous file held. + +Run it as the operator, never from the lane, and point `--out` at a directory the lane cannot +write. Which host runs it, under which identity and on what schedule is the operator's +deployment choice. The resolver does not read the sidecar. + +The post-merge check dispatches its full run under the lane identity. Where the permission that +dispatch needs also lets that identity delete a red run before the publisher reads it, either the +dispatch moves to an identity the lane cannot use, or a deleted red run stays a named residual +risk. Which permission covers dispatch and which covers run deletion: GitHub's +[Actions repository permissions table](https://docs.github.com/en/rest/authentication/permissions-required-for-github-apps#repository-permissions-for-actions) +(as of 2026-10-13; recheck when GitHub adds or splits an Actions permission). + ### `promotion_evidence_checker` The `check-security-binding.mjs` the helper runs (the autonomy plugin's diff --git a/plugins/source-control/skills/babysit-loop/scripts/fixtures/publish-promotion-evidence/recorded.json b/plugins/source-control/skills/babysit-loop/scripts/fixtures/publish-promotion-evidence/recorded.json new file mode 100644 index 0000000000..bf6cef188d --- /dev/null +++ b/plugins/source-control/skills/babysit-loop/scripts/fixtures/publish-promotion-evidence/recorded.json @@ -0,0 +1,260 @@ +{ + "pullListItem": { + "base": { + "ref": "main" + }, + "body": "Refs: #7339", + "labels": [], + "created_at": "2026-10-13T00:54:35Z", + "merge_commit_sha": "aad0b2b69bc22154c61b0857307a183d0c7821a8", + "merged_at": "2026-10-13T02:42:17Z", + "number": 7352, + "title": "fix(source-control): settle unattended merge residuals before Phase 15", + "updated_at": "2026-10-13T02:42:19Z", + "user": { + "login": "kyle-sexton", + "type": "User" + } + }, + "checkRun": { + "completed_at": "2026-10-13T02:38:54Z", + "conclusion": "success", + "name": "lint-repo", + "status": "completed" + }, + "status": { + "context": "ci-lanes", + "state": "success", + "created_at": "2026-10-13T02:42:00Z", + "updated_at": "2026-10-13T02:42:00Z" + }, + "commit": { + "commit": { + "committer": { + "date": "2026-10-13T02:35:38Z" + }, + "message": "fix(source-control): settle unattended merge residuals before Phase 15 (#7352)" + }, + "sha": "aad0b2b69bc22154c61b0857307a183d0c7821a8" + }, + "issueEvent": { + "actor": { + "login": "kyle-sexton", + "type": "User" + }, + "created_at": "2026-10-13T01:37:37Z", + "event": "ready_for_review", + "label": null + }, + "review": { + "state": "COMMENTED", + "user": { + "login": "chatgpt-codex-connector[bot]", + "type": "Bot" + } + }, + "pullGraph": { + "data": { + "repository": { + "pullRequest": { + "number": 7063, + "headRefOid": "b6888664c45a19abc91c9cd819770406a6e7616d", + "mergedAt": "2026-10-13T01:47:44Z", + "lastEditedAt": "2026-10-13T01:38:02Z", + "mergeCommit": { + "oid": "6fa4f210dd0eb904a9ca1504344d8b66294ee6f5" + }, + "mergedBy": { + "__typename": "User", + "login": "kyle-sexton" + }, + "labels": { + "pageInfo": { + "hasNextPage": false + }, + "nodes": [] + }, + "closingIssuesReferences": { + "pageInfo": { + "hasNextPage": false + }, + "nodes": [ + { + "number": 6007, + "labels": { + "pageInfo": { + "hasNextPage": false + }, + "nodes": [ + { + "name": "priority: medium" + }, + { + "name": "status: ready" + }, + { + "name": "needs-human" + }, + { + "name": "work-class: scoped" + } + ] + } + } + ] + }, + "timelineItems": { + "pageInfo": { + "hasNextPage": false + }, + "nodes": [ + { + "__typename": "AddedToMergeQueueEvent", + "createdAt": "2026-10-13T01:42:44Z", + "actor": { + "__typename": "User", + "login": "kyle-sexton" + }, + "enqueuer": { + "__typename": "User", + "login": "kyle-sexton" + } + } + ] + } + } + } + } + }, + "issue": { + "labels": [ + { + "name": "priority: medium" + }, + { + "name": "status: ready" + }, + { + "name": "needs-human" + }, + { + "name": "work-class: scoped" + } + ], + "number": 6007 + }, + "repo": { + "default_branch": "main" + }, + "commentsPage": { + "data": { + "repository": { + "pullRequest": { + "comments": { + "pageInfo": { + "hasNextPage": true, + "endCursor": "Y3Vyc29yOnYyOpHPAAAAAWv4HUQ=" + }, + "nodes": [ + { + "author": { + "__typename": "User", + "login": "kyle-sexton" + }, + "body": "`ci-status` is red because this PR is a draft, not because of the change.", + "lastEditedAt": null + } + ] + } + } + } + } + }, + "connectedEventFromCliCli14616": { + "__typename": "ConnectedEvent", + "createdAt": "2026-10-07T08:24:54Z", + "source": { + "__typename": "PullRequest", + "number": 14616 + }, + "subject": { + "__typename": "Issue", + "number": 14614 + } + }, + "commitNotFound": { + "stdout": "{\"message\":\"No commit found for SHA: 0123456789abcdef0123456789abcdef01234567\",\"documentation_url\":\"https://docs.github.com/rest/commits/commits#list-pull-requests-associated-with-a-commit\",\"status\":\"422\"}", + "stderr": "gh: No commit found for SHA: 0123456789abcdef0123456789abcdef01234567 (HTTP 422)\n" + }, + "repoNotFound": { + "data": { + "repository": null + }, + "errors": [ + { + "type": "NOT_FOUND", + "path": [ + "repository" + ], + "locations": [ + { + "line": 1, + "column": 2 + } + ], + "message": "Could not resolve to a Repository with the name 'melodic-software/no-such-repo-xyz'." + } + ] + }, + "pullNotFound": { + "data": { + "repository": { + "pullRequest": null + } + }, + "errors": [ + { + "type": "NOT_FOUND", + "path": [ + "repository", + "pullRequest" + ], + "locations": [ + { + "line": 1, + "column": 66 + } + ], + "message": "Could not resolve to a PullRequest with the number of 999999." + } + ] + }, + "workflowRun": { + "actor": { + "login": "github-merge-queue[bot]" + }, + "conclusion": "failure", + "created_at": "2026-10-12T16:32:14Z", + "event": "schedule", + "head_branch": "main", + "head_sha": "2dc6ef1cef19870b4f15ba796f970980bcb48cf8", + "html_url": "https://github.com/melodic-software/claude-code-plugins/actions/runs/38278079868", + "id": 38278079868, + "run_attempt": 1, + "status": "completed", + "triggering_actor": { + "login": "github-merge-queue[bot]" + }, + "updated_at": "2026-10-12T16:40:40Z" + }, + "compare": { + "ahead_by": 1, + "commits": [ + { + "sha": "aad0b2b69bc22154c61b0857307a183d0c7821a8" + } + ], + "status": "ahead", + "total_commits": 1 + } +} diff --git a/plugins/source-control/skills/babysit-loop/scripts/publish-promotion-evidence.mjs b/plugins/source-control/skills/babysit-loop/scripts/publish-promotion-evidence.mjs new file mode 100755 index 0000000000..28b5ac6b08 --- /dev/null +++ b/plugins/source-control/skills/babysit-loop/scripts/publish-promotion-evidence.mjs @@ -0,0 +1,795 @@ +#!/usr/bin/env node + +// Operator-side publisher for the promotion-evidence events file +// (promotion_evidence_source in reference/promotion-evidence-bootstrap.md) and the C3 +// promotion-evidence report. It reads GitHub only, through `gh api` GET calls and one read-only +// GraphQL query, and never runs inside the lane: the lane must not be able to write its output. +// +// Usage: node publish-promotion-evidence.mjs --repo --since --lane-login [--lane-login ...] +// --gate-check [--gate-check ...] [--through ] [--ai-reviewer ...] [--missed-blocking-label ] [--out ] +// Exit 0 = report printed (and published with --out); 1 = publish refused; 2 = usage or GitHub read error. +// +// Report mode (no --out) prints {repo, since, through, events, unresolvedReverts, gateFailures, unclassedAutonomous, cells} to +// stdout and writes nothing. --through defaults to, and is clamped to, now. With --out it also +// writes /evidence.json (the events array) and /evidence.meta.json ({generated_at, +// through, sha256, prev_sha256}). Both temp files are written before either rename, the events +// file renamed first. It refuses when an existing evidence.json is not a JSON array, has no +// sidecar or does not match the sidecar's sha256, or holds an event the new array lacks. +// +// Cost: every run pages the repository's whole closed-PR history (one REST page per 100 closed +// PRs, created ascending, so no merge is skipped while paging), then reads, per merged PR in the +// window, one GraphQL query and the issue events of the PR and each linked issue, and per +// autonomous merge its check runs and statuses. +// +// Autonomy follows babysit_merge_policy.py derive_auto_merge: the latest AddedToMergeQueueEvent's +// actor and enqueuer, or merged_by when the merge bypassed the queue; a merge is autonomous when +// any of them is null or not human (a User outside --lane-login, not `github-actions`, not a +// `[bot]` login; derive_auto_merge's write-permission read is not made), or merged_by is a +// --lane-login. Logins compare case-insensitively and exactly in REST spelling: a GraphQL `Bot` +// actor gains `[bot]`, so a User owning an App's bare slug is never that App. GitHub records the +// enqueuer as merged_by (probed 2026-10-12 on a queue-merged pull: merged_by.login matched the +// AddedToMergeQueueEvent enqueuer), so --lane-login names any lane identity that runs as a User +// (a machine user); an App lane is a Bot and counts without it. Recheck if merged_by stops +// matching the enqueuer. For an App lane pass --lane-login as `name[bot]`, or its dispatch runs +// and pins are not found. +// +// A red workflow run deleted before this script reads it is never seen. babysit_postmerge.py +// dispatches the post-merge full run under the lane identity; whether that identity can also +// delete runs is the deployment question promotion-evidence-bootstrap.md records with its pointer. +// Not decided here. +// +// Anything a lane can write into a PR or commit degrades to a listing or to unclassed-autonomous, +// never to a refused publish. What refuses is GitHub itself being unreadable: a failed read, an +// unparsable date, a compare that is neither ahead, identical nor behind. +// +// What counts (judgment where the contract names no GitHub source): +// - A merged PR's work class is the highest `work-class:` label (the label set +// babysit-prs/scripts/babysit_merge_policy.py WORK_CLASS_LABELS maps) on the issues it linked +// at merge and on the PR, each label set rebuilt as of merged_at by undoing later +// labeled/unlabeled events, so relabeling after the merge changes nothing. The linked set is +// closingIssuesReferences with later ConnectedEvent/DisconnectedEvent undone; when the PR body +// was edited after the merge, its closing-keyword links at merge are unknown and the class is +// none. A label, linked-issue or timeline list that is unreadable or has another page leaves +// the merge unclassed and counts it as autonomous. +// - A C2/C3 merge is counted. An autonomous merge with no class is listed in +// unclassedAutonomous, never counted toward a positive term, and each of its contrary events +// is emitted against every promotable cell. +// - reverted-merge: an autonomous C2/C3 merge named by a merged PR body or a default-branch +// commit as `Reverts #` (GitHub's revert button) or `This reverts commit +// ` (git revert); one event per reverted merge, at the earliest revert seen. A merged PR +// titled revert... with neither form is listed in unresolvedReverts for a human to read, as is +// a named target that is not a merged pull request or a SHA no commit has, so a bad reference +// cannot stop a publish. Every revert counts, whoever merged or enqueued the revert PR (a +// human, a bot or the lane) and whoever pushed a revert commit straight to the default branch +// (its git author can be forged anyway). The trade-off: a bot able to merge reverts can hold a +// cell down, which fails safe (it can only demote). +// - gate-failure: only the post-merge gate counts: check runs and commit-status contexts on an +// autonomous merge's merge commit whose name is a --gate-check (required; none named is a +// usage error, never an empty gate). In this repository babysit-prs/scripts/babysit_postmerge.py +// reads the post-merge gate as the silent-revert canary (post-merge-detect-silent-revert.yml, +// check run `Silent-revert canary` on the merge commit) and the full pr-require-checks.yml run +// (aggregate job `ci-status`), scheduled or dispatched on a later SHA containing the merge. This +// script reads those workflow runs the same way (postMergeRuns) and, on the merge commit itself, +// the --gate-check check runs and contexts. Among the latter, every check run (filter=all, so a run later +// re-run green still counts) concluding failure, timed_out or startup_failure, and every +// status in state failure or error, whenever it completed. A run concluding cancelled, +// action_required, stale or anything else not green also counts, unless a later run of the +// same check name on that commit succeeded. The gate-pass term also fails while a run is in +// progress or a context's latest status is not success. +// - verification-divergence: a human (a User actor outside --lane-login) applied +// --missed-blocking-label to a merged C3 PR (cell C3-ai-review-blocking) or an autonomous C2 +// PR (cell C2-auto-merge), read from the label timeline, so removing the label later does not +// undo it. +// - A cell's window runs from its first counted merge to its last. +// +// Event shape and vocabulary: plugins/autonomy/skills/setup/scripts/check-security-binding.mjs, +// CONTRARY_EVIDENCE_EVENTS and resolveEffectivePromotion; restated here, not imported, because +// no plugin reads another plugin's files. Predicate terms and suggested default thresholds: +// plugins/autonomy/reference/guardrails/work-classes.md "Suggested default predicates". + +import { spawnSync } from "node:child_process"; +import { createHash } from "node:crypto"; +import { readFileSync, renameSync, rmSync, writeFileSync } from "node:fs"; +import { join } from "node:path"; +import process from "node:process"; +import { fileURLToPath } from "node:url"; +import { parseArgs } from "node:util"; + +const CLASS_LABELS = { + "work-class: read-only": "C1", + "work-class: mechanical": "C2", + "work-class: scoped": "C3", + "work-class: structural": "C4", + "work-class: untrusted-provenance": "C5", +}; +const GATE_FAILED = new Set(["failure", "timed_out", "startup_failure"]); +const GATE_OK = new Set(["success", "neutral", "skipped"]); +const STATUS_FAILED = new Set(["failure", "error"]); +const PAGE = 100; +const DAY_MS = 86_400_000; +const ISO = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(\.\d+)?(Z|[+-]\d{2}:\d{2})$/; + +export class PublishError extends Error {} + +function instant(value, name) { + const ms = typeof value === "string" && ISO.test(value) ? Date.parse(value) : Number.NaN; + if (Number.isNaN(ms)) throw new PublishError(`${name}: not an ISO 8601 instant with an offset: ${JSON.stringify(value)}`); + return ms; +} + +// A REST path is read with GET; "graphql" runs a query (read-only: PULL_QUERY is the only one). +function realGh(path, graphql) { + const args = graphql + ? ["api", "graphql", "-f", `query=${graphql.query}`, ...Object.entries(graphql.variables).flatMap(([key, value]) => ["-F", `${key}=${value}`])] + : ["api", "--method", "GET", path]; + const run = spawnSync("gh", args, { encoding: "utf8", maxBuffer: 64 * 1024 * 1024 }); + if (run.status !== 0) return ghFailure(path, graphql, run.stdout, run.stderr || run.error?.message || ""); + return JSON.parse(run.stdout); +} + +// A reference a lane can write (a PR number, a commit SHA) that names nothing: listed, never fatal. +export class UnresolvableError extends PublishError {} +// A page that is unreadable or has another page: the merge's class is unknown. +class IncompleteError extends PublishError {} + +// A failed `gh api` call. Only two answers are not read failures: GraphQL NOT_FOUND on the pull +// request itself (data.repository.pullRequest null, every error path ending in pullRequest), which +// is returned as the answer, and REST 422 "No commit found for SHA", which is unresolvable. Any +// other failure, a repository NOT_FOUND included, fails the publish. +export function ghFailure(path, graphql, stdout, stderr) { + let body = null; + try { + body = JSON.parse(stdout); + } catch {} + const errors = body?.errors; + if ( + graphql && + body?.data?.repository?.pullRequest === null && + Array.isArray(errors) && + errors.length > 0 && + errors.every((e) => e?.type === "NOT_FOUND" && Array.isArray(e.path) && e.path.at(-1) === "pullRequest") + ) { + return body; + } + if (!graphql && String(body?.status) === "422" && /^No commit found for SHA/.test(body?.message ?? "")) throw new UnresolvableError(body.message); + throw new PublishError(`gh api ${path} failed: ${String(stderr).trim()}`); +} + +// A date GitHub returned; one that does not parse refuses the publish. +function ms(value, what) { + const parsed = typeof value === "string" ? Date.parse(value) : Number.NaN; + if (Number.isNaN(parsed)) throw new PublishError(`refusing to publish: ${what} has an unreadable date ${JSON.stringify(value)}`); + return parsed; +} + +const PULL_QUERY = + "query($owner:String!,$name:String!,$number:Int!){repository(owner:$owner,name:$name){pullRequest(number:$number){" + + "number headRefOid mergedAt lastEditedAt mergeCommit{oid} mergedBy{__typename login} labels(first:100){pageInfo{hasNextPage} nodes{name}} " + + "closingIssuesReferences(first:50){pageInfo{hasNextPage} nodes{number labels(first:100){pageInfo{hasNextPage} nodes{name}}}} " + + "timelineItems(first:100,itemTypes:[CONNECTED_EVENT,DISCONNECTED_EVENT,ADDED_TO_MERGE_QUEUE_EVENT]){pageInfo{hasNextPage} nodes{__typename " + + "... on ConnectedEvent{createdAt subject{__typename ... on Issue{number}}} ... on DisconnectedEvent{createdAt subject{__typename ... on Issue{number}}} " + + "... on AddedToMergeQueueEvent{createdAt actor{__typename login} enqueuer{__typename login}}}}}}}"; +const CELLS = ["C2-auto-merge", "C3-ai-review-blocking", "C3-auto-merge"]; + +// A REST login (`app[bot]` for an App), compared exactly and case-insensitively. +function loginKey(login) { + return String(login ?? "") + .trim() + .toLowerCase(); +} + +// A GraphQL actor in REST spelling, as babysit_merge_policy.py actor_key: a `Bot` gains the +// `[bot]` GraphQL omits, so a User who owns the bare slug `app` is never the App `app[bot]`. +function actorKey(actor) { + const login = loginKey(actor?.login); + return actor?.__typename === "Bot" && login && !login.endsWith("[bot]") ? `${login}[bot]` : login; +} + +function onePage(connection, what) { + if (!Array.isArray(connection?.nodes) || connection.pageInfo?.hasNextPage !== false) { + throw new IncompleteError(`gh api graphql: ${what} unreadable or hasNextPage`); + } + return connection.nodes; +} + +function pages(gh, path, key) { + const out = []; + for (let page = 1; ; page += 1) { + const body = gh(`${path}${path.includes("?") ? "&" : "?"}per_page=${PAGE}&page=${page}`); + const items = key ? body?.[key] : body; + if (!Array.isArray(items)) throw new PublishError(`gh api ${path}: expected an array${key ? ` at .${key}` : ""}`); + out.push(...items); + if (items.length < PAGE) return out; + } +} + +function workClass(names) { + return names.map((name) => CLASS_LABELS[name]).filter(Boolean).sort().at(-1) ?? null; +} + +// The label set as of `atMs`: the current set with every later labeled/unlabeled event undone. +function labelsAsOf(current, events, atMs) { + const set = new Set(current); + const later = events + .filter((e) => (e.event === "labeled" || e.event === "unlabeled") && ms(e.created_at, "a label event") > atMs) + .sort((a, b) => ms(b.created_at, "a label event") - ms(a.created_at, "a label event")); + for (const e of later) e.event === "labeled" ? set.delete(e.label?.name) : set.add(e.label?.name); + return [...set]; +} + +function eventKey(event) { + return JSON.stringify([event.cell, event.event, event.at]); +} + +// The post-merge gate's workflow runs, read as babysit-prs/scripts/babysit_postmerge.py check() +// and _judge() read them (same plugin, mirrored rather than imported across languages): +// FULL_RUN_WORKFLOW runs from `schedule` and `workflow_dispatch`, and CANARY_WORKFLOW runs from +// `push`, on the default branch, created at or after a merge (no upper bound), on a SHA that +// contains it: compare ..., as _Context.contains does. Attempt 1 decides; a red +// full run is a flake only when a lane identity's attempt 2 passed. A merge is verified at the +// later of its first green lane dispatch run and first green canary run, and runs created after +// that are not its evidence; a compare that is behind means not contained. Only a verified merge +// with nothing red or pending passes the gate, and only a lane dispatch created within +// DISPATCH_DEADLINE of the merge verifies it, so a merge with no timely lane dispatch run +// (check()'s 6-hour failure) is not passed. Departures, all toward a failure: a compare that is +// diverged or unknown refuses the publish (check() reads it as not containing); a red full run +// whose re-run has not come counts as red once DISPATCH_DEADLINE has passed since it completed +// (check() waits on it); a red run covering several merges is a failure of each. +// +// Where check() files a Failure, this emits a gate-failure, so demotion counts match: a red full +// or canary run (at the run's update time); no dispatch run by a lane identity created within +// DISPATCH_DEADLINE of the merge, once that deadline has passed (at the deadline); and, for a +// merge a lane identity enqueued, the pin read-back (read_pin): no unedited lane pin once the +// deadline has passed (at the deadline), or a pinned SHA that is not headRefOid (at the merge). +const FULL_RUN_WORKFLOW = "pr-require-checks.yml"; +const CANARY_WORKFLOW = "post-merge-detect-silent-revert.yml"; +const DISPATCH_DEADLINE_MS = 6 * 3_600_000; + +function isoSeconds(msValue) { + return new Date(msValue).toISOString().replace(/\.000Z$/, "Z"); +} + +// babysit_postmerge.py read_pin: the newest comment by the identity whose first line is the pin +// marker, read through at most PIN_PAGE_LIMIT pages; none, an edited one, or a longer thread is +// no pin. +const PIN_RE = /^$/; +const PIN_QUERY = + "query($owner:String!,$name:String!,$number:Int!,$c:String){repository(owner:$owner,name:$name){pullRequest(number:$number){" + + "comments(first:100,after:$c){pageInfo{hasNextPage endCursor} nodes{author{__typename login} body lastEditedAt}}}}}"; +const PIN_PAGE_LIMIT = 30; + +function readPin(gh, owner, name, number, lane) { + const firstLine = (text) => String(text ?? "").trimStart().split("\n", 1)[0].trim(); + const pins = []; + let cursor = null; + for (let page = 0; page < PIN_PAGE_LIMIT; page += 1) { + const variables = { owner, name, number, ...(cursor ? { c: cursor } : {}) }; + const comments = gh("graphql", { query: PIN_QUERY, variables })?.data?.repository?.pullRequest?.comments; + if (!Array.isArray(comments?.nodes) || typeof comments.pageInfo?.hasNextPage !== "boolean") throw new PublishError(`#${number} pin: no comment page`); + pins.push(...comments.nodes.filter((c) => lane.has(actorKey(c?.author)) && PIN_RE.test(firstLine(c?.body)))); + if (!comments.pageInfo.hasNextPage) { + const last = pins.at(-1); + return last && !last.lastEditedAt ? PIN_RE.exec(firstLine(last.body))[1] : null; + } + cursor = comments.pageInfo.endCursor; + if (typeof cursor !== "string" || !cursor) throw new PublishError(`#${number} pin: a next page with no cursor`); + } + return null; +} + +function postMergeRuns(gh, repo, branch, merges, { lane, throughMs }) { + if (merges.length === 0) return () => ({ failures: [], unsettled: false, verified: false }); + const floor = Math.min(...merges.map((m) => ms(m.mergedAt, `#${m.number} mergedAt`))); + const created = encodeURIComponent(`>=${new Date(floor).toISOString().replace(/\.\d{3}Z$/, "Z")}`); + const byCreated = (a, b) => a.createdMs - b.createdMs; + const list = (workflow, event) => + pages(gh, `repos/${repo}/actions/workflows/${workflow}/runs?event=${event}&branch=${encodeURIComponent(branch)}&created=${created}`, "workflow_runs") + .map((run) => ({ ...run, createdMs: ms(run.created_at, `run ${run.id}`), check: `${workflow} ${event}` })) + .filter((run) => run.head_branch === branch && run.createdMs >= floor); + const fullRuns = [...list(FULL_RUN_WORKFLOW, "schedule"), ...list(FULL_RUN_WORKFLOW, "workflow_dispatch")].sort(byCreated); + const canaryRuns = list(CANARY_WORKFLOW, "push").sort(byCreated); + + const compares = new Map(); + const contains = (sha, runSha) => { + const key = `${sha}...${runSha}`; + if (!compares.has(key)) { + const status = gh(`repos/${repo}/compare/${key}?per_page=1&page=1`)?.status; + // behind: the run's SHA is an ancestor of the merge, so it does not contain it (as check()). + if (status !== "ahead" && status !== "identical" && status !== "behind") { + throw new PublishError(`refusing to publish: compare ${key} is ${JSON.stringify(status)}, so whether that post-merge run contains the merge is unknown`); + } + compares.set(key, status !== "behind"); + } + return compares.get(key); + }; + + const outcomes = new Map(); + const outcome = (run, attributable) => { + const key = `${run.id}:${attributable}`; + if (!outcomes.has(key)) { + const attempt = (n) => (run.run_attempt === n ? run : gh(`repos/${repo}/actions/runs/${run.id}/attempts/${n}`)); + const first = attempt(1); + let state; + if (!first?.status) throw new PublishError(`run ${run.id}: attempt 1 is missing`); + if (first.status !== "completed") state = "pending"; + else if (first.conclusion === "success") state = "green"; + else if (!attributable) state = "red"; + else { + const second = run.run_attempt >= 2 ? attempt(2) : null; + if (!second) state = throughMs - ms(first.updated_at, `run ${run.id} attempt 1`) > DISPATCH_DEADLINE_MS ? "red" : "pending"; + else if (!lane.has(loginKey(second.triggering_actor?.login))) state = "red"; + else if (second.status !== "completed") state = "pending"; + else state = second.conclusion === "success" ? "flake" : "red"; + } + outcomes.set(key, { state, conclusion: first.conclusion }); + } + return outcomes.get(key); + }; + + // Compares run only where the answer can change the verdict: a red or pending run, a green lane + // dispatch run, and a green canary run once a lane dispatch has verified half the merge. + return (merge) => { + const mergedMs = ms(merge.mergedAt, `#${merge.number} mergedAt`); + const after = (runs) => runs.filter((run) => run.createdMs >= mergedMs); + const relevant = (run) => contains(merge.sha, run.head_sha); + const fulls = after(fullRuns); + const canaries = after(canaryRuns); + const dispatch = fulls.find( + (run) => + run.event === "workflow_dispatch" && + run.createdMs <= mergedMs + DISPATCH_DEADLINE_MS && + lane.has(loginKey(run.actor?.login)) && + ["green", "flake"].includes(outcome(run, true).state) && + relevant(run), + ); + const canary = dispatch && canaries.find((run) => outcome(run, false).state === "green" && relevant(run)); + const verifiedAt = canary ? Math.max(dispatch.createdMs, canary.createdMs) : Number.POSITIVE_INFINITY; + const deadline = mergedMs + DISPATCH_DEADLINE_MS; + const timely = fulls.some((run) => run.event === "workflow_dispatch" && run.createdMs <= deadline && lane.has(loginKey(run.actor?.login)) && relevant(run)); + const result = { failures: [], unsettled: false, verified: Boolean(canary) }; + if (!timely && throughMs > deadline) { + result.failures.push({ check: "dispatch deadline", conclusion: "no dispatch run by a lane identity within 6 h", at: isoSeconds(deadline) }); + } + for (const [runs, attributable] of [ + [fulls, true], + [canaries, false], + ]) { + for (const run of runs.filter((r) => r.createdMs <= verifiedAt)) { + const { state, conclusion } = outcome(run, attributable); + if (state === "red" && relevant(run)) result.failures.push({ check: run.check, conclusion, at: run.updated_at }); + else if (state === "pending" && relevant(run)) result.unsettled = true; + } + } + return result; + }; +} + +export function collect({ repo, since, through, laneLogins, gateChecks, aiReviewers = [], missedBlockingLabel, gh = realGh }) { + const gate = new Set(gateChecks ?? []); + if (gate.size === 0) throw new PublishError("at least one --gate-check naming the post-merge gate is required"); + const sinceMs = instant(since, "--since"); + const throughMs = Math.min(instant(through, "--through"), Date.now()); + const lane = new Set(laneLogins.map(loginKey)); + // babysit_merge_policy.py is_human, without its permission read: a User that is not a lane + // login, `github-actions` or a `[bot]` login. A null actor is not human. + const human = (actor) => + actor?.__typename === "User" && !lane.has(actorKey(actor)) && loginKey(actor.login) !== "github-actions" && !/\[bot\]$/i.test(actor.login ?? ""); + const inWindow = (at) => typeof at === "string" && Date.parse(at) >= sinceMs && Date.parse(at) <= throughMs; + const events = new Map(); + const addEvent = (cell, event, at) => { + ms(at, `a ${event} event`); + const record = { cell, event, at }; + events.set(eventKey(record), record); + }; + + const branch = gh(`repos/${repo}`)?.default_branch; + if (!branch) throw new PublishError(`gh api repos/${repo}: no default_branch`); + // Merges into the default branch only (babysit_postmerge.py _merged_since). Created-ascending + // order is stable while paging: an update moves no PR between pages. + const seen = new Set(); + const merged = pages(gh, `repos/${repo}/pulls?state=closed&base=${encodeURIComponent(branch)}&sort=created&direction=asc`).filter( + (pr) => inWindow(pr.merged_at) && !seen.has(pr.number) && seen.add(pr.number), + ); + + const issueEvents = new Map(); + const eventsOf = (number) => { + if (!issueEvents.has(number)) issueEvents.set(number, pages(gh, `repos/${repo}/issues/${number}/events`)); + return issueEvents.get(number); + }; + const [owner, name] = repo.split("/"); + + // The class at merge: labels of the PR and of the issues linked at merge, each rebuilt as of + // mergedAt. Links: the current set with later Connected/Disconnected events undone; a body + // edited after the merge leaves the closing-keyword links unknown, so the class is too. + const classAtMerge = (g, timeline) => { + const atMs = ms(g.mergedAt, `#${g.number} mergedAt`); + if (g.lastEditedAt && ms(g.lastEditedAt, `#${g.number} lastEditedAt`) > atMs) return { cls: null, reason: "PR body edited after the merge: closing-keyword links unknown" }; + const linked = new Map(); + for (const issue of onePage(g.closingIssuesReferences, `closingIssuesReferences of #${g.number}`)) { + linked.set(issue.number, onePage(issue.labels, `labels of issue #${issue.number}`).map((label) => label?.name)); + } + // A ConnectedEvent on a PR timeline names the issue as `subject` (recorded from cli/cli #14616); + // any other shape after the merge leaves the links unknown. + const laterLinks = timeline + .filter((e) => ["ConnectedEvent", "DisconnectedEvent"].includes(e.__typename) && ms(e.createdAt, "a link event") > atMs) + .sort((a, b) => ms(b.createdAt, "a link event") - ms(a.createdAt, "a link event")); + if (laterLinks.some((e) => e.subject?.__typename !== "Issue" || !Number.isInteger(e.subject.number))) { + return { cls: null, reason: "a link event after the merge has an unrecognized shape: links unknown" }; + } + for (const e of laterLinks) { + if (e.__typename === "ConnectedEvent") linked.delete(e.subject.number); + else if (!linked.has(e.subject.number)) { + const labels = gh(`repos/${repo}/issues/${e.subject.number}`)?.labels; + if (!Array.isArray(labels)) throw new IncompleteError(`labels of issue #${e.subject.number} unreadable`); + linked.set(e.subject.number, labels.map((label) => label?.name)); + } + } + const names = labelsAsOf(onePage(g.labels, `labels of #${g.number}`).map((label) => label?.name), eventsOf(g.number), atMs); + for (const [issue, labels] of linked) names.push(...labelsAsOf(labels, eventsOf(issue), atMs)); + const cls = workClass(names); + return { cls, reason: cls ? null : "no work-class label at merge" }; + }; + + // Autonomy as babysit_merge_policy.py derive_auto_merge reads it: the latest + // AddedToMergeQueueEvent's actor and enqueuer, or merged_by when the merge bypassed the + // queue; any of them null or not human makes the merge autonomous. + const autonomousOf = (g, timeline) => { + const queued = timeline.filter((e) => e.__typename === "AddedToMergeQueueEvent").sort((a, b) => ms(a.createdAt, "a merge-queue event") - ms(b.createdAt, "a merge-queue event")); + const actors = queued.length > 0 ? [queued.at(-1).actor, queued.at(-1).enqueuer] : [g.mergedBy]; + const byIdentity = actors.some((actor) => actor && lane.has(actorKey(actor))); + return { autonomous: actors.some((actor) => !human(actor)) || lane.has(actorKey(g.mergedBy)), byIdentity }; + }; + + const details = new Map(); + const record = (number) => { + if (!details.has(number)) { + const g = gh("graphql", { query: PULL_QUERY, variables: { owner, name, number } })?.data?.repository?.pullRequest; + if (!g?.mergedAt) throw new UnresolvableError(`#${number} is not a merged pull request`); + ms(g.mergedAt, `#${number} mergedAt`); + const base = { number, mergedAt: g.mergedAt, sha: g.mergeCommit?.oid, head: g.headRefOid, reverted: false }; + // A page a lane can make incomplete or unreadable leaves the merge unclassed-autonomous. + try { + const timeline = onePage(g.timelineItems, `timeline of #${number}`); + details.set(number, { ...base, ...classAtMerge(g, timeline), ...autonomousOf(g, timeline) }); + } catch (error) { + if (!(error instanceof IncompleteError)) throw error; + // Who enqueued it is unknown too, so it also owes a pin. + details.set(number, { ...base, cls: null, reason: "a linked-issue, label or timeline page is unreadable or incomplete: class unknown", autonomous: true, byIdentity: true }); + } + } + return details.get(number); + }; + // Tracked: C2/C3 merges, and autonomous merges with no class, which are never counted toward + // a positive term but whose contrary events go against every cell. + const tracked = merged.map((pr) => record(pr.number)).filter((r) => ["C2", "C3"].includes(r.cls) || (r.autonomous && r.cls === null)); + const cellsOf = (r) => (r.cls ? [`${r.cls}-auto-merge`] : CELLS); + const bySha = new Map(tracked.map((r) => [r.sha, r])); + + // Reverts, from merged PR bodies and default-branch commit messages; earliest per target. + const own = repo.toLowerCase(); + const revertsOf = (text) => { + const targets = []; + for (const m of (text ?? "").matchAll(/\bReverts\s+(?:([\w.-]+\/[\w.-]+))?#(\d+)/g)) { + // A GraphQL Int is at most 2^31 - 1; a longer digit string is never parsed into a Number. + if (!m[1] || m[1].toLowerCase() === own) { + const valid = /^[1-9]\d{0,9}$/.test(m[2]) && BigInt(m[2]) <= 2147483647n; + targets.push(valid ? { number: Number(m[2]) } : { invalid: m[2] }); + } + } + for (const m of (text ?? "").matchAll(/This reverts commit ([0-9a-f]{40})/g)) targets.push({ sha: m[1] }); + return targets; + }; + const reverts = []; + const unresolvedReverts = []; + const gateFailures = []; + // Every revert counts, whoever merged or enqueued it (owner decision, #4588). + for (const pr of merged) { + const targets = revertsOf(pr.body); + for (const target of targets) reverts.push({ ...target, at: pr.merged_at }); + if (targets.length === 0 && /^revert\b/i.test(pr.title ?? "")) unresolvedReverts.push({ number: pr.number, title: pr.title }); + } + const commits = pages(gh, `repos/${repo}/commits?since=${new Date(sinceMs).toISOString()}&until=${new Date(throughMs).toISOString()}`); + for (const commit of commits) { + for (const target of revertsOf(commit.commit?.message)) reverts.push({ ...target, at: commit.commit?.committer?.date }); + } + const firstRevert = new Map(); + for (const revert of reverts) { + if (!inWindow(revert.at)) continue; + if (revert.invalid !== undefined) { + unresolvedReverts.push({ target: revert.invalid, reason: "not a pull request number (a positive GraphQL Int)" }); + continue; + } + let number = revert.number; + if (number === undefined) { + const known = bySha.get(revert.sha); + try { + number = known ? known.number : gh(`repos/${repo}/commits/${revert.sha}/pulls`).find((pr) => pr.merged_at)?.number; + } catch (error) { + if (!(error instanceof UnresolvableError)) throw error; + unresolvedReverts.push({ target: revert.sha, reason: "no commit found for this SHA" }); + continue; + } + if (number === undefined) { + unresolvedReverts.push({ target: revert.sha, reason: "no merged pull request for this commit" }); + continue; + } + } + if (!firstRevert.has(number) || Date.parse(revert.at) < Date.parse(firstRevert.get(number))) firstRevert.set(number, revert.at); + } + for (const [number, at] of firstRevert) { + let target; + try { + target = record(number); + } catch (error) { + if (!(error instanceof UnresolvableError)) throw error; + unresolvedReverts.push({ target: number, reason: "not a merged pull request" }); + continue; + } + if (target.autonomous && (["C2", "C3"].includes(target.cls) || target.cls === null)) { + target.reverted = true; + for (const cell of cellsOf(target)) addEvent(cell, "reverted-merge", at); + } + } + + const judgeWorkflows = postMergeRuns(gh, repo, branch, tracked.filter((c) => c.autonomous), { lane, throughMs }); + + // check()'s read-back for a merge a lane identity enqueued. + const readBack = (r) => { + const deadline = ms(r.mergedAt, `#${r.number} mergedAt`) + DISPATCH_DEADLINE_MS; + const pinned = readPin(gh, owner, name, r.number, lane); + if (pinned === null) { + return throughMs > deadline + ? { failure: { check: "pin read-back", conclusion: "no unedited pin comment by a lane identity", at: isoSeconds(deadline) } } + : { pending: true }; + } + return pinned === r.head ? {} : { failure: { check: "pin read-back", conclusion: `headRefOid ${r.head} is not the pinned ${pinned}`, at: r.mergedAt } }; + }; + + // Post-merge gate on each autonomous merge commit: the --gate-check runs and status contexts. + // A run that did not finish green or red (cancelled, action_required, stale) fails the gate + // unless a later run of the same check succeeded, so cancelling a red run cannot erase it. + // Then the post-merge workflow runs on later SHAs containing the merge (postMergeRuns). + for (const r of tracked.filter((c) => c.autonomous)) { + const wf = judgeWorkflows(r); + const pin = r.byIdentity ? readBack(r) : {}; + if (pin.failure) wf.failures.push(pin.failure); + if (pin.pending) wf.unsettled = true; + for (const failure of wf.failures) { + for (const cell of cellsOf(r)) addEvent(cell, "gate-failure", failure.at); + gateFailures.push({ number: r.number, ...failure }); + } + const runs = pages(gh, `repos/${repo}/commits/${r.sha}/check-runs?filter=all`, "check_runs").filter((run) => gate.has(run.name)); + const statuses = pages(gh, `repos/${repo}/commits/${r.sha}/statuses`).filter((status) => gate.has(status.context)); + const outcome = (run) => { + if (run.status !== "completed") return "unsettled"; + if (GATE_FAILED.has(run.conclusion)) return "failed"; + if (GATE_OK.has(run.conclusion)) return "ok"; + const rerunGreen = runs.some( + (o) => o !== run && o.name === run.name && o.status === "completed" && o.conclusion === "success" && Date.parse(o.completed_at) > Date.parse(run.completed_at ?? 0), + ); + return rerunGreen ? "ok" : "failed"; + }; + const failedRuns = runs.filter((run) => outcome(run) === "failed"); + const failedStatuses = statuses.filter((status) => STATUS_FAILED.has(status.state)); + for (const cell of cellsOf(r)) { + for (const run of failedRuns) addEvent(cell, "gate-failure", run.completed_at ?? r.mergedAt); + for (const status of failedStatuses) addEvent(cell, "gate-failure", status.created_at); + } + for (const run of failedRuns) gateFailures.push({ number: r.number, check: run.name, conclusion: run.conclusion, at: run.completed_at ?? r.mergedAt }); + for (const status of failedStatuses) gateFailures.push({ number: r.number, check: status.context, conclusion: status.state, at: status.created_at }); + const latest = new Map(); + for (const status of [...statuses].sort((a, b) => Date.parse(b.created_at) - Date.parse(a.created_at))) { + if (!latest.has(status.context)) latest.set(status.context, status.state); + } + // Passed only where check() would call the merge verified, with nothing pending. + const settled = + wf.verified && + !wf.unsettled && + runs.every((run) => outcome(run) === "ok") && + [...latest.values()].every((state) => state === "success"); + r.gate = failedRuns.length + failedStatuses.length + wf.failures.length > 0 ? "failed" : settled ? "passed" : "unsettled"; + } + + // Human-confirmed missed-blocking AI-review findings, from the label timeline. + for (const r of tracked.filter((c) => c.cls === "C3" || c.autonomous)) { + const confirmed = eventsOf(r.number).filter( + (e) => e.event === "labeled" && e.label?.name === missedBlockingLabel && e.actor?.type === "User" && !lane.has(loginKey(e.actor.login)) && inWindow(e.created_at), + ); + r.missedBlocking = confirmed.length > 0 ? 1 : 0; + const cells = r.cls === "C3" ? ["C3-ai-review-blocking"] : r.cls === "C2" ? ["C2-auto-merge"] : CELLS; + for (const e of confirmed) for (const cell of cells) addEvent(cell, "verification-divergence", e.created_at); + } + const reviewers = new Set(aiReviewers); + if (reviewers.size > 0) { + for (const r of tracked.filter((c) => c.cls === "C3")) { + r.aiReviewed = pages(gh, `repos/${repo}/pulls/${r.number}/reviews`).some((review) => reviewers.has(review.user?.login)); + } + } + + const sorted = [...events.values()].sort((a, b) => Date.parse(a.at) - Date.parse(b.at) || eventKey(a).localeCompare(eventKey(b))); + return { + repo, + since, + through: new Date(throughMs).toISOString(), + events: sorted, + unresolvedReverts, + gateFailures, + unclassedAutonomous: [...details.values()] + .filter((r) => r.autonomous && r.cls === null && (tracked.includes(r) || r.reverted)) + .map((r) => ({ number: r.number, reason: r.reason })), + cells: report(tracked, sorted, reviewers.size > 0), + }; +} + +function term(name, value, threshold, pass, detail) { + return { term: name, value, threshold, pass, ...(detail ? { detail } : {}) }; +} + +function windowOf(records) { + if (records.length === 0) return { from: null, to: null, days: 0 }; + const times = records.map((r) => Date.parse(r.mergedAt)); + const [from, to] = [Math.min(...times), Math.max(...times)]; + return { from: new Date(from).toISOString(), to: new Date(to).toISOString(), days: Math.floor(((to - from) / DAY_MS) * 100) / 100 }; +} + +function gateTerm(name, records) { + const detail = { passed: 0, failed: 0, unsettled: 0, total: records.length }; + for (const r of records) detail[r.gate] += 1; + const value = records.length === 0 ? null : Math.floor((detail.passed / records.length) * 10000) / 100; + return term(name, value, "100%", records.length > 0 && detail.passed === records.length, detail); +} + +function cell(window, terms) { + return { window, pass: terms.every((t) => t.pass), terms }; +} + +// Thresholds are work-classes.md's suggested defaults. +function report(counted, events, reviewsMeasured) { + const c2 = counted.filter((r) => r.cls === "C2" && r.autonomous); + const c3 = counted.filter((r) => r.cls === "C3" && r.autonomous); + const c3All = counted.filter((r) => r.cls === "C3"); + const c2Window = windowOf(c2); + const reverted = (records) => records.filter((r) => r.reverted).length; + const missed = (records) => records.reduce((sum, r) => sum + (r.missedBlocking ?? 0), 0); + const c2Demotions = events.filter((e) => e.cell === "C2-auto-merge").length; + const reviewed = c3All.filter((r) => r.aiReviewed); + return { + "C2-auto-merge": cell(c2Window, [ + term("autonomous C2 completions", c2.length, ">= 20", c2.length >= 20), + term("window days", c2Window.days, ">= 14", c2Window.days >= 14), + gateTerm("deterministic-gate pass", c2), + term("reverted merges", reverted(c2), "0", reverted(c2) === 0), + ]), + "C3-auto-merge": cell(windowOf([...c2, ...c3]), [ + term("autonomous C2 merges", c2.length, ">= 20", c2.length >= 20), + term("C2 window days", c2Window.days, ">= 14", c2Window.days >= 14), + term("C2 demotion events", c2Demotions, "0", c2Demotions === 0), + term("autonomous C3 completions", c3.length, ">= 10", c3.length >= 10), + gateTerm("C3 deterministic-gate pass", c3), + term("C3 reverted merges", reverted(c3), "0", reverted(c3) === 0), + term("C3 human-confirmed missed-blocking findings", missed(c3), "0", missed(c3) === 0), + ]), + "C3-ai-review-blocking": cell(windowOf(c3All), [ + reviewsMeasured + ? term("advisory AI reviews on C3", reviewed.length, ">= 30", reviewed.length >= 30) + : term("advisory AI reviews on C3", null, ">= 30", false, { unmeasured: "no --ai-reviewer given" }), + term("human-confirmed missed-blocking findings", missed(c3All), "0", missed(c3All) === 0), + ]), + }; +} + +function sha256(text) { + return createHash("sha256").update(text).digest("hex"); +} + +// "wx" refuses a temp file that already exists rather than overwrite or remove it. +function writeTemp(path, text) { + const temp = `${path}.${process.pid}.tmp`; + writeFileSync(temp, text, { flag: "wx" }); + return temp; +} + +function readIfPresent(path) { + try { + return readFileSync(path, "utf8"); + } catch (error) { + if (error.code === "ENOENT") return null; + throw new PublishError(`cannot read ${path}: ${error.message}`); + } +} + +export function publish(outDir, events, through, now = new Date(), { rename = renameSync } = {}) { + const file = join(outDir, "evidence.json"); + const metaFile = join(outDir, "evidence.meta.json"); + const previous = readIfPresent(file); + if (previous !== null) { + let prior; + try { + prior = JSON.parse(previous); + } catch (error) { + throw new PublishError(`refusing to publish: ${file} is not JSON (${error.message}), so the superset check cannot run`); + } + if (!Array.isArray(prior)) throw new PublishError(`refusing to publish: ${file} is not a JSON array, so the superset check cannot run`); + let priorMeta; + try { + priorMeta = JSON.parse(readIfPresent(metaFile)); + } catch { + priorMeta = null; + } + if (priorMeta?.sha256 !== sha256(previous)) { + throw new PublishError(`refusing to publish: ${file} does not match the sha256 in its sidecar ${metaFile} (missing, unreadable or different)`); + } + const keys = new Set(events.map(eventKey)); + const dropped = prior.filter((event) => !keys.has(eventKey(event))); + if (dropped.length > 0) { + throw new PublishError(`refusing to publish: the new events drop ${dropped.length} previously published event(s): ${dropped.map(eventKey).join(", ")}`); + } + } + const text = `${JSON.stringify(events, null, 2)}\n`; + const meta = { generated_at: now.toISOString(), through, sha256: sha256(text), prev_sha256: previous === null ? null : sha256(previous) }; + const eventsTemp = writeTemp(file, text); + let metaTemp; + let backup; + try { + metaTemp = writeTemp(metaFile, `${JSON.stringify(meta, null, 2)}\n`); + if (previous !== null) backup = writeTemp(`${file}.prev`, previous); + rename(eventsTemp, file); + } catch (error) { + for (const temp of [eventsTemp, metaTemp, backup]) if (temp) rmSync(temp, { force: true }); + throw error; + } + try { + rename(metaTemp, metaFile); + } catch (error) { + // Put the pair back as it was. An interruption before this point still leaves a pair the + // next run refuses on the sha256 check: fail closed. + if (backup) rename(backup, file); + else rmSync(file, { force: true }); + rmSync(metaTemp, { force: true }); + throw error; + } + if (backup) rmSync(backup, { force: true }); + return meta; +} + +function main() { + let values; + try { + ({ values } = parseArgs({ + options: { + repo: { type: "string" }, + since: { type: "string" }, + through: { type: "string" }, + "lane-login": { type: "string", multiple: true }, + "gate-check": { type: "string", multiple: true }, + "ai-reviewer": { type: "string", multiple: true }, + "missed-blocking-label": { type: "string", default: "ai-review: missed-blocking" }, + out: { type: "string" }, + }, + })); + if (!/^[\w.-]+\/[\w.-]+$/.test(values.repo ?? "") || !values.since || !values["lane-login"]?.length || !values["gate-check"]?.length) { + throw new PublishError("--repo , --since , at least one --lane-login and at least one --gate-check are required"); + } + } catch (error) { + process.stderr.write(`usage: ${error.message}\n`); + process.exit(2); + } + try { + const result = collect({ + repo: values.repo, + since: values.since, + through: values.through ?? new Date().toISOString(), + laneLogins: values["lane-login"], + gateChecks: values["gate-check"], + aiReviewers: values["ai-reviewer"] ?? [], + missedBlockingLabel: values["missed-blocking-label"], + }); + if (values.out) publish(values.out, result.events, result.through); + process.stdout.write(`${JSON.stringify(result, null, 2)}\n`); + } catch (error) { + process.stderr.write(`publish-promotion-evidence: ${error.message}\n`); + process.exit(error instanceof PublishError && error.message.startsWith("refusing") ? 1 : 2); + } +} + +if (process.argv[1] && fileURLToPath(import.meta.url) === process.argv[1]) main(); diff --git a/plugins/source-control/skills/babysit-loop/scripts/publish-promotion-evidence.test.mjs b/plugins/source-control/skills/babysit-loop/scripts/publish-promotion-evidence.test.mjs new file mode 100755 index 0000000000..763aa3a5c4 --- /dev/null +++ b/plugins/source-control/skills/babysit-loop/scripts/publish-promotion-evidence.test.mjs @@ -0,0 +1,815 @@ +#!/usr/bin/env node +import assert from "node:assert/strict"; +import { spawnSync } from "node:child_process"; +import { createHash } from "node:crypto"; +import { mkdtempSync, readdirSync, readFileSync, renameSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { dirname, join } from "node:path"; +import { after, test } from "node:test"; +import { fileURLToPath } from "node:url"; +import * as publisher from "./publish-promotion-evidence.mjs"; +import { collect, PublishError, publish } from "./publish-promotion-evidence.mjs"; + +const HERE = dirname(fileURLToPath(import.meta.url)); +// Real `gh api` responses, recorded 2026-10-12 and 2026-10-13 and trimmed to the fields the +// publisher reads: from this repository (PRs #7352 and #7063, a merge commit, a check run, a +// status, an issue event, a review, a comment page, workflow runs, a compare, NOT_FOUND and 422 +// answers) and one ConnectedEvent from cli/cli #14616. The comment page's body is trimmed to its +// first line. Each case copies a record and varies number, labels, merger and timestamps. +const REC = JSON.parse(readFileSync(join(HERE, "fixtures", "publish-promotion-evidence", "recorded.json"), "utf8")); +const REPO = "melodic-software/claude-code-plugins"; +const LANE = "lane-bot[bot]"; +const SINCE = "2026-08-01T00:00:00Z"; +const THROUGH = "2026-10-05T00:00:00Z"; +const MECHANICAL = "work-class: mechanical"; +const SCOPED = "work-class: scoped"; +const MISSED = "ai-review: missed-blocking"; + +const base = mkdtempSync(join(tmpdir(), "publish-promotion-evidence-")); +after(() => rmSync(base, { recursive: true, force: true })); + +const shaOf = (number) => number.toString(16).padStart(40, "0"); +const later = (at, minutes) => new Date(Date.parse(at) + minutes * 60_000).toISOString().replace(".000Z", "Z"); + +// GraphQL spells an App as `__typename: Bot` with the bare slug; REST adds `[bot]`. +const actorOf = (login) => (login === null ? null : login.endsWith("[bot]") ? { __typename: "Bot", login: login.slice(0, -5) } : { __typename: "User", login }); +const page1 = (nodes, hasNextPage = false) => ({ pageInfo: { hasNextPage }, nodes }); +// REC.pullGraph: a recorded pull whose one closing reference, number 6007, is labeled `work-class: scoped`. +const ISSUE_6007 = REC.pullGraph.data.repository.pullRequest.closingIssuesReferences.nodes; +const queued = (at, enqueuer, actor = enqueuer) => ({ __typename: "AddedToMergeQueueEvent", createdAt: at, actor: actorOf(actor), enqueuer: actorOf(enqueuer) }); +// The shape of a real ConnectedEvent on a PR timeline (cli/cli #14616): `subject` is the issue. +const LINK = REC.connectedEventFromCliCli14616; +const link = (type, issue, at) => ({ ...LINK, __typename: type, createdAt: at, subject: { ...LINK.subject, number: issue } }); + +function pr( + number, + mergedAt, + { labels = [], merger = LANE, title = REC.pullListItem.title, body = REC.pullListItem.body, linked = [], timeline = [], lastEditedAt = null, patch = {}, base = "main", head = shaOf(number + 0x100000) } = {}, +) { + const names = labels.map((name) => ({ name })); + return { + graph: { + ...REC.pullGraph.data.repository.pullRequest, + number, + headRefOid: head, + mergedAt, + lastEditedAt, + mergeCommit: { oid: shaOf(number) }, + mergedBy: actorOf(merger), + labels: page1(names), + closingIssuesReferences: page1(linked), + timelineItems: page1(timeline), + ...patch, + }, + list: { + ...REC.pullListItem, + base: { ...REC.pullListItem.base, ref: base }, + number, + title, + body, + labels: names, + created_at: later(mergedAt, -60), + merged_at: mergedAt, + updated_at: later(mergedAt, 1), + merge_commit_sha: shaOf(number), + }, + }; +} +const labelEvent = (event, name, at, login = "kyle-sexton", type = "User") => ({ ...REC.issueEvent, event, label: { name }, actor: { login, type }, created_at: at }); + +// A recorded GitHub: answers only the GET paths the publisher is expected to call, with the +// documented semantics of each (list sort order, check-runs `filter=latest` by default, statuses +// newest first), and throws on any other, so an unexpected or mutating call fails the case. +// A run marked `superseded` was re-run: only `filter=all` returns it. +// `bumpOnPage2` updates that PR (a comment, a label) between the listing's first and second page. +// `workflowRuns`: {: [run]}; `attempts`: {: {: run}}; `history`: the +// default branch's commits oldest first (linear, as a merge-queue branch is), defaulting to the +// merge commits in merge order. +// `diverged`: head SHAs a compare reports as diverged. A PR number not in `pulls` answers with the +// recorded NOT_FOUND response. +// The lane posts a pin comment on every PR it enqueues and dispatches a full run after every merge +// (babysit_merge.py --unattended, babysit_postmerge.py), so each PR gets a correct pin and a green +// lane dispatch run on its merge commit an hour after merging, unless `comments` sets that PR's +// comments or `noDefaultDispatch` is set; an explicit run with the same id replaces the default. +const pinComment = (sha) => ({ author: actorOf(LANE), body: `\nThe unattended merge path enqueued this pull request at head \`${sha}\`.`, lastEditedAt: null }); +function github({ pulls = [], commits = [], checkRuns = {}, statuses = {}, issueEvents = {}, reviews = {}, issues = {}, workflowRuns: explicitRuns = {}, attempts = {}, history, diverged = [], comments = {}, noDefaultDispatch = false, bumpOnPage2 }) { + const defaultDispatch = noDefaultDispatch ? [] : pulls.map((p) => wfRun(900000 + p.graph.number, "workflow_dispatch", p.graph.mergeCommit.oid, later(p.graph.mergedAt, 60), { actor: LANE })); + const explicitIds = new Set(Object.values(explicitRuns).flat().map((r) => r.id)); + const workflowRuns = { ...explicitRuns, [FULL]: [...defaultDispatch.filter((r) => !explicitIds.has(r.id)), ...(explicitRuns[FULL] ?? [])] }; + const branch = history ?? [...pulls].sort((a, b) => a.graph.mergedAt.localeCompare(b.graph.mergedAt)).map((p) => p.graph.mergeCommit.oid); + const page = (items, path) => { + const n = Number(/[?&]page=(\d+)/.exec(path)[1]); + return items.slice((n - 1) * 100, n * 100); + }; + const runsFor = (number) => checkRuns[number] ?? [{ ...REC.checkRun, completed_at: later(pulls.find((p) => p.graph.number === number).graph.mergedAt, 5) }]; + return (path, graphql) => { + if (path === "graphql") { + assert.deepEqual([graphql.variables.owner, graphql.variables.name], REPO.split("/")); + assert.match(graphql.query, /^query\(/); + assert.doesNotMatch(graphql.query, /mutation/); + if (/comments\(/.test(graphql.query)) { + const pull = pulls.find((p) => p.graph.number === graphql.variables.number); + const all = comments[graphql.variables.number] ?? [pinComment(pull.graph.headRefOid)]; + const n = Number(graphql.variables.c ?? 0); + const nodes = all.slice(n * 100, (n + 1) * 100); + const more = all.length > (n + 1) * 100; + return { data: { repository: { pullRequest: { comments: { ...REC.commentsPage.data.repository.pullRequest.comments, pageInfo: { hasNextPage: more, endCursor: more ? String(n + 1) : null }, nodes } } } } }; + } + const found = pulls.find((p) => p.graph.number === graphql.variables.number); + return found ? { data: { repository: { pullRequest: found.graph } } } : REC.pullNotFound; + } + if (path === `repos/${REPO}`) return REC.repo; + const prefix = `repos/${REPO}/`; + assert.ok(path.startsWith(prefix), path); + const rest = path.slice(prefix.length); + let m; + if ((m = /^pulls\?state=closed(?:&base=([\w.-]+))?&sort=(created|updated)&direction=(asc|desc)&/.exec(rest))) { + if (bumpOnPage2 && /[?&]page=2\b/.test(rest)) pulls.find((p) => p.list.number === bumpOnPage2).list.updated_at = "2026-10-04T00:00:00Z"; + const key = m[2] === "created" ? "created_at" : "updated_at"; + const sign = m[3] === "asc" ? 1 : -1; + const listed = pulls.map((p) => p.list).filter((p) => !m[1] || p.base.ref === m[1]); + return page(listed.sort((a, b) => sign * a[key].localeCompare(b[key])), rest); + } + if ((m = /^issues\/(\d+)$/.exec(rest))) return issues[m[1]]; + if ((m = /^pulls\/(\d+)\/reviews\?/.exec(rest))) return page(reviews[m[1]] ?? [], rest); + if (/^commits\?since=[^&]+&until=[^&]+&/.test(rest)) return page(commits, rest); + if ((m = /^commits\/([0-9a-f]{40})\/check-runs\?/.exec(rest))) { + const runs = runsFor(parseInt(m[1], 16)).filter((r) => /[?&]filter=all\b/.test(rest) || !r.superseded); + return { total_count: runs.length, check_runs: page(runs, rest) }; + } + if ((m = /^commits\/([0-9a-f]{40})\/pulls$/.exec(rest))) { + // A SHA no commit has: the recorded 422, as realGh hands it to ghFailure. + if (!branch.includes(m[1])) return publisher.ghFailure(path, undefined, REC.commitNotFound.stdout, REC.commitNotFound.stderr); + return pulls.filter((p) => p.graph.mergeCommit.oid === m[1]).map((p) => ({ number: p.graph.number, merged_at: p.graph.mergedAt, merge_commit_sha: m[1] })); + } + if ((m = /^commits\/([0-9a-f]{40})\/statuses\?/.exec(rest))) return page(statuses[parseInt(m[1], 16)] ?? [], rest); + if ((m = /^issues\/(\d+)\/events\?/.exec(rest))) return page(issueEvents[m[1]] ?? [], rest); + if ((m = /^actions\/workflows\/([\w.-]+)\/runs\?event=(\w+)&branch=main&created=%3E%3D[^&]+&/.exec(rest))) { + const runs = (workflowRuns[m[1]] ?? []).filter((r) => r.event === m[2]); + return { total_count: runs.length, workflow_runs: page(runs, rest) }; + } + if ((m = /^actions\/runs\/(\d+)\/attempts\/(\d+)$/.exec(rest))) return attempts[m[1]][m[2]]; + if ((m = /^compare\/([0-9a-f]{40})\.\.\.([0-9a-f]{40})\?/.exec(rest))) { + const [base, head] = [branch.indexOf(m[1]), branch.indexOf(m[2])]; + assert.ok(base >= 0 && head >= 0, `compare outside the recorded history: ${rest}`); + const status = diverged.includes(m[2]) ? "diverged" : head > base ? "ahead" : head === base ? "identical" : "behind"; + const ahead = head > base ? branch.slice(base + 1, head + 1).map((sha) => ({ sha })) : []; + return { ...REC.compare, status, ahead_by: ahead.length, total_commits: ahead.length, commits: page(ahead, rest) }; + } + throw new Error(`unrecorded gh api call: ${path}`); + }; +} + +// The recorded check run and status name the post-merge gate in every case. +const GATE = [REC.checkRun.name, REC.status.context]; + +function run(fixture, extra = {}) { + return collect({ repo: REPO, since: SINCE, through: THROUGH, laneLogins: [LANE], gateChecks: GATE, missedBlockingLabel: MISSED, gh: github(fixture), ...extra }); +} + +const termOf = (result, cell, name) => result.cells[cell].terms.find((t) => t.term === name); + +const FULL = "pr-require-checks.yml"; +const CANARY = "post-merge-detect-silent-revert.yml"; +const wfRun = (id, event, sha, created, { conclusion = "success", status = "completed", attempt = 1, actor = "github-merge-queue[bot]", triggering = actor, updated = later(created, 8) } = {}) => ({ + ...REC.workflowRun, + id, + event, + head_sha: sha, + created_at: created, + updated_at: updated, + status, + conclusion: status === "completed" ? conclusion : null, + run_attempt: attempt, + actor: { login: actor }, + triggering_actor: { login: triggering }, +}); +const gateEvents = (at, ...cells) => cells.map((cell) => ({ cell, event: "gate-failure", at })); +// A green lane dispatch run and a green canary run on `tip`: check() calls each merge they contain +// verified. +// Each merge gets them on its own merge commit `minutes` after merging (within the 6-hour dispatch +// deadline by default). +const verifiedEach = (pulls, minutes = 60) => ({ + [FULL]: pulls.map((p) => wfRun(900000 + p.graph.number, "workflow_dispatch", p.graph.mergeCommit.oid, later(p.graph.mergedAt, minutes), { actor: LANE })), + [CANARY]: pulls.map((p) => wfRun(800000 + p.graph.number, "push", p.graph.mergeCommit.oid, later(p.graph.mergedAt, minutes))), +}); + +test("a red full run on a later SHA containing two merges yields gate-failure for each", () => { + const tip = shaOf(0xa001); + const result = run({ + pulls: [pr(1301, "2026-09-01T00:00:00Z", { labels: [MECHANICAL] }), pr(1302, "2026-09-02T00:00:00Z", { labels: [SCOPED] })], + history: [shaOf(1301), shaOf(1302), tip], + workflowRuns: { [FULL]: [wfRun(1, "schedule", tip, "2026-09-03T00:00:00Z", { conclusion: "failure" })] }, + }); + assert.deepEqual(result.events, gateEvents("2026-09-03T00:08:00Z", "C2-auto-merge", "C3-auto-merge")); + assert.deepEqual( + result.gateFailures.map((g) => [g.number, g.check, g.conclusion]), + [ + [1301, `${FULL} schedule`, "failure"], + [1302, `${FULL} schedule`, "failure"], + ], + ); +}); + +test("a run created before the merge is not its evidence", () => { + // Run 3 on #1303's own SHA predates #1303's merge, so it is evidence for #1309 only. + const result = run({ + pulls: [pr(1309, "2026-09-01T00:00:00Z", { labels: [MECHANICAL] }), pr(1303, "2026-09-02T00:00:00Z", { labels: [SCOPED] })], + workflowRuns: { [FULL]: [wfRun(3, "schedule", shaOf(1303), "2026-09-01T12:00:00Z", { conclusion: "failure" })] }, + }); + assert.deepEqual(result.events, gateEvents("2026-09-01T12:08:00Z", "C2-auto-merge")); +}); + +test("a post-merge run behind the merge is not its evidence; a diverged one refuses the publish", () => { + const other = shaOf(0xa002); + const red = { [FULL]: [wfRun(2, "schedule", other, "2026-09-03T00:00:00Z", { conclusion: "failure" })] }; + const behind = run({ pulls: [pr(1310, "2026-09-02T00:00:00Z", { labels: [MECHANICAL] })], history: [other, shaOf(1310)], workflowRuns: red }); + assert.deepEqual(behind.events, []); + const diverged = { pulls: [pr(1310, "2026-09-02T00:00:00Z", { labels: [MECHANICAL] })], history: [shaOf(1310), other], diverged: [other] }; + assert.throws(() => run({ ...diverged, workflowRuns: red }), (error) => error instanceof PublishError && /diverged/.test(error.message)); +}); + +test("a merge that is an ancestor of a same-time merge listed before it is still covered by a later run", () => { + const tip = shaOf(0xa007); + const result = run({ + pulls: [pr(1401, "2026-09-01T00:00:00Z", { labels: [MECHANICAL] }), pr(1402, "2026-09-01T00:00:00Z", { labels: [SCOPED] })], + history: [shaOf(1402), shaOf(1401), tip], + workflowRuns: { [FULL]: [wfRun(11, "schedule", tip, "2026-09-02T00:00:00Z", { conclusion: "failure" })] }, + }); + assert.deepEqual(result.events, gateEvents("2026-09-02T00:08:00Z", "C2-auto-merge", "C3-auto-merge")); +}); + +test("a post-merge run created after --through still counts", () => { + const result = run({ + pulls: [pr(1403, "2026-10-04T00:00:00Z", { labels: [MECHANICAL] })], + workflowRuns: { [CANARY]: [wfRun(12, "push", shaOf(1403), "2026-10-06T00:00:00Z", { conclusion: "failure" })] }, + }); + assert.deepEqual(result.events, gateEvents("2026-10-06T00:08:00Z", "C2-auto-merge")); +}); + +const gateOf = (result, cell = "C2-auto-merge") => termOf(result, cell, "deterministic-gate pass").detail; +const PASSED = { passed: 1, failed: 0, unsettled: 0, total: 1 }; +const FAILED = { passed: 0, failed: 1, unsettled: 0, total: 1 }; +const UNSETTLED = { passed: 0, failed: 0, unsettled: 1, total: 1 }; + +test("a merge passes the gate only with a lane dispatch within 6 hours and a green canary; no timely dispatch is a gate failure", () => { + const pulls = () => [pr(1404, "2026-09-01T00:00:00Z", { labels: [MECHANICAL] })]; + const noDispatch = gateEvents("2026-09-01T06:00:00Z", "C2-auto-merge"); + const bare = run({ pulls: pulls(), noDefaultDispatch: true }); + assert.deepEqual([gateOf(bare), bare.events], [FAILED, noDispatch]); + assert.deepEqual(bare.gateFailures, [{ number: 1404, check: "dispatch deadline", conclusion: "no dispatch run by a lane identity within 6 h", at: "2026-09-01T06:00:00Z" }]); + const dispatchOnly = run({ pulls: pulls(), workflowRuns: { [FULL]: verifiedEach(pulls())[FULL] } }); + assert.deepEqual([gateOf(dispatchOnly), dispatchOnly.events], [UNSETTLED, []]); + const canaryOnly = run({ pulls: pulls(), noDefaultDispatch: true, workflowRuns: { [CANARY]: verifiedEach(pulls())[CANARY] } }); + assert.deepEqual([gateOf(canaryOnly), canaryOnly.events], [FAILED, noDispatch]); + assert.deepEqual(gateOf(run({ pulls: pulls(), workflowRuns: verifiedEach(pulls()) })), PASSED); + const late = run({ pulls: pulls(), workflowRuns: verifiedEach(pulls(), 7 * 60) }); + assert.deepEqual([gateOf(late), late.events], [FAILED, noDispatch]); +}); + +test("a lane-enqueued merge passes only when its unedited lane pin names headRefOid; otherwise a gate failure", () => { + const pulls = () => [pr(1501, "2026-09-01T00:00:00Z", { labels: [MECHANICAL] })]; + const head = shaOf(1501 + 0x100000); + const withComments = (list) => run({ pulls: pulls(), workflowRuns: verifiedEach(pulls()), comments: { 1501: list } }); + assert.deepEqual(gateOf(withComments([pinComment(head)])), PASSED); + const filler = Array.from({ length: 120 }, () => REC.commentsPage.data.repository.pullRequest.comments.nodes[0]); + assert.deepEqual(gateOf(withComments([...filler, pinComment(head)])), PASSED); + const noPin = gateEvents("2026-09-01T06:00:00Z", "C2-auto-merge"); + for (const list of [[], [{ ...pinComment(head), lastEditedAt: "2026-09-01T00:30:00Z" }], [{ ...pinComment(head), author: actorOf("kyle-sexton") }]]) { + const result = withComments(list); + assert.deepEqual([gateOf(result), result.events], [FAILED, noPin]); + } + const mismatch = withComments([pinComment(shaOf(0xbad))]); + assert.deepEqual([gateOf(mismatch), mismatch.events], [FAILED, gateEvents("2026-09-01T00:00:00Z", "C2-auto-merge")]); + // Before the deadline a missing pin is pending, as in check(). + const fresh = [pr(1502, "2026-10-04T22:00:00Z", { labels: [MECHANICAL] })]; + const pending = run({ pulls: fresh, workflowRuns: verifiedEach(fresh, 30), comments: { 1502: [] } }); + assert.deepEqual([gateOf(pending), pending.events], [UNSETTLED, []]); + // A merge another bot enqueued is autonomous but not the lane's: it has no pin to read back. + const bot = [pr(1503, "2026-09-01T00:00:00Z", { labels: [MECHANICAL], merger: "kyle-sexton", timeline: [queued("2026-08-31T23:00:00Z", null, "some-app[bot]")] })]; + assert.deepEqual(gateOf(run({ pulls: bot, workflowRuns: verifiedEach(bot), comments: { 1503: [] } })), PASSED); +}); + +test("a pin past the 30th comment page is no pin, as in read_pin", () => { + const pulls = () => [pr(1504, "2026-09-01T00:00:00Z", { labels: [MECHANICAL] })]; + const filler = (count) => Array.from({ length: count }, () => REC.commentsPage.data.repository.pullRequest.comments.nodes[0]); + const withComments = (list) => run({ pulls: pulls(), workflowRuns: verifiedEach(pulls()), comments: { 1504: list } }); + const head = shaOf(1504 + 0x100000); + assert.deepEqual(gateOf(withComments([...filler(2999), pinComment(head)])), PASSED); + const beyond = withComments([...filler(3000), pinComment(head)]); + assert.deepEqual([gateOf(beyond), beyond.events], [FAILED, gateEvents("2026-09-01T06:00:00Z", "C2-auto-merge")]); +}); + +test("a revert number that is not a GraphQL Int is listed without a lookup", () => { + const result = run({ + pulls: [ + pr(1601, "2026-09-01T00:00:00Z", { body: `Reverts ${REPO}#99999999999999999999`, merger: "kyle-sexton" }), + pr(1602, "2026-09-01T00:00:00Z", { body: `Reverts ${REPO}#2147483648`, merger: "kyle-sexton" }), + pr(1603, "2026-09-01T00:00:00Z", { body: `Reverts ${REPO}#0`, merger: "kyle-sexton" }), + ], + }); + const reason = "not a pull request number (a positive GraphQL Int)"; + assert.deepEqual(result.unresolvedReverts, [ + { target: "99999999999999999999", reason }, + { target: "2147483648", reason }, + { target: "0", reason }, + ]); +}); + +test("a bogus `This reverts commit` SHA is listed, not fatal", () => { + const bogus = "0123456789abcdef0123456789abcdef01234567"; + const result = run({ + pulls: [pr(1408, "2026-09-01T00:00:00Z", { merger: "kyle-sexton" })], + commits: [{ ...REC.commit, sha: shaOf(1408), commit: { message: `Revert "x"\n\nThis reverts commit ${bogus}.`, committer: { date: "2026-09-01T00:00:00Z" } } }], + }); + assert.deepEqual(result.unresolvedReverts, [{ target: bogus, reason: "no commit found for this SHA" }]); +}); + +test("gh failures: a missing PR is an answer, a missing commit is unresolvable, anything else fails", () => { + assert.equal(typeof publisher.ghFailure, "function"); + assert.deepEqual(publisher.ghFailure("graphql", true, JSON.stringify(REC.pullNotFound), ""), REC.pullNotFound); + assert.throws(() => publisher.ghFailure("x/pulls", false, REC.commitNotFound.stdout, REC.commitNotFound.stderr), publisher.UnresolvableError); + const fails = (body) => + assert.throws( + () => publisher.ghFailure("graphql", true, JSON.stringify(body), "gh: Could not resolve"), + (error) => error instanceof PublishError && !(error instanceof publisher.UnresolvableError), + ); + fails(REC.repoNotFound); + // Each half of the PR-NOT_FOUND test alone: a null pullRequest with a repository-level error, and + // a pullRequest-path error with the repository gone. + fails({ ...REC.pullNotFound, errors: REC.repoNotFound.errors }); + fails({ ...REC.repoNotFound, errors: REC.pullNotFound.errors }); +}); + +test("an incomplete page on a lane PR leaves it unclassed, not a refused publish", () => { + const paged = (patch) => run({ pulls: [pr(1409, "2026-09-01T00:00:00Z", { labels: [MECHANICAL], linked: ISSUE_6007, patch })] }); + const reason = "a linked-issue, label or timeline page is unreadable or incomplete: class unknown"; + for (const patch of [{ closingIssuesReferences: page1(ISSUE_6007, true) }, { timelineItems: page1([], true) }, { labels: page1([{ name: MECHANICAL }], true) }]) { + assert.deepEqual(paged(patch).unclassedAutonomous, [{ number: 1409, reason }]); + } + const issueLabelsPaged = [{ ...ISSUE_6007[0], labels: { ...ISSUE_6007[0].labels, pageInfo: { hasNextPage: true } } }]; + const noLabels = [{ number: 6007 }]; + for (const nodes of [issueLabelsPaged, noLabels]) { + assert.deepEqual(paged({ closingIssuesReferences: page1(nodes) }).unclassedAutonomous, [{ number: 1409, reason }]); + } +}); + +test("an unparsable date refuses the publish", () => { + const result = () => + run({ pulls: [pr(1410, "2026-09-01T00:00:00Z", { labels: [MECHANICAL] })], workflowRuns: { [CANARY]: [wfRun(13, "push", shaOf(1410), "not-a-date", { updated: "not-a-date" })] } }); + assert.throws(result, (error) => error instanceof PublishError && /date/.test(error.message)); +}); + +test("a PR merged into a branch other than the default is not read", () => { + const result = run({ pulls: [pr(1405, "2026-09-01T00:00:00Z", { labels: [MECHANICAL], base: "release" })] }); + assert.equal(termOf(result, "C2-auto-merge", "autonomous C2 completions").value, 0); +}); + +test("a link event after the merge in an unrecognized shape leaves the class unknown", () => { + const odd = { ...link("DisconnectedEvent", 6007, "2026-09-02T00:00:00Z"), subject: { __typename: "PullRequest", number: 5 } }; + const result = run({ pulls: [pr(1406, "2026-09-01T00:00:00Z", { labels: [MECHANICAL], timeline: [odd] })] }); + assert.equal(termOf(result, "C2-auto-merge", "autonomous C2 completions").value, 0); + assert.deepEqual(result.unclassedAutonomous, [{ number: 1406, reason: "a link event after the merge has an unrecognized shape: links unknown" }]); +}); + +test("a revert naming a PR that is not merged or does not exist is listed, not fatal", () => { + const result = run({ pulls: [pr(1407, "2026-09-01T00:00:00Z", { body: `Reverts ${REPO}#999999`, merger: "kyle-sexton" })] }); + assert.deepEqual(result.unresolvedReverts, [{ target: 999999, reason: "not a merged pull request" }]); +}); + +test("a red full run is a flake only when a lane identity's re-run passed", () => { + const tip = shaOf(0xa003); + const rerun = (id, triggering) => wfRun(id, "schedule", tip, "2026-09-02T00:00:00Z", { attempt: 2, triggering, updated: "2026-09-02T01:00:00Z" }); + const firstAttempt = (id) => ({ 1: wfRun(id, "schedule", tip, "2026-09-02T00:00:00Z", { conclusion: "failure" }) }); + const result = run({ + pulls: [pr(1304, "2026-09-01T00:00:00Z", { labels: [MECHANICAL] })], + history: [shaOf(1304), tip], + workflowRuns: { [FULL]: [rerun(4, LANE), rerun(5, "kyle-sexton")] }, + attempts: { 4: firstAttempt(4), 5: firstAttempt(5) }, + }); + assert.deepEqual(result.events, gateEvents("2026-09-02T01:00:00Z", "C2-auto-merge")); +}); + +test("a red run after a merge was verified by a green lane dispatch and canary is not that merge's", () => { + const [s1, s2] = [shaOf(0xa004), shaOf(0xa005)]; + const result = run({ + pulls: [pr(1305, "2026-09-01T00:00:00Z", { labels: [MECHANICAL] }), pr(1306, "2026-09-01T12:00:00Z", { labels: [SCOPED] })], + history: [shaOf(1305), s1, shaOf(1306), s2], + workflowRuns: { + [FULL]: [wfRun(6, "workflow_dispatch", s1, "2026-09-01T01:00:00Z", { actor: LANE }), wfRun(7, "schedule", s2, "2026-09-02T00:00:00Z", { conclusion: "failure" })], + [CANARY]: [wfRun(8, "push", s1, "2026-09-01T01:05:00Z")], + }, + }); + assert.deepEqual(result.events, gateEvents("2026-09-02T00:08:00Z", "C3-auto-merge")); +}); + +test("a red canary run on a SHA containing the merge yields gate-failure", () => { + const result = run({ + pulls: [pr(1307, "2026-09-01T00:00:00Z", { labels: [MECHANICAL] })], + workflowRuns: { [CANARY]: [wfRun(9, "push", shaOf(1307), "2026-09-01T00:01:00Z", { conclusion: "failure" })] }, + }); + assert.deepEqual(result.events, gateEvents("2026-09-01T00:09:00Z", "C2-auto-merge")); +}); + +test("a red full run awaiting its re-run is unsettled for 6 hours, then a failure", () => { + const tip = shaOf(0xa006); + const red = (id, updated) => wfRun(id, "schedule", tip, later(updated, -8), { conclusion: "failure", updated }); + const recent = run({ + pulls: [pr(1308, "2026-10-04T00:00:00Z", { labels: [MECHANICAL] })], + history: [shaOf(1308), tip], + workflowRuns: { [FULL]: [red(10, "2026-10-04T23:00:00Z")] }, + }); + assert.deepEqual(recent.events, []); + assert.equal(termOf(recent, "C2-auto-merge", "deterministic-gate pass").detail.unsettled, 1); + const stale = run({ + pulls: [pr(1308, "2026-10-04T00:00:00Z", { labels: [MECHANICAL] })], + history: [shaOf(1308), tip], + workflowRuns: { [FULL]: [red(10, "2026-10-04T17:00:00Z")] }, + }); + assert.deepEqual(stale.events, gateEvents("2026-10-04T17:00:00Z", "C2-auto-merge")); +}); + +test("only the named gate checks and status contexts are gate evidence", () => { + const other = (name, conclusion) => ({ ...REC.checkRun, name, conclusion, completed_at: "2026-09-01T00:10:00Z" }); + const pulls = [pr(1201, "2026-09-01T00:00:00Z", { labels: [MECHANICAL] })]; + const result = run({ + pulls, + checkRuns: { 1201: [REC.checkRun, other("dispatch", "failure"), other("resolve-conflicts / run", "cancelled")] }, + statuses: { 1201: [{ ...REC.status, context: "GitGuardian", state: "error" }] }, + workflowRuns: verifiedEach(pulls), + }); + assert.deepEqual(result.events, []); + assert.deepEqual(termOf(result, "C2-auto-merge", "deterministic-gate pass").detail, { passed: 1, failed: 0, unsettled: 0, total: 1 }); +}); + +test("no gate check named is refused rather than read as an empty gate", () => { + assert.throws(() => run({ pulls: [] }, { gateChecks: [] }), (error) => error instanceof PublishError && /--gate-check/.test(error.message)); + const cli = spawnSync(process.execPath, [join(HERE, "publish-promotion-evidence.mjs"), "--repo", REPO, "--since", SINCE, "--lane-login", LANE], { encoding: "utf8" }); + assert.equal(cli.status, 2); + assert.match(cli.stderr, /--gate-check/); +}); + +test("merges outside --since..--through appear in no list, count and event", () => { + const failing = (at) => [{ ...REC.checkRun, conclusion: "failure", completed_at: at }]; + const result = run({ + pulls: [pr(1202, "2026-07-31T00:00:00Z"), pr(1203, "2026-07-31T00:00:00Z", { labels: [MECHANICAL] }), pr(1204, "2026-10-06T00:00:00Z", { labels: [SCOPED] })], + checkRuns: { 1202: failing("2026-08-02T00:00:00Z"), 1203: failing("2026-08-02T00:00:00Z"), 1204: failing("2026-10-06T00:10:00Z") }, + }); + assert.deepEqual([result.events, result.unclassedAutonomous], [[], []]); + assert.equal(termOf(result, "C2-auto-merge", "autonomous C2 completions").value, 0); + assert.equal(termOf(result, "C3-auto-merge", "autonomous C3 completions").value, 0); +}); + +test("a revert-button PR of an autonomous C2 merge yields reverted-merge at the revert's merge time", () => { + const result = run({ + pulls: [ + pr(101, "2026-09-01T00:00:00Z", { labels: [MECHANICAL] }), + pr(102, "2026-09-02T00:00:00Z", { title: 'Revert "chore: bump"', body: `Reverts ${REPO}#101`, merger: "kyle-sexton" }), + ], + }); + assert.deepEqual(result.events, [{ cell: "C2-auto-merge", event: "reverted-merge", at: "2026-09-02T00:00:00Z" }]); + assert.deepEqual(termOf(result, "C2-auto-merge", "reverted merges"), { term: "reverted merges", value: 1, threshold: "0", pass: false }); +}); + +test("a git-revert commit pushed straight to the default branch yields reverted-merge for C3 auto-merge", () => { + // No PR merged the revert commit, and its git author can be forged, so it counts. + const result = run({ + pulls: [pr(201, "2026-09-01T00:00:00Z", { labels: [SCOPED] })], + history: [shaOf(201), shaOf(999)], + commits: [{ ...REC.commit, sha: shaOf(999), commit: { message: `Revert "x"\n\nThis reverts commit ${shaOf(201)}.`, committer: { date: "2026-09-03T00:00:00Z" } } }], + }); + assert.deepEqual(result.events, [{ cell: "C3-auto-merge", event: "reverted-merge", at: "2026-09-03T00:00:00Z" }]); +}); + +test("a revert commit whose REST-named PR GraphQL cannot resolve counts as a direct push, not a crash", () => { + const fixture = { + pulls: [pr(201, "2026-09-01T00:00:00Z", { labels: [SCOPED] })], + history: [shaOf(201), shaOf(999)], + commits: [{ ...REC.commit, sha: shaOf(999), commit: { message: `Revert "x"\n\nThis reverts commit ${shaOf(201)}.`, committer: { date: "2026-09-03T00:00:00Z" } } }], + }; + const base = github(fixture); + const gh = (path, graphql) => + path === `repos/${REPO}/commits/${shaOf(999)}/pulls` ? [{ number: 998, merged_at: "2026-09-03T00:00:00Z", merge_commit_sha: shaOf(999) }] : base(path, graphql); + const result = run(fixture, { gh }); + assert.deepEqual(result.events, [{ cell: "C3-auto-merge", event: "reverted-merge", at: "2026-09-03T00:00:00Z" }]); +}); + +test("a revert PR counts whoever merged or enqueued it, a lane or another bot included", () => { + const result = run({ + pulls: [ + pr(1701, "2026-09-01T00:00:00Z", { labels: [MECHANICAL] }), + pr(1702, "2026-09-02T00:00:00Z", { title: 'Revert "x"', body: `Reverts ${REPO}#1701` }), + pr(1703, "2026-09-01T00:00:00Z", { labels: [SCOPED] }), + pr(1704, "2026-09-02T00:00:00Z", { body: `Reverts ${REPO}#1703`, merger: "kyle-sexton", timeline: [queued("2026-09-01T23:00:00Z", null, "some-app[bot]")] }), + ], + }); + assert.deepEqual(result.events.filter((e) => e.event === "reverted-merge"), [ + { cell: "C2-auto-merge", event: "reverted-merge", at: "2026-09-02T00:00:00Z" }, + { cell: "C3-auto-merge", event: "reverted-merge", at: "2026-09-02T00:00:00Z" }, + ]); + assert.deepEqual(result.unresolvedReverts, []); +}); + +test("a lane-merged revert PR and its squash commit count once", () => { + const result = run({ + pulls: [pr(1705, "2026-09-01T00:00:00Z", { labels: [MECHANICAL] }), pr(1706, "2026-09-02T00:00:00Z", { body: `Reverts ${REPO}#1705` })], + commits: [{ ...REC.commit, sha: shaOf(1706), commit: { message: `Revert "x" (#1706)\n\nReverts ${REPO}#1705`, committer: { date: "2026-09-02T00:00:00Z" } } }], + }); + assert.deepEqual(result.events.filter((e) => e.event === "reverted-merge"), [{ cell: "C2-auto-merge", event: "reverted-merge", at: "2026-09-02T00:00:00Z" }]); + assert.deepEqual(result.unresolvedReverts, []); +}); + +test("a revert of a human merge, and a revert written only in prose, emit no event; prose is listed unresolved", () => { + const result = run({ + pulls: [ + pr(301, "2026-09-01T00:00:00Z", { labels: [MECHANICAL], merger: "kyle-sexton" }), + pr(302, "2026-09-02T00:00:00Z", { body: `Reverts ${REPO}#301` }), + pr(303, "2026-09-03T00:00:00Z", { title: "revert(x): drop the K3 check", body: "Reverts the bullet #301 added." }), + ], + }); + assert.deepEqual(result.events, []); + assert.deepEqual(result.unresolvedReverts, [{ number: 303, title: "revert(x): drop the K3 check" }]); +}); + +test("a failing check run on an autonomous merge commit yields gate-failure at its completion", () => { + const failing = { ...REC.checkRun, conclusion: "failure", completed_at: "2026-09-01T00:10:00Z" }; + const result = run({ pulls: [pr(401, "2026-09-01T00:00:00Z", { labels: [MECHANICAL] })], checkRuns: { 401: [REC.checkRun, failing] } }); + assert.deepEqual(result.events, [{ cell: "C2-auto-merge", event: "gate-failure", at: "2026-09-01T00:10:00Z" }]); + assert.deepEqual(termOf(result, "C2-auto-merge", "deterministic-gate pass").detail, { passed: 0, failed: 1, unsettled: 0, total: 1 }); +}); + +test("a missed-blocking label a human applied to a C3 PR yields verification-divergence; one a bot applied does not", () => { + const labeled = (login, type, at) => ({ ...REC.issueEvent, event: "labeled", label: { name: MISSED }, actor: { login, type }, created_at: at }); + const result = run({ + pulls: [ + pr(501, "2026-09-01T00:00:00Z", { labels: [SCOPED, MISSED], merger: "kyle-sexton" }), + pr(502, "2026-09-01T00:00:00Z", { labels: [SCOPED, MISSED] }), + pr(503, "2026-09-01T00:00:00Z", { labels: [SCOPED, MISSED], merger: "kyle-sexton" }), + ], + issueEvents: { + 501: [labeled("kyle-sexton", "User", "2026-09-04T00:00:00Z")], + 502: [labeled(LANE, "Bot", "2026-09-05T00:00:00Z")], + 503: [labeled("github-actions[bot]", "Bot", "2026-09-06T00:00:00Z")], + }, + }); + assert.deepEqual(result.events, [{ cell: "C3-ai-review-blocking", event: "verification-divergence", at: "2026-09-04T00:00:00Z" }]); +}); + +test("20 autonomous C2 merges over 14+ days pass the C2 term and 19 fail it", () => { + const merges = (count) => Array.from({ length: count }, (_, i) => pr(600 + i, `2026-09-${String(i + 1).padStart(2, "0")}T00:00:00Z`, { labels: [MECHANICAL] })); + const pulls = merges(20); + const twenty = run({ pulls, workflowRuns: verifiedEach(pulls) }); + assert.equal(termOf(twenty, "C2-auto-merge", "autonomous C2 completions").pass, true); + assert.equal(twenty.cells["C2-auto-merge"].pass, true); + const nineteen = run({ pulls: merges(19) }); + assert.deepEqual(termOf(nineteen, "C2-auto-merge", "autonomous C2 completions"), { term: "autonomous C2 completions", value: 19, threshold: ">= 20", pass: false }); + assert.equal(nineteen.cells["C2-auto-merge"].pass, false); +}); + +test("a class carried only by the linked issue counts, and the higher of issue and PR class wins", () => { + const result = run({ + pulls: [pr(801, "2026-09-01T00:00:00Z", { linked: ISSUE_6007 }), pr(802, "2026-09-02T00:00:00Z", { labels: [MECHANICAL], linked: ISSUE_6007 })], + }); + assert.equal(termOf(result, "C3-auto-merge", "autonomous C3 completions").value, 2); + assert.equal(termOf(result, "C2-auto-merge", "autonomous C2 completions").value, 0); +}); + +test("advisory reviews are counted from the named AI reviewers and reported unmeasured without one", () => { + const fixture = { pulls: [pr(701, "2026-09-01T00:00:00Z", { labels: [SCOPED], merger: "kyle-sexton" })], reviews: { 701: [REC.review] } }; + assert.equal(termOf(run(fixture, { aiReviewers: [REC.review.user.login] }), "C3-ai-review-blocking", "advisory AI reviews on C3").value, 1); + assert.equal(termOf(run(fixture), "C3-ai-review-blocking", "advisory AI reviews on C3").value, null); +}); + +test("a failed check run that was re-run green still yields gate-failure (filter=all)", () => { + const failed = { ...REC.checkRun, conclusion: "failure", completed_at: "2026-09-01T00:10:00Z", superseded: true }; + const result = run({ pulls: [pr(901, "2026-09-01T00:00:00Z", { labels: [MECHANICAL] })], checkRuns: { 901: [failed, REC.checkRun] } }); + assert.deepEqual(result.events, [{ cell: "C2-auto-merge", event: "gate-failure", at: "2026-09-01T00:10:00Z" }]); + assert.equal(termOf(result, "C2-auto-merge", "deterministic-gate pass").pass, false); +}); + +test("a failure or error commit status, even one later superseded by success, yields gate-failure", () => { + const statuses = [ + { ...REC.status, state: "success", created_at: "2026-09-01T00:30:00Z" }, + { ...REC.status, state: "error", created_at: "2026-09-01T00:20:00Z" }, + { ...REC.status, context: "other", state: "failure", created_at: "2026-09-01T00:15:00Z" }, + ]; + const result = run({ pulls: [pr(902, "2026-09-01T00:00:00Z", { labels: [SCOPED] })], statuses: { 902: statuses } }, { gateChecks: [...GATE, "other"] }); + assert.deepEqual(result.events, [ + { cell: "C3-auto-merge", event: "gate-failure", at: "2026-09-01T00:15:00Z" }, + { cell: "C3-auto-merge", event: "gate-failure", at: "2026-09-01T00:20:00Z" }, + ]); +}); + +test("a gate failure that completes after --through is still emitted", () => { + const failing = { ...REC.checkRun, conclusion: "failure", completed_at: "2026-10-06T00:00:00Z" }; + const result = run({ pulls: [pr(903, "2026-10-04T23:00:00Z", { labels: [MECHANICAL] })], checkRuns: { 903: [failing] } }); + assert.deepEqual(result.events, [{ cell: "C2-auto-merge", event: "gate-failure", at: "2026-10-06T00:00:00Z" }]); +}); + +test("a missed-blocking label a human applied and someone later removed still counts", () => { + const result = run({ + pulls: [pr(904, "2026-09-01T00:00:00Z", { labels: [SCOPED], merger: "kyle-sexton" })], + issueEvents: { 904: [labelEvent("labeled", MISSED, "2026-09-04T00:00:00Z"), labelEvent("unlabeled", MISSED, "2026-09-05T00:00:00Z", LANE, "Bot")] }, + }); + assert.deepEqual(result.events, [{ cell: "C3-ai-review-blocking", event: "verification-divergence", at: "2026-09-04T00:00:00Z" }]); +}); + +test("the class is read as of the merge: post-merge relabeling neither adds nor drops a merge", () => { + const result = run({ + pulls: [ + pr(905, "2026-09-01T00:00:00Z", { labels: [MECHANICAL] }), + pr(906, "2026-09-01T00:00:00Z"), + pr(907, "2026-09-01T00:00:00Z", { labels: [MECHANICAL], linked: ISSUE_6007 }), + ], + issueEvents: { + 905: [labelEvent("labeled", MECHANICAL, "2026-09-02T00:00:00Z")], + 906: [labelEvent("labeled", MECHANICAL, "2026-08-31T00:00:00Z"), labelEvent("unlabeled", MECHANICAL, "2026-09-02T00:00:00Z", LANE, "Bot")], + // The linked 6007 is labeled scoped now, but gained that label after 907 merged. + 6007: [labelEvent("labeled", SCOPED, "2026-09-03T00:00:00Z")], + }, + }); + assert.equal(termOf(result, "C2-auto-merge", "autonomous C2 completions").value, 2); + assert.equal(termOf(result, "C3-auto-merge", "autonomous C3 completions").value, 0); +}); + +test("20 merges inside one day fail the 14-day window, which ends at the last counted merge", () => { + const sameDay = Array.from({ length: 20 }, (_, i) => pr(1000 + i, later("2026-09-01T00:00:00Z", i), { labels: [MECHANICAL] })); + const result = run({ pulls: sameDay }); + assert.equal(termOf(result, "C2-auto-merge", "window days").pass, false); + assert.equal(result.cells["C2-auto-merge"].window.to, "2026-09-01T00:19:00.000Z"); +}); + +test("--through is clamped to now", () => { + const result = run({ pulls: [] }, { through: "2099-01-01T00:00:00Z" }); + assert.ok(Date.parse(result.through) <= Date.now()); +}); + +test("an issue unlinked after the merge still classes the merge; one linked after it does not", () => { + const result = run({ + pulls: [ + pr(1101, "2026-09-01T00:00:00Z", { timeline: [link("ConnectedEvent", 6007, "2026-08-31T00:00:00Z"), link("DisconnectedEvent", 6007, "2026-09-02T00:00:00Z")] }), + pr(1102, "2026-09-01T00:00:00Z", { linked: ISSUE_6007, timeline: [link("ConnectedEvent", 6007, "2026-09-02T00:00:00Z")] }), + ], + issues: { 6007: REC.issue }, + }); + assert.equal(termOf(result, "C3-auto-merge", "autonomous C3 completions").value, 1); + assert.deepEqual(result.unclassedAutonomous, [{ number: 1102, reason: "no work-class label at merge" }]); +}); + +test("an autonomous merge whose links cannot be rebuilt is unclassed, and its contrary events hit every cell", () => { + const failing = { ...REC.checkRun, conclusion: "failure", completed_at: "2026-09-01T00:10:00Z" }; + const result = run({ + pulls: [pr(1103, "2026-09-01T00:00:00Z", { labels: [MECHANICAL], lastEditedAt: "2026-09-02T00:00:00Z" })], + checkRuns: { 1103: [failing] }, + }); + assert.deepEqual(result.unclassedAutonomous, [{ number: 1103, reason: "PR body edited after the merge: closing-keyword links unknown" }]); + assert.deepEqual(result.events, ["C2-auto-merge", "C3-ai-review-blocking", "C3-auto-merge"].map((cell) => ({ cell, event: "gate-failure", at: "2026-09-01T00:10:00Z" }))); + assert.equal(termOf(result, "C2-auto-merge", "autonomous C2 completions").value, 0); +}); + +test("a revert of an unclassed autonomous merge yields reverted-merge against every cell", () => { + const result = run({ + pulls: [pr(1104, "2026-09-01T00:00:00Z"), pr(1105, "2026-09-02T00:00:00Z", { body: `Reverts ${REPO}#1104`, merger: "kyle-sexton" })], + }); + assert.deepEqual(result.events, ["C2-auto-merge", "C3-ai-review-blocking", "C3-auto-merge"].map((cell) => ({ cell, event: "reverted-merge", at: "2026-09-02T00:00:00Z" }))); +}); + +test("a User owning the bare slug of an App lane is human, not the lane", () => { + const result = run({ pulls: [pr(1110, "2026-09-01T00:00:00Z", { labels: [MECHANICAL], merger: "lane-bot", timeline: [queued("2026-08-31T23:00:00Z", "lane-bot")] })] }); + assert.equal(termOf(result, "C2-auto-merge", "autonomous C2 completions").value, 0); +}); + +test("lane logins match case-insensitively", () => { + const result = run({ pulls: [pr(1106, "2026-09-01T00:00:00Z", { labels: [MECHANICAL], merger: "Lane-User", timeline: [queued("2026-08-31T23:00:00Z", "Lane-User")] })] }, { laneLogins: ["lane-user"] }); + assert.equal(termOf(result, "C2-auto-merge", "autonomous C2 completions").value, 1); +}); + +test("a merge a bot or nobody enqueued is autonomous even under a human merged_by; a human enqueue is not", () => { + const result = run({ + pulls: [ + pr(1107, "2026-09-01T00:00:00Z", { labels: [MECHANICAL], merger: "kyle-sexton", timeline: [queued("2026-08-31T23:00:00Z", null, "some-app[bot]")] }), + pr(1108, "2026-09-01T00:00:00Z", { labels: [MECHANICAL], merger: "kyle-sexton", timeline: [queued("2026-08-31T23:00:00Z", "kyle-sexton")] }), + pr(1109, "2026-09-01T00:00:00Z", { labels: [MECHANICAL], merger: "kyle-sexton" }), + ], + }); + assert.equal(termOf(result, "C2-auto-merge", "autonomous C2 completions").value, 1); +}); + +test("a cancelled gate run counts as a failure unless a later run of the same check succeeded", () => { + const cancelled = (name, at) => ({ ...REC.checkRun, name, conclusion: "cancelled", completed_at: at }); + const pulls = [pr(1110, "2026-09-01T00:00:00Z", { labels: [MECHANICAL] }), pr(1111, "2026-09-01T00:00:00Z", { labels: [MECHANICAL] })]; + const result = run({ + pulls, + checkRuns: { + 1110: [cancelled("lint-repo", "2026-09-01T00:10:00Z")], + 1111: [cancelled("lint-repo", "2026-09-01T00:10:00Z"), { ...REC.checkRun, completed_at: "2026-09-01T00:20:00Z" }], + }, + workflowRuns: verifiedEach(pulls), + }); + assert.deepEqual(result.events, [{ cell: "C2-auto-merge", event: "gate-failure", at: "2026-09-01T00:10:00Z" }]); + assert.deepEqual(termOf(result, "C2-auto-merge", "deterministic-gate pass").detail, { passed: 1, failed: 1, unsettled: 0, total: 2 }); + assert.deepEqual(result.gateFailures, [{ number: 1110, check: "lint-repo", conclusion: "cancelled", at: "2026-09-01T00:10:00Z" }]); +}); + +test("a revert seen in both the PR body and the squash commit yields one event at the earlier time", () => { + const result = run({ + pulls: [ + pr(909, "2026-09-01T00:00:00Z", { labels: [MECHANICAL] }), + pr(910, "2026-09-02T00:00:00Z", { title: 'Revert "x"', body: `Reverts ${REPO}#909`, merger: "kyle-sexton" }), + ], + commits: [{ ...REC.commit, sha: shaOf(910), commit: { message: `Revert "x" (#910)\n\nReverts ${REPO}#909`, committer: { date: "2026-09-01T23:55:00Z" } } }], + }); + assert.deepEqual(result.events, [{ cell: "C2-auto-merge", event: "reverted-merge", at: "2026-09-01T23:55:00Z" }]); +}); + +test("a PR updated while the listing pages is not skipped", () => { + // 101 unclassed merges plus one C3 merge that sorts last by update time until it is updated. + const pulls = Array.from({ length: 101 }, (_, i) => pr(2000 + i, later("2026-09-10T00:00:00Z", i))); + pulls.push(pr(3000, "2026-09-05T00:00:00Z", { labels: [SCOPED] })); + const result = run({ pulls, bumpOnPage2: 3000 }); + assert.equal(termOf(result, "C3-auto-merge", "autonomous C3 completions").value, 1); +}); + +const A = { cell: "C2-auto-merge", event: "reverted-merge", at: "2026-09-02T00:00:00Z" }; +const B = { cell: "C3-auto-merge", event: "gate-failure", at: "2026-09-03T00:00:00Z" }; +const hash = (text) => createHash("sha256").update(text).digest("hex"); + +test("the sidecar hash matches the published file, and prev_sha256 names the file it replaced", () => { + const out = mkdtempSync(join(base, "out-")); + const first = publish(out, [A], THROUGH, new Date("2026-10-05T00:00:01Z")); + const firstBytes = readFileSync(join(out, "evidence.json"), "utf8"); + assert.deepEqual(JSON.parse(firstBytes), [A]); + assert.deepEqual(JSON.parse(readFileSync(join(out, "evidence.meta.json"), "utf8")), { + generated_at: "2026-10-05T00:00:01.000Z", + through: THROUGH, + sha256: hash(firstBytes), + prev_sha256: null, + }); + assert.equal(first.sha256, hash(firstBytes)); + const second = publish(out, [A, B], THROUGH); + assert.equal(second.sha256, hash(readFileSync(join(out, "evidence.json"), "utf8"))); + assert.equal(second.prev_sha256, hash(firstBytes)); +}); + +test("publishing a set that drops a previously published event is refused and leaves both files unchanged", () => { + const out = mkdtempSync(join(base, "out-")); + publish(out, [A, B], THROUGH); + const before = [readFileSync(join(out, "evidence.json"), "utf8"), readFileSync(join(out, "evidence.meta.json"), "utf8")]; + assert.throws(() => publish(out, [B], THROUGH), (error) => error instanceof PublishError && /refusing.*drop 1/.test(error.message)); + assert.deepEqual([readFileSync(join(out, "evidence.json"), "utf8"), readFileSync(join(out, "evidence.meta.json"), "utf8")], before); +}); + +test("an existing temp file is neither overwritten nor removed, and nothing is published", () => { + const out = mkdtempSync(join(base, "out-")); + const temp = join(out, `evidence.json.${process.pid}.tmp`); + writeFileSync(temp, "planted"); + assert.throws(() => publish(out, [A], THROUGH), { code: "EEXIST" }); + assert.equal(readFileSync(temp, "utf8"), "planted"); + assert.throws(() => readFileSync(join(out, "evidence.json")), { code: "ENOENT" }); +}); + +test("an events file that no longer matches its sidecar is refused and left unchanged", () => { + const out = mkdtempSync(join(base, "out-")); + publish(out, [A], THROUGH); + writeFileSync(join(out, "evidence.json"), `${JSON.stringify([A, B], null, 2)}\n`); + assert.throws(() => publish(out, [A, B], THROUGH), /refusing.*sidecar/); + assert.deepEqual(JSON.parse(readFileSync(join(out, "evidence.json"), "utf8")), [A, B]); +}); + +test("a sidecar write failure leaves the published events file unchanged", () => { + const out = mkdtempSync(join(base, "out-")); + publish(out, [A], THROUGH); + const before = readFileSync(join(out, "evidence.json"), "utf8"); + writeFileSync(join(out, `evidence.meta.json.${process.pid}.tmp`), "planted"); + assert.throws(() => publish(out, [A, B], THROUGH), { code: "EEXIST" }); + assert.equal(readFileSync(join(out, "evidence.json"), "utf8"), before); + assert.throws(() => readFileSync(join(out, `evidence.json.${process.pid}.tmp`)), { code: "ENOENT" }); +}); + +test("a failed sidecar rename puts the previous events file back, or removes a first one", () => { + const failMeta = (from, to) => { + if (to.endsWith("evidence.meta.json")) throw Object.assign(new Error("rename failed"), { code: "EIO" }); + renameSync(from, to); + }; + const out = mkdtempSync(join(base, "out-")); + assert.throws(() => publish(out, [A], THROUGH, new Date(), { rename: failMeta }), { code: "EIO" }); + assert.deepEqual(readdirSync(out), []); + publish(out, [A], THROUGH); + const before = readFileSync(join(out, "evidence.json"), "utf8"); + assert.throws(() => publish(out, [A, B], THROUGH, new Date(), { rename: failMeta }), { code: "EIO" }); + assert.equal(readFileSync(join(out, "evidence.json"), "utf8"), before); + assert.deepEqual(readdirSync(out).sort(), ["evidence.json", "evidence.meta.json"]); + publish(out, [A, B], THROUGH); +}); + +test("an existing events file that is not a JSON array is refused", () => { + const out = mkdtempSync(join(base, "out-")); + writeFileSync(join(out, "evidence.json"), "{}"); + assert.throws(() => publish(out, [A], THROUGH), /refusing.*not a JSON array/); +}); + +test("the CLI exits 2 on missing required arguments", () => { + const cli = spawnSync(process.execPath, [join(HERE, "publish-promotion-evidence.mjs"), "--repo", REPO], { encoding: "utf8" }); + assert.equal(cli.status, 2); + assert.match(cli.stderr, /--lane-login/); +}); diff --git a/plugins/source-control/skills/babysit-loop/scripts/publish-promotion-evidence.test.sh b/plugins/source-control/skills/babysit-loop/scripts/publish-promotion-evidence.test.sh new file mode 100755 index 0000000000..1e99296c52 --- /dev/null +++ b/plugins/source-control/skills/babysit-loop/scripts/publish-promotion-evidence.test.sh @@ -0,0 +1,13 @@ +#!/usr/bin/env bash +# Discovery wrapper: scripts/run-plugin-tests.sh finds plugins/**/*.test.sh, so +# this hands off to the Node suite. SKIPs (exit 0) when Node is unavailable. +set -uo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +if ! command -v node >/dev/null 2>&1; then + echo "SKIP: node not installed" >&2 + exit 0 +fi + +exec node --test "$SCRIPT_DIR/publish-promotion-evidence.test.mjs"