Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
bump: patch
---

### Fixed

- check-wip-cap treats a GitHub search answer with `incomplete_results: true` as unknown, not as a failed read: new admissions pause whether or not `pause-from` is set or reached, with `admission_basis` and `admission_reason` `search-incomplete` and a `check-wip-cap` warning, so work-unblock defers newly unblocked issues instead of admitting without limit on a false empty. A read that fails without that answer still fails open, the queued cap still pauses only past `pause-from`, and no run pause changes.
- check-wip-cap's over-cap warning names why the lane keeps running (the base config could not be read, `pause-from` is not set, or `pause-from` not reached), and work-unblock's deferral log names its reason instead of always citing the queued cap.
5 changes: 3 additions & 2 deletions .github/actions/check-wip-cap/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,8 @@ description: >
newest status a `save-state` run saved to the Actions cache) applies; with
none the check fails open. Set `admission-open` to `false` only when new
admissions pause (queued work at or over the queued cap on or after
`wip.pause-from`).
`wip.pause-from`, or a search answered `incomplete_results: true`, whatever
`wip.pause-from` says).
Run it from the default branch checkout, before any slot or lane token.

inputs:
Expand All @@ -36,7 +37,7 @@ outputs:
description: "`true` only while the lane pauses (over the WIP cap or the held cap on or after `wip.pause-from`, or on a failed read whose last known state paused); otherwise `false`."
value: ${{ steps.check.outputs.capped }}
admission-open:
description: "`false` only while new admissions pause (queued work at or over the queued cap on or after `wip.pause-from`); a failed queued read leaves `true`. Runs never pause on it."
description: "`false` only while new admissions pause (queued work at or over the queued cap on or after `wip.pause-from`, or a search answered `incomplete_results: true`, whatever `wip.pause-from` says); a failed queued read without that flag leaves `true`. Runs never pause on it."
value: ${{ steps.check.outputs.admission-open }}
wip:
description: "The status as one flat JSON object: countable_lane_prs, held_lane_prs, held_oldest_age_hours, held_over_alarm_count, wip_cap, wip_cap_source and the rest (pr-pipeline README, WIP cap)."
Expand Down
37 changes: 27 additions & 10 deletions .github/actions/check-wip-cap/check-wip-cap.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,9 @@
// max(limits.admission-factor x lane merges per hour, limits.admission-min-per-hour)
// x wip.merge-rate-window-hours, on or after wip.pause-from, new admissions
// (work-unblock adding agent-ready) pause; runs never pause on it, since a
// queued issue leaves the count only by running. A failed queued read leaves
// admissions open.
// queued issue leaves the count only by running. A search that answers
// incomplete_results: true closes admissions whatever wip.pause-from says. A
// failed queued read without that flag leaves admissions open.
//
// As the action's step it writes `capped` (the run pause), `admission-open`,
// `wip` (the status as one flat JSON object) and `state-saved` (`true` when a
Expand Down Expand Up @@ -77,18 +78,21 @@ const isCount = (value) => Number.isSafeInteger(value) && value >= 0;

// The open lane PRs as { createdAt, held, issue }, the merged and opened
// counts in the window, and the queued issues' numbers; each null when its
// read fails. Reads run one at a time, and
// read fails. `searchIncomplete` is whether any answer said incomplete_results
// true, which a degraded search answers with a false empty. Reads run one at a time, and
// none starts once `remaining()` (the last x-ratelimit-remaining seen) is 0 or less.
export async function readLaneWip(github, { repo, bot = DEFAULT_BOT, now, windowHours, remaining = () => null }) {
const author = `author:app/${bot.replace(/\[bot\]$/, "")}`;
const since = isoSecond(now - windowHours * HOUR_MS);
let searchIncomplete = false;
const search = async (q, page, perPage) => {
const left = remaining();
if (left !== null && left <= 0) {
throw new Error("the rate limit has no requests left");
}
const answer = await github("GET", `/search/issues?q=${encodeURIComponent(q)}&per_page=${perPage}&page=${page}`);
if (answer?.incomplete_results !== false) {
searchIncomplete ||= answer?.incomplete_results === true;
throw new Error("the search answered incomplete results");
}
return answer;
Expand Down Expand Up @@ -147,14 +151,14 @@ export async function readLaneWip(github, { repo, bot = DEFAULT_BOT, now, window
);
}
}
return { open, merged, admitted, queued: queued?.items ?? null, queuedUnlisted: queued?.unlisted ?? null };
return { open, merged, admitted, queued: queued?.items ?? null, queuedUnlisted: queued?.unlisted ?? null, searchIncomplete };
}

const round = (value, places) => (value === null ? null : Number(value.toFixed(places)));

// The status as one flat object; the field names are a published contract
// (README.md, WIP cap).
export function wipStatus({ open, merged, admitted, queued = null, queuedUnlisted = 0 }, { limits, wip, now }) {
export function wipStatus({ open, merged, admitted, queued = null, queuedUnlisted = 0, searchIncomplete = false }, { limits, wip, now }) {
const window = wip["merge-rate-window-hours"];
const held = open?.filter((pr) => pr.held) ?? null;
const ages = held?.map((pr) => (now - pr.createdAt) / HOUR_MS).filter(Number.isFinite) ?? [];
Expand All @@ -181,8 +185,16 @@ export function wipStatus({ open, merged, admitted, queued = null, queuedUnliste
const queuedIssues = queued === null || open === null ? null : queued.filter((number) => !named.has(number)).length + (queuedUnlisted ?? 0);
const queuedTotal = queuedIssues === null ? null : queuedIssues + heldCount;
const queuedOver = queuedTotal !== null && (truncated || queuedTotal >= queuedCap - 1e-9);
// Admissions left before the cap; null when nothing gates them.
const room = pausing && queuedTotal !== null ? (truncated ? 0 : Math.max(0, Math.ceil(queuedCap - queuedTotal - 1e-9))) : null;
// Admissions left before the cap; null when nothing gates them. An
// incomplete search answer leaves none whatever pause-from says: its counts
// may be false empties.
const room = searchIncomplete
? 0
: !pausing || queuedTotal === null
? null
: truncated
? 0
: Math.max(0, Math.ceil(queuedCap - queuedTotal - 1e-9));
return {
countable_lane_prs: countable,
held_lane_prs: heldCount,
Expand Down Expand Up @@ -216,9 +228,9 @@ export function wipStatus({ open, merged, admitted, queued = null, queuedUnliste
paused: pausing && reasons.length > 0,
pause_reason: pausing && reasons.length > 0 ? reasons.join(",") : null,
admission_open: room === null || room > 0,
admission_reason: room === 0 ? "queued" : null,
admission_reason: room === 0 ? (searchIncomplete ? "search-incomplete" : "queued") : null,
admission_room: room,
admission_basis: queuedTotal === null ? "fail-open" : "read",
admission_basis: searchIncomplete ? "search-incomplete" : queuedTotal === null ? "fail-open" : "read",
};
}

Expand Down Expand Up @@ -326,9 +338,14 @@ export async function main({ env = process.env, github, now = Date.now(), log =
`::warning title=check-wip-cap::${read.countable_lane_prs === null ? "the open lane PRs could not be listed" : "the base config could not be read"}, so ${status.pause_basis === "last-state" ? `the last known state applies (paused ${status.paused})` : "with no last known state the run starts"}`,
);
}
if (status.admission_basis === "search-incomplete") {
log(
`::warning title=check-wip-cap::GitHub search answered incomplete results (admission_basis search-incomplete), so its counts are unknown and no new admissions are made this run`,
);
}
if (status.alert) {
log(
`::warning title=check-wip-cap::${escape(alertText(status))}; ${status.paused ? "this run starts no work and work-unblock re-fires it once under both caps" : `the lane keeps running (${status.wip_over || status.held_over ? "pause-from not reached" : "only the WIP and held caps pause runs"})`}`,
`::warning title=check-wip-cap::${escape(alertText(status))}; ${status.paused ? "this run starts no work and work-unblock re-fires it once under both caps" : `the lane keeps running (${!(status.wip_over || status.held_over) ? "only the WIP and held caps pause runs" : configError ? "the base config could not be read, so the defaults apply" : wip["pause-from"] === "" ? "pause-from is not set" : "pause-from not reached"})`}`,
);
}
if (status.paused) {
Expand Down
53 changes: 52 additions & 1 deletion .github/actions/check-wip-cap/check-wip-cap.test.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,32 @@ test("the open listing pages until a short page", async () => {

test("a search that answers incomplete results is an unknown read", async () => {
const reads = await read({ open: [pr()], merged: 0, admitted: 3, incomplete: true });
assert.deepEqual(reads, { open: null, merged: null, admitted: null, queued: null, queuedUnlisted: null });
assert.deepEqual(reads, { open: null, merged: null, admitted: null, queued: null, queuedUnlisted: null, searchIncomplete: true });
});

// During a GitHub search incident the API answered total_count 0 with
// incomplete_results true for a repository with about 600 open issues
// (2026-10-22). A failed read keeps admissions open; an incomplete one is
// evidence the search itself is degraded, so it closes them whatever
// pause-from says (owner decision 2026-10-22); the queued cap stays as configured.
test("an incomplete search answer closes new admissions with pause-from unset, future or past", async () => {
const reads = await read({ open: [], merged: 0, admitted: 0, queued: [], incomplete: true });
for (const pauseFrom of ["", "2026-10-19", "2026-10-01"]) {
const closed = status(reads, { "pause-from": pauseFrom });
assert.equal(closed.admission_open, false, pauseFrom);
assert.equal(closed.admission_room, 0, pauseFrom);
assert.equal(closed.admission_reason, "search-incomplete", pauseFrom);
assert.equal(closed.admission_basis, "search-incomplete", pauseFrom);
assert.equal(closed.paused, false, pauseFrom);
}
});

test("a search answer with no incomplete_results field is a failed read, not an incomplete one", async () => {
const reads = await read({ open: [pr()], merged: 48, queued: null });
assert.equal(reads.searchIncomplete, false);
const result = status(reads, { "pause-from": "2026-10-01" });
assert.equal(result.admission_open, true);
assert.equal(result.admission_basis, "fail-open");
});

test("no read is made once the rate limit has nothing left; those reads are unknown", async () => {
Expand Down Expand Up @@ -632,6 +657,32 @@ test("main pauses new admissions over this repository's queued cap past pause-fr
assert.ok(!under.logs.some((l) => l.includes("admission paused")));
});

test("main closes new admissions and warns when the search answers incomplete results, under this repository's own pause-from", async () => {
const repoConfig = readFileSync(new URL(`../../../${CONFIG}`, import.meta.url), "utf8");
for (const config of [repoConfig, pastConfig()]) {
const result = await runMain({ open: [], merged: 0, admitted: 0, queued: [], incomplete: true }, {}, config);
assert.equal(result.admissionOpen, "false");
assert.equal(result.wip.admission_basis, "search-incomplete");
const warning = result.logs.find((line) => line.includes("search-incomplete"));
assert.ok(warning?.startsWith("::warning title=check-wip-cap::"), result.logs.join("\n"));
assert.ok(warning.includes("no new admissions are made this run"), warning);
assert.ok(result.logs.includes("::notice title=admission paused::admission paused: reason=search-incomplete queued=null held=null cap=24"), result.logs.join("\n"));
}
});

test("over a cap with an unreadable base config the warning names the config, not pause-from", async () => {
const held = await runMain({ open: Array.from({ length: 60 }, () => pr({ labels: ["needs-human"] })), merged: 24 });
const warning = held.logs.find((line) => line.startsWith("::warning title=check-wip-cap::60 held"));
assert.ok(warning?.includes("the lane keeps running (the base config could not be read, so the defaults apply)"), held.logs.join("\n"));
const unset = await runMain(
{ open: Array.from({ length: 60 }, () => pr({ labels: ["needs-human"] })), merged: 24 },
{},
readFileSync(new URL(`../../../${CONFIG}`, import.meta.url), "utf8"),
);
const unsetWarning = unset.logs.find((line) => line.startsWith("::warning title=check-wip-cap::"));
assert.ok(unsetWarning?.includes("the lane keeps running (pause-from is not set)"), unset.logs.join("\n"));
});

test("a queued read that fails keeps the run's read basis and leaves admissions open", async () => {
const failed = new Error("GET /search/issues failed with status 502");
const result = await runMain({ open: [pr()], merged: 24, queued: failed }, { WIP_STATE: stateFile() }, pastConfig());
Expand Down
19 changes: 12 additions & 7 deletions .github/scripts/work-unblock/label-unblocked.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,13 @@
// payload. No model.
//
// Each new agent-ready is paced: at or over the issue lane's queued cap
// (check-wip-cap.mjs, on or after wip.pause-from) a ready issue gets
// `admission-deferred` instead, and with room for k admissions only the first
// k ready issues are labeled. Every sweep revisits the open deferred issues
// oldest first, before the closed ones: one still ready is labeled agent-ready
// within the room and loses admission-deferred, a held one stays deferred, and
// any other loses admission-deferred. Restarts, re-fires and relabels below
// (check-wip-cap.mjs, on or after wip.pause-from, or whenever a search answers
// incomplete_results: true) a ready issue gets `admission-deferred` instead,
// and with room for k admissions only the first k ready issues are labeled.
// Every sweep revisits the open deferred issues oldest first, before the closed
// ones: one still ready is labeled agent-ready within the room and loses
// admission-deferred, a held one stays deferred, and any other loses
// admission-deferred. Restarts, re-fires and relabels below
// re-run admitted issues and are never paced.
//
// On `schedule` it sweeps instead: every issue closed in the last
Expand Down Expand Up @@ -310,11 +311,15 @@ async function isTrustedClose(github, repo, closedIssue, humans) {
// asked and counting each admission against its room.
function admissionGate(status) {
let room;
let reason;
return {
// Why take() last answered false.
why: () => (reason === "search-incomplete" ? "GitHub search answered incomplete results" : "queued work is at the queued cap"),
take: async () => {
if (room === undefined) {
const s = await status();
room = s.admission_room ?? Number.POSITIVE_INFINITY;
reason = s.admission_reason;
if (!s.admission_open) {
console.log(`::notice title=admission paused::${escapeCommand(admissionLine(s))}`);
}
Expand All @@ -335,7 +340,7 @@ async function admit({ github, repo, number, gate, deferred }) {
if (!deferred) {
await github("POST", `${base}/labels`, { labels: [DEFERRED_LABEL] });
}
console.log(`label-unblocked: #${number} admission deferred: queued work is at the queued cap`);
console.log(`label-unblocked: #${number} admission deferred: ${gate.why()}`);
return;
}
await github("POST", `${base}/labels`, { labels: [LABEL] });
Expand Down
26 changes: 26 additions & 0 deletions .github/scripts/work-unblock/label-unblocked.test.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,7 @@ function world({
pulls = {},
queuedIssues = [],
deferred = [],
searchIncomplete = false,
failOn,
} = {}) {
const calls = [];
Expand Down Expand Up @@ -134,6 +135,9 @@ function world({
return { workflow_runs: listed(inRange) };
}
if (method === "GET" && route === "/search/issues") {
if (searchIncomplete) {
return { total_count: 0, incomplete_results: true, items: [] };
}
const q = new URL(path, "https://x").searchParams.get("q");
if (q.startsWith(`repo:${REPO} is:issue is:open label:agent-ready `)) {
const items = queuedIssues.map((number) => ({ number }));
Expand Down Expand Up @@ -1939,6 +1943,28 @@ test("under the queued cap, or before pause-from, a newly unblocked issue is lab
}
});

// A degraded search answers a false empty (total_count 0, incomplete_results
// true): past pause-from that admits nothing, where a failed read fails open.
test("an incomplete search answer defers a newly unblocked issue, with or without pause-from, and names why", async (t) => {
const log = t.mock.method(console, "log", () => {});
for (const config of [PAUSING, {}]) {
const { github, calls } = world({ ...oneDependentWorld(), searchIncomplete: true });
assert.equal(await main({ env: envFor({}, config), github }), 0);
assert.deepEqual(writes(calls), [deferredLabel(20)], JSON.stringify(config));
}
const lines = log.mock.calls.map((call) => String(call.arguments[0]));
assert.ok(lines.includes("label-unblocked: #20 admission deferred: GitHub search answered incomplete results"), lines.join("\n"));
assert.ok(!lines.some((line) => line.includes("at the queued cap")), lines.join("\n"));
});

test("at the queued cap the deferral names the queued cap", async (t) => {
const log = t.mock.method(console, "log", () => {});
const { github } = world({ ...oneDependentWorld(), queuedIssues: queuedN(24) });
assert.equal(await main({ env: envFor({}, PAUSING), github }), 0);
const lines = log.mock.calls.map((call) => String(call.arguments[0]));
assert.ok(lines.includes("label-unblocked: #20 admission deferred: queued work is at the queued cap"), lines.join("\n"));
});

test("with room for one admission the first ready issue is labeled and the next deferred", async (t) => {
t.mock.method(console, "log", () => {});
const blocker = [issue(10, { state: "closed" })];
Expand Down
13 changes: 9 additions & 4 deletions docs/conventions/pr-pipeline/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1589,8 +1589,13 @@ the log and step summary carry one line
annotation titled `admission paused`). Under it, `admission_room` is how many more issues may be
admitted now. The queued cap gates only new `agent-ready` labels, never a run: a queued issue
leaves the count only by running, so pausing runs on it would hold the count shut. A queued read
that fails leaves admissions open (`admission_basis: fail-open`) and changes no run pause. The
search lists at most 1,000 issues: past that, the issues it counted but did not list count as
that fails leaves admissions open (`admission_basis: fail-open`) and changes no run pause. A
search that answers `incomplete_results: true` (a degraded search can answer a false empty) is
unknown, not failed: admissions pause whether or not `pause-from` is set or reached
(`admission_basis` and `admission_reason` `search-incomplete`), with a `check-wip-cap` warning,
and work-unblock's deferral log names that reason. The queued cap itself still pauses only on or
after `pause-from`, and no run pause changes.
The search lists at most 1,000 issues: past that, the issues it counted but did not list count as
queued, and admissions pause (`queued_truncated: true`).

work-unblock is the one lane that adds `agent-ready` to a new issue. Past the room it labels a
Expand Down Expand Up @@ -1643,8 +1648,8 @@ and the log line `check-wip-cap: <json>`. `capped` is `true` only while runs pau
| `pause_from`, `paused` | The pause date and whether the lane pauses now. |
| `pause_reason` | `wip`, `held-cap` or `wip,held-cap` while paused; null otherwise. |
| `pause_basis` | `read` from this run's reads; on a failed read `last-state` (the last known pause) or `fail-open` (none saved). |
| `admission_open`, `admission_reason`, `admission_room` | Whether new admissions are open; `queued` while paused, null otherwise; admissions left before the cap, null when nothing gates them. |
| `admission_basis` | `read`, or `fail-open` when the queued count could not be read. |
| `admission_open`, `admission_reason`, `admission_room` | Whether new admissions are open; `queued` or `search-incomplete` while paused, null otherwise; admissions left before the cap, null when nothing gates them. |
| `admission_basis` | `read`; `fail-open` when the queued count could not be read; `search-incomplete` when a search answered `incomplete_results: true`. |

## Adoption

Expand Down
Loading