Coverage workflow (.github/workflows/coverage.yml, landed in PR #364) currently uploads artifacts and writes a step-summary but does NOT post deltas to PRs and does NOT gate on regression. This is open-loop and won't survive cadence — the dark-factory thesis says tests carry the load human review used to, and coverage is the proxy for whether that load is doing its job. A passive HTML artifact rots; a delta-gated PR comment closes the loop so the agent reading its own coverage output adds tests or justifies the drop.
Goal
On every PR run:
- Download the base branch's most recent successful
coverage-data artifact.
- Compute lines / branches / functions delta vs head.
- Post (or update) a
## Coverage comment on the PR with base / head / delta table.
- Gate the lines delta at
>= -0.5% by default. Fail CI if lower. Bypass via [skip-coverage-gate] (case-insensitive) in the PR title.
- On
push: main, do nothing extra — the artifact is the next PR's base.
Known blockers from the first attempt
The first cut of the closed-loop logic shipped four blockers in adversarial review and was reverted. Any follow-up implementation must address all of these:
gh run download --branch <ref> does not exist. Resolve a run-id first via gh run list --branch \"\$BASE_BRANCH\" --workflow Coverage --status success --limit 1 --json databaseId -q '.[0].databaseId' then gh run download <run-id> -n coverage-data.
coverage-final.json is not emitted by the current vitest reporter set. Either add 'json' to reporter in vitest.config.ts or drop the path from upload-artifact. The current open-loop workflow already drops it (uploads coverage-summary.json + lcov.info only).
gh api -F body=@file is wrong for markdown. -F is form-encoded; the issue-comments PATCH endpoint expects a JSON body. Use --input <(jq -Rs '{body: .}' < /tmp/comment.md) or -f body=\"\$(cat /tmp/comment.md)\".
- HEREDOC terminator indentation. A non-
<<- heredoc requires the terminator at column 0. The first cut had the EOF indented inside an if/else block; bash reads past it. Either dedent or switch to <<-EOF with tabs.
Edge cases to handle
- First-run / no base. Workflow has never produced a coverage-data artifact on
main. Degrade gracefully: post an informational comment with head numbers only; skip the gate.
- 30-day retention vs long-open PRs. A PR open longer than 30 days with no main-branch coverage runs will lose its base artifact. Degrade to no-base path; mention in the comment template.
- Bot version PR.
pull_request does not fire on PRs authored by GITHUB_TOKEN (changesets/action's version PR). The version PR will not have a coverage check by default. Acceptable since the version PR doesn't change runtime behavior; document.
- Multiple bot comments. Search for existing comment by
github-actions[bot] prefix ## Coverage; update via PATCH else create. head -1 if duplicates exist.
- PR title race.
[skip-coverage-gate] evaluated at gate-step time from github.event.pull_request.title, which is captured at workflow trigger. Editing the title alone won't re-evaluate; users need a synchronize event (push) to re-gate.
Out of scope
- Per-line uncovered diff in the PR comment. Useful but expensive to compute correctly across renames; defer.
- Threshold tuning beyond -0.5% lines. Pick the floor empirically once we have a few release cycles of data.
- Coverage on Firefox/WebKit. Not measurable today (JSC and SpiderMonkey don't expose V8's coverage protocol).
Coverage workflow (
.github/workflows/coverage.yml, landed in PR #364) currently uploads artifacts and writes a step-summary but does NOT post deltas to PRs and does NOT gate on regression. This is open-loop and won't survive cadence — the dark-factory thesis says tests carry the load human review used to, and coverage is the proxy for whether that load is doing its job. A passive HTML artifact rots; a delta-gated PR comment closes the loop so the agent reading its own coverage output adds tests or justifies the drop.Goal
On every PR run:
coverage-dataartifact.## Coveragecomment on the PR with base / head / delta table.>= -0.5%by default. Fail CI if lower. Bypass via[skip-coverage-gate](case-insensitive) in the PR title.push: main, do nothing extra — the artifact is the next PR's base.Known blockers from the first attempt
The first cut of the closed-loop logic shipped four blockers in adversarial review and was reverted. Any follow-up implementation must address all of these:
gh run download --branch <ref>does not exist. Resolve a run-id first viagh run list --branch \"\$BASE_BRANCH\" --workflow Coverage --status success --limit 1 --json databaseId -q '.[0].databaseId'thengh run download <run-id> -n coverage-data.coverage-final.jsonis not emitted by the current vitest reporter set. Either add'json'toreporterinvitest.config.tsor drop the path fromupload-artifact. The current open-loop workflow already drops it (uploadscoverage-summary.json+lcov.infoonly).gh api -F body=@fileis wrong for markdown.-Fis form-encoded; the issue-comments PATCH endpoint expects a JSON body. Use--input <(jq -Rs '{body: .}' < /tmp/comment.md)or-f body=\"\$(cat /tmp/comment.md)\".<<-heredoc requires the terminator at column 0. The first cut had theEOFindented inside anif/elseblock; bash reads past it. Either dedent or switch to<<-EOFwith tabs.Edge cases to handle
main. Degrade gracefully: post an informational comment with head numbers only; skip the gate.pull_requestdoes not fire on PRs authored byGITHUB_TOKEN(changesets/action's version PR). The version PR will not have a coverage check by default. Acceptable since the version PR doesn't change runtime behavior; document.github-actions[bot]prefix## Coverage; update via PATCH else create.head -1if duplicates exist.[skip-coverage-gate]evaluated at gate-step time fromgithub.event.pull_request.title, which is captured at workflow trigger. Editing the title alone won't re-evaluate; users need asynchronizeevent (push) to re-gate.Out of scope