Skip to content

Closed-loop coverage: PR delta comment + regression gate #379

Description

@brianmhunt

Coverage workflow (.github/workflows/coverage.yml, landed in PR #364) currently uploads artifacts and writes a step-summary but does NOT post deltas to PRs and does NOT gate on regression. This is open-loop and won't survive cadence — the dark-factory thesis says tests carry the load human review used to, and coverage is the proxy for whether that load is doing its job. A passive HTML artifact rots; a delta-gated PR comment closes the loop so the agent reading its own coverage output adds tests or justifies the drop.

Goal

On every PR run:

  1. Download the base branch's most recent successful coverage-data artifact.
  2. Compute lines / branches / functions delta vs head.
  3. Post (or update) a ## Coverage comment on the PR with base / head / delta table.
  4. Gate the lines delta at >= -0.5% by default. Fail CI if lower. Bypass via [skip-coverage-gate] (case-insensitive) in the PR title.
  5. On push: main, do nothing extra — the artifact is the next PR's base.

Known blockers from the first attempt

The first cut of the closed-loop logic shipped four blockers in adversarial review and was reverted. Any follow-up implementation must address all of these:

  1. gh run download --branch <ref> does not exist. Resolve a run-id first via gh run list --branch \"\$BASE_BRANCH\" --workflow Coverage --status success --limit 1 --json databaseId -q '.[0].databaseId' then gh run download <run-id> -n coverage-data.
  2. coverage-final.json is not emitted by the current vitest reporter set. Either add 'json' to reporter in vitest.config.ts or drop the path from upload-artifact. The current open-loop workflow already drops it (uploads coverage-summary.json + lcov.info only).
  3. gh api -F body=@file is wrong for markdown. -F is form-encoded; the issue-comments PATCH endpoint expects a JSON body. Use --input <(jq -Rs '{body: .}' < /tmp/comment.md) or -f body=\"\$(cat /tmp/comment.md)\".
  4. HEREDOC terminator indentation. A non-<<- heredoc requires the terminator at column 0. The first cut had the EOF indented inside an if/else block; bash reads past it. Either dedent or switch to <<-EOF with tabs.

Edge cases to handle

  • First-run / no base. Workflow has never produced a coverage-data artifact on main. Degrade gracefully: post an informational comment with head numbers only; skip the gate.
  • 30-day retention vs long-open PRs. A PR open longer than 30 days with no main-branch coverage runs will lose its base artifact. Degrade to no-base path; mention in the comment template.
  • Bot version PR. pull_request does not fire on PRs authored by GITHUB_TOKEN (changesets/action's version PR). The version PR will not have a coverage check by default. Acceptable since the version PR doesn't change runtime behavior; document.
  • Multiple bot comments. Search for existing comment by github-actions[bot] prefix ## Coverage; update via PATCH else create. head -1 if duplicates exist.
  • PR title race. [skip-coverage-gate] evaluated at gate-step time from github.event.pull_request.title, which is captured at workflow trigger. Editing the title alone won't re-evaluate; users need a synchronize event (push) to re-gate.

Out of scope

  • Per-line uncovered diff in the PR comment. Useful but expensive to compute correctly across renames; defer.
  • Threshold tuning beyond -0.5% lines. Pick the floor empirically once we have a few release cycles of data.
  • Coverage on Firefox/WebKit. Not measurable today (JSC and SpiderMonkey don't expose V8's coverage protocol).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions