Skip to content

pr-pipeline: review-lane hold follow-ups from the #9361 security review #9369

Open

Description

@kyle-sexton

Refs: #9276

Follow-ups from the security review of #9361 (review-lane holds in ci-status). Each was rated MEDIUM or LOW and left out of that PR on purpose.

Medium

  1. Review runs are matched to a PR by head SHA only. ci-status's "Require both review lanes on this head" step, and pr-feedback-dispatch.yml's rerun-ci-status, take the newest pull_request run of each caller on the head SHA. A second PR on the same SHA, with a different base, has its own review, and that review can count for this PR. Fix: give both callers a run-name carrying the PR number, and filter on display_title. Alternatively, also require this PR's own claude-lane-review-complete marker for the head.
  2. The kill switch's pass can fail open and outlive the switch. A lane passes when its Report the kill switch step ran, and that step also runs when check-kill-switch fails closed on an unreadable variable. A head that passed while the switch was on stays green after it clears. pr-merge.yml does not read CLAUDE_REVIEW_DISABLED, so a review-only stop leaves agent PRs merging with no AI review. OD-8 chose no merge freeze. Options: pass a stopped lane only on PRs a trusted person authored, or make pr-merge refuse lane merges while a review switch is on, and re-run ci-status on open heads when the switch clears.
  3. The gate runs from the PR's own copy of pr-require-checks.yml.
    • hold-paths does not list the gate's own files: pr-require-checks.yml, both review callers, pr-feedback-dispatch.yml and pr-merge.yml.
    • The merge-group run re-checks labels only, not the review-lane requirement or who lifted a hold.
    • Options: add the workflow files to hold-paths (owner decision Q48 set the current list), and move the lift and review checks into default-branch code, such as the pr-merge prefilter or a workflow_run job posting a status.

Low

  1. An exempt author's PR skips the review wait after a pushed fix. fix-ci can push to a release-App PR, which is exempt from the review requirement by author id. Agent-written commits can then merge before their review finishes. Fix: exempt only when every commit is the exempt App's own, or keep exempt-author PRs out of fix-ci.
  2. The hold label name can diverge. ci-status reads the human-gated label from the base .work-item-tracker.json, while both callers hardcode hold-label: needs-human. Renaming the tracker label would let review holds stop holding. Fix: fail ci-status on the literal needs-human as well, or feed the callers the tracker value.
  3. The rerun listener is unproven live. rerun-ci-status runs from main, so feat(pr-pipeline): turn on review-lane holds and require both review lanes in ci-status #9361's own CI could not exercise it. Confirm it on the first PR after merge.

Activity

  1. added
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Oct 23, 2026
  2. github-actions commented on Oct 23, 2026

    @github-actions
    Contributor

    Proposed work class: work-class: scoped
    Basis: automated intake triage; the lanes App applied this class.

    Size: Medium
    Complexity: Medium
    Uncertainty: Known
    Flags: none
    Blocked by: none

    Decision recorded. The owner answered in the Refinement and Issue Triage sessions (2026-10-25), deciding the open items below on the facts:

    go with whatever the facts/data/metrics + /discovery:research says and ensure we're aligned with SOLID, DRY/SSOT principles

    [quote truncated; the full decision is at the link]

    #9369 (comment)

    The decision settles every open question: per-PR run-name matching (item 1), kill-switch pass only when read as on and a trusted person authored the PR (2), no change to hold-paths (3), exemption only when all commits are exempt (4), and a literal needs-human check (5). Item 6 is confirmed later from run history; an unattended run can't do that, so the other items are triaged as usual. No duplicates found.

    Automated triage, applied by the lanes App.

  3. added
    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.
    and removed
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Oct 23, 2026
  4. kyle-sexton commented on Oct 26, 2026

    @kyle-sexton
    MemberAuthor

    Decision recorded. The owner answered in the Refinement and Issue Triage sessions (2026-10-25), deciding the open items below on the facts:

    go with whatever the facts/data/metrics + /discovery:research says and ensure we're aligned with SOLID, DRY/SSOT principles

    I should only ever be in the loop ONLY when necessary (intent/vision changed, something came up during implementation that impacts trajectory/alignment, OR a high security risk) - I've covered this all before

    No owner ledger (.work/*/interview-checklist.md, .work/gh-monitor-owner-decisions/) names this issue. No ledger entry for OD-8 or for a Q48 about hold-paths was found; the Q48 rows in .work/*/interview-checklist.md are on other subjects. Every item below tightens a gate or leaves it as is; none widens one. The review rated each MEDIUM or LOW.

    1. Item 1: review runs matched to a PR by head SHA only.
      Decided: Give both review callers a run-name carrying the PR number, and have both readers keep only runs whose display_title names this PR. Both readers still pick by head_sha alone: ci-status (.github/workflows/pr-require-checks.yml:4217, :4297-4299) and rerun-ci-status (.github/workflows/pr-feedback-dispatch.yml:335, :472-481). The pr-unblock dispatchers already match runs by display_title (.github/scripts/pr-unblock/fix-docs-dispatch.mjs:210), so this reuses that pattern.
    2. Item 2: the kill switch's pass.
      Decided: A lane stopped by a kill switch passes only on a PR a trusted person authored, and only when the switch was read as on, not when check-kill-switch failed closed on an unreadable variable. Today a stop counts on the report step's notice (pr-require-checks.yml:4212-4213), and the report step runs whenever proceed is not true (.github/workflows/pr-review-hosted.yml:125, :151-157). Of the body's two options this is the narrower one: pr-merge is unchanged and a person's PRs still merge; only agent-authored PRs wait for a review, which is the gap the body names ("a review-only stop leaves agent PRs merging with no AI review"). The pr-merge-refuses option also stops merges of PRs a person wrote, so it is not taken. OD-8's own text was not found in this repository or .work/; the body reports it chose no merge freeze, and neither part of this decision freezes pr-merge.
    3. Item 3: the gate runs from the PR's own copy of its workflows.
      Decided: No change. hold-paths "never holds by itself" (pr-review-hosted.yml:197); adding files to it only adds review scrutiny. A rewritten gate reaches a merge only through a PR pr-merge lists, which is a lanes App, trusted-actor or Dependabot PR (.github/workflows/pr-merge.yml:620-627), and a lane patch cannot touch .github/workflows/**: every patch row refuses github-workflows (.github/actions/resolve-config/patch-rules.json:3, :14, :24, :35, :54, :64).
    4. Item 4: an exempt author's PR after a pushed fix.
      Decided: ci-status exempts a PR from the review requirement by author only when every commit on it is authored by an exempt account. Today the check reads only the PR author's id (pr-require-checks.yml:4230, :4248). This keeps fix-ci working on those PRs and makes the review wait once an agent commit lands. The review lanes do run after such a push: they skip a [bot] sender only when its id is not in allowed-bot-sender-ids, which holds the lanes App (pr-review-hosted.yml:16-18, :183), so the wait ends.
    5. Item 5: hold label name divergence.
      Decided: ci-status also fails on the literal needs-human label, besides the tracker's human-gated label. The callers hardcode hold-label: needs-human (pr-review-hosted.yml:199), and ci-status reads the tracker value (pr-require-checks.yml:4023-4034). A workflow with: cannot read the tracker file, so the literal check is the smaller change.
    6. Item 6: the rerun listener unproven live.
      Decided: Check it from run history, as a criterion below. Basis: the body says rerun-ci-status runs from main, so the PR that added it could not exercise it.

    Build: host session (every item but 6 edits .github/workflows/**, which lane patches refuse, .github/actions/resolve-config/patch-rules.json:64).

    Acceptance criteria:

    • Both review callers set a run-name with the PR number, and ci-status's review-lane step and rerun-ci-status count only runs whose display_title names this PR; a test shows a second PR's review on the same head SHA is not counted.
    • A kill-switch stop counts as a pass only when the switch was read as on and a trusted person authored the PR; a test shows an unreadable switch and an agent-authored PR both fail.
    • The author exemption applies only when every commit on the PR is authored by an exempt account; a test shows an exempt PR with a lanes App commit needs both review lanes.
    • ci-status fails on a needs-human label even when the tracker names a different human-gated label; a test covers it.
    • The PR links one pr-feedback-dispatch.yml run whose rerun-ci-status job re-ran ci-status on a PR after a review run completed.

    Builder: the Automation Lanes session builds this from the owner's machine, because the fix edits .github/workflows, which the lanes App cannot push. agent-ready stays off so the issue lane does not race the build.

  5. added
    status: confirmedA maintainer confirmed the item is real and in scope; ready to be worked.
    and removed
    status: needs-decisionAwaiting a human or maintainer judgment call.
    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.
    on Oct 26, 2026
  6. kyle-sexton commented on Oct 26, 2026

    @kyle-sexton
    MemberAuthor

    Released for re-triage under the owner's 2026-10-26 rule: a person is involved only for "1. A change in original intent/vision 2. Something comes up during implementation ... that alters intent/vision or the primary design 3. If there is a HIGH/CRITICAL security issue". None applies here, so the hold label came off and intake triage runs again now.

    To hold this issue again, add needs-human (or status: needs-decision) and name which of the three applies.

  7. melodic-automation-lanes commented on Oct 26, 2026

    @melodic-automation-lanes
    Contributor

    Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. On an issue a listed human filed, it counts as the issue's spec once posted, while unedited; it is still model text, so check it.

    Intended outcome: Tighten the review-lane gate so reviews are matched per PR, kill-switch stops and author exemptions pass only on safe cases, and the needs-human label always holds, per the owner's recorded decision on items 1, 2, 4 and 5.

    Acceptance criteria

    • Both review callers set a run-name containing the PR number, and ci-status's review-lane step (pr-require-checks.yml) and rerun-ci-status (pr-feedback-dispatch.yml) count only runs whose display_title names this PR.
    • A test shows a second PR's review run on the same head SHA is not counted for this PR.
    • A kill-switch stop counts as a lane pass only when the switch was read as on and a trusted person authored the PR.
    • A test shows an unreadable kill switch (check-kill-switch failed closed) fails the lane, and an agent-authored PR with a stopped lane also fails.
    • The author exemption from the review requirement applies only when every commit on the PR is authored by an exempt account.
    • A test shows an exempt-author PR carrying a lanes App commit needs both review lanes.
    • ci-status fails on a needs-human label even when .work-item-tracker.json names a different human-gated label, and a test covers it.
    • Item 3 (hold-paths and default-branch gate code) is left unchanged, and pr-merge.yml is not changed.
    • The PR links one pr-feedback-dispatch.yml run whose rerun-ci-status job re-ran ci-status on a PR after a review run completed (item 6, from run history).

    Dependencies

    • Host session build: every edit is under .github/workflows/**, which lane patches refuse (patch-rules.json:64), so the Automation Lanes session builds it with agent-ready off.
    • Item 6 waits on a real pr-feedback-dispatch.yml run after the run-name change merges; the unattended run cannot verify it live.

    Related work

    Sources read

    • /home/runner/work/_temp/refine-data/context.json: the owner's Decision recorded comment fixes the approach for items 1, 2, 4 and 5, leaves item 3 unchanged, and checks item 6 from run history.
    • .base/.github/workflows: only pr-refine.yml, work-resolve-issue.yml and pr-unblock.yml use run-name or display_title, so the review callers have no PR-numbered run-name yet.

    Automated refinement by the lanes App. Intake triages the issue again with it.

  8. removed
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    priority: mediumReal value, no hard deadline; normal backlog flow.status: confirmedA maintainer confirmed the item is real and in scope; ready to be worked.status: readyTriaged, unblocked, and fully specified; eligible to pick up.work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.

    Type

    Fields

    No fields configured for Task.

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions