test(source-control): add eval cases for babysit-prs, babysit-loop and setup #7148

Activity

1. kyle-sexton

   added

   needs-triageNot yet classified. Floor until a type and one priority tier are set.Not yet classified. Floor until a type and one priority tier are set.

   on Oct 11, 2026
2. github-actions commented on Oct 11, 2026

   

   github-actionsBot

   on Oct 11, 2026 – with GitHub Actions

   Contributor

   More actions

   Thanks for the writeup. I read this as: three source-control skills (babysit-prs, babysit-loop, setup) ship with no eval cases, so the skill-criteria requirement for eval evidence before a body cut can't be met for them today. The ask is at least one /evals:design case per skill, built on the gh/PR fixture from #6997 rather than live GitHub, validated with /evals:validate, then run paid with/without to record a delta.

   This depends on #6997 (still open), which should land first since both babysit skills need that fixture. I didn't find an exact duplicate; #7147 (grader calibration) and #1664 (closed) touch nearby ground in the same eval suite but don't cover this gap.

   _{Automated triage. A maintainer reviews it.}
3. github-actions

   added

   priority: mediumReal value, no hard deadline; normal backlog flow.Real value, no hard deadline; normal backlog flow.

   on Oct 11, 2026
4. kyle-sexton

   removed

   needs-triageNot yet classified. Floor until a type and one priority tier are set.Not yet classified. Floor until a type and one priority tier are set.

   on Oct 12, 2026
5. kyle-sexton

   added the

   Task

   issue type on Oct 12, 2026
6. kyle-sexton commented on Oct 13, 2026

   

   kyle-sexton

   on Oct 13, 2026

   MemberAuthor

   More actions

   Owner decision (2026-10-13): approved the recommendation.

   • Build the babysit cases now, or wait for test(source-control): put the eval gh fixture on PATH without relying on zsh's .zshenv #7176? → Build now on fixtures/gh-forge/install.sh, and in the same change extend the shared gh stand-in with the commands the babysit cycle calls (gh pr merge, gh api graphql for review threads, gh api user, gh search prs, gh pr review). setup needs no gh fixture.
     Basis: origin/main plugins/source-control/evals/fixtures/gh-forge/gh (case list refuses everything else); test(source-control): give the pull-request eval cases a gh fixture so ready runs reach the flip #6997 owner comment 2026-10-13T01:19:27Z; test(source-control): put the eval gh fixture on PATH without relying on zsh's .zshenv #7176 OPEN
   • Where and when do the paid with/without runs happen? → Approved: at least 3 skills x 2 arms x 3 runs, on the laptop WSL shell once its sandbox reports ready, or on the hosted eval-browser-dispatch lane after test(source-control): put the eval gh fixture on PATH without relying on zsh's .zshenv #7176 lands. Not on the desktop WSL host (its ~/.docker links refuse Bash-granting runs), and not on the hosted lane before test(source-control): put the eval gh fixture on PATH without relying on zsh's .zshenv #7176, because install.sh reaches PATH only through ~/.zshenv and a bash run would reach the real gh.
     Basis: origin/main fixtures/gh-forge/install.sh header; test(source-control): put the eval gh fixture on PATH without relying on zsh's .zshenv #7176 OPEN
   • Can the cycle run far enough offline? → Not yet verified; a kept-trace run after the stand-in extension settles it.
     Basis: test(source-control): give the pull-request eval cases a gh fixture so ready runs reach the flip #6997 owner comment 2026-10-13T01:19:27Z

   Next: an agent lane, via admission, builds the three cases and the stand-in extension and validates them with /evals:validate. The paid runs follow on one of the two hosts above.

   This was generated by AI.
7. kyle-sexton commented on Oct 13, 2026

   

   kyle-sexton

   on Oct 13, 2026

   MemberAuthor

   More actions

   Owner batch-ratified this admission in an attended session on 2026-10-12: work-class: scoped. Rule: class C1-C3, no trust surface, the independent check confirmed the recorded decision, and no owner step remains.
8. kyle-sexton

   added

   work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.A briefed fix or small feature; blast radius bounded by the brief, tests exist.

   agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.Fully specified and briefed; eligible for autonomous pickup from the frontier.

   and removed

   agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.Fully specified and briefed; eligible for autonomous pickup from the frontier.

   on Oct 13, 2026
9. kyle-sexton

   added sub-issues on Oct 18, 2026

   • test(source-control): serve the babysit cycle's gh commands from the gh-forge stand-in #7996
   • test(source-control): add a babysit-prs eval case that the safe tier never resolves or merges #7997
   • test(source-control): add a setup eval case that writes only what the user confirmed #7998
10. kyle-sexton commented on Oct 18, 2026

    

    kyle-sexton

    on Oct 18, 2026

    MemberAuthor

    More actions

    This was generated by AI during backlog shaping.

    This issue is split into 4 pieces, each sized for one lane PR. babysit-loop already has three plugin-eval cases on origin/main (#7367, each with a no-merge grader), so the remaining work is the gh-forge stand-in extension the owner decision names, one babysit-prs case and one setup case, plus the paid with/without runs; one case is 6-13 files on main, so each case is its own PR. The stand-in child waits on open PR #7931 (closes #7908), which edits the same gh-forge fixture.

    • filed: test(source-control): serve the babysit cycle's gh commands from the gh-forge stand-in (test(source-control): serve the babysit cycle's gh commands from the gh-forge stand-in #7996)
    • filed: test(source-control): add a babysit-prs eval case that the safe tier never resolves or merges (test(source-control): add a babysit-prs eval case that the safe tier never resolves or merges #7997)
    • filed: test(source-control): add a setup eval case that writes only what the user confirmed (test(source-control): add a setup eval case that writes only what the user confirmed #7998)
    • not filed: test(source-control): run the babysit-prs, babysit-loop and setup cases paid with and without the plugin: paid-run: at least 3 skills x 2 arms x 3 runs on the laptop WSL shell or the hosted eval-browser-dispatch lane (owner decision 2026-10-13)

    Filed pieces are sub-issues of this one, and the paced queue admits them to the lane. Unfiled pieces wait on the blocker shown. The owner approved this split in an attended backlog-shaping session on 2026-10-18.
11. kyle-sexton

    added

    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.Human-in-the-loop required; autonomous sessions must not resolve items carrying this.

    on Oct 19, 2026
12. kyle-sexton commented on Oct 19, 2026

    

    kyle-sexton

    on Oct 19, 2026

    MemberAuthor

    More actions

    This was generated by AI during triage.

    Held: the case-building pieces are filed as #7996, #7997 and #7998. What remains here is the paid with/without runs (at least 3 skills x 2 arms x 3 runs) the owner placed on the laptop WSL shell or the hosted eval lane.