Skip to content

test(source-control): add a setup eval case that writes only what the user confirmed #7998

Closed

Description

@kyle-sexton

Refs: #7148

Agent Brief

Type: Task
Summary: One plugin-eval case shows source-control setup writes only the settings the user confirmed.

Current behavior:
plugins/source-control/evals has no setup case.

Desired behavior:
A case in the layout of the existing source-control cases runs setup in a scratch repository where the prompt confirms one change and declines another; deterministic graders check that only the confirmed change is written. No gh fixture is needed.

Key interfaces:

  • The plugin-eval case layout.
  • The setup skill's confirm-before-write contract.

Acceptance criteria:

  • The case directory exists with case.yaml, prompt.md, scaffold and graders.
  • A grader fails when the declined change appears in the workspace.
  • A source-control changelog fragment is added and scripts/check-changelog-fragments.sh passes.

Out of scope:

  • babysit cases and the gh stand-in.
  • Paid runs.

This was generated by AI during backlog shaping. Split from #7148; the owner approved filing it in an attended backlog-shaping session on 2026-10-18.

Activity

  1. added
    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.
    work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.
    on Oct 18, 2026
  2. added theissue type on Oct 18, 2026
  3. added
    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.
    and removed
    needs-humanHuman-in-the-loop required; autonomous sessions must not resolve items carrying this.
    on Oct 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.work-class: scopedA briefed fix or small feature; blast radius bounded by the brief, tests exist.

    Type

    Fields

    No fields configured for Task.

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions