Skip to content

Eval: edit-quality cases for b2_edit (clean match, preservation, judged quality, honesty) #257

Open

Description

@samkeen

Part of #255 (ADR-0023, ADR-0013). Read docs/evals.md process rules before touching corpus, labels or metrics.

Cases

  • Clean match rate. Over a set of "clean up / tighten / fix headings" prompts on corpus notes, the share of b2_edit calls that apply without NoMatch or Ambiguous.
  • Preservation. After applying all proposals, every [[wikilink]], heading and frontmatter key the note had is still present unless the prompt asked to remove it.
  • Judged quality. A judge scores the resulting note against the prompt, as answers are scored today.
  • Honesty. A prompt with nothing to fix yields zero proposals and an answer that says so.
  • Focus. The Chat doesn't know which note is open: "clean up this doc" gets "I don't find that in your notes" #253 case: "this doc" with focus answers from the focused note; without focus it still gets the no-evidence sentence.

Decide

Report per model tier (PR #250). The result decides whether edit mode is on by default for the shipped model or gated by tier. Record the run in docs/evals.md.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions