[apucacao]

Why

A judge is an ordinary AI Config, so it can carry outputFormat (a JSON Schema). Two code paths read that one field with opposite intent:

• judge_scoring.py ignores it. It appends its formatting instructions and parses the verdict as { score: number 0-1, reasoning: string }. The verdict contract belongs to the SDK.
• The provider handler honors it. Every judge goes through execute_and_track, which hands the whole judge config to the handler. openai-messages sets text.format = json_schema from config.outputFormat; claude and langchain handlers read it too.

So the model is told to produce two different shapes, obeys the provider-level schema, and the judge parser then rejects the verdict. The judge produces no score and tracks no metric, silently.

This is live. A read-only sweep of 215 AI Config variations across four LaunchDarkly connections found 30 with mode: judge, and one of those sets outputFormat to { message: string, script: string }. Nothing about a score. That judge can never return a valid verdict, and nothing warns its author.

Fix

Strip outputFormat from a judge config before it can reach a handler or be stored on a JudgeTask. One helper, without_output_format, lives in judge_scoring.py next to the rest of the verdict contract, and is applied at the four places a judge config is handed off:

• run_judges, before execute_and_track
• build_judge_tasks, before the config is persisted on the task
• run_judge, defensively, so an older serialized task still behaves
• the offline evaluations runner (evaluations/runner.py), when it resolves each judge, so evals.run(...) judges are covered too

When the field was actually present, say so once per judge. The helper returns the same reference when the field is absent and never mutates the input, which may be a cached extract_variation result.

Not in scope, deliberately: the score contract, the parser, the formatting instructions, and handler selection are untouched. Making the SDK honor a judge's schema, or substituting the SDK's own { score, reasoning } schema to get structured output for judges, are both follow-ups.

🤖 Generated with Claude Code

---

Note

Overview
Fixes judges that silently produced no score when their AI Config defined outputFormat: provider handlers enforced that schema while the SDK still parsed {score, reasoning}, so verdicts could never validate.

Adds without_output_format in judge_scoring.py to drop outputFormat from judge configs (non-mutating, one warning per judge when present) and wires it through inline run_judges, deferred build_judge_tasks / run_judge, and offline evaluations judge resolution so handlers never see a judge-owned schema.

AGENTS.md now states that judge outputFormat is ignored and belongs on the config being evaluated, not the judge. Tests cover handler visibility, task serialization, defensive run_judge, and the evaluations path.

^{Reviewed by Cursor Bugbot for commit 2eb0534. Bugbot is set up for automated code reviews on this repo. Configure here.}

[apucacao]

bugbot run

[apucacao]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 627f1fa. Configure here.}

[jeffdupont]

This looks right to me, and it covers all four judge paths, including offline evaluations (evaluations/runner.py:373).

I'm holding off on approving only because the spec has no rule for this yet. §8.8.3 says a judge config has no tools but says nothing about outputFormat, and §1.9 says handlers enforce outputFormat whenever it is present. A monorepo PR adding the rule to §3.13 and §8.8.3, with a test sentence, would give js#56 (which misses the offline path) something to test against. I'll approve once that's up.

[apucacao]

@jeffdupont the spec is up in ai-sdks-monorepo#82: §3.13 has the rule and required tests for the inline and deferred paths, §8.8.3 has the offline rule (one warning per judge per run) and its test, and §1.9 notes judges as the exception to handlers enforcing outputFormat. js#56 now covers the offline path as well.