Repository navigation
[dotnet-port-api] Add AI-judge loop evaluator - #1162
PratikDhanave (PratikDhanave) wants to merge 3 commits into
Conversation
Port the .NET AIJudgeLoopEvaluator: a loop evaluator that queries a separate
judge agent after each iteration to decide whether the user's original request
was fully addressed, continuing the loop with the judge's gap analysis as
feedback while the answer is "no".
The judge is queried with the original request contents and the agent's latest
response, and asked for a structured JudgeVerdict {answered, gapAnalysis}. When
the judge does not return structured output, the verdict falls back to the
non-overlapping VERDICT: DONE / VERDICT: MORE markers (MORE wins when ambiguous
or absent, so the loop keeps running). Criteria render into the judge
instructions at {criteria}, and the feedback template substitutes the gap
analysis at {gap_analysis}.
github-actions
Bot
added
area:agent
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
Three moderate evaluator issues must be addressed before approval.
Review effort: Lite
Findings: None
What changed in this PR
Adds a Go AIJudgeEvaluator that iteratively judges agent responses, supports criteria and gap-analysis feedback, and falls back to text verdict markers.
Changes:
- Added evaluator configuration and verdict parsing.
- Added criteria and feedback-template rendering.
- Added structured, fallback, and nil-judge tests.
| File | Summary |
|---|---|
agent/harness/loop/loop_test.go |
Tests verdict handling, fallbacks, criteria, and nil judges. |
agent/harness/loop/evaluators.go |
Implements the evaluator. Three moderate findings remain: native structured output is not requested, non-text response content is omitted, and JSON extraction can reject valid verdicts surrounded by other JSON. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
github-actions
Bot
added
kind:code
This comment has been minimized.
This comment has been minimized.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Generated by Go API Consistency Review Agent · copilot · auto · 153.5 AIC · ⌖ 7.82 AIC · ⊞ 9.2K
| } | ||
| contents = append(contents, &message.TextContent{Text: "\n\n## Agent's latest response:\n" + loop.LastResponse.String()}) | ||
|
|
||
| resp, err := e.judge.Run(ctx, []*message.Message{{Role: message.RoleUser, Contents: contents}}, agent.WithInstructions(e.instructions)).Collect() |
There was a problem hiding this comment.
Parity gap vs. the upstream .NET AIJudgeLoopEvaluator:
-
No judge isolation.
NewAIJudgeEvaluatortakes a full*agent.AgentandEvaluateruns it throughe.judge.Run(...), which goes through the entire agent pipeline (default in-memory history provider, anyContextProviders,Tools,Middlewaresconfigured on that agent). Upstream'sAIJudgeLoopEvaluator(IChatClient judgeClient, ...)intentionally takes a rawIChatClientso the judge is, per its doc remarks, "queried directly (without any agent tools, session, or middleware)". Seedotnet/src/Microsoft.Agents.AI/Harness/Loop/AIJudgeLoopEvaluator.cs. If a caller passes a Go agent that has tools or session/history configured, the judge could execute tools or leak state across loop iterations — a structural guarantee the .NET type enforces at the type level that Go does not. Consider constraining the accepted judge type (e.g. a raw provider/run function) or explicitly documenting/enforcing the no-tools/no-session expectation. -
No native structured-output request. Upstream calls
judgeClient.GetResponseAsync<JudgeVerdict>(judgeMessages, LoopJsonContext.Default.Options, ...), asking the provider for structured JSON output before falling back to text-marker parsing only when the client doesn't honor it (seeTryGetResultbranch in the same file). This GoEvaluateonly passesagent.WithInstructions(e.instructions)and never usesagent.WithStructuredOutput/agent.WithResponseFormat(already available inagent/options.go/agent/structuredoutput.go), so it always falls back to manual{/}substring scraping (extractJudgeVerdict) rather than requesting the provider's native structured-output support like .NET does.
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
Warning
Firewall blocked 2 domains
The following domains were blocked by the firewall during workflow execution:
github.comraw.githubusercontent.com
To allow these domains, add them to the network.allowed list in your workflow frontmatter:
network:
allowed:
- defaults
- "github.com"
- "raw.githubusercontent.com"See Network Configuration for more information.
Generated by Go API Consistency Review Agent for #1162 · copilot · auto · 236.9 AIC · ⌖ 8.05 AIC · ⊞ 13.9K
| } | ||
| contents = append(contents, &message.TextContent{Text: "\n\n## Agent's latest response:\n" + loop.LastResponse.String()}) | ||
|
|
||
| resp, err := e.judge.Run(ctx, []*message.Message{{Role: message.RoleUser, Contents: contents}}, agent.WithInstructions(e.instructions)).Collect() |
There was a problem hiding this comment.
Judge is never asked for structured output, so the primary verdict path is effectively unreachable.
Upstream, both the .NET AIJudgeLoopEvaluator and Python AgentLoopMiddleware.with_judge explicitly request structured output from the judge client and only fall back to the VERDICT: DONE/VERDICT: MORE text markers when the client doesn't honor it:
- .NET:
this._judgeClient.GetResponseAsync<JudgeVerdict>(judgeMessages, LoopJsonContext.Default.Options, ...)(AIJudgeLoopEvaluator.cs) - Python:
judge_client.get_response(judge_messages, options={"response_format": response_format})withresponse_format: ... = JudgeVerdict(_loop.py)
Here, e.judge.Run(ctx, ..., agent.WithInstructions(e.instructions)) never passes agent.WithResponseFormat(...) (e.g. via jsonformat.MustFor[loop.JudgeVerdict](), the pattern used in examples/02-agents/agents/step05_structured_output). extractJudgeVerdict only succeeds by incidentally finding a {...} blob with an "answered" key in free-form prose — most judge models will instead follow the text-marker instructions literally, so the structured-output branch (and the richer GapAnalysis it carries) will rarely fire in practice, diverging from both upstream implementations' designed behavior.
Suggested fix: request structured output from the judge (e.g. agent.WithResponseFormat(jsonformat.MustFor[JudgeVerdict]()) or agent.WithStructuredOutput(&verdict)), falling back to extractJudgeVerdict/text markers only when the provider doesn't support it or decoding fails.
…evaluator # Conflicts: # agent/harness/loop/loop_test.go
github-actions
Bot
added
the
parity-approved
API consistency review
Changed Go contractNew exported Upstream evidence reviewed
AssessmentConstants, defaults, criteria rendering, feedback substitution and text-marker fallback match .NET, and both are opt-in. Two differences are idiomatic and not parity issues. Go takes a judge
|
Ports the .NET
AIJudgeLoopEvaluator(Harness/Loop). Go's loop harness previously shipped onlyCompletionMarkerEvaluator; the parity doc lists the AI-judge evaluator as a gap.What it does
AIJudgeEvaluatorqueries a separate judge agent after each iteration to decide whether the user's original request has been fully addressed, continuing the loop (with the judge's gap analysis as feedback) while the answer is "no".JudgeVerdict{answered, gapAnalysis}.VERDICT: DONE/VERDICT: MOREtext markers, withMOREwinning when the verdict is ambiguous or absent (so the loop keeps running rather than stopping on an incomplete answer) — matching .NET.Criteriarender into the judge instructions at{criteria}; the feedback template substitutes the gap analysis at{gap_analysis}.Security
Mirrors the .NET type's security note in the doc comment: the judge is sent the request and latest response every iteration, so only use a judge you trust as much as the primary model, and prefer a stricter
MaxIterations.Tests