Repository navigation
Python: Pass judge model in evaluate_traces sample - #9576
Merged
Eduard van Valkenburg (eavanvalkenburg) merged 1 commit intoOct 22, 2026
Conversation
evaluate_traces() requires a keyword-only `model` argument for the evaluator LLM judge. The sample omitted it, so both calls raised TypeError before reaching Foundry. Reuse the FOUNDRY_MODEL deployment the sample already reads for the chat client. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Naveen Chatlapalli (1aifanatic)
requested review from
SergeyMenshykh,
Tao Chen (TaoChenOSU),
Eduard van Valkenburg (eavanvalkenburg),
Giles Odigwe (giles17),
Jose Alvarez (jpalvarezl),
Evan Mattson (moonbox3),
Roger Barreto (rogerbarreto),
sophia-ramsey and
westey (westey-m)
as code owners
October 21, 2026 14:21
Naveen Chatlapalli (1aifanatic)
deployed
to
github-app-auth
October 21, 2026 14:21 — with
GitHub Actions
Active
Naveen Chatlapalli (1aifanatic)
deployed
to
github-app-auth
October 21, 2026 14:21 — with
GitHub Actions
Active
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
The sample-only fix matches the required API signature and existing usage, with no unresolved findings.
No open issues 🎉
What changed in this PR
Fixes the Foundry trace-evaluation sample by supplying the required judge model argument.
Changes:
- Reuses
FOUNDRY_MODELfor the chat client and both evaluation calls. - Updates the commented
agent_idexample to include the model argument.
| File | Description |
|---|---|
| python/samples/05-end-to-end/evaluation/foundry_evals/evaluate_traces_sample.py | Passes the shared model deployment to trace evaluations. |
Review effort: Balanced
Eduard van Valkenburg (eavanvalkenburg)
approved these changes
Oct 22, 2026
Eduard van Valkenburg (eavanvalkenburg)
left a comment
Member
There was a problem hiding this comment.
The sample now supplies the required judge model to every evaluate_traces() call and reuses the existing FOUNDRY_MODEL deployment consistently. This is a focused fix for a deterministic startup error.
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation & Context
samples/05-end-to-end/evaluation/foundry_evals/evaluate_traces_sample.pycannot run.evaluate_traces()declaresmodelas a required keyword-only argument (the deployment Foundry uses as the evaluator LLM judge), but both calls in the sample omit it:The error is raised before any request reaches Foundry, so every user who follows the README for this sample hits it.
05-end-to-endis excluded from the samples type-check sweep (SAMPLE_TYPING_EXCLUDES), which is why CI did not catch the drift.Description & Review Guide
FOUNDRY_MODELonce intomodeland pass it to theFoundryChatClientand to bothevaluate_traces()calls, plus the commented-outagent_idexample, so it stays correct when uncommented.evaluate_traces()signature and its docstring example.FOUNDRY_MODEL).Verified locally: Pyright on the file reports 0 errors (previously
Argument missing for parameter "model"twice);ruff check/ruff format --checkpass. Runningmain()with the Foundry evaluation call stubbed now reaches both evaluations, withmodel="gpt-4o"forwarded.Related Issue
N/A - trivial change: the sample omits a required argument and fails with
TypeErroron startup. The fix only supplies that argument as documented inevaluate_traces()'s own docstring, so the desired result is not in dispute.AI Assistance
AI assistance details: Claude Code helped find the broken sample (by type-checking the excluded
05-end-to-endsamples), implement the fix, run the local verification, and draft this description.Contribution Checklist
breaking changelabel (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.🤖 Generated with Claude Code