Skip to content

Python: Pass judge model in evaluate_traces sample - #9576

Merged
Eduard van Valkenburg (eavanvalkenburg) merged 1 commit into
microsoft:mainfrom
1aifanatic:contrib/python-evaluate-traces-sample-model
Oct 22, 2026
Merged

Eduard van Valkenburg (eavanvalkenburg) merged 1 commit into
microsoft:mainfrom
1aifanatic:contrib/python-evaluate-traces-sample-model

Conversation

@1aifanatic

Copy link
Copy Markdown
Contributor

Motivation & Context

samples/05-end-to-end/evaluation/foundry_evals/evaluate_traces_sample.py cannot run. evaluate_traces() declares model as a required keyword-only argument (the deployment Foundry uses as the evaluator LLM judge), but both calls in the sample omit it:

TypeError: evaluate_traces() missing 1 required keyword-only argument: 'model'

The error is raised before any request reaches Foundry, so every user who follows the README for this sample hits it. 05-end-to-end is excluded from the samples type-check sweep (SAMPLE_TYPING_EXCLUDES), which is why CI did not catch the drift.

Description & Review Guide

  • What are the major changes? Read FOUNDRY_MODEL once into model and pass it to the FoundryChatClient and to both evaluate_traces() calls, plus the commented-out agent_id example, so it stays correct when uncommented.
  • What is the impact of these changes? Sample only. It now matches the documented evaluate_traces() signature and its docstring example.
  • What do you want reviewers to focus on? Whether reusing the chat deployment as the judge model is the right default for this sample (it matches how the sibling Foundry eval samples use FOUNDRY_MODEL).

Verified locally: Pyright on the file reports 0 errors (previously Argument missing for parameter "model" twice); ruff check / ruff format --check pass. Running main() with the Foundry evaluation call stubbed now reaches both evaluations, with model="gpt-4o" forwarded.

Related Issue

N/A - trivial change: the sample omits a required argument and fails with TypeError on startup. The fix only supplies that argument as documented in evaluate_traces()'s own docstring, so the desired result is not in dispute.

AI Assistance

  • No material AI assistance was used.
  • This is an AI-assisted contribution. I reviewed, understood, and verified all submitted content and accept responsibility for it.

AI assistance details: Claude Code helped find the broken sample (by type-checking the excluded 05-end-to-end samples), implement the fix, run the local verification, and draft this description.

Contribution Checklist

  • The code builds clean without any errors or warnings
  • All unit tests pass, and I have added new tests where possible
  • The PR follows the Contribution Guidelines
  • This PR links to an agreed issue with no competing open PR, or the Related Issue section documents a trivial-change or repository-automation exception.
  • This is not a breaking change. If it is a breaking change, add the breaking change label (or add "[BREAKING]" to the title prefix, before or after any language prefix) — a workflow keeps the label and title prefix in sync automatically.

🤖 Generated with Claude Code

evaluate_traces() requires a keyword-only `model` argument for the
evaluator LLM judge. The sample omitted it, so both calls raised
TypeError before reaching Foundry. Reuse the FOUNDRY_MODEL deployment
the sample already reads for the chat client.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The sample-only fix matches the required API signature and existing usage, with no unresolved findings.

No open issues 🎉

What changed in this PR

Fixes the Foundry trace-evaluation sample by supplying the required judge model argument.

Changes:

  • Reuses FOUNDRY_MODEL for the chat client and both evaluation calls.
  • Updates the commented agent_id example to include the model argument.
File Description
python/​samples/​05-end-to-end/​evaluation/​foundry_evals/​evaluate_traces_sample.py Passes the shared model deployment to trace evaluations.

Review effort: Balanced

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The sample now supplies the required judge model to every evaluate_traces() call and reuses the existing FOUNDRY_MODEL deployment consistently. This is a focused fix for a deterministic startup error.

Merged via the queue into microsoft:main with commit 621e8e3 Oct 22, 2026
52 checks passed

This branch was successfully deployed

1 active deployment
github-app-auth — 2b05b33f Deployed Oct 21, 2026 by 1aifanatic via team_check #7160
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

python Usage: [Issues, PRs], Target: Python

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants