Support the current azd ai eval command surface

[placerda]

Problem

AgentOps Accelerator currently delegates execution: azd to the legacy azd ai agent eval command surface and discovers eval.yaml recipes.

Current Microsoft Foundry guidance uses the azure.ai.evaluations extension with:

• azd ai eval
• evals/azure.eval.yaml
• azd ai eval create
• azd ai eval run start
• azd ai eval run output list

This gap prevents the AgentOps VBD Evaluate lab from using agentops eval run as the evidence and release-readiness wrapper around the current Foundry cloud evaluation flow.

Related VBD work: #483

Goal

Support the current azd ai eval surface without breaking existing azd ai agent eval integrations.

Scope

• Discover legacy eval.yaml and current evals/azure.eval.yaml recipes
• Detect and parse both recipe schemas
• Add an adapter for azd ai eval create, run start, and run output list
• Normalize run metrics, rubric dimensions, failed samples, and execution errors into the existing results.json contract
• Preserve threshold binding, fail-closed behavior, baseline comparison, CI exit codes, and report.md
• Retain the raw azd result for troubleshooting and audit
• Update agentops eval analyze and initialization guidance
• Preserve the existing azd ai agent eval adapter for backward compatibility
• Update tests, documentation, examples, and changelog

Acceptance criteria

• A Foundry hosted agent can be evaluated through agentops eval run using evals/azure.eval.yaml
• The cloud evaluation runs once through the current azd ai eval surface
• AgentOps produces normalized results.json and report.md
• Thresholds and rubric dimensions bind correctly and missing metrics fail closed
• Baseline comparison and CI exit-code behavior remain available
• Raw azd output is retained for diagnostics
• Legacy azd ai agent eval recipes and tests continue to work
• Documentation identifies required azd and extension versions

[placerda]

Shipped in v0.15.0 (agentops-accelerator==0.15.0 on PyPI) via #485.

Acceptance criteria

• A Foundry hosted agent can be evaluated through agentops eval run using evals/azure.eval.yaml — implemented and covered end to end through the CLI. See the caveat below on live validation.
• The cloud evaluation runs once through the current azd ai eval surface — asserted explicitly: exactly one run start per invocation, and result retrieval never re-executes the evaluation.
• AgentOps produces normalized results.json and report.md — same schema as every other execution mode. The current surface additionally fills rows with one entry per sample, including failed and errored ones, where the legacy adapter stays aggregate-only.
• Thresholds and rubric dimensions bind correctly and missing metrics fail closed — binding is two-stage. A threshold naming a metric no evaluator in the recipe can produce is a configuration error caught before any azd command runs (exit 1), so a typo never bills a cloud run. A declared metric the run did not emit is recorded as a failed threshold (exit 2). Rubric dimension ids bind exactly like builtin metrics.
• Baseline comparison and CI exit-code behavior remain available — --baseline works unchanged, and the full 0/2/1 matrix is asserted at CLI level.
• Raw azd output is retained for diagnostics — the run payload, per-sample items, and both command streams are written for successful and failed runs, including failures at poll, retrieval, or normalization.
• Legacy azd ai agent eval recipes and tests continue to work — every pre-existing test passes unmodified (191 lines added to existing test files, 0 removed), and a dedicated regression guard proves a legacy workspace behaves identically and never dispatches to the new surface.
• Documentation identifies required azd and extension versions — docs/evaluation.md, docs/how-it-works.md, and docs/tutorial-hosted-agent.md all state the extension ids and the azd 1.27.1 floor.

How the surface is selected

The recipe decides. No new command, flag, or config field:

Recipe	azd commands	Extension
evals/azure.eval.yaml	azd ai eval	azure.ai.evaluations (azd 1.27.1+)
eval.yaml, src/<agent>/eval.yaml	azd ai agent eval	azure.ai.agents

Classification is by file content, not filename. When both are discoverable the current surface wins and the run reports which recipe it skipped.

Important caveat: the upstream extension is not published yet

azure.ai.evaluations still exists only as an unmerged pull request against Azure/azure-dev (#9500). It is absent from the azd extension registry, so azd extension install azure.ai.evaluations does not resolve today.

Every command name, flag, and JSON field was read from that PR branch's source and its formal JSON schema rather than guessed, and all of it is confined to src/agentops/pipeline/azd_eval_runner.py so an upstream change stays a localized fix. But live end-to-end validation against the real extension was not possible, and the surface can still change before it merges. Everything shipped is verified through a mocked subprocess boundary across 6 platform/version combinations.

The new path is strictly opt-in: it activates only when a current-surface recipe exists. A workspace without evals/azure.eval.yaml behaves exactly as before, and agentops eval init keeps generating a legacy recipe while the extension is unavailable, so nobody ends up with a workspace that fails on a dependency they cannot install.

To exercise it today, the extension has to be built from the PR branch and published into a local azd extension source — see the walkthrough in docs/tutorial-hosted-agent.md.

Incidental fix included

azd extension detection was reading the human-readable azd extension list output, which lists every registry extension including uninstalled ones. Verified live: 9 extensions installed vs 15 in the registry, with azure.ai.finetune matching the text scan while not being installed. Detection now parses azd extension list --installed -o json and matches on extension id.

---

Design rationale, verified upstream facts with sources, the interface contract, and the validation guide are under specs/011-azd-ai-eval-surface/.

Closing as completed. Please reopen if the upstream surface changes and the integration needs adjusting.