feat(eval): support the current azd ai eval command surface - #485
Merged
Conversation
Adds the current-surface recipe model (evals/azure.eval.yaml, azure.ai.evaluations) alongside the existing legacy one, plus content-based classification and cross-surface discovery. Recipes are classified by content, never by filename: a sequence-valued 'evals' root is the current surface, a mapping-valued 'agent' or a 'dataset_reference' key is legacy. Discovery precedence is explicit eval_recipe, then the current surface (recording what it skipped), then legacy; more than one candidate within a surface stays ambiguous as before. find_eval_yaml keeps its legacy-only behavior so the existing adapter is untouched. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
azd extension list lists every registry extension, including uninstalled ones marked 'Not installed', so scanning its text reported an extension as available before the user had installed it. Verified live: this machine has 9 extensions installed but 15 in the registry, and azure.ai.finetune matched the text scan while not being installed. Detection now parses 'azd extension list --installed -o json' and matches on extension id, falling back to the text scan only when the structured form is unsupported. azd_available() delegates to the new shared probe and keeps its signature and answers. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Adds the current-surface adapter and dispatches to it from the orchestrator when the resolved recipe belongs to that surface. The legacy adapter keeps serving legacy recipes unchanged. The adapter reconciles the evaluation, submits with --no-wait, polls to a terminal state under the AgentOps timeout, then reads the run object and every per-sample item. Submitting without waiting is deliberate: azd's blocking mode can exit zero with an unfinished run when its internal wait budget expires, and it returns a differently shaped payload when it does. The azd run object exposes only counts, so aggregate metrics are computed as the mean of per-sample scores. Absent scores are excluded rather than read as zero, null verdicts are never collapsed to false, and a run with zero samples, no decodable metrics, or a non-completed status can never report a pass. AgentOps never passes a failure-gating flag; the release gate stays here so a gate breach remains distinguishable from an operational failure. Thresholds bind against the recipe's declared metrics before anything runs, so a misconfigured threshold fails with exit code 1 without consuming a cloud evaluation. A declared metric the run never emitted is recorded as a failed threshold, exit code 2. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ninstallable one eval analyze now reports the resolved recipe path and its surface for azd workspaces, including which recipe was skipped when both surfaces are present, and turns any resolution error into a reported gap instead of raising. eval init probes for the current-surface extension and only targets it when it is actually installed. azure.ai.evaluations is preview and absent from the default azd registry, so generating evals/azure.eval.yaml today would produce a workspace whose very next command fails on a dependency the user cannot install. Availability is fail-safe: any inability to determine it means 'not available', which keeps initialization on the legacy surface that is published and installable. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Proves the current surface is strictly opt-in. A workspace with a legacy eval.yaml and no evals/ directory runs and analyzes exactly as before, emits no current-surface text, and dispatches only to azd ai agent eval commands. Legacy normalization stays aggregate-only with no surface marker in its provenance block, so an absent surface key means legacy. Surface assertions use ordered argv slices because azd ai agent eval and azd ai eval share the ai and eval tokens. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Covers extension ids, azd version floors, discovery precedence, the threshold binding table, and what the current surface adds. States plainly that azure.ai.evaluations is preview and not yet in the default azd extension registry, and that the surface is opt-in so existing workspaces are unaffected. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Specification, implementation plan, phase 0 research, data model, interface contract, quickstart validation guide, and the task breakdown, following the existing specs/001-010 pattern. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The recipe variable was assigned Path in the current-surface branch and Optional[Path] in the legacy branch, so mypy inferred Path from the first assignment and rejected the second. Declares the type up front and narrows the discovered value before assigning. Verified with the CI toolchain: uv run mypy src/agentops/ --ignore-missing-imports -> no issues in 109 source files. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This was referenced Sep 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds support for the current
azd ai evalcommand surface (azure.ai.evaluations) toexecution: azd, alongside the existingazd ai agent evalsurface, which is left behaviorally untouched.Closes #484. Unblocks the VBD Evaluate lab (#483).
Why the recipe decides, not a setting
No new command, flag, or config field. The discovered recipe determines which azd surface runs:
evals/azure.eval.yamlazd ai evalazure.ai.evaluations(preview, azd 1.27.1+)eval.yaml,src/<agent>/eval.yamlazd ai agent evalazure.ai.agentsClassification is by file content, never filename: a sequence-valued
evals:root is current, a mapping-valuedagent:or adataset_reference:key is legacy. When both are discoverable the current surface wins and the run reports which recipe it skipped.Read this before reviewing: the upstream extension is not published
azure.ai.evaluationsexists today only as an unmerged pull request (Azure/azure-dev#9500). It is absent from bothregistry.jsonandregistry.dev.json, andazd extension install azure.ai.evaluationsdoes not resolve. Verified live on azd 1.31.0.Every command name, flag, and JSON field in this PR was read from that PR branch's source and formal JSON schema, not guessed. But the surface can still change before it merges, which is why all of it is confined to one module.
Practical consequence: live end-to-end validation is not possible for anyone today. Everything here is verified through a mocked subprocess boundary, which the constitution requires anyway.
Nobody is opted in by accident
This is the property I most want reviewed. The new surface activates only when a current-surface recipe exists.
eval.yamland noevals/: identical behavior, no new output, no new warning, no new error. Guarded bytests/integration/test_azd_surface_compat.py.agentops eval initkeeps generating a legacy recipe while the current extension is unavailable. Generatingevals/azure.eval.yamltoday would produce a workspace whose next command fails on a dependency the user cannot install (FR-016, SC-010).surfacekey added. Absent means legacy.Design decisions worth a reviewer's attention
Submitted with
--no-wait, polled by AgentOps. azd's blocking mode has an internal wait budget; when it expires it exits zero with an unfinished run and returns a differently shaped payload. Trusting the exit code would report an incomplete run as success.Aggregate metrics are computed, not read. The azd run object exposes only counts (
result_counts,per_testing_criteria_results). There are no aggregate numeric metrics anywhere on it. Each metric is the mean of its per-sample scores.The gate is never delegated. No
--fail-onis passed. azd collapses extension exit codes, so a gate breach and an operational failure would be indistinguishable. AgentOps thresholds stay the only gate, preserving the 0/2/1 contract.Threshold binding is two-stage. A threshold naming a metric no evaluator can produce is a configuration error caught before any azd command runs (exit 1, asserted by a zero-invocation test), so a typo never bills a cloud run. A declared metric the run did not emit is a gate failure (exit 2).
Parsing is deliberately tolerant.
scoreaccepts number or string;passedis three-valued andnullis never collapsed tofalse; an absent score is excluded from the mean rather than read as0.0; the always-presenterrorobject is tested by message content, not existence.Fail-closed guarantees
A run never reports a pass when it produced zero samples, no decodable metrics, a non-
completedterminal status, a non-empty run-level error, or an unbound threshold. Partial retrieval cannot inflate the pass rate.Incidental fix: a real extension-detection false positive
azd extension listlists every registry extension, including uninstalled ones markedNot installed. The previous substring scan therefore reported an extension as available before it was installed.Verified on this machine: 9 extensions installed, 15 in the registry.
azure.ai.finetunematches the text scan but is not installed, and the new structured probe correctly answersFalse.Detection now parses
azd extension list --installed -o jsonand matches onid, falling back to the text scan only when the structured form is unsupported.Testing
The single failure is pre-existing and environmental:
test_cockpit.py::test_deployments_diagnostic_no_github_remotefails ongit initwithmissing config value GIT_CONFIG_VALUE_2in this sandbox, identically on the base commit.+70 tests. No test requires azd, Azure credentials, or network. 191 lines added to pre-existing test files, 0 removed — no existing test was modified to accommodate this change.
Beyond the mocks, these paths were exercised against the real azd 1.31.0: the unavailable-extension message, the missing-recipe message, and the extension probe against real installed/registry data.
Reviewer notes, including things I would push back on myself
src/agentops/pipeline/azd_eval_runner.pyby design, but that is mitigation, not immunity.datasource_itemwere not verifiable from source. Extraction never affects the gate, since metric scores come from the results list, but rows could show empty input/response text against the real service.run output exportwas not used. It would collapse two calls into one, but its document shape was unverified. Two individually verified calls were preferred.Not in this PR
developis currently 5 commits behindmain(never synced after v0.14.0). Per.github/skills/release-management, cutting a release from a staledevelopcorrupts the CHANGELOG. That needs fixing before the next release, but it is unrelated to this change and deserves its own PR.Spec Kit artifacts under
specs/011-azd-ai-eval-surface/document the specification, plan, research (including every verified upstream fact and its source), data model, interface contract, and quickstart validation guide.