Release v0.15.0

.claude-plugin/marketplace.json

@@ -13,7 +13,7 @@
       "name": "agentops-accelerator",
       "source": "../../plugins/agentops",
       "description": "Copilot agent skills for running standardized evaluation workflows with AgentOps Toolkit and Microsoft Foundry agents.",
-      "version": "0.14.0",
+      "version": "0.15.0",
       "keywords": [
         "agentops",
         "evaluation",

.github/plugin/marketplace.json

@@ -13,7 +13,7 @@
       "name": "agentops-accelerator",
       "source": "../../plugins/agentops",
       "description": "Copilot agent skills for running standardized evaluation workflows with AgentOps Toolkit and Microsoft Foundry agents.",
-      "version": "0.14.0",
+      "version": "0.15.0",
       "keywords": [
         "agentops",
         "evaluation",

CHANGELOG.md

@@ -5,6 +5,40 @@ This format follows [Keep a Changelog](https://keepachangelog.com/) and adheres
 
 ## [Unreleased]
 
+## [0.15.0] - 2026-09-06
+
+### Added
+- **`execution: azd` supports the current `azd ai eval` command surface.**
+  AgentOps now discovers `evals/azure.eval.yaml` alongside the legacy
+  `eval.yaml`, classifies the recipe by content, and delegates to the matching
+  azd surface: `azd ai eval` via the `azure.ai.evaluations` extension (azd
+  1.27.1+), or the existing `azd ai agent eval` via `azure.ai.agents`. The
+  evaluation is created, started, polled to a terminal state, and read back
+  per-sample, then normalized into the same `results.json` and `report.md`
+  contract as every other execution mode. Threshold binding, fail-closed
+  behavior, baseline comparison, and the `0`/`2`/`1` exit codes are unchanged.
+  Raw azd output is retained for successful and failed runs alike.
+
+  The current surface fills `results.json` rows with one entry per sample,
+  including failed and errored samples, and computes aggregate metrics as the
+  mean of the per-sample scores, because the azd run object exposes only counts.
+  Threshold keys that name a metric the recipe cannot produce are now rejected
+  *before* the evaluation is submitted, so a typo never consumes a cloud run.
+
+  `azure.ai.evaluations` is in preview and is not yet published to the default
+  azd extension registry. The new surface is strictly opt-in: it activates only
+  when a current-surface recipe exists, and `agentops eval init` continues to
+  generate a legacy recipe while the extension is unavailable, so existing
+  workspaces and fresh clones are unaffected.
+
+### Changed
+- **azd extension detection now reads structured output.** Availability is
+  determined from `azd extension list --installed -o json` and matched on
+  extension id, falling back to the previous text scan only when the structured
+  form is unsupported. The human-readable table also lists uninstalled registry
+  entries, so scanning it could report an extension as available before it was
+  actually installed.
+
 ## [0.14.0] - 2026-09-01
 
 ### Added

docs/evaluation.md

@@ -66,7 +66,7 @@ separate from the Evaluations page.
 
 !!! note "The azd dataset remains recipe-owned"
     When `execution: azd` is selected, azd continues to read the dataset declared
-    in `eval.yaml`. AgentOps does not rewrite that external recipe from the
+    in the azd recipe. AgentOps does not rewrite that external recipe from the
     `dataset` value in `agentops.yaml`.
 
 ```mermaid
@@ -284,7 +284,9 @@ eval assets.
 
 The `execution:` field decides where the evaluation actually runs. Local is the
 default and works for every target. Cloud runs a Foundry agent server-side. The
-azd recipe path delegates to an existing `azd ai agent eval` flow.
+azd recipe path delegates to an azd evaluation flow — either the current
+`azd ai eval` surface or the legacy `azd ai agent eval` one, chosen by the
+recipe. See [Delegating to azd](#delegating-to-azd) below.
 
 | Target | Cloud (`execution: cloud`) | Local runner | Recommended default |
 |---|---|---|---|
@@ -297,12 +299,77 @@ For prompt-agent CI pipelines that need a merge or deploy gate, prefer cloud
 eval. Foundry executes the managed evaluation and AgentOps enforces thresholds,
 baselines, Doctor readiness, and release evidence.
 
-!!! info "Reusing an azd eval recipe"
-    If a Foundry project already uses the public-preview `azd ai agent eval`
-    recipe, set `execution: azd` and `eval_recipe: eval.yaml`. AgentOps
-    delegates execution to azd, normalizes the metrics, binds thresholds, writes
-    `results.json`, and fails closed for any threshold that has no emitted
-    metric. Rubric evaluator dimensions are treated as first-class metric names.
+## Delegating to azd
+
+`execution: azd` hands the evaluation to the Azure Developer CLI and keeps the
+AgentOps contract around it: normalized `results.json`, `report.md`, threshold
+gating, baseline comparison, and the same exit codes. Foundry runs the
+evaluation; AgentOps decides whether the release is ready.
+
+Two azd evaluation surfaces are supported. **The recipe decides which one is
+used** — there is no extra setting.
+
+| Surface | Recipe location | azd commands | Extension | azd version |
+|---|---|---|---|---|
+| Current | `evals/azure.eval.yaml` | `azd ai eval` | `azure.ai.evaluations` | 1.27.1+ |
+| Legacy | `eval.yaml`, or `src/<agent>/eval.yaml` | `azd ai agent eval` | `azure.ai.agents` | as previously documented |
+
+Classification is by file content, not by filename: a recipe whose root has a
+sequence-valued `evals:` key is current, while a mapping-valued `agent:` key or
+a `dataset_reference:` key is legacy.
+
+!!! warning "`azure.ai.evaluations` is preview and not yet published"
+    As of this writing the extension exists only as an unmerged pull request
+    against `Azure/azure-dev`. It is absent from the default azd extension
+    registry, so `azd extension install azure.ai.evaluations` does not resolve
+    yet. Until it ships, the current surface is only usable where the extension
+    has been built and published into a local azd extension source.
+
+    This affects nobody who does not opt in. The current surface activates only
+    when `evals/azure.eval.yaml` exists. A workspace without it behaves exactly
+    as it did before, and `agentops eval init` keeps generating a legacy recipe
+    while the extension is unavailable.
+
+### Discovery precedence
+
+1. `eval_recipe:` in `agentops.yaml`, when set, wins outright.
+2. If recipes from both surfaces are discoverable, the current surface wins and
+   the run reports which recipe it skipped.
+3. More than one recipe within a single surface is rejected as ambiguous; set
+   `eval_recipe:` to choose.
+4. No recipe is a configuration error naming both supported locations.
+
+### Thresholds against an azd run
+
+Threshold keys bind to the metric names the recipe declares — builtin evaluator
+references such as `builtin.task_adherence` (the bare `task_adherence` alias
+works), evaluator labels, and rubric dimension ids. Binding uses narrow aliases
+only; nothing is fuzzy-matched, because a wrong match here would create a
+false-green gate.
+
+For the current surface, binding happens in two stages:
+
+| Condition | When | Result | Exit code |
+|---|---|---|---|
+| Threshold names a metric no evaluator declares | before any azd command runs | configuration error | `1` |
+| Threshold is ambiguous across declared metrics | before any azd command runs | configuration error | `1` |
+| Declared metric produced no score in the run | after the run | threshold recorded as failed | `2` |
+| Bound threshold not satisfied | after the run | gate failure | `2` |
+
+Catching a typo before submission means a misconfigured threshold never consumes
+a cloud evaluation.
+
+### What the current surface adds
+
+The current surface exposes per-sample output, so AgentOps populates
+`results.json` rows with one entry per sample, including failed and errored
+samples, and computes each aggregate metric as the mean of its per-sample
+scores. The run object itself carries only counts. The legacy surface remains
+aggregate-only.
+
+A run that produced zero samples, no decodable metrics, or a non-`completed`
+terminal status never reports a pass. Raw azd output is retained in the run's
+artifact directory for successful and failed runs alike.
 
 ## Input mapping
 

docs/how-it-works.md

@@ -150,7 +150,7 @@ flowchart TD
     RS --> F
     F --> G["Invoke target per row or submit Foundry cloud eval"]
     G --> H["Run/collect evaluator scores"]
-    E -->|azd| AZD["Call azd ai agent eval using eval.yaml"]
+    E -->|azd| AZD["Call azd (ai eval or ai agent eval) using the discovered recipe"]
     AZD --> H
     H --> I["Evaluate thresholds"]
     I --> J["Write results.json + report.md"]
@@ -181,7 +181,31 @@ federated, workload, managed, or service-principal identity. The identity needs
 tokens, SAS, account keys, connection strings, query strings, fragments, and
 embedded credentials are unsupported. Storage firewall and private-endpoint
 connectivity remain runner responsibilities. `execution: azd` is unchanged and
-continues to use the dataset declared by `eval.yaml`.
+continues to use the dataset declared by the azd recipe.
+
+### The two azd evaluation surfaces
+
+`execution: azd` supports two azd command families. The discovered recipe
+decides which one runs; there is no extra setting.
+
+| Recipe | azd commands | Extension |
+|---|---|---|
+| `evals/azure.eval.yaml` | `azd ai eval` | `azure.ai.evaluations` (preview, azd 1.27.1+) |
+| `eval.yaml`, `src/<agent>/eval.yaml` | `azd ai agent eval` | `azure.ai.agents` |
+
+Recipes are classified by content rather than filename, and when both surfaces
+are discoverable the current one wins with the skipped recipe reported.
+
+For the current surface AgentOps reconciles the evaluation definition, submits
+the run without waiting, polls it to a terminal state under its own timeout, and
+then reads the run object plus every per-sample output item. Submitting without
+waiting is deliberate: azd's blocking mode has an internal wait budget that, on
+expiry, exits zero with an unfinished run and a differently shaped payload.
+
+AgentOps computes aggregate metrics itself for that surface, because the azd run
+object exposes only counts, never scores. It also never passes a failure-gating
+flag to azd — the release gate stays in AgentOps so a gate breach remains
+distinguishable from an operational failure.
 
 ## POC-to-production readiness flow
 

docs/tutorial-hosted-agent.md

@@ -274,6 +274,60 @@ That is expected. A hosted endpoint is evaluated with AgentOps local eval so the
 repo can invoke the endpoint, normalize results, apply thresholds, and keep a
 stable `results.json` contract, in sandbox and in CI alike.
 
+### Optional: delegate the run to azd instead
+
+If your Foundry project already drives evaluations through the Azure Developer
+CLI, you can keep that flow and still get the AgentOps gate around it. Set
+`execution: azd` in `agentops.yaml` and commit an azd recipe. Which azd surface
+runs is decided by the recipe, not by a setting:
+
+| Recipe you commit | azd commands | Extension |
+|---|---|---|
+| `evals/azure.eval.yaml` | `azd ai eval` | `azure.ai.evaluations` (preview, azd 1.27.1+) |
+| `eval.yaml` | `azd ai agent eval` | `azure.ai.agents` |
+
+A minimal current-surface recipe:
+
+```yaml
+# evals/azure.eval.yaml
+datasets:
+  - name: smoke
+    file: ../.agentops/data/smoke.jsonl
+
+evals:
+  - name: hosted-agent-regression
+    dataset: smoke
+    evaluation_level: turn
+    evaluators:
+      - evaluator: builtin.task_adherence
+        initialization_parameters:
+          model: gpt-4o
+    target:
+      type: agent
+      name: <your-agent-name>
+```
+
+Thresholds in `agentops.yaml` bind to the metric names the recipe declares, so
+`task_adherence: ">=4"` gates `builtin.task_adherence`. AgentOps checks that
+binding *before* submitting, so a typo fails immediately with exit code `1`
+instead of burning a cloud run.
+
+!!! warning "The current-surface extension is not published yet"
+    `azure.ai.evaluations` exists today only as an unmerged pull request against
+    `Azure/azure-dev`, so `azd extension install azure.ai.evaluations` does not
+    resolve from the default registry. To exercise this path now you must build
+    and publish the extension into a local azd extension source:
+
+    ```powershell
+    azd extension install microsoft.azd.extensions
+    azd x build; azd x pack; azd x publish
+    azd extension install azure.ai.evaluations --source local
+    ```
+
+    Skip this section entirely if that is not set up. Nothing else in the
+    tutorial depends on it, and a workspace without `evals/azure.eval.yaml` is
+    completely unaffected.
+
 ## 8. Run a local eval
 
 Replay the dataset against the sandbox endpoint and score it:

plugins/agentops/package.json

@@ -2,7 +2,7 @@
   "name": "agentops-accelerator",
   "displayName": "AgentOps Accelerator — Skills for GitHub Copilot",
   "description": "Copilot agent skills for running standardized evaluation workflows with AgentOps Accelerator and Microsoft Foundry agents.",
-  "version": "0.14.0",
+  "version": "0.15.0",
   "publisher": "AgentOpsAccelerator",
   "icon": "icon.png",
   "license": "MIT",

plugins/agentops/plugin.json

@@ -1,7 +1,7 @@
 {
   "name": "agentops-accelerator",
   "description": "Copilot agent skills for running standardized evaluation workflows with AgentOps Accelerator and Microsoft Foundry agents.",
-  "version": "0.14.0",
+  "version": "0.15.0",
   "author": {
     "name": "AgentOps Accelerator",
     "url": "https://github.com/Azure/agentops"

requirements-docs.txt

@@ -4,4 +4,4 @@ mkdocs-material-extensions==1.3.1
 
 # Needed for dynamic release version in docs
 mkdocs-macros-plugin==1.5.0
-requests==2.33.0
+requests==2.34.2

specs/011-azd-ai-eval-surface/checklists/requirements.md

@@ -0,0 +1,44 @@
+# Specification Quality Checklist: Current azd AI Evaluation Surface Support
+
+**Purpose**: Validate specification completeness and quality before proceeding to planning
+**Created**: 2026-09-06
+**Feature**: [spec.md](../spec.md)
+
+## Content Quality
+
+- [x] No implementation details (languages, frameworks, APIs)
+- [x] Focused on user value and business needs
+- [x] Written for non-technical stakeholders
+- [x] All mandatory sections completed
+
+## Requirement Completeness
+
+- [x] No [NEEDS CLARIFICATION] markers remain
+- [x] Requirements are testable and unambiguous
+- [x] Success criteria are measurable
+- [x] Success criteria are technology-agnostic (no implementation details)
+- [x] All acceptance scenarios are defined
+- [x] Edge cases are identified
+- [x] Scope is clearly bounded
+- [x] Dependencies and assumptions identified
+
+## Feature Readiness
+
+- [x] All functional requirements have clear acceptance criteria
+- [x] User scenarios cover primary flows
+- [x] Feature meets measurable outcomes defined in Success Criteria
+- [x] No implementation details leak into specification
+
+## Notes
+
+- Iteration 1: One open [NEEDS CLARIFICATION] marker in Edge Cases, covering recipe resolution
+  when both a legacy and a current-surface recipe are discoverable and no explicit `eval_recipe`
+  is set.
+- Iteration 2: Resolved. The user chose "prefer the current surface and report the choice".
+  Encoded as FR-004a (cross-surface precedence), FR-004b (same-surface ambiguity is still
+  rejected), SC-009 (deterministic, always-reported resolution), an Edge Cases entry, and an
+  Assumptions entry recording the rationale and the `eval_recipe` escape hatch. All checklist
+  items now pass.
+- Command names, file paths, and configuration field names retained in the spec are existing
+  public contracts of this product, not implementation choices, and are required for the
+  requirements to be testable.