Release v0.15.0 - #487
Merged
Merged
Conversation
Bumps [requests](https://github.com/psf/requests) from 2.33.0 to 2.34.2. - [Release notes](https://github.com/psf/requests/releases) - [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md) - [Commits](psf/requests@v2.33.0...v2.34.2) --- updated-dependencies: - dependency-name: requests dependency-version: 2.34.2 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com>
chore: sync v0.14.0 release back to develop
…34.2 chore(deps): bump requests from 2.33.0 to 2.34.2
chore: sync develop with main after v0.14.0
* feat(eval): classify and discover both azd evaluation recipe schemas Adds the current-surface recipe model (evals/azure.eval.yaml, azure.ai.evaluations) alongside the existing legacy one, plus content-based classification and cross-surface discovery. Recipes are classified by content, never by filename: a sequence-valued 'evals' root is the current surface, a mapping-valued 'agent' or a 'dataset_reference' key is legacy. Discovery precedence is explicit eval_recipe, then the current surface (recording what it skipped), then legacy; more than one candidate within a surface stays ambiguous as before. find_eval_yaml keeps its legacy-only behavior so the existing adapter is untouched. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(azd): detect installed extensions from structured output azd extension list lists every registry extension, including uninstalled ones marked 'Not installed', so scanning its text reported an extension as available before the user had installed it. Verified live: this machine has 9 extensions installed but 15 in the registry, and azure.ai.finetune matched the text scan while not being installed. Detection now parses 'azd extension list --installed -o json' and matches on extension id, falling back to the text scan only when the structured form is unsupported. azd_available() delegates to the new shared probe and keeps its signature and answers. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * feat(eval): run evaluations through the current azd ai eval surface Adds the current-surface adapter and dispatches to it from the orchestrator when the resolved recipe belongs to that surface. The legacy adapter keeps serving legacy recipes unchanged. The adapter reconciles the evaluation, submits with --no-wait, polls to a terminal state under the AgentOps timeout, then reads the run object and every per-sample item. Submitting without waiting is deliberate: azd's blocking mode can exit zero with an unfinished run when its internal wait budget expires, and it returns a differently shaped payload when it does. The azd run object exposes only counts, so aggregate metrics are computed as the mean of per-sample scores. Absent scores are excluded rather than read as zero, null verdicts are never collapsed to false, and a run with zero samples, no decodable metrics, or a non-completed status can never report a pass. AgentOps never passes a failure-gating flag; the release gate stays here so a gate breach remains distinguishable from an operational failure. Thresholds bind against the recipe's declared metrics before anything runs, so a misconfigured threshold fails with exit code 1 without consuming a cloud evaluation. A declared metric the run never emitted is recorded as a failed threshold, exit code 2. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * feat(eval): report the resolved azd surface and never initialize an uninstallable one eval analyze now reports the resolved recipe path and its surface for azd workspaces, including which recipe was skipped when both surfaces are present, and turns any resolution error into a reported gap instead of raising. eval init probes for the current-surface extension and only targets it when it is actually installed. azure.ai.evaluations is preview and absent from the default azd registry, so generating evals/azure.eval.yaml today would produce a workspace whose very next command fails on a dependency the user cannot install. Availability is fail-safe: any inability to determine it means 'not available', which keeps initialization on the legacy surface that is published and installable. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * test(eval): guard that legacy azd workspaces are untouched Proves the current surface is strictly opt-in. A workspace with a legacy eval.yaml and no evals/ directory runs and analyzes exactly as before, emits no current-surface text, and dispatches only to azd ai agent eval commands. Legacy normalization stays aggregate-only with no surface marker in its provenance block, so an absent surface key means legacy. Surface assertions use ordered argv slices because azd ai agent eval and azd ai eval share the ai and eval tokens. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * docs(eval): document both azd evaluation surfaces Covers extension ids, azd version floors, discovery precedence, the threshold binding table, and what the current surface adds. States plainly that azure.ai.evaluations is preview and not yet in the default azd extension registry, and that the surface is opt-in so existing workspaces are unaffected. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * docs(spec): add spec kit artifacts for the azd eval surface feature Specification, implementation plan, phase 0 research, data model, interface contract, quickstart validation guide, and the task breakdown, following the existing specs/001-010 pattern. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> * fix(eval): annotate the resolved recipe path for mypy The recipe variable was assigned Path in the current-surface branch and Optional[Path] in the legacy branch, so mypy inferred Path from the first assignment and rejected the second. Declares the type up front and narrows the discovered value before assigning. Verified with the CI toolchain: uv run mypy src/agentops/ --ignore-missing-imports -> no issues in 109 source files. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release v0.15.0
Automated release branch created from
develop.What happened
release/v0.15.0created fromdevelopCHANGELOG.mdupdated: versioned section[0.15.0]added0.15.0(package.json, plugin.json, marketplace.json)Next steps
maingit tag v0.15.0 && git push origin v0.15.0git checkout develop && git merge main && git push origin developgit fetch origin && git log --oneline origin/develop..origin/mainmust print nothingChecklist
v0.15.0pushed (publishes to PyPI, irreversible)git log --oneline origin/develop..origin/mainprints nothing