Skip to content

ci: add an AI changelog impact review to the framework release watch - #1112

Merged
tommasobolis merged 1 commit into
developfrom
chore/1067-ai-changelog-review
Oct 12, 2026
Merged

tommasobolis merged 1 commit into
developfrom
chore/1067-ai-changelog-review

Conversation

@tommasobolis

Copy link
Copy Markdown
Collaborator

Closes #1067

Summary

  • New ai-review job in framework-release-watch.yml: one leg per release, in parallel with evaluate, fail-fast: false, 15 min. anthropics/claude-code-action pinned by SHA (v1.0.248), model claude-sonnet-5-5. alert embeds the result in the issue under "Impact review (AI, advisory)": Risk (none/low/high), affected symbols with upstream citations, recommended action (bump / adapt adapter / record §8.1 / watch) and confidence.
  • scripts/review_framework_release.py builds the prompt (package, versions, release notes, GitHub compare link, adapter module, FRAMEWORKS rows, §8 row plus all of §8.1/§8.2) and validates the structured output (--json-schema). The answer is untrusted: only the schema's fields are kept, Markdown is escaped, HTML and comment markers are stripped, mentions and # refs are neutralised, only http(s) citations are kept, and an answer with credential-shaped text is dropped.
  • Hardening: contents: read, no write token; Read/Grep/Glob plus WebFetch limited to github.com, raw.githubusercontent.com and pypi.org; Bash/edit/subagent tools disallowed, reads under /proc denied, --setting-sources user. Only alert writes.
  • The key is an ai-review environment secret (develop only, no repo-level secret). The job is skipped on pull_request and on non-develop refs. No key, a failed action or an invalid answer gives a notice plus "Impact review unavailable", and the watch stays green.
  • Cost bound: one review per (extra, package, version) via actions/cache (artifacts don't persist across runs).
  • New replay dispatch input (LOCKED..LATEST with package) re-runs one past release as a dry run, so the review can be checked against a known break.
  • Advisory only: never verification (§0.3), never flips a §8 row.

Test plan

  • nox -s typecheck-and-lint test-3.11 new-dependencies commit-identities workflow-lint green; actionlint and zizmor clean
  • New tests/unit/test_review_framework_release.py: prompt contents, schema, GitHub-output delimiter, validation, rendering (block markers, HTML, mentions, Markdown, table-cell breakout, credential-shaped text), record and load
  • test_check_framework_releases.py: compare_url, replay, impact precedence (this run → earlier review → unavailable), --impacts
  • test_ci_workflows.py: ai-review is the only job on environment: ai-review in any workflow, never runs on pull_request, contents: read, read/fetch-only tools, key named only in that job; replay is a dry run
  • Local claude -p with the job's exact flags: anthropic 0.116.0..1.0.0 gave Risk: high (record §8.1); crewai 1.15.23..1.15.27 gave Risk: low (bump)

Post-deploy steps

After merge, on develop:

  1. gh workflow run framework-release-watch.yml --ref develop -f package=anthropic -f replay=0.116.0..1.0.0 → summary shows Risk: high.
  2. gh workflow run framework-release-watch.yml --ref develop -f package=crewai → none/low.
  3. Temporarily remove ANTHROPIC_API_KEY from the ai-review environment and dispatch once → run green, issue says "Impact review unavailable"; then restore the key.

🤖 Generated with Claude Code

…(§0.3, #1067)

A new `ai-review` job in framework-release-watch.yml runs one leg per
release, beside `evaluate`. It asks claude-code-action (pinned by SHA,
claude-sonnet-5-5) whether the upstream notes touch what the adapter uses,
and `alert` embeds the answer in the release's issue under "Impact review
(AI, advisory)": Risk, the affected symbols with citations, a recommended
action and the confidence.

- scripts/review_framework_release.py builds the prompt (package, versions,
  release notes, compare link, adapter module, FRAMEWORKS rows, the §8 row
  and all of §8.1/§8.2) and validates the structured output. The renderer
  treats the answer as untrusted: schema fields only, Markdown escaped, no
  HTML or comment markers, no mentions, http(s) citations only, and no
  credential-shaped text.
- Read-only: `contents: read`, Read/Grep/Glob plus WebFetch on github.com,
  raw.githubusercontent.com and pypi.org; no Bash or edit tools, /proc
  denied, in-repo Claude settings ignored. Only `alert` writes.
- The key is an `ai-review` environment secret (develop only). The job is
  skipped on pull requests and never fails the run: no key, a failed
  action or an invalid answer leaves "impact review unavailable".
- One review per (extra, package, version) via actions/cache.
- A `replay` dispatch input (LOCKED..LATEST) reruns one past release as a
  dry run, to check the review against a known break.
- Advisory only: never verification (§0.3), never a §8 row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions github-actions Bot added the chore Build/tooling/refactor, no user-facing behavior change label Oct 12, 2026
@tommasobolis
tommasobolis merged commit f88dbe8 into develop Oct 12, 2026
57 checks passed
@tommasobolis
tommasobolis deleted the chore/1067-ai-changelog-review branch October 12, 2026 16:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

chore Build/tooling/refactor, no user-facing behavior change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: add an AI changelog impact review to the framework release watch

1 participant