ci: add an AI changelog impact review to the framework release watch

.github/workflows/framework-release-watch.yml

@@ -1,4 +1,4 @@
-# Framework release watch (#1066).
+# Framework release watch (#1066, AI impact review #1067).
 #
 # PR CI installs each framework from its committed lock (ADR 0007 rule 3) and
 # the nightly resolves everything fresh at once (rule 4), so neither says which
@@ -14,13 +14,25 @@
 #                --require-installed, the whole suite with the skip guard armed,
 #                and the example smoke. Each check is continue-on-error, so a
 #                red check is a result, recorded in result.json, not a red run.
+#   `ai-review` — one leg per release, beside `evaluate`: a model reads the
+#                upstream notes against the adapter and returns Risk, the
+#                affected symbols with citations, a recommended action and its
+#                confidence (scripts/review_framework_release.py builds the
+#                prompt and validates the answer into impact.json). Advisory
+#                only: never verification (§0.3), never a §8 row. It reads files
+#                and fetches pages, nothing else, holds no write scope, and
+#                cannot fail the run: no key or a failed review is "impact
+#                review unavailable". One review per release, cached.
 #   `alert`    — the only job that writes. It opens or updates ONE issue per
-#                release, labelled `framework-release`, and closes the ones the
-#                lock has caught up with. A rerun with unchanged results edits
-#                the issue but adds no comment.
+#                release, labelled `framework-release`, with the impact review
+#                embedded, and closes the ones the lock has caught up with. A
+#                rerun with unchanged results edits the issue but adds no comment.
 #
-# On a pull request touching this workflow or the script it is a dry run: the
-# planned issue bodies go to the job summary and nothing is written.
+# On a pull request touching this workflow or its scripts it is a dry run: the
+# planned issue bodies go to the job summary, nothing is written, and the review
+# does not run (its key is an `ai-review` environment secret). A dispatch with
+# `replay` reruns one past release of `package`, e.g. a break §8.1 records, as a
+# dry run too.
 #
 # Alerting (docs/ci.md convention 4): a failed run (PyPI down, a broken plan)
 # opens or updates ONE open issue labelled `framework-release-watch-failure`.
@@ -40,10 +52,15 @@ on:
         description: "Check only this watched package (e.g. crewai); empty checks all"
         type: string
         default: ""
+      replay:
+        description: "LOCKED..LATEST: rerun one past release of `package` as a dry run (e.g. 0.116.0..1.0.0)"
+        type: string
+        default: ""
   pull_request:
     paths:
       - .github/workflows/framework-release-watch.yml
       - python/scripts/check_framework_releases.py
+      - python/scripts/review_framework_release.py
 
 permissions:
   contents: read
@@ -75,10 +92,12 @@ jobs:
         id: detect
         env:
           PACKAGE: ${{ inputs.package }}
+          REPLAY: ${{ inputs.replay }}
         run: |
           set -euo pipefail
           args=(--json)
           if [ -n "${PACKAGE:-}" ]; then args+=(--package "$PACKAGE"); fi
+          if [ -n "${REPLAY:-}" ]; then args+=(--replay "$REPLAY"); fi
           matrix=$(python scripts/check_framework_releases.py "${args[@]}")
           echo "$matrix"
           echo "matrix=$matrix" >> "$GITHUB_OUTPUT"
@@ -179,11 +198,114 @@ jobs:
           path: ${{ runner.temp }}/result/result.json
           retention-days: 14
 
+  ai-review:
+    # The `ai-review` environment holds ANTHROPIC_API_KEY and admits only
+    # develop, so the job is skipped (not failed) on pull requests and on
+    # dispatches from other branches. No step here may fail the run: the
+    # review and its record are continue-on-error, and a leg without a review
+    # leaves "impact review unavailable" in the issue.
+    needs: [detect]
+    if: >-
+      needs.detect.outputs.count != '0' &&
+      github.event_name != 'pull_request' &&
+      github.ref == 'refs/heads/develop'
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    environment: ai-review
+    permissions:
+      contents: read
+    strategy:
+      fail-fast: false
+      matrix:
+        include: ${{ fromJSON(needs.detect.outputs.matrix) }}
+    name: ai-review (${{ matrix.package }} ${{ matrix.latest }})
+    env:
+      MODEL: claude-sonnet-5-5
+      RELEASE: ${{ toJSON(matrix) }}
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.12"
+      - name: Is the review key set?
+        id: key
+        env:
+          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
+        run: |
+          if [ -n "${ANTHROPIC_API_KEY:-}" ]; then
+            echo "available=true" >> "$GITHUB_OUTPUT"
+          else
+            echo "::notice::No ANTHROPIC_API_KEY in the ai-review environment: impact review unavailable."
+          fi
+      # At most one review per release: artifacts do not outlive their run, the
+      # cache does (and the daily restore keeps it from expiring).
+      - uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
+        id: cache
+        with:
+          path: ${{ runner.temp }}/impact
+          key: ai-review-v1-${{ matrix.extra }}-${{ matrix.package }}-${{ matrix.latest }}
+      - name: Build the prompt
+        id: prompt
+        if: steps.cache.outputs.cache-hit != 'true' && steps.key.outputs.available == 'true'
+        run: >-
+          python scripts/review_framework_release.py --prompt
+          --release "$RELEASE" --github-output "$GITHUB_OUTPUT"
+      # Read-only: files on the runner (never /proc, where the key lives) and
+      # pages on the hosts the release notes are on. No shell, no edits, no
+      # subagents, no GitHub tools (agent mode installs none unless allowed),
+      # and no in-repo Claude settings that could widen any of that.
+      - name: Review the release (AI, advisory)
+        id: review
+        if: steps.prompt.outcome == 'success'
+        continue-on-error: true
+        uses: anthropics/claude-code-action@1d6de8cb0c237e7c15e9e1bdf973826ebae490cc # v1.0.248
+        with:
+          anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
+          github_token: ${{ github.token }}
+          prompt: ${{ steps.prompt.outputs.prompt }}
+          claude_args: >-
+            --model ${{ env.MODEL }}
+            --max-turns 30
+            --setting-sources user
+            --allowedTools "Read,Grep,Glob,WebFetch(domain:github.com),WebFetch(domain:raw.githubusercontent.com),WebFetch(domain:pypi.org)"
+            --disallowedTools "Bash,Edit,Write,MultiEdit,NotebookEdit,Task,Agent,WebSearch,Read(//proc/**)"
+            --json-schema '${{ steps.prompt.outputs.schema }}'
+      - name: Record the review
+        id: record
+        if: steps.review.outcome == 'success'
+        continue-on-error: true
+        env:
+          REVIEW: ${{ steps.review.outputs.structured_output }}
+          RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
+        run: |
+          set -euo pipefail
+          python scripts/review_framework_release.py --record "$RUNNER_TEMP/impact/impact.json" \
+            --release "$RELEASE" --review "$REVIEW" --model "$MODEL" --run-url "$RUN_URL"
+          if [ -f "$RUNNER_TEMP/impact/impact.json" ]; then echo "saved=true" >> "$GITHUB_OUTPUT"; fi
+      - name: Impact review unavailable
+        if: steps.review.outcome == 'failure' || steps.record.outcome == 'failure'
+        run: echo "::notice::The review did not complete; the issue says impact review unavailable."
+      - uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
+        if: steps.record.outputs.saved == 'true'
+        continue-on-error: true
+        with:
+          path: ${{ runner.temp }}/impact
+          key: ${{ steps.cache.outputs.cache-primary-key }}
+      - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        if: always()
+        with:
+          name: impact-${{ matrix.extra }}-${{ matrix.package }}
+          path: ${{ runner.temp }}/impact/impact.json
+          if-no-files-found: ignore
+          retention-days: 14
+
   alert:
     # `needs:` lists every other job (tests/unit/test_ci_workflows.py asserts
     # it); `always()` so stale issues are closed even when nothing is new.
     if: always()
-    needs: [detect, evaluate]
+    needs: [detect, evaluate, ai-review]
     runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:
@@ -203,8 +325,15 @@ jobs:
         with:
           pattern: result-*
           path: ${{ runner.temp }}/results
+      # The reviews are advisory: none at all is "impact review unavailable".
+      - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
+        if: needs.detect.outputs.count != '0' && needs.detect.outputs.count != ''
+        continue-on-error: true
+        with:
+          pattern: impact-*
+          path: ${{ runner.temp }}/impacts
       - name: List the open framework-release issues
-        if: github.event_name != 'pull_request'
+        if: github.event_name != 'pull_request' && !inputs.replay
         env:
           GH_TOKEN: ${{ github.token }}
           REPO: ${{ github.repository }}
@@ -221,20 +350,21 @@ jobs:
       - name: Plan the issue writes
         env:
           EVENT: ${{ github.event_name }}
+          REPLAY: ${{ inputs.replay }}
           RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
           REPO_URL: ${{ github.server_url }}/${{ github.repository }}
         run: |
           set -euo pipefail
           open="$RUNNER_TEMP/open-issues.json"
           [ -f "$open" ] || echo '[]' > "$open"
           dry=()
-          if [ "$EVENT" = pull_request ]; then dry=(--dry-run); fi
+          if [ "$EVENT" = pull_request ] || [ -n "${REPLAY:-}" ]; then dry=(--dry-run); fi
           python scripts/check_framework_releases.py --plan-issues \
-            --results "$RUNNER_TEMP/results" --open-issues "$open" \
+            --results "$RUNNER_TEMP/results" --impacts "$RUNNER_TEMP/impacts" --open-issues "$open" \
             --run-url "$RUN_URL" --repo-url "$REPO_URL" \
             --out "$RUNNER_TEMP/plan.json" "${dry[@]}" >> "$GITHUB_STEP_SUMMARY"
       - name: Open, update and close the framework-release issues
-        if: github.event_name != 'pull_request'
+        if: github.event_name != 'pull_request' && !inputs.replay
         env:
           GH_TOKEN: ${{ github.token }}
           REPO: ${{ github.repository }}

docs/ci.md

@@ -51,7 +51,7 @@ should be split.
 | Did this change break something? | `ci.yml` (PR-gating jobs, resolving from the committed lock) | Every PR; pushes to `main` and `develop` |
 | Did an upstream release break us? | `nightly-matrix.yml`, plus the fresh-resolve jobs in `ci.yml` (`all-extra-resolves`, `anthropic-stacks`, `adk-stacks`) | Daily 06:00 UTC; every PR for the `ci.yml` jobs |
 | Can the lock move to the newest releases? | `lock-refresh.yml` (opens one bump PR; its CI answers) | After each green nightly on `develop`; manual |
-| Which new framework release breaks which adapter? | `framework-release-watch.yml` (one `framework-release` issue per release ahead of the lock) | Daily 05:30 UTC; manual, optionally for one package; a dry run on PRs that change it |
+| Which new framework release breaks which adapter? | `framework-release-watch.yml` (one `framework-release` issue per release ahead of the lock) | Daily 05:30 UTC; manual, optionally for one package or a replay of one past release; a dry run on PRs that change it |
 | Did the live gateway's contract drift? | `live-contract-check.yml` (the `openai-model-routing` proxy only) | Mondays 07:00 UTC |
 | Did a secret get committed? | `secret-scan.yml` (gitleaks over the full history) | Every PR; pushes to `main` and `develop` |
 | Is the published site current? | `docs.yml` (build and deploy to GitHub Pages) | Pushes to `main` under `website/**`; every 6 hours; manual |
@@ -179,11 +179,60 @@ What to do with an issue:
   incompatibility in [`verified-apis.md` §8.1](verified-apis.md). Never add a
   ceiling (`§8.4`).
 
+**Impact review (AI, advisory).** The checks say *whether* a release breaks the
+adapter; they don't say what in the changelog did. Beside `evaluate`, the
+`ai-review` job runs one leg per release (#1067). It gives
+`anthropics/claude-code-action` (pinned by SHA, model `claude-sonnet-5-5`) a
+prompt that `python scripts/review_framework_release.py --prompt` builds: the
+package and both versions, the release notes and the GitHub compare link, the
+adapter module, its `FRAMEWORKS` rows in `verify_frameworks.py`, and its
+[`verified-apis.md`](verified-apis.md) §8 rows with all of §8.1 and §8.2. The
+model answers in a fixed schema: **Risk** (none, low or high), the **affected
+symbols**, each with an upstream citation, the **recommended action** (bump,
+adapt adapter, record §8.1, or watch) and its **confidence**. `alert` puts the
+answer in the issue under "Impact review (AI, advisory)".
+
+- *It is advisory.* It is never verification (§0.3), and it never changes a §8
+  row or the next action the checks give. Confirm a finding against the cited
+  source before acting on it.
+- *It is read-only.* The job has `contents: read` and no write token. The model
+  can read files (never `/proc`, where its key lives) and fetch pages from
+  github.com, raw.githubusercontent.com and pypi.org, nothing else: no shell,
+  no edits, no subagents and no GitHub tools, and in-repo Claude settings are
+  ignored. The prompt says fetched pages are data, not instructions. Only
+  `alert` writes, and the answer is untrusted: the script keeps only the
+  schema's fields, escapes Markdown, strips HTML and comment markers,
+  neutralises `@` mentions and `#` references, keeps only `http(s)` citations,
+  and drops an answer that holds anything shaped like a credential.
+- *It never fails the run.* No key, a failed action or an answer outside the
+  schema leaves "Impact review unavailable" in the issue and a notice in the
+  run. A rerun without a review keeps the review an earlier run wrote.
+- *It runs once per release.* The review is cached by extra, package and
+  version (`actions/cache`), so the daily rerun of an open issue costs nothing.
+  To review a release again, delete its `ai-review-v1-…` cache under
+  **Actions → Caches**.
+
+**One-time setup for the review.** The key is an environment secret, never a
+repository secret, and the environment admits only `develop`:
+
+1. Create an `ai-review` environment in **Settings → Environments**. Under
+   deployment branches, allow only `develop`; leave required reviewers off,
+   because the daily run is unattended.
+2. Add the environment secret `ANTHROPIC_API_KEY`:
+   `gh secret set ANTHROPIC_API_KEY --env ai-review` prompts for the value.
+   `tests/unit/test_ci_workflows.py` holds `ai-review` as the only job on that
+   environment, and keeps it off pull requests.
+
 Run it for one package with
 `gh workflow run framework-release-watch.yml --ref develop -f package=crewai`.
-A PR that changes the workflow or the script gets a dry run: the planned issue
-bodies go to the job summary and nothing is written. A failed run opens or
-updates one issue labelled `framework-release-watch-failure`.
+To check the review against a release with a known outcome, replay that
+release as a dry run:
+`gh workflow run framework-release-watch.yml --ref develop -f package=anthropic -f replay=0.116.0..1.0.0`
+evaluates and reviews `anthropic` 1.0.0 as if the lock still pinned 0.116.0,
+and puts the planned issue in the job summary without writing it. A PR that
+changes the workflow or either script gets a dry run too: the planned issue
+bodies go to the job summary, nothing is written, and the review doesn't run. A
+failed run opens or updates one issue labelled `framework-release-watch-failure`.
 
 ### `live-contract-check.yml`: the gateway contract
 
@@ -496,6 +545,10 @@ into a `run:` body. In place:
   `constraints/release-py3.11.txt` and build with `--no-isolation` (#763).
 - `docs.yml` always builds `main`, so a manual run from another branch can't
   publish unreleased docs (#771).
+- The model call in `framework-release-watch.yml`'s `ai-review` job reads its
+  key from the `ai-review` environment, which admits only `develop`. The job has
+  `contents: read`, and the model gets read and fetch tools only; its output
+  reaches an issue only through `alert`, after validation (#1067).
 
 ### 8. The site is built on PRs, on a supported Node. **In place** (#759)