tommasobolis · Oct 12, 2026 · Oct 12, 2026
@@ -0,0 +1,284 @@
+# Framework release watch (#1066).
+#
+# PR CI installs each framework from its committed lock (ADR 0007 rule 3) and
+# the nightly resolves everything fresh at once (rule 4), so neither says which
+# upstream release breaks which adapter. This workflow tests each new release
+# on its own: everything else stays at the PR lock and only that package moves.
+#
+#   `detect`   — scripts/check_framework_releases.py lists every framework
+#                extra's direct requirement whose newest stable PyPI release is
+#                ahead of python/constraints/adapter-contract-<extra>-py3.12.txt.
+#   `evaluate` — one leg per release: install from the lock with only that pin
+#                moved (or, when that does not resolve, unconstrained: "needs
+#                co-bump"), then the adapter-contract checks: verify_frameworks.py
+#                --require-installed, the whole suite with the skip guard armed,
+#                and the example smoke. Each check is continue-on-error, so a
+#                red check is a result, recorded in result.json, not a red run.
+#   `alert`    — the only job that writes. It opens or updates ONE issue per
+#                release, labelled `framework-release`, and closes the ones the
+#                lock has caught up with. A rerun with unchanged results edits
+#                the issue but adds no comment.
+#
+# On a pull request touching this workflow or the script it is a dry run: the
+# planned issue bodies go to the job summary and nothing is written.
+#
+# Alerting (docs/ci.md convention 4): a failed run (PyPI down, a broken plan)
+# opens or updates ONE open issue labelled `framework-release-watch-failure`.
+#
+# Security: matrix values and inputs reach run: bodies through env:, never
+# interpolated. The scheduled and dispatch runs check out the default branch,
+# so they watch develop's lock.
+
+name: Framework release watch
+
+on:
+  schedule:
+    - cron: "30 5 * * *" # 05:30 UTC daily, before the 06:00 nightly
+  workflow_dispatch:
+    inputs:
+      package:
+        description: "Check only this watched package (e.g. crewai); empty checks all"
+        type: string
+        default: ""
+  pull_request:
+    paths:
+      - .github/workflows/framework-release-watch.yml
+      - python/scripts/check_framework_releases.py
+
+permissions:
+  contents: read
+
+concurrency:
+  group: framework-release-watch-${{ github.ref }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+
+defaults:
+  run:
+    working-directory: python
+
+jobs:
+  detect:
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    outputs:
+      matrix: ${{ steps.detect.outputs.matrix }}
+      count: ${{ steps.detect.outputs.count }}
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.12"
+      # Standard library only: nothing to install.
+      - name: List releases ahead of the PR lock
+        id: detect
+        env:
+          PACKAGE: ${{ inputs.package }}
+        run: |
+          set -euo pipefail
+          args=(--json)
+          if [ -n "${PACKAGE:-}" ]; then args+=(--package "$PACKAGE"); fi
+          matrix=$(python scripts/check_framework_releases.py "${args[@]}")
+          echo "$matrix"
+          echo "matrix=$matrix" >> "$GITHUB_OUTPUT"
+          echo "count=$(jq length <<<"$matrix")" >> "$GITHUB_OUTPUT"
+
+  evaluate:
+    needs: [detect]
+    if: needs.detect.outputs.count != '0'
+    runs-on: ubuntu-latest
+    timeout-minutes: 30
+    strategy:
+      fail-fast: false
+      matrix:
+        include: ${{ fromJSON(needs.detect.outputs.matrix) }}
+    name: evaluate (${{ matrix.package }} ${{ matrix.latest }})
+    env:
+      FRAMEWORK: ${{ matrix.framework }}
+      EXTRA: ${{ matrix.extra }}
+      PACKAGE: ${{ matrix.package }}
+      LATEST: ${{ matrix.latest }}
+      RELEASE: ${{ toJSON(matrix) }}
+      # tests/conftest.py's skip guard: an importorskip of this framework fails.
+      DONKEY_CONTRACT_EXTRA: ${{ matrix.extra }}
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.12"
+      - name: Write the PR lock with only this package moved
+        run: >-
+          python scripts/check_framework_releases.py
+          --constraints-for "$PACKAGE==$LATEST" --extra "$EXTRA"
+          --out "$RUNNER_TEMP/constraints.txt"
+      # The adapter-contract job's install line exactly (ci.yml), so every
+      # other package is held at the pins compiled for it.
+      - name: Does the release resolve against the lock?
+        id: mode
+        run: |
+          set -euo pipefail
+          python -m pip install --upgrade pip
+          if pip install --dry-run -e ".[llm,local,$EXTRA]" --group dev \
+              -c "$RUNNER_TEMP/constraints.txt" > "$RUNNER_TEMP/resolve.log" 2>&1; then
+            echo "mode=locked" >> "$GITHUB_OUTPUT"
+          else
+            tail -n 20 "$RUNNER_TEMP/resolve.log"
+            echo "mode=co-bump" >> "$GITHUB_OUTPUT"
+          fi
+      - name: Install
+        id: install
+        continue-on-error: true
+        env:
+          MODE: ${{ steps.mode.outputs.mode }}
+        run: |
+          set -euo pipefail
+          if [ "$MODE" = locked ]; then
+            pip install -e ".[llm,local,$EXTRA]" --group dev -c "$RUNNER_TEMP/constraints.txt"
+          else
+            pip install -e ".[llm,local,$EXTRA]" --group dev "$PACKAGE==$LATEST"
+          fi
+      - name: Signature verification (§8), fail if not installed
+        id: verify
+        if: steps.install.outcome == 'success'
+        continue-on-error: true
+        run: |
+          set -euo pipefail
+          python scripts/verify_frameworks.py --only "$FRAMEWORK" --require-installed --json \
+            | tee "$RUNNER_TEMP/verify.json"
+      - name: Whole suite, skip guard armed
+        id: pytest
+        if: steps.install.outcome == 'success'
+        continue-on-error: true
+        run: python -m pytest -q
+      - name: Strict example smoke
+        id: smoke
+        if: steps.install.outcome == 'success'
+        continue-on-error: true
+        run: timeout 120 python scripts/smoke_example.py "$FRAMEWORK"
+      - name: Record the result
+        if: always()
+        env:
+          MODE: ${{ steps.mode.outputs.mode }}
+          INSTALL: ${{ steps.install.outcome }}
+          VERIFY: ${{ steps.verify.outcome }}
+          PYTEST: ${{ steps.pytest.outcome }}
+          SMOKE: ${{ steps.smoke.outcome }}
+        run: >-
+          python scripts/check_framework_releases.py
+          --record-result "$RUNNER_TEMP/result/result.json"
+          --release "$RELEASE" --mode "${MODE:-co-bump}"
+          --verify-json "$RUNNER_TEMP/verify.json"
+          --outcome "install=$INSTALL" "verify=$VERIFY" "pytest=$PYTEST" "smoke=$SMOKE"
+      - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
+        if: always()
+        with:
+          name: result-${{ matrix.extra }}-${{ matrix.package }}
+          path: ${{ runner.temp }}/result/result.json
+          retention-days: 14
+
+  alert:
+    # `needs:` lists every other job (tests/unit/test_ci_workflows.py asserts
+    # it); `always()` so stale issues are closed even when nothing is new.
+    if: always()
+    needs: [detect, evaluate]
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    permissions:
+      contents: read
+      issues: write
+    steps:
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+          # The close comment links the commit that last moved each lock.
+          fetch-depth: 0
+      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.12"
+      - uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
+        if: needs.detect.outputs.count != '0' && needs.detect.outputs.count != ''
+        with:
+          pattern: result-*
+          path: ${{ runner.temp }}/results
+      - name: List the open framework-release issues
+        if: github.event_name != 'pull_request'
+        env:
+          GH_TOKEN: ${{ github.token }}
+          REPO: ${{ github.repository }}
+        run: |
+          set -euo pipefail
+          # `gh issue create --label` fails when a label does not exist.
+          gh label create framework-release --repo "$REPO" --force \
+            --color 1D76DB --description "A framework release is ahead of the PR lock (#1066)"
+          # `dependencies` may already exist with its own colour: never overwrite it.
+          gh label create dependencies --repo "$REPO" --color 0366D6 \
+            --description "Pull requests that update a dependency file" 2>/dev/null || true
+          gh issue list --repo "$REPO" --label framework-release --state open \
+            --limit 200 --json number,body > "$RUNNER_TEMP/open-issues.json"
+      - name: Plan the issue writes
+        env:
+          EVENT: ${{ github.event_name }}
+          RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
+          REPO_URL: ${{ github.server_url }}/${{ github.repository }}
+        run: |
+          set -euo pipefail
+          open="$RUNNER_TEMP/open-issues.json"
+          [ -f "$open" ] || echo '[]' > "$open"
+          dry=()
+          if [ "$EVENT" = pull_request ]; then dry=(--dry-run); fi
+          python scripts/check_framework_releases.py --plan-issues \
+            --results "$RUNNER_TEMP/results" --open-issues "$open" \
+            --run-url "$RUN_URL" --repo-url "$REPO_URL" \
+            --out "$RUNNER_TEMP/plan.json" "${dry[@]}" >> "$GITHUB_STEP_SUMMARY"
+      - name: Open, update and close the framework-release issues
+        if: github.event_name != 'pull_request'
+        env:
+          GH_TOKEN: ${{ github.token }}
+          REPO: ${{ github.repository }}
+        run: |
+          set -euo pipefail
+          while read -r op; do
+            kind=$(jq -r .op <<<"$op")
+            number=$(jq -r '.number // empty' <<<"$op")
+            case "$kind" in
+              create)
+                jq -r .body <<<"$op" | gh issue create --repo "$REPO" \
+                  --title "$(jq -r .title <<<"$op")" \
+                  --label framework-release --label dependencies --body-file - ;;
+              edit)
+                jq -r .body <<<"$op" | gh issue edit "$number" --repo "$REPO" \
+                  --title "$(jq -r .title <<<"$op")" --body-file - ;;
+              comment)
+                jq -r .body <<<"$op" | gh issue comment "$number" --repo "$REPO" --body-file - ;;
+              close)
+                jq -r .comment <<<"$op" | gh issue comment "$number" --repo "$REPO" --body-file -
+                gh issue close "$number" --repo "$REPO" \
+                  --reason "$(jq -r .reason <<<"$op")" < /dev/null ;;
+            esac
+          done < <(jq -c '.[]' "$RUNNER_TEMP/plan.json")
+      - name: Open or update the framework-release-watch-failure tracking issue
+        if: >-
+          github.event_name != 'pull_request' &&
+          (failure() || contains(needs.*.result, 'failure'))
+        env:
+          GH_TOKEN: ${{ github.token }}
+          REPO: ${{ github.repository }}
+          RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
+        run: |
+          set -euo pipefail
+          gh label create framework-release-watch-failure --repo "$REPO" --force \
+            --color B60205 --description "The framework release watch failed (#1066)"
+          existing=$(gh issue list --repo "$REPO" --label framework-release-watch-failure \
+            --state open --json number --jq '.[0].number // empty')
+          body="The framework release watch failed: ${RUN_URL}
+
+          New framework releases are not being tested against the PR lock until this is fixed (docs/ci.md, \`framework-release-watch.yml\`)."
+          if [ -n "$existing" ]; then
+            gh issue comment "$existing" --repo "$REPO" --body "$body"
+          else
+            gh issue create --repo "$REPO" --title "Framework release watch failing" \
+              --label framework-release-watch-failure --body "$body"
+          fi
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -751,8 +751,13 @@ build plan has the rationale behind each rule:
   `.github/workflows/lock-refresh.yml` recompiles every file and opens or
   updates one `chore(deps): refresh the PR CI lock` PR
   ([ADR 0011](docs/adr/0011-nightly-lock-refresh-pr.md),
-  [`docs/ci.md`](docs/ci.md#lock-refreshyml-the-lock-bump)). **Refresh the
-  lock by hand** in the PR that changes a dependency:
+  [`docs/ci.md`](docs/ci.md#lock-refreshyml-the-lock-bump)). An open
+  `framework-release` issue is the other trigger: when
+  `.github/workflows/framework-release-watch.yml` marks a framework release
+  safe to bump and no bump PR carries it yet, refresh the lock for it
+  ([`docs/ci.md`](docs/ci.md#framework-release-watchyml-one-release-at-a-time),
+  #1066). **Refresh the lock by hand** in the PR that changes a dependency, or
+  for a `framework-release` issue:
   ```bash
   cd python
   python scripts/compile_constraints.py   # needs `uv` on PATH (dev-time only; CI still uses plain pip)

diff --git a/docs/ci.md b/docs/ci.md
@@ -20,6 +20,7 @@ policy behind the locks and the nightly run, see
   - [ci.yml: the PR gate](#ciyml-the-pr-gate)
   - [nightly-matrix.yml: the canary](#nightly-matrixyml-the-canary)
   - [lock-refresh.yml: the lock bump](#lock-refreshyml-the-lock-bump)
+  - [framework-release-watch.yml: one release at a time](#framework-release-watchyml-one-release-at-a-time)
   - [live-contract-check.yml: the gateway contract](#live-contract-checkyml-the-gateway-contract)
 - [Scenario maps](#scenario-maps)
   - [Feature or fix (normal path)](#feature-or-fix-normal-path)
@@ -50,6 +51,7 @@ should be split.
 | Did this change break something? | `ci.yml` (PR-gating jobs, resolving from the committed lock) | Every PR; pushes to `main` and `develop` |
 | Did an upstream release break us? | `nightly-matrix.yml`, plus the fresh-resolve jobs in `ci.yml` (`all-extra-resolves`, `anthropic-stacks`, `adk-stacks`) | Daily 06:00 UTC; every PR for the `ci.yml` jobs |
 | Can the lock move to the newest releases? | `lock-refresh.yml` (opens one bump PR; its CI answers) | After each green nightly on `develop`; manual |
+| Which new framework release breaks which adapter? | `framework-release-watch.yml` (one `framework-release` issue per release ahead of the lock) | Daily 05:30 UTC; manual, optionally for one package; a dry run on PRs that change it |
 | Did the live gateway's contract drift? | `live-contract-check.yml` (the `openai-model-routing` proxy only) | Mondays 07:00 UTC |
 | Did a secret get committed? | `secret-scan.yml` (gitleaks over the full history) | Every PR; pushes to `main` and `develop` |
 | Is the published site current? | `docs.yml` (build and deploy to GitHub Pages) | Pushes to `main` under `website/**`; every 6 hours; manual |
@@ -142,6 +144,47 @@ Without either, the PR step fails and the run files the
 `lock-refresh-failure` issue. Check the setup with
 `gh workflow run lock-refresh.yml --ref develop`.
 
+### `framework-release-watch.yml`: one release at a time
+
+The nightly resolves everything fresh at once, so a red nightly doesn't say
+which upstream release broke which adapter. This workflow tests each new
+release on its own (#1066). `detect` runs
+`python scripts/check_framework_releases.py`, which lists every direct
+requirement of each framework extra whose newest stable PyPI release is ahead
+of its pin in `python/constraints/adapter-contract-<extra>-py3.12.txt`.
+Pre-releases and yanked releases don't count. `evaluate` runs one leg per
+release. It installs the `adapter-contract` job's line with that one pin moved.
+If that doesn't resolve, it installs the release unconstrained and marks it
+"needs co-bump". Then it runs the `adapter-contract` checks:
+`verify_frameworks.py --require-installed`, the whole suite with the skip guard
+armed, and the example smoke. A red check is a result, not a red run.
+
+`alert` keeps one open issue per release, titled
+`framework-release: <package> <locked> → <latest>` and labelled
+`framework-release` and `dependencies`. The issue lists the versions, the bump
+kind, the release date and notes, the result of each check, the run link and
+the next action. A rerun refreshes the body and comments only when the results
+changed. A newer release of the same package closes the older issue as
+superseded. Once the lock reaches the release, the next run closes the issue
+with a link to the lock commit.
+
+What to do with an issue:
+
+- **All green: safe to bump.** The next `lock-refresh.yml` PR may already
+  carry it. If not, from `python/` run `python scripts/compile_constraints.py`,
+  then `python scripts/generate_docs.py`, then
+  `python scripts/verify_frameworks.py --only <framework> --emit-verified`, and
+  then the pre-PR gate.
+- **Anything red.** Adapt the adapter to the release, or record the
+  incompatibility in [`verified-apis.md` §8.1](verified-apis.md). Never add a
+  ceiling (`§8.4`).
+
+Run it for one package with
+`gh workflow run framework-release-watch.yml --ref develop -f package=crewai`.
+A PR that changes the workflow or the script gets a dry run: the planned issue
+bodies go to the job summary and nothing is written. A failed run opens or
+updates one issue labelled `framework-release-watch-failure`.
+
 ### `live-contract-check.yml`: the gateway contract
 
 Every Monday this workflow runs the `sandbox` suite (`tests/sandbox/`) against
@@ -363,8 +406,10 @@ Each cron workflow ends with an `if: failure()` job or step that opens or
 updates one labelled tracking issue (never one issue per run), using
 `issues: write` on that job only. `live-contract-check.yml` does this with the
 `live-contract-drift` label (#753), `nightly-matrix.yml` with the
-`nightly-failure` label (#755), and `lock-refresh.yml`, which follows the
-nightly, with the `lock-refresh-failure` label (#763). `docs.yml`'s cron still needs the same step.
+`nightly-failure` label (#755), `lock-refresh.yml`, which follows the
+nightly, with the `lock-refresh-failure` label (#763), and
+`framework-release-watch.yml` with the `framework-release-watch-failure` label
+(#1066). `docs.yml`'s cron still needs the same step.
 
 ### 5. One task runner mirrors CI. **In place** (#765)