Skip to content

ci: test each new framework release in isolation against the PR lock - #1100

Merged
tommasobolis merged 1 commit into
developfrom
chore/1066-framework-release-watch
Oct 12, 2026
Merged

tommasobolis merged 1 commit into
developfrom
chore/1066-framework-release-watch

Conversation

@tommasobolis

Copy link
Copy Markdown
Collaborator

Closes #1066

Part of #1065 (Phase 1.3). The scheduled run watches develop's lock (ADR 0007 rules 3–4, §8.4).

Summary

  • python/scripts/check_framework_releases.py (stdlib only; reuses fetch_project/canonicalize from check_new_dependencies.py):
    • Watches every framework extra's direct requirements, one extra per adapter-contract-<extra>-py3.12.txt lock, and lists the ones whose newest stable PyPI release is ahead of the lock pin.
    • Pre-releases, dev releases and fully yanked releases are skipped.
    • Each bump is classified as major, minor or patch.
    • --json emits the workflow matrix, or [] when nothing is new. --package limits the check to one package.
    • --constraints-for PKG==VER --extra EXTRA --out PATH writes the lock with only that one pin moved.
    • --record-result writes a test leg's result.json: the outcome of each check, and any other locked packages that moved.
    • --plan-issues plans the issue create, edit, comment and close actions and prints them as Markdown for the job summary.
  • .github/workflows/framework-release-watch.yml:
    • Runs daily at 05:30 UTC, on dispatch (with an optional package input), and as a dry run on PRs that change the workflow or the script.
    • detect lists the new releases. evaluate runs one leg per release (fail-fast: false, 30 min).
    • Each leg installs the adapter-contract install line from the lock with one pin moved. If that doesn't resolve, it installs the release unconstrained and marks it "needs co-bump".
    • Each leg then runs verify_frameworks.py --require-installed --json, the whole suite with the skip guard armed, and smoke_example.py, all continue-on-error.
    • alert runs on always() and is the only job that writes (issues: write). It opens or updates one issue per release, framework-release: <pkg> <locked> → <latest>, labelled framework-release and dependencies.
    • The issue body has the versions, bump kind, release date, notes link, results table, run link, an impact placeholder for ci: add an AI changelog impact review to the framework release watch #1067, and the next action.
  • Docs: a docs/ci.md section, a signals-table row and the workflow added to convention 4's list. CONTRIBUTING.md's lock-refresh step names framework-release issues as a trigger.

Changes from the issue's spec

  1. Install line: it is .[llm,local,$EXTRA] plus the dev group, exactly what the adapter-contract job installs and the line the lock was compiled for, so everything except the bumped package stays pinned. With otel added, the otel packages would install unpinned.
  2. Install mode: the workflow runs pip install --dry-run against the rewritten lock to choose between a locked install and "needs co-bump".
  3. No --stale-issues option: --plan-issues replaces it and plans every issue write.
    • A hidden marker in each issue stores a hash of its results, so a re-run with unchanged results refreshes the body but adds no comment.
    • A newer release of the same package closes the older issue as superseded, to keep daily releases from piling up issues.
    • When the lock reaches a release, its issue is closed with a link to the lock commit.
    • The ci: add an AI changelog impact review to the framework release watch #1067 impact section is kept when an issue is edited.
  4. --constraints-for also needs --extra, because a package can be pinned in more than one lock.
  5. Failure alert: following docs/ci.md convention 4, a failed run opens one framework-release-watch-failure issue. The dependencies label is created only if it doesn't exist yet, so its colour is never overwritten.

Test plan

  • nox -s typecheck-and-lint test-3.11 new-dependencies commit-identities workflow-lint docs-llms-drift (actionlint and zizmor are clean)
  • tests/unit/test_check_framework_releases.py (45 tests, fake PyPI lookup). It covers:
    • pre-releases and yanked versions skipped
    • major/minor/patch classification
    • lock parsing and the single-pin rewrite
    • planning: create, edit without a comment, comment only on changed results, superseded, stale and unwatched
    • the watched set matching ADAPTERS × each extra's direct requirements
  • test_ci_workflows.py registers the workflow and checks that only alert has write access and that a PR run writes nothing. test_workflow_bounds.py covers it automatically.
  • Live run of the script on 2026-10-12 finds 8 packages ahead of the lock: langchain-openai, langchain-core, litellm, strands-agents, agent-framework, anthropic, crewai, plus openai 3.24.0 → 3.29.0 (released 2026-10-11).
  • This PR's dry run lists the same releases in the alert job summary.

ADR needed?

  • No: this PR changes none of those. It adds a signal on top of ADR 0007's lock policy and doesn't change the policy.

Post-deploy steps

After merge to develop:

  1. gh workflow run framework-release-watch.yml --ref develop -f package=crewai should create one framework-release: crewai 1.15.23 → 1.15.27 issue.
  2. Running it again should update that issue without adding a comment, as long as the results are unchanged.
  3. Once the lock bump to crewai ≥ 1.15.27 merges, the next run should close the issue with a link to the lock commit.

🤖 Generated with Claude Code

framework-release-watch.yml runs daily (and on dispatch, optionally for one
package). scripts/check_framework_releases.py lists every framework extra's
direct requirement whose newest stable PyPI release is ahead of its
adapter-contract lock pin; each release is then installed with only that pin
moved (or unconstrained, "needs co-bump") and run through the
adapter-contract checks. The alert job keeps one framework-release issue per
release and closes it once the lock catches up. PRs that change the workflow
or script get a dry run into the job summary.

Closes #1066

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tommasobolis tommasobolis self-assigned this Oct 12, 2026
@github-actions github-actions Bot added chore Build/tooling/refactor, no user-facing behavior change dependencies Pull requests that update a dependency file labels Oct 12, 2026
@tommasobolis
tommasobolis enabled auto-merge (squash) October 12, 2026 09:18
@tommasobolis
tommasobolis merged commit 9698576 into develop Oct 12, 2026
56 checks passed
@tommasobolis
tommasobolis deleted the chore/1066-framework-release-watch branch October 12, 2026 14:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

chore Build/tooling/refactor, no user-facing behavior change dependencies Pull requests that update a dependency file

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: detect new framework releases and test each one in isolation against the PR lock

1 participant