[FEATURE] Branching strategy Phase 3 - PyPI and CodeArtifact publish workflows

[John McCall (lowlydba)]

Summary

Phase 3 of the branching strategy (#509): the publish side of the pipeline Phase 2.B (#557) computed versions for. Internal builds go to CodeArtifact on every push to main; human-cut releases go to public PyPI.

What lands

Internal builds → CodeArtifact

main-publish.yaml, on push to main: detect-affected-packages (new composite action, using tj-actions/changed-files pinned by commit SHA rather than a hand-rolled git diff --name-only, for correct filename quoting and its dir_names-to-package-name folding) diffs the push against packages/** to find packages touched without a version bump (bumped packages release via the path below instead). Each affected package gets a .postN+main.<sha> version stamped in via compute-version, built, and published to CodeArtifact.

The same workflow doubles as a build-only smoke test on PRs touching the composite actions or itself: it discovers every package unconditionally instead of diffing (a PR touching .github/actions/** never touches packages/**, so the diff would always return zero and skip the job it's meant to exercise), builds, and stops short of publishing. Replaces compute-versions-dry-run.yaml.

Releases → PyPI

release-publish.yaml, on release: published: parses <package>-v<version> off the tag, builds, verifies the wheel matches (and, for overture-schema-pyspark, that the generated expressions are in it), and publishes via pypa/gh-action-pypi-publish with OIDC trusted publishing and attestations. Gated by the pypi-release environment's required reviewers, so a release only reaches PyPI after a human approves it there (Slack subscription to that environment's approval requests covers the notification side, no extra code).

release-trigger.yaml creates releases with an overture-release-publisher app installation token instead of GITHUB_TOKEN: GitHub doesn't fire release: published for releases created with the default token, so release-publish.yaml would never see them. Token scoped to contents: write only. App provisioning was tracked on #637 (now done).

A package's first version counts as a bump too: detect_version_bumps.py no longer skips before: null entries, so a brand-new package flows through release-trigger to PyPI on its first version, same as any other bump. detect_affected_packages.py's bumped set was widened to match (any package with a non-null after version), so the same push doesn't also queue it for an internal .postN build.

Mechanical follow-through: diff-package-versions action, deleted workflows, docs

package_versions.py diff was inlined at three call sites (detect-version-bumps, the new detect-affected-packages, and the pre-existing reusable-check-python-package-versions.yaml), each reimplementing the temp-file-then-$GITHUB_OUTPUT plumbing slightly differently. Pulled it into its own diff-package-versions action exposing count/diff outputs; the three consumers now just pipe diff into their own filter logic instead of each knowing the script's path and output convention.

package_versions.py itself moved from .github/workflows/scripts/ into diff-package-versions/, its only remaining caller, so every action in the repo now carries its own script(s) instead of reaching outside its directory.

Deletes compute-versions-dry-run.yaml and publish-python-packages.yaml, both superseded by the two workflows above.

docs/versioning.md gets a workflow-name reference table and the pypi-release approval-gate mechanics under the release walkthrough. CONTRIBUTING.md drops the "rolling out in phases" banner (this is the last phase) and corrects two release-flow descriptions that predated the approval gate.

Testing

detect-affected-packages's exclusion logic is tested locally against real repo commit ranges plus synthetic bumped/removed/new-package cases, including through the diff-package-versions wiring and after the script relocation. The uv version <post-release-string> --package <pkg> --frozen --dry-run stamping technique is confirmed to accept PEP 440 local-label strings. zizmor is clean on every touched/new workflow and action file. The workflows themselves can't run end to end until the external prerequisites below land.

Still open (tracked on #509, all external/manual)

• PyPI Trusted Publisher config per package
• pypi-release environment + required reviewers
• p3-dev-builds-ca (vnext dev builds), blocked on ops-team#299
• a resolver smoke test against real CodeArtifact once the app and PyPI config are live

Closes #509

[github-actions]

🗺️ Schema reference docs preview is live!

🌍 Preview	https://staging.overturemaps.org/schema/pr/638/schema/index.html
🕐 Updated	Aug 14, 2026 14:22 UTC
📝 Commit	3dd502d
🔧 env SCHEMA_PREVIEW	true

Note

♻️ This preview updates automatically with each push to this PR.

[Copilot]

Pull request overview

Implements Phase 3 of the branching strategy by adding dedicated publish workflows: internal .postN+main.<sha> builds to AWS CodeArtifact on no-bump merges to main, and human-approved GitHub Releases that publish released versions to public PyPI via Trusted Publishing.

Changes:

• Add main-publish.yaml to detect “changed without version bump” packages and publish .postN builds to CodeArtifact on pushes to main (plus a PR smoke-test mode).
• Add release-publish.yaml to build and publish a single released package to PyPI on release: published, gated via the pypi-release environment.
• Factor version-diffing into a reusable diff-package-versions composite action, update consumers, and retire superseded workflows.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
docs/versioning.md	Documents the new workflow matrix and the Environment approval gate for PyPI publishing.
CONTRIBUTING.md	Updates contributor-facing release flow descriptions to reflect the maintainer-gated PyPI publish.
.github/workflows/reusable-check-python-package-versions.yaml	Switches to the new diff-package-versions action for version-diff plumbing.
.github/workflows/release-trigger.yaml	Creates releases using an app token (and AWS Secrets Manager retrieval) so release: published downstream workflows fire.
.github/workflows/release-publish.yaml	New workflow: builds and publishes the released package to public PyPI with OIDC + attestations and environment gating.
.github/workflows/publish-python-packages.yaml	Deleted legacy publish workflow superseded by the new release/main publish split.
.github/workflows/main-publish.yaml	New workflow: computes .postN versions and publishes internal builds to CodeArtifact for no-bump changes on main.
.github/workflows/compute-versions-dry-run.yaml	Deleted dry-run workflow superseded by main-publish.yaml (PR smoke-test path) and real publish flows.
.github/actions/diff-package-versions/package_versions.py	Updates script docs to reflect new location/invocation from the composite action.
.github/actions/diff-package-versions/action.yml	New composite action exposing count/diff outputs for per-package version diffs.
.github/actions/detect-version-bumps/detect_version_bumps.py	Treats newly added packages as releasable bumps (removed packages still skipped).
.github/actions/detect-version-bumps/action.yml	Migrates to diff-package-versions output wiring (no temp file plumbing).
.github/actions/detect-affected-packages/detect_affected_packages.py	New script: combines version diff + changed directories to find no-bump packages needing .postN builds.
.github/actions/detect-affected-packages/action.yml	New composite action: changed-dir detection + version diff + filtering into an affected package matrix.

Suppressed comments (2)

.github/workflows/release-publish.yaml:79

• github.event.release.target_commitish can be a branch name (e.g. main) rather than the immutable tag/commit for the release, which risks publishing a different commit than the one the release tag points to if the branch has moved. Check out the release tag itself to ensure the build exactly matches what was released.

          set -euo pipefail
          if [[ "$EVENT_NAME" == "workflow_dispatch" ]]; then
            if [[ ! -d "packages/${INPUT_PACKAGE}" ]]; then
              echo "::error::No such package: packages/${INPUT_PACKAGE}"
              exit 1

.github/actions/detect-affected-packages/action.yml:62

• all_modified_files does not include deletions, so removing a file from a package without a version bump would be missed even though it changes the built wheel. Use all_changed_files (with json: true) so deletions are included in the affected-package detection.

        # all_modified_files (not all_changed_files) so deletions count too: a
        # file removed from a package without a version bump still changes
        # the built wheel.
        CHANGED_DIRS: ${{ steps.changed-dirs.outputs.all_modified_files }}
        DIFF: ${{ steps.diff.outputs.diff }}

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Victor Schappert (vcschapp)]

Raised a few minor nittish review comments. Nothing big. I'm excited to merge this, once we do it seems we're in the business of publishing Python packages!