diff --git a/.agents/skills/release-openshell/SKILL.md b/.agents/skills/release-openshell/SKILL.md index 4682c8438f..aca2a98f63 100644 --- a/.agents/skills/release-openshell/SKILL.md +++ b/.agents/skills/release-openshell/SKILL.md @@ -86,8 +86,15 @@ Watch the stable release run and report its result. ## PyPI publishing -Stable releases publish the built Python wheel directly to PyPI from the `publish-pypi` job in `release.yml`, after the other release publishers succeed. The job downloads the wheel from the stable GitHub Release, not the run's build artifact, and checks the stable tag's source commit and the wheel's release checksum. It uses Trusted Publishing with the `pypi` GitHub environment. Configure the `openshell` PyPI project's GitHub publisher with owner `NVIDIA`, repository `OpenShell`, workflow `release.yml`, and environment `pypi`. No PyPI API token or GitLab trigger credentials are required. +Stable releases publish the Python wheel to PyPI from `wheel-publish.yml`, which the `publish-pypi` job in `release-publish.yml` calls after the GitHub Release is created. The workflow downloads the wheel from the stable GitHub Release, not a run's build artifact, and checks the stable tag's source commit and the wheel's release checksum. It uses Trusted Publishing with the `pypi` GitHub environment. PyPI matches the workflow file that contains the publishing job, so configure the `openshell` PyPI project's GitHub publisher with owner `NVIDIA`, repository `OpenShell`, workflow `wheel-publish.yml`, and environment `pypi`. No PyPI API token or GitLab trigger credentials are required. -The environment must allow the branch used to dispatch Release. For a support release, explicitly allow its support branch before dispatching. Required reviewers, if configured, must approve the PyPI job. Stable docs publish after PyPI succeeds. An existing wheel on PyPI is skipped when retrying Release; PyPI does not allow replacing a published wheel. Because the job publishes the release asset, a new Release run for the same tag can finish a failed PyPI upload. +The environment must allow the branch used to dispatch Release or `wheel-publish.yml`. For a support release, explicitly allow its support branch before dispatching. Required reviewers, if configured, must approve the PyPI job. An existing wheel on PyPI is skipped; PyPI does not allow replacing a published wheel. + +If a stable release did not reach PyPI, publish the wheel without rerunning Release: + +```shell +gh workflow run wheel-publish.yml -R NVIDIA/OpenShell --ref main \ + -f release-tag=vX.Y.Z -f source-sha= -f python-version=X.Y.Z +``` Development and prerelease wheels remain GitHub Release assets and do not publish to PyPI. diff --git a/.agents/skills/sync-agent-infra/SKILL.md b/.agents/skills/sync-agent-infra/SKILL.md index ec12469d61..7cd31ad722 100644 --- a/.agents/skills/sync-agent-infra/SKILL.md +++ b/.agents/skills/sync-agent-infra/SKILL.md @@ -28,6 +28,7 @@ Run after adding, removing, or renaming a skill or crate; changing issue or PR c | TUI behavior | `tui-development` | | Release artifacts or smoke coverage | `test-release-canary` | | Release cadence, tagging, or qualification workflows | `release-captain` | +| Release workflows or publishers | `release-openshell`, `release-captain`; keep the Release workflows table and the manual-trigger rule in `CI.md` current | | CI diagnostics | `watch-github-actions` | | Native Windows builds, runtime, or release branch guidance | `build-openshell-mxc-windows`; keep Windows implementation links on `windows` even in docs merged to `main` | | Gator supervision | `launch-openshell-gator` | diff --git a/.github/workflows/release-prerelease.yml b/.github/workflows/release-prerelease.yml index 6dc3b51378..2d54d02b23 100644 --- a/.github/workflows/release-prerelease.yml +++ b/.github/workflows/release-prerelease.yml @@ -124,6 +124,7 @@ jobs: release-tag: ${{ needs.resolve.outputs.tag }} source-sha: ${{ needs.build.outputs.source_sha }} semver: ${{ needs.build.outputs.semver }} + python-version: ${{ needs.build.outputs.python_version }} npm-version: ${{ needs.build.outputs.npm_version }} is-prerelease: true secrets: diff --git a/.github/workflows/release-publish.yml b/.github/workflows/release-publish.yml index 32c1363a91..ed6e63827c 100644 --- a/.github/workflows/release-publish.yml +++ b/.github/workflows/release-publish.yml @@ -27,6 +27,10 @@ on: description: Release version without the v prefix required: true type: string + python-version: + description: Python package version + required: true + type: string npm-version: description: TypeScript package version required: true @@ -518,6 +522,19 @@ jobs: buf push proto --label "${RELEASE_TAG}" \ --source-control-url "${SOURCE_CONTROL_URL}" + publish-pypi: + name: Publish Python Wheel + needs: release + if: ${{ !inputs.is-prerelease }} + permissions: + contents: read + id-token: write + uses: ./.github/workflows/wheel-publish.yml + with: + release-tag: ${{ inputs.release-tag }} + source-sha: ${{ inputs.source-sha }} + python-version: ${{ inputs.python-version }} + publish-snap: name: Publish Snap needs: release @@ -583,6 +600,7 @@ jobs: - tag-ghcr-release - release-helm - release + - publish-pypi - publish-snap - publish-prerelease-snap - publish-sdk-typescript @@ -602,7 +620,7 @@ jobs: if [[ "${IS_PRERELEASE}" == "true" ]]; then expected+=(publish-prerelease-snap) else - expected+=(publish-sdk-typescript tag-sdk-go) + expected+=(publish-pypi publish-sdk-typescript tag-sdk-go) if [[ "${MAKE_LATEST}" == "true" ]]; then expected+=(publish-snap) fi diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index f20d97fc1c..6ec1e1914f 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -153,6 +153,7 @@ jobs: release-tag: ${{ needs.promote.outputs.tag }} source-sha: ${{ needs.build.outputs.source_sha }} semver: ${{ needs.build.outputs.semver }} + python-version: ${{ needs.build.outputs.python_version }} npm-version: ${{ needs.build.outputs.npm_version }} is-prerelease: false make-latest: ${{ needs.promote.outputs.make-latest == 'true' }} @@ -160,64 +161,9 @@ jobs: BUF_TOKEN: ${{ secrets.BUF_TOKEN }} SNAPCRAFT_STORE_CREDENTIALS: ${{ secrets.SNAPCRAFT_STORE_CREDENTIALS }} - publish-pypi: - name: Publish Python Wheel to PyPI - needs: [promote, build, publish] - runs-on: ubuntu-latest - timeout-minutes: 10 - environment: - name: pypi - url: https://pypi.org/project/openshell/ - permissions: - contents: read - id-token: write - steps: - # Publish the wheel attached to the GitHub Release rather than this run's - # build artifact, so a later run can finish publishing the same bytes. - - name: Download and verify the released wheel - shell: bash - env: - GH_TOKEN: ${{ github.token }} - RELEASE_TAG: ${{ needs.promote.outputs.tag }} - SOURCE_SHA: ${{ needs.build.outputs.source_sha }} - PYTHON_VERSION: ${{ needs.build.outputs.python_version }} - run: | - set -euo pipefail - tag_sha=$(gh api "repos/${GITHUB_REPOSITORY}/commits/${RELEASE_TAG}" --jq .sha) - if [[ "${tag_sha}" != "${SOURCE_SHA}" ]]; then - echo "${RELEASE_TAG} points to ${tag_sha}, not ${SOURCE_SHA}" >&2 - exit 1 - fi - gh release download "${RELEASE_TAG}" -R "${GITHUB_REPOSITORY}" \ - --pattern '*.whl' --pattern openshell-checksums-sha256.txt --dir release - shopt -s nullglob - wheels=(release/*.whl) - if [[ "${#wheels[@]}" -ne 1 ]]; then - echo "Expected exactly one Python wheel on ${RELEASE_TAG}, found ${#wheels[@]}" >&2 - exit 1 - fi - wheel=$(basename "${wheels[0]}") - if [[ "${wheel}" != openshell-"${PYTHON_VERSION}"-*.whl ]]; then - echo "${wheel} is not an openshell ${PYTHON_VERSION} wheel" >&2 - exit 1 - fi - checksum=$(awk -v wheel="${wheel}" '$2 == wheel { print $1 " " $2 }' release/openshell-checksums-sha256.txt) - if [[ -z "${checksum}" ]]; then - echo "${wheel} is missing from the release checksums" >&2 - exit 1 - fi - (cd release && sha256sum -c - <<<"${checksum}") - mkdir dist - mv "release/${wheel}" dist/ - - - name: Publish wheel to PyPI - uses: pypa/gh-action-pypi-publish@7245906fb61fa33a13554f9d5fb856fe61758086 # v1.14.3 - with: - skip-existing: true - publish-fern-docs: name: Sync and Publish Fern Docs - needs: [build, publish, publish-pypi] + needs: [build, publish] permissions: contents: write uses: ./.github/workflows/sync-docs.yml @@ -234,7 +180,7 @@ jobs: check-publication: name: Check Stable Publication - needs: [promote, build, release-notes, publish, publish-pypi, publish-fern-docs] + needs: [promote, build, release-notes, publish, publish-fern-docs] if: ${{ always() && !inputs.notes-only }} runs-on: ubuntu-latest timeout-minutes: 2 @@ -250,7 +196,7 @@ jobs: notify-release-failure: name: Notify Release Failure - needs: [promote, build, release-notes, publish, publish-pypi, publish-fern-docs, check-publication] + needs: [promote, build, release-notes, publish, publish-fern-docs, check-publication] if: failure() runs-on: ubuntu-latest timeout-minutes: 2 diff --git a/.github/workflows/wheel-publish.yml b/.github/workflows/wheel-publish.yml new file mode 100644 index 0000000000..15512ee639 --- /dev/null +++ b/.github/workflows/wheel-publish.yml @@ -0,0 +1,92 @@ +# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +name: Publish Wheel + +# PyPI Trusted Publishing matches the `job_workflow_ref` claim, which names this +# file for both calls and dispatches. The `openshell` PyPI project trusts +# workflow `wheel-publish.yml` with environment `pypi`. +on: + workflow_call: + inputs: + release-tag: + required: true + type: string + source-sha: + required: true + type: string + python-version: + required: true + type: string + workflow_dispatch: + inputs: + release-tag: + description: Stable tag whose GitHub Release holds the wheel, e.g. v0.1.7 + required: true + type: string + source-sha: + description: Commit the tag points to + required: true + type: string + python-version: + description: Python package version, e.g. 0.1.7 + required: true + type: string + +permissions: + contents: read + +jobs: + publish: + name: Publish Python Wheel to PyPI + runs-on: ubuntu-latest + timeout-minutes: 10 + environment: + name: pypi + url: https://pypi.org/project/openshell/ + permissions: + contents: read + id-token: write + steps: + # Publish the wheel attached to the GitHub Release rather than a run's + # build artifact, so a retry or dispatch publishes the same bytes. + - name: Download and verify the released wheel + shell: bash + env: + GH_TOKEN: ${{ github.token }} + RELEASE_TAG: ${{ inputs.release-tag }} + SOURCE_SHA: ${{ inputs.source-sha }} + PYTHON_VERSION: ${{ inputs.python-version }} + run: | + set -euo pipefail + tag_sha=$(gh api "repos/${GITHUB_REPOSITORY}/commits/${RELEASE_TAG}" --jq .sha) + if [[ "${tag_sha}" != "${SOURCE_SHA}" ]]; then + echo "${RELEASE_TAG} points to ${tag_sha}, not ${SOURCE_SHA}" >&2 + exit 1 + fi + gh release download "${RELEASE_TAG}" -R "${GITHUB_REPOSITORY}" \ + --pattern '*.whl' --pattern openshell-checksums-sha256.txt --dir release + shopt -s nullglob + wheels=(release/*.whl) + if [[ "${#wheels[@]}" -ne 1 ]]; then + echo "Expected exactly one Python wheel on ${RELEASE_TAG}, found ${#wheels[@]}" >&2 + exit 1 + fi + wheel=$(basename "${wheels[0]}") + if [[ "${wheel}" != openshell-"${PYTHON_VERSION}"-*.whl ]]; then + echo "${wheel} is not an openshell ${PYTHON_VERSION} wheel" >&2 + exit 1 + fi + checksum=$(awk -v wheel="${wheel}" '$2 == wheel { print $1 " " $2 }' release/openshell-checksums-sha256.txt) + if [[ -z "${checksum}" ]]; then + echo "${wheel} is missing from the release checksums" >&2 + exit 1 + fi + (cd release && sha256sum -c - <<<"${checksum}") + mkdir dist + mv "release/${wheel}" dist/ + + - name: Publish wheel to PyPI + uses: pypa/gh-action-pypi-publish@7245906fb61fa33a13554f9d5fb856fe61758086 # v1.14.3 + with: + skip-existing: true diff --git a/CI.md b/CI.md index 9f7aae8cde..31647818a3 100644 --- a/CI.md +++ b/CI.md @@ -694,6 +694,8 @@ The bot's full administrator documentation is internal to NVIDIA. The only comma These workflows run after merge to publish dev/tagged artifacts and verify them. They are not PR-gated. +Each step that publishes to an external registry, store, or site lives in its own workflow with both `workflow_call` and `workflow_dispatch`. Release calls it, and a maintainer can dispatch it to finish one missing publication for an existing release without rerunning Release. Dispatch inputs identify an already-published release, such as its tag and source commit; the workflow verifies them before publishing and should skip what is already published. Keep the manual trigger when changing or moving a publisher. + | File | Role | |---|---| | `.github/workflows/release-dev.yml` | Publishes the rolling `dev` build on every push to `main`. Builds gateway, sandbox, and supervisor images and binaries, packages, wheels, and pushes the Helm chart as `oci://ghcr.io/nvidia/openshell/helm-chart:0.0.0-dev` (plus an immutable `0.0.0-dev.` pin). Also dispatchable manually. | @@ -701,14 +703,14 @@ These workflows run after merge to publish dev/tagged artifacts and verify them. | `.github/workflows/release-auto-tag.yml` | Weekdays at 14:00 Europe/Zurich, tags the next `-pre.N` on `main` when commits landed since the latest one, then dispatches Pre-release. It skips when the train has no `-pre.1` or its version is already stable, so push each train's `-pre.1` tag by hand. | | `.github/workflows/release-prerelease.yml` | Pre-release orchestrator, triggered by prerelease tag pushes or dispatched by Release Auto-Tag or manually. A tagless dispatch creates the next candidate on its branch; explicit tags select existing candidates. Builds, then publishes and qualifies in parallel. Qualification failure fails the run and sends the failure notification; published artifacts remain available. | | `.github/workflows/release-build.yml` | Shared version computation, binary and image builds, and package builds. Validates that the tag matches the caller's release kind. | -| `.github/workflows/release-publish.yml` | Shared artifact assembly, OCI image tags and Helm chart publication, and package publication. Stable releases also publish release-labeled public schemas to Buf (`BUF_TOKEN` required), GitHub Releases, snap, npm, PyPI, and Fern. | +| `.github/workflows/release-publish.yml` | Shared artifact assembly, OCI image tags and Helm chart publication, and package publication. Stable releases also publish release-labeled public schemas to Buf (`BUF_TOKEN` required) and GitHub Releases, and call the PyPI, snap, npm, and Go SDK publishers. | | `.github/workflows/release-notes.yml` | Shared prompt-driven release notes generation through Inference Hub, used by prerelease previews, notes-only previews, and stable releases. Stable releases reuse the promoted prerelease's preview when available. | | `.github/workflows/release-qualification.yml` | Reusable qualification suites and result aggregation. Runs security scans, conformance and feature integration, Docker and VM E2E, protobuf compatibility, and advisory compatibility review. | | `.github/workflows/snap-package.yml` | Builds snap and component artifacts for Release Dev, Release, and Pre-release without Store credentials or publication. | | `.github/workflows/snap-publish.yml` | Uploads existing snap and component artifacts directly to a Store channel without rebuilding. Release Dev uses `latest/edge`, Pre-release uses `latest/beta`, and Release uses `latest/stable` for the stable rebuild. Uploads queue per channel and architecture, so tagged beta and stable jobs recheck repository tags after earlier publication finishes and before updating their floating channels. Manual dispatch uploads the `snap-linux-*` artifacts from `artifact-run-id`. | | `.github/workflows/sdk-typescript-publish.yml` | Publishes the TypeScript SDK to GitHub Packages. Called by stable Release; manual dispatch publishes a given tag and npm version. | | `.github/workflows/sdk-go-tag.yml` | Creates the `sdk/go/vX.Y.Z` tag and asks the Go module proxy to fetch it. Called by stable Release; manual dispatch tags a given version and commit. | -| `.github/workflows/wheel-publish.yml` | Triggers the GitLab pipeline that publishes wheels from a GitHub Release to PyPI. Called by stable Release; manual dispatch republishes a given tag. | +| `.github/workflows/wheel-publish.yml` | Publishes the wheel attached to a stable GitHub Release to PyPI with Trusted Publishing in the `pypi` environment. Called by stable Release; manual dispatch publishes a given tag. | | `.github/workflows/release-canary.yml` | Smoke-tests published dev artifacts in the `macos`, `ubuntu-deb`, `ubuntu-snap-system-docker`, `fedora`, and `kubernetes` (kind + Helm) jobs. Each job reaches its gateway and creates, exercises, and deletes a sandbox. The `ubuntu-snap-system-docker` job verifies a compatible system Docker lifecycle, while `ubuntu-snap-docker-preflight` tests fail-fast behavior when Docker is absent or supplied by the Docker snap. The positive snap lane also runs a local policy containment check with the packaged prover. It runs automatically after `Release Dev` succeeds and supports manual dispatch (`gh workflow run release-canary.yml --ref `). See the `test-release-canary` skill for the playbook and local kind reproduction. | ## Required status contexts diff --git a/tasks/scripts/sync_docs_website_test.py b/tasks/scripts/sync_docs_website_test.py index fdc6beb906..f1eb06248d 100644 --- a/tasks/scripts/sync_docs_website_test.py +++ b/tasks/scripts/sync_docs_website_test.py @@ -52,7 +52,6 @@ def test_release_workflows_sync_and_publish_docs_once() -> None: assert tag_job["needs"] == [ "build", "publish", - "publish-pypi", ] assert tag_job["uses"] == "./.github/workflows/sync-docs.yml" assert tag_job["with"]["channel"] == "stable"