Use trusted publishing

Now that all the builds are on github CI, we can move to trusted publishing. Help welcome. I guess the publishing should be triggered by merging a PR to main and/or maintenance/* branches.

@andyfaff

@andyfaff, could you help out with trusted publishing? Can we do it when we still have 3 workflows: posix, win-x86_64, win-arm64?

I can help. Only one workflow (e.g. publish.yml) can work with trusted publishing. That workflow can then use other workflows from the repo, in this case the ones that build the artefacts for each platform.

Out of interest where do the mac wheels get built?

The macos ones are built in the posix ci run. A bit of a misnomer 🤷‍♂️

Underway. Can you see if you can grab the testpypi name? It'd be good to practice on there before th real thing.

Done,
https://test.pypi.org/project/scipy-openblas64/
https://test.pypi.org/project/scipy-openblas32/

Let me know what to fill in for trusted publishing

If that image is for testpypi then the environment name is 'testpypi'. The workflow will be called 'publish.yml'.

What's the typical workflow for a release? I want to make a check in the workflow to ensure what has been built is definitely a release, and not a nightly. Do you normally push a tag and then do a manual build from the tag?

THere's a few ways to do that, e.g:

check the name of a wheel
if [ -n "$(git tag --points-at HEAD)" ]; then. This checks if the current HEAD has a tag, but is less useful if there have been commits since.

You also need to configure the settings for the MacPython/openblas-libs repo.

Navigate to Settings --> Environments.
Make two new environments. One called 'testpypi' and the other called 'pypi'.
If necessary you can specify protection rules that limit the branches that you can deploy from. e.g. on scipy we could limit deployments to branches called maintenance/**.

If you have access to numpy/numpy-release then you can see what's done there.

When you configure each environment you also need to add 'required reviewers', i.e. "Specify people or teams that may approve workflow runs when they access this environment."

I suggest that you add whoever deals with releases. For scipy and numpy there are numpy-release and scipy-release teams. You can of course add individual user handles.

These reviewers get a notification for when the artefacts are ready to deploy. At that point the release manager looks at all the wheels then does a manual approval for the deployment to pypi.

What's the typical workflow for a release?

I am the only committer here, rarely someone else will pitch in. So my workflow has been

create a feature branch, bump the pyproject.toml version number manually following the OpenBLAS commit in github_commit.txt. If the github_commit.txt is unchanged, then increment the build number on the version, so 0.3.31.126.0 would become 0.3.31.126.1
Once I am happy with CI I merge to main
pull main, create a tag from the pyproject.toml version, and push to main.
Download the wheels from anaconda.org's storage with tools/download-wheels.py
use twine to upload the wheels to PyPI

So I think it makes sense to push to PyPI with trusted publishing on every merge to main. Is there a way to automate the tag creation on merging to main?

Since I am the only committer, I would also be the 'required reviewer'.

There is still a need for the cron job to run once a week to build wheels off OpenBLAS/develop and upload them to anaconda.org, as well as run test CI against PRs with no uploading.

There is still a need for the cron job to run once a week to build wheels off OpenBLAS/develop and upload them to anaconda.org, as well as run test CI against PRs with no uploading.

That will still occur on the individual workflows.

It seems to me that when you want to release you push a tag, which at that moment in time is pointing to the HEAD commit.
How about one of these two possibilities:

You could start a release run manually, selecting the pypi or testpypi environment, then approve deployment when wheels are built and CI passes. A check is done to see if HEAD corresponds to a tag.
This option is more automated. As soon as you push a tag to main the wheels+CI are run, then deployment automatically occurs. It's already know at this point that it should be a release because the run was activated by a tag. You could still have a deployment approval if you wanted to.

I have created the testpypi and pypi environments and configured testpypi.org/project/scipy-openblas64 and testpypi.org/project/scipy-openblas32 to use the testpypi environment, publish.yml workflow.

Once we get that going I will configure pypi.org/project/* as well.

#256 was merged, thanks. I triggered a run with testpypi.

What happens if i trigger a run with “none” as the environment?

Nothing will upload. Only if you trigger with testpypi as the environment.
If you trigger with testpypi as an environment you may or may not get a request to approve the deployment. That will depend on how you set up the environment in the repo settings.

Before you make changes to the publish.yml workflow to deploy to pypi it needs some additional changes. Viz, checking that the commit it's building against is associated with a tag.

The testpypi trigger is working. Thanks for the help, i think i can take it from here: activating the tag check and enabling pypi upload

It looks a bit weird. 0.3.31.126.2 looks less complete in terms of number of wheels, compared to 0.3.31.126.
I think the 0.3.31.126.2 release has all the windows wheels, but the 0.3.31.126 has the posix wheels. Is there a naming issue with the two builds?

Ahh, right, good catch. The logic around NIGHTLY (which only happens on "posix") in .github/workflow/posix.yml is wrong for a publish workflow:

    env:
      NIGHTLY: ${{ github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' }}

It should be more complicated and distinguish between a "publish" workflow and a "manual dispatch" workflow.

Can I add something to the workflow call, like an env?

# Workflows to build wheels
  call-posix:
    env:
      PUBLISH: 1
    uses: ./.github/workflows/posix.yml

Ahh, there is with. Claude says I can do

# called-workflow.yml
on:
  workflow_call:
    inputs:
      publish:
        type: boolean
        default: false
        required: false

jobs:
  build:
    runs-on: ubuntu-latest
    env:
      PUBLISH: ${{ inputs.publish == true && 'true' || 'false' }}
    steps:
      - run: echo "PUBLISH is $PUBLISH"

and

# caller-workflow.yml
jobs:
  call-workflow:
    uses: octo-org/example-repo/.github/workflows/called-workflow.yml@main
    with:
      publish: true

That looks like a good avenue to explore.

It's probably obvious, but the reason there's testpypi-publish32 and testpypi-publish64 jobs is because a single trusted publishing action can't upload wheels to two different projects.

Right, so eventually we will have 4 publishing jobs.

Update: the NIGHTLY check seems fixed in #259. I see the correct number of uploads in the testpypi sites. I want to test the "regular" workflow triggering, which should build against the HEAD of upstream OpenBLAS (as opposed to the openblas_commit.txt commit), but right now the HEAD is the same as the commit used here. I will activate/add the pypi workflows once I can verify that the logic is working correctly.

@andyfaff

Update2: I enabled PyPI publishing in #260. If it works we can close this issue. Things I learned: it is better to use annotated tags

git tag -a v0.3.31.126.4 -m"use trusted publishing"

than bare tags

git tag v0.3.31.126.4

I will update the README.md in the next round of commits.

Thanks again @andyfaff for the help

Rather than the manual start to the run you could have it such that if an annotated tag is pushed to the repo then the publish workflow happens.
When I do a release on refnx/refnx I make a PR to check that all the wheels build. Once CI passes and I merge the PR I then push an annotated tag.

Thanks, I will open a new issue to automate the release.

Closing. It all works as designed.

Navigation Menu

Repository navigation

Use trusted publishing #244

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Use trusted publishing #244

Description

Activity

mattip commented on Mar 6, 2026

andyfaff commented on Mar 6, 2026

andyfaff commented on Mar 6, 2026

mattip commented on Mar 6, 2026

andyfaff commented on Mar 7, 2026

mattip commented on Mar 7, 2026

mattip commented on Mar 7, 2026

andyfaff commented on Mar 7, 2026

andyfaff commented on Mar 7, 2026

andyfaff commented on Mar 7, 2026

mattip commented on Mar 7, 2026

mattip commented on Mar 7, 2026

andyfaff commented on Mar 7, 2026

mattip commented on Mar 9, 2026

mattip commented on Mar 10, 2026

andyfaff commented on Mar 10, 2026

mattip commented on Mar 10, 2026

andyfaff commented on Mar 10, 2026

mattip commented on Mar 10, 2026

mattip commented on Mar 10, 2026

mattip commented on Mar 10, 2026

andyfaff commented on Mar 10, 2026

andyfaff commented on Mar 10, 2026

mattip commented on Mar 10, 2026

mattip commented on Mar 10, 2026

mattip commented on Mar 12, 2026

andyfaff commented on Mar 12, 2026

mattip commented on Mar 12, 2026

mattip commented on Mar 12, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions