fix(ci): run release jobs on hosted runners and gate the PyPI publish (ENG-2000)

[lucas-koontz]

User story

As a maintainer who publishes minds-sdk releases
I want the release tests and the PyPI upload to run on GitHub-hosted runners, with the upload waiting for the tests to pass
So that PyPI gets only versions whose integration tests passed, built from the commit those tests ran on

Why this matters

When a maintainer publishes a minds-sdk release, PyPI gets it whether or not its release tests pass, so users can install a version that failed them. Versions 2.0.0 and 2.0.1 both reached PyPI after their release test runs failed. The 2.0.1 run could not have passed: the workflow sets MINDS_API_KEY and BASE_URL, but tests/integration/config.py reads MINDS_API_TOKEN and MINDS_API_BASE_URL, so pytest exits 4 at import. Both jobs also run on the self-hosted mdb-dev runner, and GitHub recommends GitHub-hosted runners for public repositories.

What happens today

flowchart TD
    R["Maintainer publishes a release"] --> T["test_on_deploy.yml runs on mdb-dev"]
    T --> X["config.py fails at import, pytest exits 4"]
    X --> W["deploy.yml runs without checking the result"]
    W --> P["deploy_to_pypi on mdb-dev uploads the default branch head"]

Loading

What should happen

flowchart TD
    R["Maintainer publishes a release"] --> T["test_on_deploy.yml runs on ubuntu-latest"]
    T --> X["Integration tests run against mdb.ai"]
    X --> W{"Succeeded, from a release, head in this repository?"}
    W -- yes --> P["deploy_to_pypi on ubuntu-latest uploads the tested commit"]
    W -- no --> N["Publish skipped"]

Loading

Acceptance criteria

• The test job in test_on_deploy.yml and deploy_to_pypi in deploy.yml run on ubuntu-latest. No workflow in this repository names mdb-dev, mdb-prod or self-hosted, in any letter case.
• deploy_to_pypi runs only when the triggering run concluded success, came from a release event and has its head in mindsdb/minds_python_sdk, and deploy.yml has no trigger besides workflow_run. Negative: a failed release test run, or a run of a same-named workflow on another event or from another repository, skips the publish.
• deploy_to_pypi checks out github.event.workflow_run.head_sha, the commit the release tests ran on.
• The release test step passes MINDS_API_TOKEN and MINDS_API_BASE_URL, and with them pytest tests/integration --collect-only collects 5 tests. Negative: neither the test job nor its pytest step sets continue-on-error, and the pytest command has no ||, so a failing test fails the release run.
• Every action in both release workflows is pinned to a full commit SHA with a # vX.Y.Z comment, and both checkouts set persist-credentials: false.
• tests/unit/test_workflows.py fails when any criterion above regresses, except the collect-only count, and the PR workflow runs it on Python 3.8 to 3.11.

How to test

Start in a clean checkout of this branch, with Python 3.8 or newer, no .env file and no MINDS_* variables in your shell.

1. Run pip install -r requirements.txt -r requirements_test.txt, then PYTHONPATH=./ pytest tests/unit -q. Expect 24 passed, 14 of them in tests/unit/test_workflows.py.
2. Run PYTHONPATH=./ MINDS_API_TOKEN=dummy MINDS_API_BASE_URL=https://mdb.ai pytest tests/integration --collect-only -q. Expect 5 tests collected. Repeat with MINDS_API_KEY=dummy BASE_URL=https://mdb.ai instead, and expect exit code 4 with AttributeError: 'NoneType' object has no attribute 'strip' at tests/integration/config.py:22.
3. Run actionlint at the repository root, and expect no output. Run zizmor --offline --persona=regular .github/workflows/deploy.yml .github/workflows/test_on_deploy.yml, and expect one informational finding, use-trusted-publishing.
4. Check that the tests depend on the change. Run git checkout origin/main -- .github/workflows/deploy.yml .github/workflows/test_on_deploy.yml, then PYTHONPATH=./ pytest tests/unit/test_workflows.py -q. Expect 9 failed, 5 passed. Restore the files with git checkout HEAD -- .github/workflows/.
5. After the merge, publish the next release and open both runs. Expect the Run Integration Tests on Release job on a GitHub-hosted runner, with test_sdk_happy_path_lifecycle passed, not skipped. Expect Publish to PyPI to run only after that run succeeds, and PyPI to get the version in minds/__about__.py at the release's commit. This first release is the step most likely to find a problem.

Notes for the reviewer

This PR depends on no other PR and can merge now. It sits in step 4 of the merge order under Ships with, beside mindsdb/engine, mindsdb/data-vault and mindsdb/hashnode-starter-kit. It has to be on main before the final step in that list.

It needs no operator steps. Both jobs read the secrets and the variable they read today: MINDS_API_KEY, PYPI_PASSWORD and CI_PYTHON_VERSION. No environment, secret or variable changes.

Rollback: revert this commit on main. The release jobs then request mdb-dev again, and the publish goes back to uploading the default branch's head whatever the test result.

Watch the first release after this merges. The next release reaches PyPI only if the integration suite passes against https://mdb.ai. No release run of the suite has passed yet. I could not run it end to end here, because it needs MINDS_API_KEY and creates and drops datasources and minds on mdb.ai.

A re-run of a failed release run tests the same commit. Re-runs reuse the original event's commit. So a re-run helps only when the cause was outside the code: a flaky answer from the mind, a rotated MINDS_API_KEY or a fix on mdb.ai. Re-run the test run, not the skipped publish run. The publish re-run reuses its original event and skips again, while a passing test re-run starts a new publish run. A fix to the tests or the package needs a release at the fixed commit. Delete the failed release and its tag and publish it again at that commit, or bump the version and cut a new release. The failed version never reached PyPI, so publishing it again is safe.

A skipped end-to-end test does not block the publish. test_sdk_happy_path_lifecycle is the only test that asks a mind a question. Its db_ground_truth fixture connects to samples.mindsdb.com:5432 and skips the test on any connection error, and pytest still exits 0. This change moves that connection onto GitHub-hosted runners, so check that the first release's log shows the test passed.

The guard also checks that the triggering run came from a release. workflow_run matches the triggering workflow by name only. Without the event check, a successful run of a same-named workflow on another event, such as a push, would publish its branch. Every condition in the if: reads a field GitHub sets on the triggering run.

workflow_run stays, behind the guard and an inline zizmor ignore. zizmor reports dangerous-triggers on every workflow_run trigger and anchors the finding on the on: key, so the ignore covers every trigger in the file. tests/unit/test_workflows.py makes up for that: it pins the if:, and it fails when deploy.yml gains a second trigger. A publish job in test_on_deploy.yml behind needs: test would remove workflow_run altogether. That move fits best with PyPI trusted publishing, which binds the publisher to one workflow file.

The pins change nothing that runs today. actions/checkout v4 and v4.4.0 both point at 11d5960a, and actions/setup-python v5.6.0 points at a26af69b.

Deliberate omissions.

• The upload still authenticates with the PYPI_PASSWORD secret. PyPI trusted publishing needs a publisher registered on PyPI first, so it is a separate change, and zizmor's informational use-trusted-publishing finding stays.
• test_on_pr.yml and codeql.yml keep their action references. The pin test covers only the two release workflows, which hold the publishing and API secrets.
• This repository has no Dependabot config for GitHub Actions, so nothing bumps the new SHA pins.
• The || check is a heuristic. ; exit 0, set +e or a wrapper script would still swallow a pytest failure.
• Two tests read raw text on purpose. The runner test fails on any workflow comment that names mdb-dev, mdb-prod or self-hosted, and the pin test fails on a comment containing uses: in either release workflow.
• tests/unit/test_workflows.py reads the workflow YAML as plain dicts. Each test reads two or three keys, so a typed model of GitHub's workflow schema would add more code than the checks it serves.
• pytest.ini's happy_path marker line still produces Unknown config option: happy_path, as it does on main.

Verified locally

Check	Result
pytest tests/unit -q in local venvs with requirements.txt and requirements_test.txt installed, on Python 3.8.20, 3.9.6, 3.10.13 and 3.11.13	24 passed on each. Each run warns Unknown config option: happy_path, as main does. 3.9.6 is macOS's system Python and adds urllib3's LibreSSL warning
pytest tests/integration --collect-only -q with dummy MINDS_API_TOKEN and MINDS_API_BASE_URL	5 tests collected, exit 0
Same with MINDS_API_KEY and BASE_URL, the names main passes	AttributeError at tests/integration/config.py:22, exit 4
22 single regressions of the two release workflows, each in a scratch copy, run against tests/unit/test_workflows.py	Each fails exactly the one test that guards it. The unchanged copy passes 14 of 14
main's two release workflows against tests/unit/test_workflows.py	9 of 14 fail
actionlint 1.7.12	No findings. On main: unknown runner label mdb-dev in both release workflows, and shellcheck SC2035 on the clean step
zizmor 1.28.0 --offline --persona=regular on both release workflows	1 informational finding, use-trusted-publishing. On main: 15 findings, including dangerous-triggers, 4 unpinned-uses and 2 artipacked
ruff check 0.15.20 on tests/unit/test_workflows.py	No findings
gh api on the pinned action tags, 2026-10-03	actions/checkout v4 and v4.4.0 both point at 11d5960a. actions/setup-python v5 and v5.6.0 both point at a26af69b
gh api on the v2.0.0 and v2.0.1 release runs, and the PyPI JSON API	Both test runs have event release and conclusion failure. Both publish runs concluded success, and PyPI shows each version uploaded after its failed test run
Pre-PR sweep against origin/main	No whitespace errors, ticket ids in added comments or debug leftovers

Not run: the integration suite end to end, which needs MINDS_API_KEY and writes to mdb.ai, and python setup.py sdist with twine upload, which would publish.

Ships with

Merge order

1. mindsdb/terraform#239 merges, and the operator applies it: the GitHub OIDC providers, the image build and installer upload roles, the dev-tier image repositories and the deployment environment settings.
2. mindsdb/github-actions#71 merges.
3. The operator creates mindsdb/deployer, a new private repository, and the deployer PR opens then. That PR merges once it re-pins argocd-pr-env-deploy to the github-actions merge commit from step 2. The operator also creates the deployer's GitHub App and sets its client ID and private key in the staging and prod environments of cowork and cowork-server.
4. mindsdb/anton#526, mindsdb/cowork#1117 and mindsdb/cowork-server#621 merge into staging. This PR, mindsdb/engine#7, mindsdb/data-vault#4 and mindsdb/hashnode-starter-kit#39 merge into main. These four depend on no other step.
5. Once each dev-tier image repository holds its staging tag, mindsdb/scratchpad-controller#80 and mindsdb/argocd-envs#25 merge.
6. anton, cowork and cowork-server each promote staging to main in their next release.
7. mindsdb/terraform#240 applies once the main builds push through the prod writer roles.
8. mindsdb/Kubernetes-Foundational-Services#166 merges, and the operator upgrades it on both clusters, once cowork-server's change is on main and staging.
9. The operator finishes with one step outside these repositories.

Sibling PRs, in merge order:

• mindsdb/terraform#239: creates the GitHub OIDC providers, the image build and installer upload roles and the dev-tier image repositories, and sets up the deployment environments.
• mindsdb/github-actions#71: build-push-ecr gains builder: local for GitHub-hosted builds, and argocd-pr-env-deploy takes the pull request as inputs and checks its image in the dev tier.
• mindsdb/deployer, a new private repository whose PR opens in step 3: rolls cowork and cowork-server out to staging and prod, and syncs their PR environments.
• mindsdb/anton#526: builds the scratchpad image on GitHub-hosted runners, and pushes pull request and staging builds to the dev tier.
• mindsdb/cowork#1117: runs every cowork job on GitHub-hosted runners, and rolls staging and prod out through mindsdb/deployer.
• mindsdb/cowork-server#621: runs every job on GitHub-hosted runners, and deploys staging and prod through mindsdb/deployer.
• mindsdb/engine#7: deletes the jobs that ran on self-hosted runners, and keeps the pull request unit tests on GitHub-hosted runners.
• mindsdb/data-vault#4: deletes the jobs that ran on self-hosted runners, and keeps the pull request unit tests on GitHub-hosted runners.
• mindsdb/hashnode-starter-kit#39: deletes the two workflows that build and deploy the blog.
• mindsdb/scratchpad-controller#80: points the dev and staging scratchpad workers at the dev-tier image.
• mindsdb/argocd-envs#25: points PR environments at the dev-tier images of cowork, cowork-server and the scratchpad worker.
• mindsdb/terraform#240, the second terraform change: sets the prod-tier image repository policies.
• mindsdb/Kubernetes-Foundational-Services#166: updates the self-hosted runner chart on both clusters.

Refs: ENG-2000

[github-actions]

Thank you for your submission, we really appreciate it. Like many open-source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution. You can sign the CLA by just posting a Pull Request Comment same as the below format.

---

I have read the CLA Document and I hereby sign the CLA

---

_{You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot.}