🌱 ci: build and push the toolchain runner image from CI (#7289)

[Danathar]

Refs #7289

Summary

#7206 wrote the Dockerfile that bakes gcc/libc6-dev/tmux into the self-hosted runner image so race shards stop failing at Prepare cgo toolchain when the cluster's apt egress drops. It merged on 2026-09-16 — and a day later the image still did not exist on GHCR and two more test (hub 2/3) shards had died the same way, because publishing the image was left to "an operator with docker and GHCR write access".

This PR removes that half of the rollout blocker: a CI Runner Image workflow builds the Dockerfile on a GitHub-hosted runner and pushes it to ghcr.io/hivecommons/hive-ci-runner with the job's own GITHUB_TOKEN. The only step left for an operator is the one CI genuinely cannot do — kubectl patch the RunnerDeployment in arc-systems to the tag the run prints — and the job summary hands them that command with the digest filled in.

How it works

• workflow_dispatch builds and pushes. Inputs: runner_base_image (default matches the live runners) and tag_suffix. The pushed tag is <runner version>-<suffix>, e.g. v2.337.0-ubuntu-24.04-toolchain-1, so the tag always records which runner release it was built on (what the [ci] self-hosted runners lost egress to archive.ubuntu.com — every -race shard fails at 'Prepare cgo toolchain' since ~07:00Z 2026-09-11 #6648 suite caveat needs an operator to read off the cluster). An existing tag is refused, never overwritten — ARC does not re-pull an unchanged tag.
• pull_request, path-filtered to the Dockerfile and the workflow, builds only. The Dockerfile's own gcc --version / tmux -V verification becomes the gate that keeps a Dockerfile change from shipping an image that quietly sends CI back to the mirrors.
• runs-on: ubuntu-latest, not HIVE_RUNNER_LABELS — the image exists to remove CI's dependency on the cluster's egress, so building it behind that egress would defeat it.
• No type=gha layer cache — the Actions cache is the resource 🌱 [ci] bake gcc/libc6-dev into the ARC runner image — root fix for the recurring apt-egress failure class #7206 measured full (9.99/10 GB, 96.6% buildkit blobs), the reason the 🐛 cache cgo apt packages for race tests #7009 offline .deb cache never survived.
• No push: trigger / no branches: filter — rolling a new tag restarts every runner pod, so when stays an operator decision; the release-line guard has nothing to classify (verified with check-release-lines.sh, check-action-pins.sh).
• Attestations off (provenance: false, sbom: false) so the pushed tag is a plain image manifest, per the stable/candidate/edge images: hive binary fails exec with EPERM from /usr/local/bin/hive (works when copied elsewhere) #3760 contract.

Also

• src/deploy/ci-runners/README.md now leads with gh workflow run ci-runner-image.yml --ref v4 …, keeps the laptop docker build as a fallback, marks the kubectl patch as the step that still needs vllm-d access, and adds the first-push caveat: GHCR creates the package private and ARC pulls anonymously, so hive-ci-runner must be made public (like hive/hive-hub/hive-contributor) before patching or the pods land in ImagePullBackOff.
• pkg/config/ci_runner_image_workflow_test.go pins the properties above (off-cluster, no gha cache, attestations off, SHA-pinned actions, push only on dispatch, README documents the workflow and the visibility caveat).
• changelog.d/changed-7289-ci-runner-image-workflow.md.

Testing

• go test ./pkg/config/ -run CIRunner — all pass (existing Dockerfile guards + 6 new).
• src/scripts/check-release-lines.sh, check-action-pins.sh, check-no-image-attestations.sh — PASS.
• The tag-resolution step was exercised locally for dispatch, PR, untagged base (rejected), and bad suffix (rejected).
• The pull_request lane of the new workflow should run on this PR itself (it touches the workflow file), which is the first real build of the Dockerfile.

After merge

gh workflow run ci-runner-image.yml --ref v4 -f tag_suffix=toolchain-1

then follow the run's job summary for the kubectl patch. Close #7289 on the first green race shard whose Prepare cgo toolchain step reports the tool already present.

— hive: backend=claude model=claude-opus-5

🤖 Generated with Claude Code

[kubestellar-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign hanthor for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[clubanderson]

CI-built runner image off-cluster with immutable tags and a build-only PR gate — this closes the gap that left the cgo-toolchain shards flaking since #7206. LGTM.

[kubestellar-prow]

LGTM label has been added.

Details

Git tree hash: 7754da08612265d6a53a93c80c31bcd1b3c3829d

[github-actions]

Thank you for your contribution! Your PR has been merged.

We'd love to hear how your experience was: share feedback

[clubanderson]

Dispatched the workflow after merge — run 35215122771 built and pushed ghcr.io/hivecommons/hive-ci-runner:v2.337.0-ubuntu-24.04-toolchain-1. Remaining operator step: kubectl patch the RunnerDeployment in arc-systems to that tag per src/deploy/ci-runners/README.md — the cgo-toolchain flakes persist until the runner pods roll onto it.