fix: drain the runner before replacing its container mid-job

[compscidr]

Problem

Replacing a runner container while its runner is executing a CI job destroys the job: GitHub waits out the runner heartbeat (~10 min), then fails every remaining step. Observed concretely on BumpApp/bump 2026-08-10, where one deploy replaced two runners mid-run and killed both instrumentation suites (BumpApp/bump#5223).

Design

• Pull split out: docker_image_pull runs first — always safe while a job runs, and its changed is half the "replace imminent" signal. The container task now runs pull: false; a fresh digest still triggers the replace via the module's image-ID comparison.
• Plan pass: the same container task in check mode (pull: false) detects config-driven recreation (env, volumes, network mode, …) without touching anything. Idempotent deploys skip the drain entirely — unchanged from today, they remain safe to run concurrently with CI.
• Drain gate: when a replace is imminent and the container exists, poll the runners API until no runner matching ^<name>(-|$) is busy (the image registers <RUNNER_NAME_PREFIX>-<random>; the bare-name alternative covers a suffix-less future). Org and repo scopes both supported; the busy-check uses the same PAT registration already requires, so no new credentials. On timeout the deploy fails loudly with guidance instead of killing the job.
• Escape hatches: github_runner_drain_before_replace: false restores the old always-replace behavior; github_runner_force_replace: true replaces immediately in an emergency.

New variables (documented in README + CHANGELOG, folded into the pending 0.1.4): github_runner_drain_before_replace (true), github_runner_drain_timeout_minutes (45), github_runner_drain_poll_seconds (30), github_runner_force_replace (false), github_runner_api_base (derived for github.com/GHES).

Shared container args moved to vars/main.yml so the plan and deploy passes can't drift; renamed with the role prefix per ansible-lint's var-naming[no-role-prefix]. community.docker floor raised to >=3.6.0 for docker_image_pull.

Testing

• ansible-lint roles/github_runner: 0 failures, 0 warnings (production profile).
• molecule test --scenario-name default: converge, idempotence, verify, destroy all pass. The drain gate is exercised as "skipped" in molecule (no replace imminent on the idempotence pass, which is the correct behavior); live busy-path verification needs a real busy runner — suggested check after release: run a deploy during a long bump instrumentation suite and watch the drain task poll.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR prevents destructive runner container replacement during active GitHub Actions jobs by adding a “drain before replace” gate that waits for the runner to go idle when a replacement is imminent (image digest change or config-driven recreation).

Changes:

• Split image pulling into a separate docker_image_pull step and run the container task with pull: false to safely detect “replace imminent”.
• Add a drain gate that polls the GitHub runners API and blocks replacement while the matching runner is busy, failing loudly on timeout (with escape hatches).
• Centralize shared container arguments in roles/github_runner/vars/main.yml and document new variables in defaults/README/CHANGELOG; raise community.docker minimum version.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
roles/github_runner/vars/main.yml	Adds internal computed vars for container args and constructs the runners API URL/name-matching regex used by the drain gate.
roles/github_runner/tasks/main.yml	Adds image pull + “plan” pass and introduces the drain-before-replace polling gate before deploying the container.
roles/github_runner/defaults/main.yml	Introduces new drain/force-replace/API-base configuration defaults.
README.md	Documents newly added role variables for drain behavior and API base configuration.
galaxy.yml	Raises community.docker dependency floor to support docker_image_pull.
CHANGELOG.md	Records the drain-before-replace behavior, new vars, and dependency change for 0.1.4.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.