fix: drain the runner before replacing its container mid-job

CHANGELOG.md

@@ -1,5 +1,17 @@
 # Changelog
 
+## [0.1.5] - 2026-08-12
+### Fixed
+- The deploy no longer replaces a runner container while its runner is mid-job (which destroyed the CI job: GitHub waits out the runner heartbeat ~10 min, then fails every remaining step). When a replace is imminent — new image digest or config change — and the existing runner is busy, the role now polls the GitHub runners API until the runner is idle before replacing, failing the deploy loudly on timeout instead of killing the job. The busy-check fails loudly on a paginated (>100 runners) listing, treats malformed/rate-limited API responses as still-waiting, and clamps the poll interval to >=1s.
+
+### Added
+- `github_runner_drain_before_replace` (default `true`), `github_runner_drain_timeout_minutes` (default `45`), `github_runner_drain_poll_seconds` (default `30`): drain-before-replace guard configuration.
+- `github_runner_force_replace` (default `false`): emergency override — replace immediately even if a job is mid-flight.
+- `github_runner_api_base`: GitHub API base for the busy-check, derived for github.com and GHES.
+
+### Changed
+- The image pull is now a separate `docker_image_pull` task and the container task runs with `pull: false`; behavior is unchanged (a new digest still triggers the replace via image-ID comparison), but the pull result now feeds the drain guard. Requires `community.docker` >= 3.6.0.
+
 ## [0.1.4] - 2026-08-10
 ### Fixed
 - First playbook run no longer fails with `UnixHTTPConnectionPool ... Read timed out (read timeout=60)` when replacing an existing runner container. Stopping and removing a crash-looping or wedged runner can exceed the Docker SDK's 60s default client timeout; the daemon finished the removal in the background, which is why an immediate rerun succeeded.

README.md

@@ -78,6 +78,11 @@ github_runner_github_host                   | The GITHUB_HOST used for registeri
 github_runner_persist_config                | whether to persist runner configuration across container restarts using a named volume (defaults to true)
 github_runner_stop_timeout                  | seconds docker waits after SIGTERM before SIGKILL when stopping the runner container, e.g. while replacing it on an image bump (defaults to 10)
 github_runner_docker_timeout                | Docker API client timeout in seconds for the deploy task; stopping and removing a crash-looping runner can exceed the SDK's 60s default (defaults to 180)
+github_runner_drain_before_replace          | when a deploy is about to replace the runner container while the runner is executing a CI job, wait for the job to finish instead of destroying it (defaults to true)
+github_runner_drain_timeout_minutes         | how long to wait for a busy runner to go idle before failing the deploy (defaults to 45)
+github_runner_drain_poll_seconds            | polling interval for the busy-check against the GitHub runners API (defaults to 30)
+github_runner_force_replace                 | emergency override: replace the container immediately even if a job is mid-flight (defaults to false)
+github_runner_api_base                      | GitHub API base URL used by the busy-check; derived automatically for github.com and GHES (/api/v3)
 
 Notes: the env file lets you do things like set site-specific credentials into the runner that can be built into the code
 at build time, for instance, Wi-Fi credentials that can be built into tests that are specific to the location of

galaxy.yml

@@ -8,7 +8,7 @@ description: "Installs github self hosted repo or org runners within a docker co
   can be a vanilla runner, or one with java / android installed."
 license_file: LICENSE
 readme: README.md
-version: 0.1.4
+version: 0.1.5
 repository: https://github.com/compscidr/ansible-github-runner
 tags:
     - github
@@ -18,4 +18,4 @@ tags:
     - hosted
     - infrastructure
 dependencies:
-    community.docker: "*"
+    community.docker: ">=3.6.0"  # docker_image_pull

roles/github_runner/defaults/main.yml

@@ -35,3 +35,20 @@ github_runner_stop_timeout: 10
 # crash-looping or wedged runner container can exceed the SDK's 60s default,
 # which fails the first playbook run and succeeds on the rerun.
 github_runner_docker_timeout: 180
+
+# Drain-before-replace: when the deploy is about to replace the runner
+# container (new image digest or config change) while the existing runner is
+# executing a CI job, wait for the job to finish instead of destroying it
+# (replacing mid-job makes GitHub wait out the runner heartbeat ~10 min and
+# fail every remaining step). If the runner is still busy when the timeout
+# expires the deploy fails loudly rather than killing the job.
+github_runner_drain_before_replace: true
+github_runner_drain_timeout_minutes: 45
+github_runner_drain_poll_seconds: 30
+
+# Emergency override: replace immediately even if a job is mid-flight.
+github_runner_force_replace: false
+
+# GitHub API base for the drain busy-check. Derived for github.com and GHES
+# (/api/v3); override if your API lives elsewhere.
+github_runner_api_base: "{{ 'https://api.github.com' if github_runner_github_host == 'github.com' else 'https://' ~ github_runner_github_host ~ '/api/v3' }}"

roles/github_runner/tasks/main.yml

@@ -1,14 +1,138 @@
 ---
+# The pull is separated from the container task for the drain guard below:
+# pulling is always safe while the runner is mid-job (it never touches the
+# container), and whether the pull actually fetched a new digest is half the
+# signal for "the container is about to be replaced". The deploy task then
+# runs with pull: false — a freshly pulled image still triggers the replace
+# via the module's image-ID comparison.
+- name: Pull runner image
+  tags: github_runner
+  become: true
+  community.docker.docker_image_pull:
+    name: "{{ github_runner_java_image if github_runner_java else github_runner_non_java_image }}"
+  register: github_runner_image_pull
+
+# Check-mode pass with identical arguments (minus the pull): detects
+# config-driven recreation (env, volumes, network mode, ...) without touching
+# anything. Together with the pull result this decides whether the drain
+# guard needs to run at all — an idempotent deploy skips straight through.
+- name: Check whether the runner container would change
+  tags: github_runner
+  become: true
+  check_mode: true
+  community.docker.docker_container:
+    name: "{{ github_runner_name }}"
+    image: "{{ github_runner_java_image if github_runner_java else github_runner_non_java_image }}"
+    pull: false
+    devices: "{{ ['/dev/bus/usb:/dev/bus/usb:rwm'] if (github_runner_java and github_runner_java_mount_usb) else omit }}"
+    device_cgroup_rules: "{{ ['c 189:* rmw'] if (github_runner_java and github_runner_java_mount_usb) else omit }}"
+    volumes: "{{ github_runner_volumes }}"
+    restart_policy: unless-stopped
+    stop_timeout: "{{ github_runner_stop_timeout }}"
+    timeout: "{{ github_runner_docker_timeout }}"
+    network_mode: "{{ github_runner_network_mode }}"
+    ports: "{{ github_runner_ports }}"
+    env: "{{ github_runner_env }}"
+    env_file: "{{ github_runner_env_filename if github_runner_env_file else omit }}"
+  register: github_runner_container_plan
+
+- name: Look up existing runner container
+  tags: github_runner
+  become: true
+  community.docker.docker_container_info:
+    name: "{{ github_runner_name }}"
+  register: github_runner_existing
+
+# Drain-before-replace: replacing the container while its runner is executing
+# a CI job destroys the job (GitHub waits out the runner heartbeat ~10 min,
+# then fails every remaining step). When a replace is imminent and the
+# existing runner is busy, poll the runners API until it goes idle. A timeout
+# fails the deploy loudly — rerun with github_runner_force_replace=true to
+# replace anyway (killing the in-flight job) in an emergency.
+- name: Wait for the runner to finish its current job before replacing
+  tags: github_runner
+  check_mode: false
+  when:
+    - github_runner_drain_before_replace
+    - not github_runner_force_replace
+    - github_runner_existing.exists
+    - github_runner_image_pull is changed or github_runner_container_plan is changed
+  block:
+    # The busy-check reads a single page. A truncated listing could hide the
+    # target runner on a later page and let the drain conclude "idle" while a
+    # job is mid-flight — the exact failure this gate exists to prevent — so a
+    # paginated listing fails loudly instead of proceeding.
+    - name: Probe the runners listing for the drain check
+      ansible.builtin.uri:
+        url: "{{ github_runner_api_runners_url }}"
+        headers:
+          Authorization: "Bearer {{ github_runner_personal_access_token }}"
+          Accept: "application/vnd.github+json"
+        return_content: true
+      register: github_runner_listing_probe
+      # The Authorization header carries the PAT — keep it out of logs.
+      no_log: true
+
+    - name: Fail if the runners listing is paginated beyond one page
+      ansible.builtin.assert:
+        that:
+          - github_runner_listing_probe.json.total_count == (github_runner_listing_probe.json.runners | length)
+        fail_msg: >-
+          The runners listing has
+          {{ github_runner_listing_probe.json.total_count }} runners but only
+          {{ github_runner_listing_probe.json.runners | length }} were
+          returned — the drain busy-check could miss the target runner on a
+          later page and replace it mid-job. Implement pagination in the
+          drain gate before deploying to this account.
+        quiet: true
+
+    - name: Poll the runners API until this runner is idle
+      ansible.builtin.uri:
+        url: "{{ github_runner_api_runners_url }}"
+        headers:
+          Authorization: "Bearer {{ github_runner_personal_access_token }}"
+          Accept: "application/vnd.github+json"
+        return_content: true
+      register: github_runner_busy_check
+      # The Authorization header carries the PAT — keep it out of logs.
+      no_log: true
+      retries: "{{ ((github_runner_drain_timeout_minutes | int) * 60 / (github_runner_drain_poll_seconds_clamped | int)) | round(0, 'ceil') | int }}"
+      delay: "{{ github_runner_drain_poll_seconds_clamped }}"
+      # Guard the shape before reading it: a rate-limit or transient non-JSON
+      # response must count as "still waiting" (retry), not blow up the until
+      # expression and abort the drain a retry might have saved.
+      until: >-
+        github_runner_busy_check.json is defined
+        and github_runner_busy_check.json.runners is defined
+        and (github_runner_busy_check.json.runners
+        | selectattr('name', 'match', github_runner_name_match)
+        | selectattr('busy')
+        | list | length == 0)
+  rescue:
+    - name: Fail with drain guidance
+      ansible.builtin.fail:
+        msg: >-
+          Not replacing {{ github_runner_name }}'s container: the drain gate
+          did not confirm the runner idle within
+          {{ github_runner_drain_timeout_minutes }} minutes.
+          Underlying failure: {{ ansible_failed_result.msg | default('(none recorded)') }}.
+          Rerun with -e github_runner_force_replace=true to replace anyway
+          (this kills any in-flight CI job), or with
+          -e github_runner_drain_before_replace=false to restore the old
+          always-replace behavior.
+
 - name: Deploy Github Runner
   tags: github_runner
   become: true
   community.docker.docker_container:
     name: "{{ github_runner_name }}"
     image: "{{ github_runner_java_image if github_runner_java else github_runner_non_java_image }}"
-    pull: true
+    # Pulled above; the image-ID comparison against the running container
+    # still triggers the replace when the pull fetched a new digest.
+    pull: false
     devices: "{{ ['/dev/bus/usb:/dev/bus/usb:rwm'] if (github_runner_java and github_runner_java_mount_usb) else omit }}"
     device_cgroup_rules: "{{ ['c 189:* rmw'] if (github_runner_java and github_runner_java_mount_usb) else omit }}"
-    volumes: "{{ runner_volumes }}"
+    volumes: "{{ github_runner_volumes }}"
     restart_policy: unless-stopped
     # Replacing an existing container (image bump) stops and removes it first.
     # A crash-looping or wedged runner can take well over the Docker SDK's
@@ -19,42 +143,6 @@
     stop_timeout: "{{ github_runner_stop_timeout }}"
     timeout: "{{ github_runner_docker_timeout }}"
     network_mode: "{{ github_runner_network_mode }}"
-    ports: "{{ runner_ports }}"
-    env: "{{ runner_env }}"
+    ports: "{{ github_runner_ports }}"
+    env: "{{ github_runner_env }}"
     env_file: "{{ github_runner_env_filename if github_runner_env_file else omit }}"
-  vars:
-    # ports: is invalid together with network_mode: host, so omit it in host mode.
-    runner_ports: >-
-      {{
-        [github_runner_adb_port | string ~ ':5037']
-        if (github_runner_android and github_runner_android_expose_adb_ports
-            and github_runner_network_mode != 'host')
-        else omit
-      }}
-    runner_volumes: >-
-      {{
-        ['/var/run/docker.sock:/var/run/docker.sock'] +
-        (['/root/.android:/root/.android'] if github_runner_android else []) +
-        ([github_runner_name ~ '-runner-data:/runner-data'] if github_runner_persist_config else [])
-      }}
-    runner_env: >-
-      {{
-        {
-          'ACCESS_TOKEN': github_runner_personal_access_token,
-          'DISABLE_AUTO_UPDATE': 'true',
-          'DISABLE_AUTOMATIC_DEREGISTRATION': 'true',
-          'RUNNER_NAME_PREFIX': github_runner_name,
-          'LABELS': github_runner_labels,
-          'HOST_NAME': inventory_hostname,
-          'GITHUB_HOST': github_runner_github_host
-        } | combine(
-          {'REPO_URL': 'https://' ~ github_runner_github_host ~ '/' ~ github_runner_repo}
-          if not github_runner_org else {}
-        ) | combine(
-          {'RUNNER_SCOPE': 'org', 'ORG_NAME': github_runner_org_name}
-          if github_runner_org else {}
-        ) | combine(
-          {'CONFIGURED_ACTIONS_RUNNER_FILES_DIR': '/runner-data'}
-          if github_runner_persist_config else {}
-        )
-      }}

roles/github_runner/vars/main.yml

@@ -0,0 +1,56 @@
+---
+# Internal computed values shared by the plan (check-mode) and deploy passes in
+# tasks/main.yml. Not intended for consumer override — tune the github_runner_*
+# variables in defaults/main.yml instead.
+
+# ports: is invalid together with network_mode: host, so omit it in host mode.
+github_runner_ports: >-
+  {{
+    [github_runner_adb_port | string ~ ':5037']
+    if (github_runner_android and github_runner_android_expose_adb_ports
+        and github_runner_network_mode != 'host')
+    else omit
+  }}
+github_runner_volumes: >-
+  {{
+    ['/var/run/docker.sock:/var/run/docker.sock'] +
+    (['/root/.android:/root/.android'] if github_runner_android else []) +
+    ([github_runner_name ~ '-runner-data:/runner-data'] if github_runner_persist_config else [])
+  }}
+github_runner_env: >-
+  {{
+    {
+      'ACCESS_TOKEN': github_runner_personal_access_token,
+      'DISABLE_AUTO_UPDATE': 'true',
+      'DISABLE_AUTOMATIC_DEREGISTRATION': 'true',
+      'RUNNER_NAME_PREFIX': github_runner_name,
+      'LABELS': github_runner_labels,
+      'HOST_NAME': inventory_hostname,
+      'GITHUB_HOST': github_runner_github_host
+    } | combine(
+      {'REPO_URL': 'https://' ~ github_runner_github_host ~ '/' ~ github_runner_repo}
+      if not github_runner_org else {}
+    ) | combine(
+      {'RUNNER_SCOPE': 'org', 'ORG_NAME': github_runner_org_name}
+      if github_runner_org else {}
+    ) | combine(
+      {'CONFIGURED_ACTIONS_RUNNER_FILES_DIR': '/runner-data'}
+      if github_runner_persist_config else {}
+    )
+  }}
+
+# Runners API endpoint for the drain busy-check: org runners or repo runners
+# depending on scope. The runners register as "<github_runner_name>-<random>"
+# (RUNNER_NAME_PREFIX), so the busy-check matches that prefix — and a bare
+# "<github_runner_name>" too, in case a future image drops the suffix.
+github_runner_api_runners_url: >-
+  {{
+    github_runner_api_base ~ '/' ~
+    ('orgs/' ~ github_runner_org_name if github_runner_org else 'repos/' ~ github_runner_repo) ~
+    '/actions/runners?per_page=100'
+  }}
+github_runner_name_match: "^{{ github_runner_name | regex_escape }}(-|$)"
+
+# Poll interval clamped to >=1s so a consumer setting 0 (or a value that casts
+# to 0) can't divide the retry computation by zero or spin the API.
+github_runner_drain_poll_seconds_clamped: "{{ [github_runner_drain_poll_seconds | int, 1] | max }}"