Skip to content

runner: drain in-flight jobs on SIGTERM instead of dying - #79

Merged
rcurranmoz merged 1 commit into
mainfrom
runner-graceful-drain
Sep 24, 2026
Merged

rcurranmoz merged 1 commit into
mainfrom
runner-graceful-drain

Conversation

@rcurranmoz

Copy link
Copy Markdown
Collaborator

Why

launchd sends SIGTERM to the runner on every restart: the step-ca cert renewal (step ca renew --daemon --exec launchctl kickstart -k …), a puppet reload's bootout, and a manual reload. The runner died immediately, and any in-flight job's reprovision subprocess died with it. Mid-EACS, that strands the host quarantined and leaves its Hangar job open forever.

What

  • On SIGTERM the runner stops claiming, finishes its in-flight jobs, and then exits. launchd's KeepAlive restarts it onto the new cert or code.
  • stop is re-checked before every claim, so a SIGTERM that lands mid fill-pass doesn't start new work. One of the new tests caught that gap in the first version.
  • The main loop moves into a testable _serve(client, cfg, pool, stop). Behaviour is otherwise unchanged.

Needs the ronin side: launchd only waits ExitTimeOut (default 20s) before SIGKILL. The companion ronin PR raises it for the runner plist. The two are independent and either can merge first: each alone behaves exactly like today.

Verified

  • launchd behaviour, on a throwaway LaunchAgent: kickstart -k delivers SIGTERM to the main process only, blocks until it exits (up to ExitTimeOut), and the child keeps running during the drain.
  • Real runner, locally: pointed at a dead API URL and sent SIGTERM, it logged "no new claims; draining" and then "drained … exiting", and exited cleanly.
  • 3 new tests:
    • no claims after stop, and _serve stays alive until the in-flight job finishes (5/5 repeat runs)
    • idle plus stop returns promptly
    • main() installs the handler
  • 285 tests pass with op removed from PATH; ruff is clean.

🤖 Generated with Claude Code

launchd SIGTERMs the runner on every restart: the step-ca renewal's
`kickstart -k`, a puppet reload's bootout. The runner died with it, and so
did any job's `reprovision` subprocess -- mid-EACS that strands the host,
quarantined, with its Hangar job never completed.

SIGTERM now stops new claims (re-checked per claim, so a signal mid fill pass
takes nothing new) and returns once in-flight jobs finish. launchd waits up to
the plist's ExitTimeOut before SIGKILL; the ronin side raises that to cover a
reprovision. Verified locally that `kickstart -k` delivers SIGTERM to the main
process only and waits, leaving the child running.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@rcurranmoz
rcurranmoz requested a review from a team as a code owner September 24, 2026 13:36
@rcurranmoz
rcurranmoz merged commit be9468a into main Sep 24, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant