Skip to content

Don't EACS the macOS 26 minis mid-task: know the 2600 pool, fail closed - #80

Merged
rcurranmoz merged 1 commit into
mainfrom
orchestrator-2600-pool-safety
Sep 24, 2026
Merged

rcurranmoz merged 1 commit into
mainfrom
orchestrator-2600-pool-safety

Conversation

@rcurranmoz

Copy link
Copy Markdown
Collaborator

The bug

macmini-m4-130 and m4-131 (macOS 26) run the gecko_t_osx_1500_m4 role but register in gecko-t-osx-2600-m4. candidate_pools() only listed the 1500 pools, so for these hosts:

  1. resolve() found them in no pool and set registered=False.
  2. For an unregistered host, the reprovision flow skips quarantine and drain.
  3. step_wipe's busy gate then asked the 1500 pool, and is_currently_busy treats that 404 as idle.

A Hangar reprovision of either host could EACS it mid-task. It's the same failure mode as the 2026-07-10 incident, reached by a different route. Found while probing m4-130 for RELOPS-2454, when a quarantine 404'd.

The fix: two layers

  • Know the pool. A role now maps to every prod pool it can be registered in, with the primary last so the unregistered fallback is unchanged. gecko_t_osx_1500_m4 → 2600-m4, 1500-m4, plus their -staging variants, probed staging first.
  • Fail closed if we still don't. For an unregistered host, step_wipe now runs pgrep for generic-worker and refuses to wipe if it's running, because a live worker means it is registered in a pool this map is missing. It also refuses if the check itself fails. A genuinely fresh host has no generic-worker, so it still wipes.

Tests

5 new tests, and 291 pass with op stripped from PATH; ruff is clean.

  • the pool map includes 2600 and keeps 1500-m4 as the fallback
  • resolve() finds a macOS 26 host in 2600
  • the wipe refuses when generic-worker is running
  • the wipe refuses when pgrep can't be run
  • the wipe proceeds on a clean unregistered host, and registered hosts skip the extra check

Mutation-checked: dropping the 2600 pool fails 2 tests, and disabling the gate fails 2.

Verified live (read-only resolve())

host main this branch
macmini-m4-130 registered=False, pool 1500-m4 registered=True, pool 2600-m4
macmini-m4-112 (staging) registered, 1500-m4-staging unchanged
macmini-m4-80 (prod) registered, 1500-m4 unchanged

🤖 Generated with Claude Code

macmini-m4-130/131 run the gecko_t_osx_1500_m4 role but register in
gecko-t-osx-2600-m4, which candidate_pools() didn't list. resolve() therefore
called them unregistered, and for an unregistered host the reprovision flow
skips quarantine and drain; step_wipe's busy check then asked the 1500 pool,
whose 404 is_currently_busy reads as idle. A Hangar reprovision of either host
could have wiped it mid-task.

- A role now maps to every prod pool it can be registered in (primary last,
  so the fallback is unchanged); gecko_t_osx_1500_m4 includes 2600-m4.
- step_wipe refuses an unregistered host if generic-worker is running on it,
  or if that can't be checked: a live worker means it IS registered, in a pool
  the orchestrator doesn't know. A genuinely fresh host still wipes.

Verified live (read-only resolve): main -> m4-130 registered=False; this
branch -> registered in gecko-t-osx-2600-m4; m4-112 / m4-80 unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@rcurranmoz
rcurranmoz requested a review from a team as a code owner September 24, 2026 18:53
@rcurranmoz
rcurranmoz merged commit 411aac8 into main Sep 24, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant