Roadmap: capability wave — cost (spot/Graviton/warm pools), reliability (fallback/reaper/diagnostics), reach (matrix/disk/distros)

[kurok]

Where the 10x is

The Hardening & Modernization plan (#15, phases 1–8, plus #20) is done: aws-sdk v3, OIDC, IMDSv2, encrypted EBS, ephemeral non-root runners, structured logging, retries, unit tests. The action is now solid — the next multiplier is capable. A codebase review shows the ceilings:

1. Cost: on-demand only (src/aws.js builds plain RunInstances — no spot), x64 only (runner-checksums.js pins one arch), full cold boot every job. Spot + Graviton + warm pools compound to a 3–5× cost/latency improvement for the same CI minutes.
2. Reliability: one subnet, one instance type — any capacity hiccup fails the workflow; bootstrap failures "fail silently inside cloud-init, and the action will eventually time out" (README's own words); leaked instances (cancelled workflows, stop-job failures) bill until a human notices.
3. Reach: one runner per invocation (matrix builds need hand-wired duplicates), 8 GiB default disk kills Docker jobs, non-yum distros are told to fork.
4. Toil: runner-version bumps are a documented manual multi-file ritual (fix: bump default actions/runner to 2.335.1 #36 being the latest performance of it).

Phase 1 — Reliability first

• Bootstrap diagnostics: phone-home status tags, fast-fail, console-output capture, cleanup on failed start #41 Bootstrap diagnostics: phone-home status tags, fast-fail, console-output capture, terminate-on-failed-start — high (biggest MTTR win; several later items depend on its attribution)
• Orphan protection: instance TTL self-destruct + mode:cleanup reaper for leaked runners #42 Orphan protection: TTL self-destruct + mode: cleanup reaper — high (closes every instance-leak path)
• Capacity resilience: ordered fallback across subnets/AZs and instance types #40 Capacity resilience: ordered subnet + instance-type fallback chains — high

Phase 2 — Cost

• Spot instance support with on-demand fallback (market-type input) #39 Spot with on-demand fallback — high (60–90% off; ephemeral runners already fit the interruption model)
• ARM64/Graviton support (architecture input, per-arch runner checksums, AMI-arch validation) #43 ARM64/Graviton (architecture input, per-arch checksums, AMI-arch fail-fast) — medium
• Warm pools: reuse stopped instances (stop/start lifecycle) for 60%+ faster job starts #48 Warm pools: stop/start reuse for 60%+ faster job starts — medium (most design-sensitive; last in this phase)

Phase 3 — Capability & toil

• Root volume configuration: volume-size / volume-type / iops / throughput inputs #44 Root volume sizing (volume-size/type/iops/throughput) — medium (small, high frequency-of-pain; good early ship despite phase placement)
• Multi-runner batches: count input for matrix builds (N instances, one label) #45 Multi-runner batches (count: N, one label) for matrix builds — medium
• Pluggable bootstrap: pre-runner-script injection and user-data-template override (unfork Ubuntu et al.) #46 Pluggable bootstrap: pre-runner-script + user-data-template (converts "fork for Ubuntu" into "copy an example") — medium
• Automate actions/runner version bumps: scheduled PR workflow + bump script #47 Automated runner-version bump PRs (script + scheduled workflow) — medium (pure toil elimination; also feeds ARM64/Graviton support (architecture input, per-arch runner checksums, AMI-arch validation) #43's second checksum set)

Sequencing & dependencies

#41 diagnostics ─▶ #45 count (N-way wait visibility) · #46 templates (failure attribution) · #48 reuse (re-registration debugging)
#42 reaper ─▶ #48 warm pools (drain semantics extend the reaper)
#40 capacity ↔ #39 spot (independent, multiply together; spot makes capacity errors common, fallback makes them survivable)
#43 arm64 ─▶ #47 automation maintains both checksum sets (either order works; #47 is forward-compatible)
#44 volume · #47 automation — independent, can ship anytime

Suggested order: #41 → #42 → #44 → #40 → #39 → #47 → #43 → #45 → #46 → #48.

Considered and deferred

• Queue-depth autoscaling / webhook-driven pools (à la actions-runner-controller or philips-labs runners): a fundamentally different architecture (long-running control plane vs. a workflow-step action). The action's start/stop-inside-the-workflow model is its simplicity advantage — kept. Warm pools (Warm pools: reuse stopped instances (stop/start lifecycle) for 60%+ faster job starts #48) capture most of the latency benefit without a control plane.
• Windows runners: different bootstrap universe (PowerShell user-data); revisit if demand materializes — Pluggable bootstrap: pre-runner-script injection and user-data-template override (unfork Ubuntu et al.) #46's template contract is the likely vehicle.
• CreateFleet-based provisioning: would subsume Spot instance support with on-demand fallback (market-type input) #39/Capacity resilience: ordered fallback across subnets/AZs and instance types #40 natively but changes semantics and IAM surface; single-instance RunInstances model deliberately preserved.
• Runtime runner-version: latest: rejected — unpinned versions would defeat checksum verification; Automate actions/runner version bumps: scheduled PR workflow + bump script #47 (fast automated bumps) is the answer.

Cross-cutting rules (apply to every item)

• Back-compat: existing inputs keep working with byte-identical default behavior (regression tests on generated params/user-data are the standard); new inputs optional with safe defaults.
• No new leak paths: every launched resource is tagged with the action's signature, covered by TTL + reaper, and DeleteOnTermination where applicable.
• Secrets: registration tokens never appear in logs or captured console output (redaction tests extend with each feature).
• Repo standards: npm ci && npm run package with dist committed (verify-dist green), unit tests per the existing jest patterns, SHA-pinned workflow actions, IAM examples updated per feature (least-privilege with tag-scoped conditions), README inputs/outputs tables + action.yml descriptions in the same PR.

Individual issues carry full designs and acceptance criteria — review and adjust priorities there.

[kurok]

Roadmap complete ✅

All 10 items shipped, each as one PR, merged on green CI:

Phase 1 — Reliability

• Bootstrap diagnostics: phone-home status tags, fast-fail, console-output capture, cleanup on failed start #41 Bootstrap diagnostics (phone-home tags, fast-fail, console capture, cleanup-on-failure) — feat: bootstrap diagnostics (phone-home tags, fast-fail, console capture, cleanup on failed start) #50
• Orphan protection: instance TTL self-destruct + mode:cleanup reaper for leaked runners #42 Orphan protection (TTL self-destruct + mode: cleanup reaper) — feat: orphan protection — TTL self-destruct + mode:cleanup reaper #51
• Capacity resilience: ordered fallback across subnets/AZs and instance types #40 Capacity resilience (ordered subnet/AZ + instance-type fallback) — feat: capacity resilience — ordered subnet/AZ + instance-type fallback #53

Phase 2 — Cost

• Spot instance support with on-demand fallback (market-type input) #39 Spot instances with on-demand fallback — feat: spot instance support with on-demand fallback #54
• ARM64/Graviton support (architecture input, per-arch runner checksums, AMI-arch validation) #43 ARM64/Graviton support (+ AMI-arch fail-fast) — feat: ARM64/Graviton support (architecture input + AMI-arch validation) #56
• Warm pools: reuse stopped instances (stop/start lifecycle) for 60%+ faster job starts #48 Warm pools (reuse: stop, stop/start reuse) — feat: warm pools — reuse stopped instances (reuse: stop) #59

Phase 3 — Capability & toil

• Root volume configuration: volume-size / volume-type / iops / throughput inputs #44 Root volume configuration (volume-size/type/iops/throughput) — feat: root volume configuration (volume-size/type/iops/throughput) #52
• Automate actions/runner version bumps: scheduled PR workflow + bump script #47 Automated runner-version bump script + scheduled PR workflow — feat: automate actions/runner version bumps (bump script + scheduled PR) #55
• Multi-runner batches: count input for matrix builds (N instances, one label) #45 Multi-runner batches (count for matrix builds) — feat: multi-runner batches — count input for matrix builds #57
• Pluggable bootstrap: pre-runner-script injection and user-data-template override (unfork Ubuntu et al.) #46 Pluggable bootstrap (pre-runner-script + user-data-template; unforks Ubuntu) — feat: pluggable bootstrap — pre-runner-script + user-data-template #58

Test suite grew from 52 → 224 unit tests; lint + verify-dist green throughout.

Manual verification still owed (couldn't provision EC2 in CI): the real-infra e2e checklists in #43 (Graviton boot), #46 (Ubuntu template boot), and #48 (warm-start ≥60% timing). Warm pools (#48) also carries the design-sensitivity note this epic flagged — worth a design-review pass before heavy production use.

Closing as complete.