diff --git a/README.md b/README.md index 96e18584..6644b2e5 100644 --- a/README.md +++ b/README.md @@ -314,7 +314,10 @@ Now you're ready to go! | `spot-fallback` | Optional. Used only with `start` + `market-type: spot`. | What to do when spot capacity is unavailable: `on-demand` (default) retries the launch on-demand; `fail` surfaces the error. | | `spot-max-price` | Optional. Used only with `start` + `market-type: spot`. | Max spot price in USD/hour (e.g. `0.05`). Empty (default) caps at the on-demand price. | | `label` | Required if you use the `stop` mode. | Name of the unique label assigned to the runner.

The label is provided by the output of the action in the `start` mode.

The label is used to remove the runner from GitHub when the runner is not needed anymore. | -| `ec2-instance-id` | Required if you use the `stop` mode. | EC2 Instance Id of the created runner.

The id is provided by the output of the action in the `start` mode.

The id is used to terminate the EC2 instance when the runner is not needed anymore. | +| `count` | Optional. Used only with the `start` mode. | Number of runner instances to launch behind the single shared label (default `1`). Enables matrix builds — see [Matrix builds](#matrix-builds-multiple-runners). | +| `allow-partial` | Optional. Used only with the `start` mode (`count` > 1). | When `false` (default) the batch is all-or-nothing; when `true`, as few as 1 instance may launch, with the realized set in `ec2-instance-ids` and a warning. | +| `ec2-instance-id` | Required for `stop` mode (or `ec2-instance-ids`). | EC2 Instance Id of the created runner.

The id is provided by the output of the action in the `start` mode.

The id is used to terminate the EC2 instance when the runner is not needed anymore. | +| `ec2-instance-ids` | Optional. Used with the `stop` mode. | JSON array of instance ids to terminate, from the `ec2-instance-ids` output of a batched `start` (e.g. `["i-aaa","i-bbb"]`). Either this or `ec2-instance-id` is required to stop. | | `iam-role-name` | Optional. Used only with the `start` mode. | IAM role name to attach to the created EC2 runner.

This allows the runner to have permissions to run additional actions within the AWS account, without having to manage additional GitHub secrets and AWS users.

Setting this requires additional AWS permissions for the role launching the instance (see above). | | `aws-resource-tags` | Optional. Used only with the `start` mode. | Specifies tags to add to the EC2 instance and any attached storage.

This field is a stringified JSON array of tag objects, each containing a `Key` and `Value` field (see example below).

Setting this requires additional AWS permissions for the role launching the instance (see above). | | `eip-allocation-id` | Optional. Used only with the `start` mode. | Allocation Id of an Elastic IP to associate with the runner instance once it is running. | @@ -348,7 +351,8 @@ We recommend using [aws-actions/configure-aws-credentials](https://github.com/aw |               Name               | Description | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `label` | Name of the unique label assigned to the runner.

The label is used in two cases:
- to use as the input of `runs-on` property for the following jobs;
- to remove the runner from GitHub when it is not needed anymore. | -| `ec2-instance-id` | EC2 Instance Id of the created runner.

The id is used to terminate the EC2 instance when the runner is not needed anymore. | +| `ec2-instance-id` | EC2 Instance Id of the created runner (the first instance when `count` > 1, kept for compatibility).

Used to terminate the EC2 instance when the runner is not needed anymore. | +| `ec2-instance-ids` | JSON array of all instance ids launched by `start` (e.g. `["i-aaa","i-bbb"]`; a single-element array when `count` is 1). Pass to the `stop` mode to terminate the whole batch. | | `instance-type-used` | The EC2 instance type actually launched. With a [capacity-fallback](#capacity-fallback-across-azs-and-instance-types) list this may differ from your first choice. | | `subnet-id-used` | The subnet the runner was actually launched into. With a capacity-fallback list this may differ from your first choice. | | `market-type-used` | The market the runner launched in: `spot` or `on-demand`. Differs from `market-type` when spot fell back to on-demand. | @@ -426,6 +430,63 @@ In [this discussion](https://github.com/machulav/ec2-github-runner/discussions/1 If you use this action in your workflow, feel free to add your story there as well 🙌 +## Matrix builds (multiple runners) + +Matrix workflows need N runners. Instead of hand-wiring N start/stop jobs, launch a batch with `count` — all N register under the one shared label, and GitHub distributes the matrix jobs across them: + +```yml +jobs: + start-runners: + runs-on: ubuntu-latest + outputs: + label: ${{ steps.start.outputs.label }} + ids: ${{ steps.start.outputs.ec2-instance-ids }} + steps: + - uses: aws-actions/configure-aws-credentials@v4 + with: + aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} + aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} + aws-region: ${{ secrets.AWS_REGION }} + - id: start + uses: namecheap/ec2-github-runner@v3 + with: + mode: start + github-token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }} + ec2-image-id: ami-123 + ec2-instance-type: c7i.4xlarge + subnet-id: subnet-123 + security-group-id: sg-123 + count: 4 # launch 4 runners behind one label + + build: + needs: start-runners + runs-on: ${{ needs.start-runners.outputs.label }} # 4 jobs spread across the 4 runners + strategy: + matrix: + shard: [1, 2, 3, 4] + steps: + - run: echo "shard ${{ matrix.shard }}" + + stop-runners: + needs: [start-runners, build] + if: ${{ always() }} + runs-on: ubuntu-latest + steps: + - uses: aws-actions/configure-aws-credentials@v4 + with: + aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} + aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} + aws-region: ${{ secrets.AWS_REGION }} + - uses: namecheap/ec2-github-runner@v3 + with: + mode: stop + github-token: ${{ secrets.GH_PERSONAL_ACCESS_TOKEN }} + label: ${{ needs.start-runners.outputs.label }} + ec2-instance-ids: ${{ needs.start-runners.outputs.ids }} +``` + +The whole batch launches in one `RunInstances` call (all-or-nothing by default; `allow-partial: true` opts into a best-effort count). The start waits until **all N** runners register — if any instance fails to bootstrap, the start fails and **all** launched instances are terminated (no half-fleet leaks). `stop` deregisters all N runners and terminates all N instances, reporting per-instance outcomes. The [capacity-fallback](#capacity-fallback-across-azs-and-instance-types) chain retries placement for the whole batch (partial placement across subnets is out of scope). + ## Saving costs with spot CI runners are a textbook spot workload — short-lived, ephemeral (registered with `--ephemeral`), and restartable — and spot pricing is typically 60–90% below on-demand. Opt in with `market-type: spot`: diff --git a/action.yml b/action.yml index 634d9c4a..ab03f7ad 100644 --- a/action.yml +++ b/action.yml @@ -60,12 +60,34 @@ inputs: The runner doesn't require any inbound traffic. However, outbound traffic should be allowed. This input is required if you use the 'start' mode. required: false + count: + description: >- + Used only with the 'start' mode. Number of runner instances to launch + behind the single shared label (default 1). GitHub distributes matrix + jobs across runners sharing a label. One RunInstances call launches all + N; by default it is all-or-nothing (see allow-partial). + required: false + default: '1' + allow-partial: + description: >- + Used only with the 'start' mode with count > 1. When 'false' (default) + the batch is all-or-nothing (MinCount = count). When 'true', as few as + 1 instance may launch; the realized set is reported in the + 'ec2-instance-ids' output and a warning is logged. + required: false + default: 'false' label: description: >- Name of the unique label assigned to the runner. The label is used to remove the runner from GitHub when the runner is not needed anymore. This input is required if you use the 'stop' mode. required: false + ec2-instance-ids: + description: >- + JSON array of EC2 instance ids to terminate, as emitted by a batched + 'start' (the 'ec2-instance-ids' output). Used with the 'stop' mode; + either this or 'ec2-instance-id' is required to stop. + required: false ec2-instance-id: description: >- EC2 Instance Id of the created runner. @@ -229,6 +251,12 @@ outputs: description: >- EC2 Instance Id of the created runner. The id is used to terminate the EC2 instance when the runner is not needed anymore. + ec2-instance-ids: + description: >- + JSON array of all EC2 instance ids launched by a batched 'start' + (e.g. '["i-aaa","i-bbb"]'). For count=1 this is a single-element array; + 'ec2-instance-id' remains the first id for compatibility. Pass this to + the 'stop' mode to terminate the whole batch. instance-type-used: description: >- The EC2 instance type that was actually launched (start mode). With a diff --git a/dist/index.js b/dist/index.js index 987b0ded..c2f7f35f 100644 --- a/dist/index.js +++ b/dist/index.js @@ -104727,8 +104727,8 @@ async function launchWithFallback(attempt, instanceTypes, subnetIds) { for (const instanceType of instanceTypes) { for (const subnetId of subnetIds) { try { - const instanceId = await attempt(instanceType, subnetId); - return { instanceId, instanceType, subnetId }; + const instanceIds = await attempt(instanceType, subnetId); + return { instanceIds, instanceType, subnetId }; } catch (error) { const kind = classifyRunError(error); if (kind === 'capacity') { @@ -105162,12 +105162,18 @@ async function startEc2Instance(label, githubRegistrationToken) { log.warn('ami_architecture', { applied: false, reason: 'AMI did not report an architecture — skipping arch validation', ami_id: resolved.id }); } + // Batch size: MaxCount instances behind one shared label. Default all-or- + // nothing (MinCount == MaxCount) for clean matrix capacity semantics; + // allow-partial opts into MinCount 1 (realized count reported in outputs). + const count = Number(config.input.count); + const allowPartial = config.input.allowPartial === 'true'; + // InstanceType and SubnetId are injected per attempt by the fallback // chain (see below), so they are intentionally absent from the base. const params = { ImageId: config.input.ec2ImageId, - MinCount: 1, - MaxCount: 1, + MinCount: allowPartial ? 1 : count, + MaxCount: count, UserData: Buffer.from(userData).toString('base64'), SecurityGroupIds: [config.input.securityGroupId], IamInstanceProfile: { Name: config.input.iamRoleName }, @@ -105239,7 +105245,7 @@ async function startEc2Instance(label, githubRegistrationToken) { () => client.send(new RunInstancesCommand(attemptParams)), { shouldRetry: (error) => classifyRunError(error) === 'transient' }, ); - return result.Instances[0].InstanceId; + return result.Instances.map((instance) => instance.InstanceId); }; const marketPlan = buildMarketPlan(config.input.marketType, config.input.spotFallback); @@ -105257,26 +105263,36 @@ async function startEc2Instance(label, githubRegistrationToken) { core.error('AWS EC2 instance starting error'); throw error; } - const ec2InstanceId = placement.instanceId; - log.info('run_instance', { instance_id: ec2InstanceId, instance_type: placement.instanceType, subnet_id: placement.subnetId, market_type: placement.marketType, elapsed_ms: Date.now() - runStart }); - core.info(`AWS EC2 instance ${ec2InstanceId} is started (type ${placement.instanceType}, subnet ${placement.subnetId}, ${placement.marketType})`); + const instanceIds = placement.instanceIds; + if (instanceIds.length < count) { + log.warn('run_instance', { requested: count, realized: instanceIds.length, allow_partial: allowPartial }); + core.warning(`Requested ${count} instance(s) but only ${instanceIds.length} launched (allow-partial).`); + } + log.info('run_instance', { instance_ids: instanceIds, instance_type: placement.instanceType, subnet_id: placement.subnetId, market_type: placement.marketType, elapsed_ms: Date.now() - runStart }); + core.info(`Started ${instanceIds.length} EC2 instance(s) [${instanceIds.join(', ')}] (type ${placement.instanceType}, subnet ${placement.subnetId}, ${placement.marketType})`); + // Elastic IP association only makes sense for a single instance — a lone + // EIP can't attach to N runners. if (config.input.eipAllocationId) { - await waitForInstanceRunning(ec2InstanceId); - - try { - log.info('associate_address', { allocation_id: config.input.eipAllocationId, instance_id: ec2InstanceId }); - await client.send(new AssociateAddressCommand({ - AllocationId: config.input.eipAllocationId, - InstanceId: ec2InstanceId, - })); - } catch (error) { - log.warn('associate_address', { allocation_id: config.input.eipAllocationId, instance_id: ec2InstanceId, error: error.name, message: error.message }); - core.warning(`Elastic IP association error, trying to proceed w/o EIP: ${error.message}`); + if (instanceIds.length !== 1) { + log.warn('associate_address', { skipped: true, reason: 'eip-allocation-id is ignored for multi-instance batches', count: instanceIds.length }); + core.warning('eip-allocation-id is ignored when count > 1 (a single EIP cannot attach to multiple instances).'); + } else { + await waitForInstanceRunning(instanceIds[0]); + try { + log.info('associate_address', { allocation_id: config.input.eipAllocationId, instance_id: instanceIds[0] }); + await client.send(new AssociateAddressCommand({ + AllocationId: config.input.eipAllocationId, + InstanceId: instanceIds[0], + })); + } catch (error) { + log.warn('associate_address', { allocation_id: config.input.eipAllocationId, instance_id: instanceIds[0], error: error.name, message: error.message }); + core.warning(`Elastic IP association error, trying to proceed w/o EIP: ${error.message}`); + } } } - return { instanceId: ec2InstanceId, instanceType: placement.instanceType, subnetId: placement.subnetId, marketType: placement.marketType }; + return { instanceIds, instanceType: placement.instanceType, subnetId: placement.subnetId, marketType: placement.marketType }; } async function terminateInstanceById(ec2InstanceId) { @@ -105289,6 +105305,9 @@ async function terminateInstanceById(ec2InstanceId) { client.send(new TerminateInstancesCommand({ InstanceIds: [ec2InstanceId], })), + // An already-gone instance (InvalidInstanceID.NotFound) is terminal — + // don't burn retries; the caller treats it as already-terminated. + { shouldRetry: (error) => !(error.name && error.name.includes('NotFound')) }, ); log.info('terminate_instance', { instance_id: ec2InstanceId, elapsed_ms: Date.now() - start }); core.info(`AWS EC2 instance ${ec2InstanceId} is terminated`); @@ -105299,10 +105318,6 @@ async function terminateInstanceById(ec2InstanceId) { } } -async function terminateEc2Instance() { - return terminateInstanceById(config.input.ec2InstanceId); -} - // Read the instance's bootstrap phone-home tag. Returns the tag value // (e.g. 'downloading', 'failed:configuring') or null when the tag is not // yet set. Missing ec2:DescribeTags permission (or a transient API error) @@ -105325,6 +105340,20 @@ async function getBootstrapStatus(ec2InstanceId) { } } +// For a batch, return the first instance's `failed:` status (so the +// wait loop can fail fast if ANY instance's bootstrap aborts), or null when +// none have failed. Makes N-way waits fail as fast as single ones. +async function getBatchBootstrapStatus(ec2InstanceIds) { + for (const id of ec2InstanceIds) { + const status = await getBootstrapStatus(id); + if (typeof status === 'string' && status.startsWith('failed:')) { + log.warn('bootstrap_status', { instance_id: id, status }); + return status; + } + } + return null; +} + // Replace every occurrence of each secret value with '***'. Uses literal // (non-regex) replacement so tokens containing regex metacharacters are // still fully scrubbed. @@ -105372,31 +105401,39 @@ async function getConsoleOutputTail(ec2InstanceId, opts = {}) { return redactSecrets(tail, opts.redactValues); } -// Handle a failed start: capture and print the instance's console output -// (collapsible group, secrets redacted), then either terminate the -// instance (default, so failed starts don't leak billing) or preserve it -// for interactive debugging when cleanup-on-start-failure is 'false'. -// Order is capture-then-terminate so the diagnostics survive the cleanup. -async function handleStartFailure(ec2InstanceId, opts = {}) { - const tail = await getConsoleOutputTail(ec2InstanceId, { redactValues: opts.redactValues }); - core.startGroup(`EC2 instance ${ec2InstanceId} console output (last ${CONSOLE_TAIL_LINES} lines)`); - core.info(tail || '(no console output was available yet — the instance may have failed before cloud-init produced output)'); - core.endGroup(); +// Handle a failed start: for every launched instance, capture and print its +// console output (collapsible group, secrets redacted), then either +// terminate all of them (default, so a failed start — including one bad +// instance in a batch — never leaks billing) or preserve them for +// interactive debugging when cleanup-on-start-failure is 'false'. Capture +// happens before termination so the diagnostics survive the cleanup. +// Accepts a single id or an array (single-instance callers unchanged). +async function handleStartFailure(ec2InstanceIds, opts = {}) { + const ids = Array.isArray(ec2InstanceIds) ? ec2InstanceIds : [ec2InstanceIds]; + + for (const id of ids) { + const tail = await getConsoleOutputTail(id, { redactValues: opts.redactValues }); + core.startGroup(`EC2 instance ${id} console output (last ${CONSOLE_TAIL_LINES} lines)`); + core.info(tail || '(no console output was available yet — the instance may have failed before cloud-init produced output)'); + core.endGroup(); + } if (config.input.cleanupOnStartFailure === 'true') { - log.info('cleanup_on_start_failure', { instance_id: ec2InstanceId, action: 'terminate' }); - try { - await terminateInstanceById(ec2InstanceId); - } catch (error) { - log.error('cleanup_on_start_failure', { instance_id: ec2InstanceId, error: error.name, message: error.message }); - core.warning(`Could not terminate failed instance ${ec2InstanceId}: ${error.message}. Terminate it manually to avoid charges.`); + for (const id of ids) { + log.info('cleanup_on_start_failure', { instance_id: id, action: 'terminate' }); + try { + await terminateInstanceById(id); + } catch (error) { + log.error('cleanup_on_start_failure', { instance_id: id, error: error.name, message: error.message }); + core.warning(`Could not terminate failed instance ${id}: ${error.message}. Terminate it manually to avoid charges.`); + } } } else { - log.info('cleanup_on_start_failure', { instance_id: ec2InstanceId, action: 'preserve' }); + log.info('cleanup_on_start_failure', { instance_ids: ids, action: 'preserve' }); core.warning( - `Instance ${ec2InstanceId} was left running for debugging (cleanup-on-start-failure: false).\n` + - `Inspect it with:\n aws ec2 get-console-output --latest --instance-id ${ec2InstanceId}\n` + - `Terminate it when done:\n aws ec2 terminate-instances --instance-ids ${ec2InstanceId}`, + `Instance(s) ${ids.join(', ')} left running for debugging (cleanup-on-start-failure: false).\n` + + `Inspect with:\n aws ec2 get-console-output --latest --instance-id \n` + + `Terminate when done:\n aws ec2 terminate-instances --instance-ids ${ids.join(' ')}`, ); } } @@ -105445,10 +105482,10 @@ async function listManagedInstances(repo) { module.exports = { startEc2Instance, - terminateEc2Instance, terminateInstanceById, waitForInstanceRunning, getBootstrapStatus, + getBatchBootstrapStatus, getConsoleOutputTail, handleStartFailure, listManagedInstances, @@ -105642,6 +105679,9 @@ class Config { eipAllocationId: core.getInput('eip-allocation-id'), label: core.getInput('label'), ec2InstanceId: core.getInput('ec2-instance-id'), + ec2InstanceIds: core.getInput('ec2-instance-ids') ? JSON.parse(core.getInput('ec2-instance-ids')) : null, + count: core.getInput('count') || '1', + allowPartial: core.getInput('allow-partial') || 'false', iamRoleName: core.getInput('iam-role-name'), runnerVersion: core.getInput('runner-version') || '2.335.1', architecture: core.getInput('architecture') || 'x64', @@ -105694,8 +105734,11 @@ class Config { this.validateVolumeInputs(); this.validateMarketInputs(); this.validateArchitectureInputs(); + this.validateCountInput(); } else if (this.input.mode === 'stop') { - if (!this.input.label || !this.input.ec2InstanceId) { + // A stop needs the shared label plus at least one instance id — either + // the compat scalar or the JSON array from a batched start. + if (!this.input.label || (!this.input.ec2InstanceId && !(this.input.ec2InstanceIds && this.input.ec2InstanceIds.length))) { throw new Error(`Not all the required inputs are provided for the 'stop' mode`); } } else if (this.input.mode === 'cleanup') { @@ -105770,6 +105813,13 @@ class Config { } } + // Validate the multi-runner batch size. + validateCountInput() { + if (!(/^[0-9]+$/.test(this.input.count) && Number(this.input.count) >= 1)) { + throw new Error(`'count' must be a positive integer`); + } + } + generateUniqueLabel() { return Math.random().toString(36).substr(2, 5); } @@ -105835,30 +105885,6 @@ async function deregisterRunner(runnerId) { ); } -async function removeRunner() { - const runner = await getRunner(config.input.label); - - // skip the runner removal process if the runner is not found - if (!runner) { - log.info('remove_runner', { label: config.input.label, skipped: true, reason: 'not_found' }); - core.info(`GitHub self-hosted runner with label ${config.input.label} is not found, so the removal is skipped`); - return; - } - - const start = Date.now(); - log.info('remove_runner', { runner_id: runner.id, label: config.input.label }); - try { - await deregisterRunner(runner.id); - log.info('remove_runner', { runner_id: runner.id, label: config.input.label, elapsed_ms: Date.now() - start }); - core.info(`GitHub self-hosted runner ${runner.name} is removed`); - return; - } catch (error) { - log.error('remove_runner', { runner_id: runner.id, label: config.input.label, error: error.name, message: error.message }); - core.error('GitHub self-hosted runner removal error'); - throw error; - } -} - // True once the runner for `label` has registered with GitHub and reports // as online. Used as the success signal by the start action's wait loop // (see src/wait.js), polled alongside the instance's bootstrap tag. @@ -105868,11 +105894,59 @@ async function isRunnerOnline(label) { return !!(runner && runner.status === 'online'); } +// Count how many runners carrying `label` are registered and online. Used by +// the batch wait loop, which succeeds only once all N runners are up. +async function countOnlineRunners(label) { + const octokit = github.getOctokit(config.input.githubToken); + try { + const runners = await octokit.paginate('GET /repos/{owner}/{repo}/actions/runners', config.githubContext); + const online = runners.filter((r) => r.labels.some((l) => l.name === label) && r.status === 'online'); + log.debug('runner_status', { label, online: online.length }); + return online.length; + } catch (_error) { + return 0; + } +} + +// Deregister EVERY runner carrying `label` (a batch registers N runners under +// one shared label). Returns per-runner outcomes; a single failure does not +// abort the rest. Idempotent — no matching runners is a clean no-op. +async function removeAllRunners(label) { + const octokit = github.getOctokit(config.input.githubToken); + let runners; + try { + const all = await octokit.paginate('GET /repos/{owner}/{repo}/actions/runners', config.githubContext); + runners = all.filter((r) => r.labels.some((l) => l.name === label)); + } catch (error) { + throw new Error(`could not list runners for label ${label}: ${error.message}`, { cause: error }); + } + + if (runners.length === 0) { + log.info('remove_runner', { label, skipped: true, reason: 'not_found' }); + core.info(`No GitHub self-hosted runners found for label ${label}; removal skipped`); + return { removed: 0, failures: [] }; + } + + const failures = []; + for (const runner of runners) { + try { + await deregisterRunner(runner.id); + log.info('remove_runner', { runner_id: runner.id, label }); + } catch (error) { + log.error('remove_runner', { runner_id: runner.id, label, error: error.name, message: error.message }); + failures.push({ runnerId: runner.id, message: error.message }); + } + } + core.info(`Removed ${runners.length - failures.length}/${runners.length} GitHub self-hosted runners for label ${label}`); + return { removed: runners.length - failures.length, failures }; +} + module.exports = { getRegistrationToken, - removeRunner, deregisterRunner, isRunnerOnline, + countOnlineRunners, + removeAllRunners, getRunner, }; @@ -111085,17 +111159,21 @@ const core = __nccwpck_require__(7484); // command; GitHub runners now surface that as a warning. Bypass the // legacy path — modern runners always set GITHUB_OUTPUT. function setOutput(label, placement) { - const { instanceId, instanceType, subnetId, marketType } = placement; + const { instanceIds, instanceType, subnetId, marketType } = placement; + // Compat scalar: the first instance id. Batch consumers use the JSON array. + const firstId = instanceIds[0]; + const idsJson = JSON.stringify(instanceIds); const outputFile = process.env.GITHUB_OUTPUT; if (outputFile) { fs.appendFileSync( outputFile, - `label=${label}${os.EOL}ec2-instance-id=${instanceId}${os.EOL}instance-type-used=${instanceType}${os.EOL}subnet-id-used=${subnetId}${os.EOL}market-type-used=${marketType}${os.EOL}`, + `label=${label}${os.EOL}ec2-instance-id=${firstId}${os.EOL}ec2-instance-ids=${idsJson}${os.EOL}instance-type-used=${instanceType}${os.EOL}subnet-id-used=${subnetId}${os.EOL}market-type-used=${marketType}${os.EOL}`, ); return; } core.setOutput('label', label); - core.setOutput('ec2-instance-id', instanceId); + core.setOutput('ec2-instance-id', firstId); + core.setOutput('ec2-instance-ids', idsJson); core.setOutput('instance-type-used', instanceType); core.setOutput('subnet-id-used', subnetId); core.setOutput('market-type-used', marketType); @@ -111108,25 +111186,27 @@ async function start() { const label = config.generateUniqueLabel(); const githubRegistrationToken = await gh.getRegistrationToken(); const placement = await aws.startEc2Instance(label, githubRegistrationToken); - const ec2InstanceId = placement.instanceId; + const instanceIds = placement.instanceIds; setOutput(label, placement); - await aws.waitForInstanceRunning(ec2InstanceId); + for (const id of instanceIds) { + await aws.waitForInstanceRunning(id); + } - // Watch the bootstrap phone-home tag and GitHub registration together. - // On a bootstrap failure or registration timeout, capture the console - // output and (by default) terminate the instance so failed starts - // don't leak billing. The registration token is redacted from the - // captured output. + // Watch the bootstrap phone-home tags and GitHub registration together. + // The batch is ready only when ALL N runners are online; a bootstrap + // failure on ANY instance fails fast. On failure or timeout, capture the + // console output of every instance and (by default) terminate them all — + // no half-fleet is left leaking billing. The token is redacted. try { await waitForRunnerReady({ - getBootstrapStatus: () => aws.getBootstrapStatus(ec2InstanceId), - isRunnerOnline: () => gh.isRunnerOnline(label), + getBootstrapStatus: () => aws.getBatchBootstrapStatus(instanceIds), + isRunnerOnline: async () => (await gh.countOnlineRunners(label)) >= instanceIds.length, }); } catch (waitError) { - await aws.handleStartFailure(ec2InstanceId, { redactValues: [githubRegistrationToken] }); + await aws.handleStartFailure(instanceIds, { redactValues: [githubRegistrationToken] }); throw waitError; } - log.info('start', { label, instance_id: ec2InstanceId, instance_type: placement.instanceType, subnet_id: placement.subnetId, outcome: 'registered' }); + log.info('start', { label, instance_ids: instanceIds, instance_type: placement.instanceType, subnet_id: placement.subnetId, outcome: 'registered' }); } finally { core.endGroup(); } @@ -111138,17 +111218,32 @@ async function stop() { try { log.debug('stop_inputs', config.input); - // Attempt both cleanups independently — neither should short-circuit - // the other. A GitHub API failure must not prevent EC2 termination - // (billing) and vice versa. Both have internal retries via - // withRetry(); catch here is the last line of defense. - try { - await aws.terminateEc2Instance(); - } catch (error) { - failures.push({ step: 'terminate_instance', error: error.name, message: error.message }); + // Terminate every instance from the batch (the JSON array), or the single + // compat scalar. Independent per instance — one failure doesn't stop the + // rest — and idempotent: an already-gone instance is not a failure. + const instanceIds = (config.input.ec2InstanceIds && config.input.ec2InstanceIds.length) + ? config.input.ec2InstanceIds + : [config.input.ec2InstanceId]; + + for (const id of instanceIds) { + try { + await aws.terminateInstanceById(id); + } catch (error) { + if (error.name && error.name.includes('NotFound')) { + log.info('terminate_instance', { instance_id: id, skipped: true, reason: 'already_gone' }); + } else { + failures.push({ step: `terminate_instance:${id}`, error: error.name, message: error.message }); + } + } } + + // Deregister ALL runners sharing the label (a batch registers N). A + // GitHub failure must not have prevented the terminations above. try { - await gh.removeRunner(); + const result = await gh.removeAllRunners(config.input.label); + for (const f of result.failures) { + failures.push({ step: `remove_runner:${f.runnerId}`, message: f.message }); + } } catch (error) { failures.push({ step: 'remove_runner', error: error.name, message: error.message }); } @@ -111158,7 +111253,7 @@ async function stop() { const summary = failures.map((f) => `${f.step}: ${f.message}`).join('; '); throw new Error(`stop mode completed with ${failures.length} cleanup failure(s): ${summary}`); } - log.info('stop', { instance_id: config.input.ec2InstanceId, label: config.input.label, outcome: 'ok' }); + log.info('stop', { instance_ids: instanceIds, label: config.input.label, outcome: 'ok' }); } finally { core.endGroup(); } diff --git a/src/aws.js b/src/aws.js index 362a03bc..9e378986 100644 --- a/src/aws.js +++ b/src/aws.js @@ -77,8 +77,8 @@ async function launchWithFallback(attempt, instanceTypes, subnetIds) { for (const instanceType of instanceTypes) { for (const subnetId of subnetIds) { try { - const instanceId = await attempt(instanceType, subnetId); - return { instanceId, instanceType, subnetId }; + const instanceIds = await attempt(instanceType, subnetId); + return { instanceIds, instanceType, subnetId }; } catch (error) { const kind = classifyRunError(error); if (kind === 'capacity') { @@ -512,12 +512,18 @@ async function startEc2Instance(label, githubRegistrationToken) { log.warn('ami_architecture', { applied: false, reason: 'AMI did not report an architecture — skipping arch validation', ami_id: resolved.id }); } + // Batch size: MaxCount instances behind one shared label. Default all-or- + // nothing (MinCount == MaxCount) for clean matrix capacity semantics; + // allow-partial opts into MinCount 1 (realized count reported in outputs). + const count = Number(config.input.count); + const allowPartial = config.input.allowPartial === 'true'; + // InstanceType and SubnetId are injected per attempt by the fallback // chain (see below), so they are intentionally absent from the base. const params = { ImageId: config.input.ec2ImageId, - MinCount: 1, - MaxCount: 1, + MinCount: allowPartial ? 1 : count, + MaxCount: count, UserData: Buffer.from(userData).toString('base64'), SecurityGroupIds: [config.input.securityGroupId], IamInstanceProfile: { Name: config.input.iamRoleName }, @@ -589,7 +595,7 @@ async function startEc2Instance(label, githubRegistrationToken) { () => client.send(new RunInstancesCommand(attemptParams)), { shouldRetry: (error) => classifyRunError(error) === 'transient' }, ); - return result.Instances[0].InstanceId; + return result.Instances.map((instance) => instance.InstanceId); }; const marketPlan = buildMarketPlan(config.input.marketType, config.input.spotFallback); @@ -607,26 +613,36 @@ async function startEc2Instance(label, githubRegistrationToken) { core.error('AWS EC2 instance starting error'); throw error; } - const ec2InstanceId = placement.instanceId; - log.info('run_instance', { instance_id: ec2InstanceId, instance_type: placement.instanceType, subnet_id: placement.subnetId, market_type: placement.marketType, elapsed_ms: Date.now() - runStart }); - core.info(`AWS EC2 instance ${ec2InstanceId} is started (type ${placement.instanceType}, subnet ${placement.subnetId}, ${placement.marketType})`); + const instanceIds = placement.instanceIds; + if (instanceIds.length < count) { + log.warn('run_instance', { requested: count, realized: instanceIds.length, allow_partial: allowPartial }); + core.warning(`Requested ${count} instance(s) but only ${instanceIds.length} launched (allow-partial).`); + } + log.info('run_instance', { instance_ids: instanceIds, instance_type: placement.instanceType, subnet_id: placement.subnetId, market_type: placement.marketType, elapsed_ms: Date.now() - runStart }); + core.info(`Started ${instanceIds.length} EC2 instance(s) [${instanceIds.join(', ')}] (type ${placement.instanceType}, subnet ${placement.subnetId}, ${placement.marketType})`); + // Elastic IP association only makes sense for a single instance — a lone + // EIP can't attach to N runners. if (config.input.eipAllocationId) { - await waitForInstanceRunning(ec2InstanceId); - - try { - log.info('associate_address', { allocation_id: config.input.eipAllocationId, instance_id: ec2InstanceId }); - await client.send(new AssociateAddressCommand({ - AllocationId: config.input.eipAllocationId, - InstanceId: ec2InstanceId, - })); - } catch (error) { - log.warn('associate_address', { allocation_id: config.input.eipAllocationId, instance_id: ec2InstanceId, error: error.name, message: error.message }); - core.warning(`Elastic IP association error, trying to proceed w/o EIP: ${error.message}`); + if (instanceIds.length !== 1) { + log.warn('associate_address', { skipped: true, reason: 'eip-allocation-id is ignored for multi-instance batches', count: instanceIds.length }); + core.warning('eip-allocation-id is ignored when count > 1 (a single EIP cannot attach to multiple instances).'); + } else { + await waitForInstanceRunning(instanceIds[0]); + try { + log.info('associate_address', { allocation_id: config.input.eipAllocationId, instance_id: instanceIds[0] }); + await client.send(new AssociateAddressCommand({ + AllocationId: config.input.eipAllocationId, + InstanceId: instanceIds[0], + })); + } catch (error) { + log.warn('associate_address', { allocation_id: config.input.eipAllocationId, instance_id: instanceIds[0], error: error.name, message: error.message }); + core.warning(`Elastic IP association error, trying to proceed w/o EIP: ${error.message}`); + } } } - return { instanceId: ec2InstanceId, instanceType: placement.instanceType, subnetId: placement.subnetId, marketType: placement.marketType }; + return { instanceIds, instanceType: placement.instanceType, subnetId: placement.subnetId, marketType: placement.marketType }; } async function terminateInstanceById(ec2InstanceId) { @@ -639,6 +655,9 @@ async function terminateInstanceById(ec2InstanceId) { client.send(new TerminateInstancesCommand({ InstanceIds: [ec2InstanceId], })), + // An already-gone instance (InvalidInstanceID.NotFound) is terminal — + // don't burn retries; the caller treats it as already-terminated. + { shouldRetry: (error) => !(error.name && error.name.includes('NotFound')) }, ); log.info('terminate_instance', { instance_id: ec2InstanceId, elapsed_ms: Date.now() - start }); core.info(`AWS EC2 instance ${ec2InstanceId} is terminated`); @@ -649,10 +668,6 @@ async function terminateInstanceById(ec2InstanceId) { } } -async function terminateEc2Instance() { - return terminateInstanceById(config.input.ec2InstanceId); -} - // Read the instance's bootstrap phone-home tag. Returns the tag value // (e.g. 'downloading', 'failed:configuring') or null when the tag is not // yet set. Missing ec2:DescribeTags permission (or a transient API error) @@ -675,6 +690,20 @@ async function getBootstrapStatus(ec2InstanceId) { } } +// For a batch, return the first instance's `failed:` status (so the +// wait loop can fail fast if ANY instance's bootstrap aborts), or null when +// none have failed. Makes N-way waits fail as fast as single ones. +async function getBatchBootstrapStatus(ec2InstanceIds) { + for (const id of ec2InstanceIds) { + const status = await getBootstrapStatus(id); + if (typeof status === 'string' && status.startsWith('failed:')) { + log.warn('bootstrap_status', { instance_id: id, status }); + return status; + } + } + return null; +} + // Replace every occurrence of each secret value with '***'. Uses literal // (non-regex) replacement so tokens containing regex metacharacters are // still fully scrubbed. @@ -722,31 +751,39 @@ async function getConsoleOutputTail(ec2InstanceId, opts = {}) { return redactSecrets(tail, opts.redactValues); } -// Handle a failed start: capture and print the instance's console output -// (collapsible group, secrets redacted), then either terminate the -// instance (default, so failed starts don't leak billing) or preserve it -// for interactive debugging when cleanup-on-start-failure is 'false'. -// Order is capture-then-terminate so the diagnostics survive the cleanup. -async function handleStartFailure(ec2InstanceId, opts = {}) { - const tail = await getConsoleOutputTail(ec2InstanceId, { redactValues: opts.redactValues }); - core.startGroup(`EC2 instance ${ec2InstanceId} console output (last ${CONSOLE_TAIL_LINES} lines)`); - core.info(tail || '(no console output was available yet — the instance may have failed before cloud-init produced output)'); - core.endGroup(); +// Handle a failed start: for every launched instance, capture and print its +// console output (collapsible group, secrets redacted), then either +// terminate all of them (default, so a failed start — including one bad +// instance in a batch — never leaks billing) or preserve them for +// interactive debugging when cleanup-on-start-failure is 'false'. Capture +// happens before termination so the diagnostics survive the cleanup. +// Accepts a single id or an array (single-instance callers unchanged). +async function handleStartFailure(ec2InstanceIds, opts = {}) { + const ids = Array.isArray(ec2InstanceIds) ? ec2InstanceIds : [ec2InstanceIds]; + + for (const id of ids) { + const tail = await getConsoleOutputTail(id, { redactValues: opts.redactValues }); + core.startGroup(`EC2 instance ${id} console output (last ${CONSOLE_TAIL_LINES} lines)`); + core.info(tail || '(no console output was available yet — the instance may have failed before cloud-init produced output)'); + core.endGroup(); + } if (config.input.cleanupOnStartFailure === 'true') { - log.info('cleanup_on_start_failure', { instance_id: ec2InstanceId, action: 'terminate' }); - try { - await terminateInstanceById(ec2InstanceId); - } catch (error) { - log.error('cleanup_on_start_failure', { instance_id: ec2InstanceId, error: error.name, message: error.message }); - core.warning(`Could not terminate failed instance ${ec2InstanceId}: ${error.message}. Terminate it manually to avoid charges.`); + for (const id of ids) { + log.info('cleanup_on_start_failure', { instance_id: id, action: 'terminate' }); + try { + await terminateInstanceById(id); + } catch (error) { + log.error('cleanup_on_start_failure', { instance_id: id, error: error.name, message: error.message }); + core.warning(`Could not terminate failed instance ${id}: ${error.message}. Terminate it manually to avoid charges.`); + } } } else { - log.info('cleanup_on_start_failure', { instance_id: ec2InstanceId, action: 'preserve' }); + log.info('cleanup_on_start_failure', { instance_ids: ids, action: 'preserve' }); core.warning( - `Instance ${ec2InstanceId} was left running for debugging (cleanup-on-start-failure: false).\n` + - `Inspect it with:\n aws ec2 get-console-output --latest --instance-id ${ec2InstanceId}\n` + - `Terminate it when done:\n aws ec2 terminate-instances --instance-ids ${ec2InstanceId}`, + `Instance(s) ${ids.join(', ')} left running for debugging (cleanup-on-start-failure: false).\n` + + `Inspect with:\n aws ec2 get-console-output --latest --instance-id \n` + + `Terminate when done:\n aws ec2 terminate-instances --instance-ids ${ids.join(' ')}`, ); } } @@ -795,10 +832,10 @@ async function listManagedInstances(repo) { module.exports = { startEc2Instance, - terminateEc2Instance, terminateInstanceById, waitForInstanceRunning, getBootstrapStatus, + getBatchBootstrapStatus, getConsoleOutputTail, handleStartFailure, listManagedInstances, diff --git a/src/config.js b/src/config.js index 933ac417..af9ca41e 100644 --- a/src/config.js +++ b/src/config.js @@ -19,6 +19,9 @@ class Config { eipAllocationId: core.getInput('eip-allocation-id'), label: core.getInput('label'), ec2InstanceId: core.getInput('ec2-instance-id'), + ec2InstanceIds: core.getInput('ec2-instance-ids') ? JSON.parse(core.getInput('ec2-instance-ids')) : null, + count: core.getInput('count') || '1', + allowPartial: core.getInput('allow-partial') || 'false', iamRoleName: core.getInput('iam-role-name'), runnerVersion: core.getInput('runner-version') || '2.335.1', architecture: core.getInput('architecture') || 'x64', @@ -71,8 +74,11 @@ class Config { this.validateVolumeInputs(); this.validateMarketInputs(); this.validateArchitectureInputs(); + this.validateCountInput(); } else if (this.input.mode === 'stop') { - if (!this.input.label || !this.input.ec2InstanceId) { + // A stop needs the shared label plus at least one instance id — either + // the compat scalar or the JSON array from a batched start. + if (!this.input.label || (!this.input.ec2InstanceId && !(this.input.ec2InstanceIds && this.input.ec2InstanceIds.length))) { throw new Error(`Not all the required inputs are provided for the 'stop' mode`); } } else if (this.input.mode === 'cleanup') { @@ -147,6 +153,13 @@ class Config { } } + // Validate the multi-runner batch size. + validateCountInput() { + if (!(/^[0-9]+$/.test(this.input.count) && Number(this.input.count) >= 1)) { + throw new Error(`'count' must be a positive integer`); + } + } + generateUniqueLabel() { return Math.random().toString(36).substr(2, 5); } diff --git a/src/gh.js b/src/gh.js index 35b1913d..de392c7e 100644 --- a/src/gh.js +++ b/src/gh.js @@ -45,30 +45,6 @@ async function deregisterRunner(runnerId) { ); } -async function removeRunner() { - const runner = await getRunner(config.input.label); - - // skip the runner removal process if the runner is not found - if (!runner) { - log.info('remove_runner', { label: config.input.label, skipped: true, reason: 'not_found' }); - core.info(`GitHub self-hosted runner with label ${config.input.label} is not found, so the removal is skipped`); - return; - } - - const start = Date.now(); - log.info('remove_runner', { runner_id: runner.id, label: config.input.label }); - try { - await deregisterRunner(runner.id); - log.info('remove_runner', { runner_id: runner.id, label: config.input.label, elapsed_ms: Date.now() - start }); - core.info(`GitHub self-hosted runner ${runner.name} is removed`); - return; - } catch (error) { - log.error('remove_runner', { runner_id: runner.id, label: config.input.label, error: error.name, message: error.message }); - core.error('GitHub self-hosted runner removal error'); - throw error; - } -} - // True once the runner for `label` has registered with GitHub and reports // as online. Used as the success signal by the start action's wait loop // (see src/wait.js), polled alongside the instance's bootstrap tag. @@ -78,10 +54,58 @@ async function isRunnerOnline(label) { return !!(runner && runner.status === 'online'); } +// Count how many runners carrying `label` are registered and online. Used by +// the batch wait loop, which succeeds only once all N runners are up. +async function countOnlineRunners(label) { + const octokit = github.getOctokit(config.input.githubToken); + try { + const runners = await octokit.paginate('GET /repos/{owner}/{repo}/actions/runners', config.githubContext); + const online = runners.filter((r) => r.labels.some((l) => l.name === label) && r.status === 'online'); + log.debug('runner_status', { label, online: online.length }); + return online.length; + } catch (_error) { + return 0; + } +} + +// Deregister EVERY runner carrying `label` (a batch registers N runners under +// one shared label). Returns per-runner outcomes; a single failure does not +// abort the rest. Idempotent — no matching runners is a clean no-op. +async function removeAllRunners(label) { + const octokit = github.getOctokit(config.input.githubToken); + let runners; + try { + const all = await octokit.paginate('GET /repos/{owner}/{repo}/actions/runners', config.githubContext); + runners = all.filter((r) => r.labels.some((l) => l.name === label)); + } catch (error) { + throw new Error(`could not list runners for label ${label}: ${error.message}`, { cause: error }); + } + + if (runners.length === 0) { + log.info('remove_runner', { label, skipped: true, reason: 'not_found' }); + core.info(`No GitHub self-hosted runners found for label ${label}; removal skipped`); + return { removed: 0, failures: [] }; + } + + const failures = []; + for (const runner of runners) { + try { + await deregisterRunner(runner.id); + log.info('remove_runner', { runner_id: runner.id, label }); + } catch (error) { + log.error('remove_runner', { runner_id: runner.id, label, error: error.name, message: error.message }); + failures.push({ runnerId: runner.id, message: error.message }); + } + } + core.info(`Removed ${runners.length - failures.length}/${runners.length} GitHub self-hosted runners for label ${label}`); + return { removed: runners.length - failures.length, failures }; +} + module.exports = { getRegistrationToken, - removeRunner, deregisterRunner, isRunnerOnline, + countOnlineRunners, + removeAllRunners, getRunner, }; diff --git a/src/index.js b/src/index.js index 31c5df3d..09f54777 100644 --- a/src/index.js +++ b/src/index.js @@ -28,17 +28,21 @@ const core = require('@actions/core'); // command; GitHub runners now surface that as a warning. Bypass the // legacy path — modern runners always set GITHUB_OUTPUT. function setOutput(label, placement) { - const { instanceId, instanceType, subnetId, marketType } = placement; + const { instanceIds, instanceType, subnetId, marketType } = placement; + // Compat scalar: the first instance id. Batch consumers use the JSON array. + const firstId = instanceIds[0]; + const idsJson = JSON.stringify(instanceIds); const outputFile = process.env.GITHUB_OUTPUT; if (outputFile) { fs.appendFileSync( outputFile, - `label=${label}${os.EOL}ec2-instance-id=${instanceId}${os.EOL}instance-type-used=${instanceType}${os.EOL}subnet-id-used=${subnetId}${os.EOL}market-type-used=${marketType}${os.EOL}`, + `label=${label}${os.EOL}ec2-instance-id=${firstId}${os.EOL}ec2-instance-ids=${idsJson}${os.EOL}instance-type-used=${instanceType}${os.EOL}subnet-id-used=${subnetId}${os.EOL}market-type-used=${marketType}${os.EOL}`, ); return; } core.setOutput('label', label); - core.setOutput('ec2-instance-id', instanceId); + core.setOutput('ec2-instance-id', firstId); + core.setOutput('ec2-instance-ids', idsJson); core.setOutput('instance-type-used', instanceType); core.setOutput('subnet-id-used', subnetId); core.setOutput('market-type-used', marketType); @@ -51,25 +55,27 @@ async function start() { const label = config.generateUniqueLabel(); const githubRegistrationToken = await gh.getRegistrationToken(); const placement = await aws.startEc2Instance(label, githubRegistrationToken); - const ec2InstanceId = placement.instanceId; + const instanceIds = placement.instanceIds; setOutput(label, placement); - await aws.waitForInstanceRunning(ec2InstanceId); + for (const id of instanceIds) { + await aws.waitForInstanceRunning(id); + } - // Watch the bootstrap phone-home tag and GitHub registration together. - // On a bootstrap failure or registration timeout, capture the console - // output and (by default) terminate the instance so failed starts - // don't leak billing. The registration token is redacted from the - // captured output. + // Watch the bootstrap phone-home tags and GitHub registration together. + // The batch is ready only when ALL N runners are online; a bootstrap + // failure on ANY instance fails fast. On failure or timeout, capture the + // console output of every instance and (by default) terminate them all — + // no half-fleet is left leaking billing. The token is redacted. try { await waitForRunnerReady({ - getBootstrapStatus: () => aws.getBootstrapStatus(ec2InstanceId), - isRunnerOnline: () => gh.isRunnerOnline(label), + getBootstrapStatus: () => aws.getBatchBootstrapStatus(instanceIds), + isRunnerOnline: async () => (await gh.countOnlineRunners(label)) >= instanceIds.length, }); } catch (waitError) { - await aws.handleStartFailure(ec2InstanceId, { redactValues: [githubRegistrationToken] }); + await aws.handleStartFailure(instanceIds, { redactValues: [githubRegistrationToken] }); throw waitError; } - log.info('start', { label, instance_id: ec2InstanceId, instance_type: placement.instanceType, subnet_id: placement.subnetId, outcome: 'registered' }); + log.info('start', { label, instance_ids: instanceIds, instance_type: placement.instanceType, subnet_id: placement.subnetId, outcome: 'registered' }); } finally { core.endGroup(); } @@ -81,17 +87,32 @@ async function stop() { try { log.debug('stop_inputs', config.input); - // Attempt both cleanups independently — neither should short-circuit - // the other. A GitHub API failure must not prevent EC2 termination - // (billing) and vice versa. Both have internal retries via - // withRetry(); catch here is the last line of defense. - try { - await aws.terminateEc2Instance(); - } catch (error) { - failures.push({ step: 'terminate_instance', error: error.name, message: error.message }); + // Terminate every instance from the batch (the JSON array), or the single + // compat scalar. Independent per instance — one failure doesn't stop the + // rest — and idempotent: an already-gone instance is not a failure. + const instanceIds = (config.input.ec2InstanceIds && config.input.ec2InstanceIds.length) + ? config.input.ec2InstanceIds + : [config.input.ec2InstanceId]; + + for (const id of instanceIds) { + try { + await aws.terminateInstanceById(id); + } catch (error) { + if (error.name && error.name.includes('NotFound')) { + log.info('terminate_instance', { instance_id: id, skipped: true, reason: 'already_gone' }); + } else { + failures.push({ step: `terminate_instance:${id}`, error: error.name, message: error.message }); + } + } } + + // Deregister ALL runners sharing the label (a batch registers N). A + // GitHub failure must not have prevented the terminations above. try { - await gh.removeRunner(); + const result = await gh.removeAllRunners(config.input.label); + for (const f of result.failures) { + failures.push({ step: `remove_runner:${f.runnerId}`, message: f.message }); + } } catch (error) { failures.push({ step: 'remove_runner', error: error.name, message: error.message }); } @@ -101,7 +122,7 @@ async function stop() { const summary = failures.map((f) => `${f.step}: ${f.message}`).join('; '); throw new Error(`stop mode completed with ${failures.length} cleanup failure(s): ${summary}`); } - log.info('stop', { instance_id: config.input.ec2InstanceId, label: config.input.label, outcome: 'ok' }); + log.info('stop', { instance_ids: instanceIds, label: config.input.label, outcome: 'ok' }); } finally { core.endGroup(); } diff --git a/tests/config.test.js b/tests/config.test.js index f28b704a..f6425032 100644 --- a/tests/config.test.js +++ b/tests/config.test.js @@ -192,6 +192,52 @@ describe('Config — root volume inputs', () => { }); }); +describe('Config — count / batch inputs', () => { + test('defaults count to 1 and allow-partial to false', () => { + const config = loadConfig(startModeInputs); + expect(config.input.count).toBe('1'); + expect(config.input.allowPartial).toBe('false'); + }); + + test('accepts count > 1 with allow-partial', () => { + const config = loadConfig({ ...startModeInputs, 'count': '4', 'allow-partial': 'true' }); + expect(config.input.count).toBe('4'); + expect(config.input.allowPartial).toBe('true'); + }); + + test('rejects a non-integer count', () => { + expectValidationFailure({ ...startModeInputs, 'count': 'many' }, /'count' must be a positive integer/); + }); + + test('rejects count 0', () => { + expectValidationFailure({ ...startModeInputs, 'count': '0' }, /'count' must be a positive integer/); + }); +}); + +describe('Config — stop mode with instance batch', () => { + const stopBatch = { + 'mode': 'stop', + 'github-token': 'ghs_testtoken', + 'label': 'runner-xyz', + 'ec2-instance-ids': '["i-aaa","i-bbb"]', + 'aws-resource-tags': '[]', + }; + + test('accepts ec2-instance-ids array instead of the scalar', () => { + const config = loadConfig(stopBatch); + expect(config.input.ec2InstanceIds).toEqual(['i-aaa', 'i-bbb']); + }); + + test('still accepts the scalar ec2-instance-id (compat)', () => { + const config = loadConfig({ ...stopModeInputs }); + expect(config.input.ec2InstanceId).toBe('i-abc'); + }); + + test('fails when neither ec2-instance-id nor ec2-instance-ids is provided', () => { + expectValidationFailure({ 'mode': 'stop', 'github-token': 'ghs_testtoken', 'label': 'l', 'aws-resource-tags': '[]' }, /required inputs are provided for the 'stop' mode/); + }); +}); + describe('Config — architecture input', () => { test('defaults to x64', () => { expect(loadConfig(startModeInputs).input.architecture).toBe('x64'); diff --git a/tests/diagnostics.test.js b/tests/diagnostics.test.js index aea53082..94778487 100644 --- a/tests/diagnostics.test.js +++ b/tests/diagnostics.test.js @@ -91,6 +91,23 @@ describe('getBootstrapStatus', () => { }); }); +describe('getBatchBootstrapStatus', () => { + const tagsFor = (value) => ({ Tags: value ? [{ Key: aws.BOOTSTRAP_TAG_KEY, Value: value }] : [] }); + + test('returns the first failed: across the batch', async () => { + mockSend.mockImplementation((cmd) => { + const id = cmd.Filters.find((f) => f.Name === 'resource-id').Values[0]; + return Promise.resolve(id === 'i-2' ? tagsFor('failed:configuring') : tagsFor('downloading')); + }); + await expect(aws.getBatchBootstrapStatus(['i-1', 'i-2', 'i-3'])).resolves.toBe('failed:configuring'); + }); + + test('returns null when no instance has failed', async () => { + mockSend.mockImplementation(() => Promise.resolve(tagsFor('registered'))); + await expect(aws.getBatchBootstrapStatus(['i-1', 'i-2'])).resolves.toBeNull(); + }); +}); + describe('getConsoleOutputTail', () => { test('decodes base64 and returns the last N lines', async () => { const lines = Array.from({ length: 10 }, (_, i) => `line${i + 1}`).join('\n'); @@ -168,6 +185,34 @@ describe('handleStartFailure', () => { expect(warnings).toContain('get-console-output'); }); + test('captures ALL instances then terminates ALL (batch, no half-fleet)', async () => { + mockSend.mockImplementation((cmd) => { + if (cmd.__command === 'GetConsoleOutput') return Promise.resolve({ Output: b64('boot log') }); + return Promise.resolve({}); + }); + + await aws.handleStartFailure(['i-1', 'i-2'], { redactValues: [] }); + + // Every instance's console captured before any termination. + expect(commandsSent()).toEqual(['GetConsoleOutput', 'GetConsoleOutput', 'TerminateInstances', 'TerminateInstances']); + expect(core.startGroup).toHaveBeenCalledTimes(2); + }); + + test('preserves ALL instances when cleanup is disabled (batch)', async () => { + config.input.cleanupOnStartFailure = 'false'; + mockSend.mockImplementation((cmd) => { + if (cmd.__command === 'GetConsoleOutput') return Promise.resolve({ Output: b64('x') }); + return Promise.resolve({}); + }); + + await aws.handleStartFailure(['i-1', 'i-2'], { redactValues: [] }); + + expect(commandsSent()).toEqual(['GetConsoleOutput', 'GetConsoleOutput']); + const warnings = core.warning.mock.calls.map((c) => String(c[0])).join('\n'); + expect(warnings).toContain('i-1'); + expect(warnings).toContain('i-2'); + }); + test('does not throw if termination itself fails after capture', async () => { mockSend.mockImplementation((cmd) => { if (cmd.__command === 'GetConsoleOutput') return Promise.resolve({ Output: b64('x') }); diff --git a/tests/fallback.test.js b/tests/fallback.test.js index d87a3708..f6c9d2e8 100644 --- a/tests/fallback.test.js +++ b/tests/fallback.test.js @@ -38,9 +38,9 @@ describe('classifyRunError', () => { describe('launchWithFallback', () => { test('returns on the first cell when it succeeds (no fallback)', async () => { - const attempt = jest.fn().mockResolvedValue('i-1'); + const attempt = jest.fn().mockResolvedValue(['i-1']); const result = await launchWithFallback(attempt, ['t1', 't2'], ['s1', 's2']); - expect(result).toEqual({ instanceId: 'i-1', instanceType: 't1', subnetId: 's1' }); + expect(result).toEqual({ instanceIds: ['i-1'], instanceType: 't1', subnetId: 's1' }); expect(attempt).toHaveBeenCalledTimes(1); }); @@ -48,9 +48,9 @@ describe('launchWithFallback', () => { const attempt = jest.fn() .mockRejectedValueOnce(capacity()) // t1/s1 .mockRejectedValueOnce(capacity()) // t1/s2 - .mockResolvedValueOnce('i-9'); // t2/s1 + .mockResolvedValueOnce(['i-9']); // t2/s1 const result = await launchWithFallback(attempt, ['t1', 't2'], ['s1', 's2']); - expect(result).toEqual({ instanceId: 'i-9', instanceType: 't2', subnetId: 's1' }); + expect(result).toEqual({ instanceIds: ['i-9'], instanceType: 't2', subnetId: 's1' }); expect(attempt.mock.calls).toEqual([['t1', 's1'], ['t1', 's2'], ['t2', 's1']]); }); diff --git a/tests/spot.test.js b/tests/spot.test.js index e03c78ef..dad867d5 100644 --- a/tests/spot.test.js +++ b/tests/spot.test.js @@ -49,21 +49,21 @@ describe('buildMarketPlan', () => { describe('launchAcrossMarkets', () => { test('returns the spot placement when spot succeeds (no on-demand attempt)', async () => { - const attemptFor = jest.fn((market) => jest.fn().mockResolvedValue(`i-${market}`)); + const attemptFor = jest.fn((market) => jest.fn().mockResolvedValue([`i-${market}`])); const result = await launchAcrossMarkets(attemptFor, ['spot', 'on-demand'], ['t1'], ['s1']); - expect(result).toMatchObject({ instanceId: 'i-spot', marketType: 'spot' }); + expect(result).toMatchObject({ instanceIds: ['i-spot'], marketType: 'spot' }); expect(attemptFor).toHaveBeenCalledTimes(1); // on-demand attempt fn never built }); test('falls back to on-demand exactly once when spot capacity is exhausted', async () => { const spotAttempt = jest.fn().mockRejectedValue(capacity()); - const onDemandAttempt = jest.fn().mockResolvedValue('i-od'); + const onDemandAttempt = jest.fn().mockResolvedValue(['i-od']); const attemptFor = (market) => (market === 'spot' ? spotAttempt : onDemandAttempt); const onDowngrade = jest.fn(); const result = await launchAcrossMarkets(attemptFor, ['spot', 'on-demand'], ['t1'], ['s1'], { onDowngrade }); - expect(result).toMatchObject({ instanceId: 'i-od', marketType: 'on-demand' }); + expect(result).toMatchObject({ instanceIds: ['i-od'], marketType: 'on-demand' }); expect(spotAttempt).toHaveBeenCalledTimes(1); expect(onDemandAttempt).toHaveBeenCalledTimes(1); expect(onDowngrade).toHaveBeenCalledWith('spot', 'on-demand', expect.anything()); @@ -71,7 +71,7 @@ describe('launchAcrossMarkets', () => { test('falls back on SpotMaxPriceTooLow (price is a capacity-class code)', async () => { const spotAttempt = jest.fn().mockRejectedValue(err('SpotMaxPriceTooLow')); - const onDemandAttempt = jest.fn().mockResolvedValue('i-od'); + const onDemandAttempt = jest.fn().mockResolvedValue(['i-od']); const attemptFor = (market) => (market === 'spot' ? spotAttempt : onDemandAttempt); const result = await launchAcrossMarkets(attemptFor, ['spot', 'on-demand'], ['t1'], ['s1']); expect(result.marketType).toBe('on-demand');