Skip to content

fix: pass raw user-data bytes to ModifyInstanceAttribute on warm start#66

Merged
kurok merged 1 commit into
mainfrom
fix/warm-pool-userdata-double-base64
Jul 6, 2026
Merged

fix: pass raw user-data bytes to ModifyInstanceAttribute on warm start#66
kurok merged 1 commit into
mainfrom
fix/warm-pool-userdata-double-base64

Conversation

@kurok

@kurok kurok commented Jul 6, 2026

Copy link
Copy Markdown

Summary

Fixes warm-pool (reuse: stop) registration, which failed 100% of the time on every warm restart with:

EC2 runner bootstrap failed during the "configuring" step —
Invalid configuration provided for url. Terminating unattended configuration.

Root cause — double base64 encoding

warmStartInstance() rewrites a stopped pool instance's user-data via ModifyInstanceAttribute before restarting it, so the per-boot register service re-registers with a fresh token/label.

That field is a blob (BlobAttributeValue.Value: Uint8Array), unlike the plain string UserData that RunInstances takes. The EC2 query protocol base64-encodes a blob's bytes on the wire, and EC2 base64-decodes them once before storing what IMDS serves.

The code handed the SDK an already-base64'd string:

UserData: { Value: Buffer.from(userData).toString('base64') }  // ❌ double-encodes

So the wire carried base64(base64(userData)); EC2 decoded only the outer layer and IMDS served the base64 text of the script instead of the script. The per-boot register step reads user-data back from IMDS and sed-greps GH_REPO_URL='...' out of it — against a single base64 blob line there is no match, GH_REPO_URL is empty, and config.sh --url "" aborts with the error above.

Cold launches were unaffected: RunInstances' UserData is a string the caller correctly pre-encodes (startEc2Instance).

Fix

UserData: { Value: Buffer.from(userData) }  // ✅ raw bytes; SDK does the single base64 layer

Verification

Captured the actual serialized ModifyInstanceAttribute request body through a real EC2Client:

UserData.Value what IMDS serves (base64-decoded once)
Buffer.from(userData).toString('base64') (old) IyEvYmluL2Jhc2gK… — base64 text, no GH_REPO_URL line
Buffer.from(userData) (new) #!/bin/bash\nGH_REPO_URL='…' — the real script ✅

Tests

  • Strengthened the warmStartInstance unit test: it now asserts the SDK receives the raw script bytes (Buffer/Uint8Array), not a pre-encoded string. The previous assertion base64-decoded the value, which only passed because the value was wrongly pre-encoded — it now fails against the bug.
  • Added tests/warmpool-userdata-encoding.test.js: runs a real EC2Client through a capturing request handler and asserts raw bytes round-trip to the script verbatim, while a pre-base64'd string double-encodes and loses it. This pins the SDK wire contract independently of our code.

npm run lint, npm test (239 passing), and npm run package (dist rebuilt) all clean.

#66)

warmStartInstance() rewrites a stopped pool instance's user-data via
ModifyInstanceAttribute before restarting it, so the per-boot register
service re-registers with a fresh token/label. That field is a BLOB
(BlobAttributeValue.Value: Uint8Array), unlike the plain-string UserData
that RunInstances takes. The EC2 query protocol base64-encodes a blob's
bytes on the wire, and EC2 base64-decodes them once before storing what
IMDS serves.

The code handed the SDK an already-base64'd STRING
(Buffer.from(userData).toString('base64')), so the wire carried
base64(base64(userData)); EC2 decoded only the outer layer and IMDS ended
up serving the base64 TEXT of the script instead of the script. The
per-boot register step reads user-data back from IMDS and greps out
GH_REPO_URL with sed; against a single base64 blob line there is no
GH_REPO_URL='...' match, so GH_REPO_URL is empty and config.sh aborts with
"Invalid configuration provided for url. Terminating unattended
configuration." Every reuse: stop warm start failed registration this way;
cold launches were unaffected because RunInstances' UserData is a string
that the caller correctly pre-encodes (startEc2Instance).

Hand ModifyInstanceAttribute the raw user-data bytes and let the SDK do
the single base64 layer.

Tests:
- Strengthen the warmStartInstance unit test to assert the SDK receives the
  raw script bytes (Buffer/Uint8Array), not a pre-encoded string. The old
  assertion base64-decoded the value, which only passed because the value
  was wrongly pre-encoded; it now fails against that bug.
- Add tests/warmpool-userdata-encoding.test.js: runs a real EC2Client
  through a capturing requestHandler and asserts raw bytes round-trip to the
  script verbatim while a pre-base64'd string double-encodes and loses it.

Signed-off-by: kurok <22548029+kurok@users.noreply.github.com>
@kurok kurok merged commit 65fbe4f into main Jul 6, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant