fix(isolated-vm): drain worker host deploys without failing in-flight executions by waleedlatif1 · Pull Request #9389 · simstudioai/sim

waleedlatif1 · 2026-10-27T00:05:55Z

Summary

A load balancer deregistering a worker host (deploy or scale-in) keeps its open WebSocket connections for the deregistration delay and sends the host nothing until SIGTERM, so the app kept dispatching new executions onto those connections until they were cut, and every execution still running failed with "Code execution failed unexpectedly". The app now retires each remote worker connection at a max age (IVM_REMOTE_WORKER_MAX_AGE_MS, default 2 min, jittered down by up to 20%): it takes no new work, finishes what it runs, then closes. New connections land on a healthy host, and the old host's connections age out before its deregistration delay, which infra sizes as max age + the longest execution a release keeps alive + margin (companion infra PR)
An execution whose connection was already closing when it was dispatched never left the app, so it moves to another worker (at most 3 times). An execution that may have reached the host is never re-run; if its connection drops it fails with "Code execution was interrupted because the connection to its sandbox host closed"
Host: a connection may send at most 16 KiB before it authenticates, counted on the raw TCP socket, so an unauthenticated client can no longer make the host buffer a full maxPayload frame on each pending connection
executeInIsolatedVM returns an already-cancelled or misconfigured call before reading the worker-target flag, instead of waiting on that read first
Protocol unchanged in both directions: no new frame types; the max age and the pre-auth cap are each enforced on one side only, so app N works with host N-1 and the reverse

Type of Change

Bug fix

Testing

scripts/test-isolated-vm-remote-e2e.ts (real hosts over real WebSockets) now routes the app through a TCP balancer that behaves like the internal ALB. New check: executions start, the balancer stops routing new connections to the host while keeping open ones, new executions keep arriving until the delay is nearly over, then the balancer cuts what is left and the host gets SIGTERM. Every execution must succeed, nothing may still be open on the old host at expiry, and work must move to the replacement. On the pre-fix sources it fails with 6 of 20 executions reporting "Code execution failed unexpectedly"; with the fix all 19 checks pass
New e2e check: a raw socket completes the WebSocket handshake, declares a 32 MiB frame and streams 1 MiB without authenticating. The host must close it within 1s. Fails on the pre-fix host (connection still open), passes with the fix
The two worker-inspection checks now hold an execution open while they look, since an idle worker can retire at any time
lib/execution/isolated-vm.test.ts: a pre-aborted call with a never-resolving flag read returns cancelled (hangs before the fix), and an execution dispatched onto a closing connection completes on a new worker (timed out before the fix). Both reverted-guard runs went red
biome, check:comment-hygiene, check:utils, check:test-patterns, check:explicit-any, check:source-conventions, check:unused-exports; CI runs the rest

Checklist

Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing (new tests pass the test-audit authoring gate)
No new warnings introduced
I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

… executions - Retire each remote worker connection at a max age (IVM_REMOTE_WORKER_MAX_AGE_MS, default 2 min) so a host the load balancer deregistered stops getting new executions and drains before its deregistration delay cuts the connections - Re-dispatch an execution whose connection closed before it was sent; one that may have reached the host is never re-run and fails with a clear error - Cap raw bytes a host connection may send before authenticating (16 KiB) - Return a cancelled call before reading the worker-target flag

vercel · 2026-10-27T00:06:00Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
docs	Skipped		Oct 27, 2026 12:39am UTC

cubic-dev-ai

All reported issues were addressed across 6 files

_{This PR changes concurrency-sensitive code such as locks, queues, or retries. Ultrareviews find 2.4x more serious bugs than standard reviews. Comment @cubic-dev-ai ultrareview to run one.

Turn on auto-fix | Re-trigger cubic}

greptile-apps · 2026-10-27T00:10:35Z

Confidence Score: 5/5

[Medium impact] The PR appears safe to merge; no blocking findings remain.

Summary

Remote workers stop taking new work when their connections reach the age limit, finish active executions, then close.

Remote worker connections retire gradually so hosts can drain during deploys.
Executions retry on a new worker only when the request never left the app.
The worker host closes connections that send over 16 KiB before authentication.
Cancelled and misconfigured calls return before waiting for the worker-target flag.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Execution queued] --> B[Choose worker]
  B --> C{Connection open?}
  C -->|No, request not sent| D[Queue again, up to three times]
  D --> B
  C -->|Yes| E[Send execution]
  E --> F[Finish execution]
  G[Connection reaches age limit] --> H[Stop assigning new work]
  H --> I[Wait for active executions]
  F --> I
  I --> J[Close retired connection]

_{Reviews (3) · Last reviewed commit: "test(isolated-vm): report an execution t..." · Reviewed by Greptile}

- Keep counting a refused connection's bytes until its socket closes - Skip a redispatch the caller cancelled in between - Look up the owner's state after the admission waits - Parse IVM_REMOTE_WORKER_MAX_AGE_MS strictly, falling back below 1s - Hold the inspected execution open until the worker check finishes

waleedlatif1 · 2026-10-27T00:24:22Z

@greptile

waleedlatif1 · 2026-10-27T00:24:23Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-27T00:24:28Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 6 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Turn on auto-fix | Re-trigger cubic}

waleedlatif1 · 2026-10-27T00:31:50Z

@greptile review

waleedlatif1 · 2026-10-27T00:43:26Z

@greptile

waleedlatif1 · 2026-10-27T00:43:27Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-27T00:43:34Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 6 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Turn on auto-fix | Re-trigger cubic}

cubic-dev-ai Bot reviewed Oct 27, 2026

View reviewed changes

Comment thread apps/sim/lib/core/config/env.ts Outdated

Comment thread apps/sim/lib/execution/isolated-vm-host.cjs Outdated

Comment thread apps/sim/lib/execution/isolated-vm.ts Outdated

greptile-apps Bot reviewed Oct 27, 2026

View reviewed changes

Comment thread apps/sim/lib/execution/isolated-vm.ts

Comment thread apps/sim/scripts/test-isolated-vm-remote-e2e.ts Outdated

vercel Bot temporarily deployed to Preview October 27, 2026 00:16 Inactive

cubic-dev-ai Bot reviewed Oct 27, 2026

View reviewed changes

greptile-apps Bot reviewed Oct 27, 2026

View reviewed changes

Comment thread apps/sim/scripts/test-isolated-vm-remote-e2e.ts Outdated

test(isolated-vm): report an execution that fails before it is held

aa88523

vercel Bot temporarily deployed to Preview October 27, 2026 00:39 Inactive

cubic-dev-ai Bot reviewed Oct 27, 2026

View reviewed changes

waleedlatif1 merged commit 5a98a62 into staging Oct 27, 2026
68 checks passed

waleedlatif1 deleted the fix/ivm-host-drain branch October 27, 2026 01:35

waleedlatif1 mentioned this pull request Oct 27, 2026

v0.9.34: oauth reconnect conflicts, sandbox host drain, token sealing backfill, migrate guards #9396

Merged

Conversation

waleedlatif1 commented Oct 27, 2026

Summary

Type of Change

Testing

Checklist

Uh oh!

vercel Bot commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

greptile-apps Bot commented Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Diagram

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Oct 27, 2026

Uh oh!

waleedlatif1 commented Oct 27, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 27, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

waleedlatif1 commented Oct 27, 2026

Uh oh!

Uh oh!

waleedlatif1 commented Oct 27, 2026

Uh oh!

waleedlatif1 commented Oct 27, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 27, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This branch was previously deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

vercel Bot commented Oct 27, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

greptile-apps Bot commented Oct 27, 2026 •

edited

Loading