improvement(execution): run sandbox workers on a dedicated worker host (#9348)

* improvement(execution): run sandbox workers on a dedicated worker host

- Isolated-vm pool can open workers as connections to a dedicated worker host
  (isolated-vm-host.cjs, shipped in the app image) instead of local child
  processes, behind IVM_REMOTE_WORKER_URL/TOKEN and the
  isolated-vm-remote-workers flag. Remote mode never falls back to local
  workers; a target change retires old workers without interrupting runs.
- The host relays the unchanged worker protocol; fetch and brokers stay in the
  app, and the app sends its worker limits so both modes enforce the same values.
- ffmpeg/ffprobe run with a minimal environment and a file-only input
  protocol whitelist.

* fix(execution): address review on the dedicated worker host

- Host: reject an unparseable upgrade target instead of throwing; bound
  unauthenticated connections separately so they never hold a worker slot.
- Pool: re-enqueue an execution whose selected worker was retired before
  dispatch; forward TZ/LANG/LC_ALL with the limits so remote and local workers
  produce the same Date/Intl results.
- Keep the child_process import in its original form so the bundler does not
  trace the worker spawn; give ffmpeg's env NODE_ENV for ProcessEnv.
- E2E: stop the suite when a check cannot settle; assert the exact truncation
  bound; cover the unparseable target.

* improvement(execution): replace the remote-workers flag and its env fallback with a local-workers kill switch

A configured worker host is now used by default; the AppConfig-only
isolated-vm-local-workers kill switch forces local workers during rollout.
Drops IVM_REMOTE_WORKERS, and an unreachable AppConfig keeps the host in use.

* fix(execution): re-queue work selected onto a retiring worker; stop E2E hosts with their workers

- dispatchToWorker re-queues when the selected worker retired during a target
  switch (not only when it was removed), and cleans it up if idle.
- E2E hosts run in their own process group so cleanup also stops their
  worker children.
- Sandbox rule notes the URL needs its token.

Author: waleedlatif1

.claude/rules/sim-sandbox.md

@@ -2,6 +2,8 @@
 description: Isolated-vm sandbox worker security policy. Hard rules for anything that lives in the worker child process that runs user code.
 paths:
   - "apps/sim/lib/execution/isolated-vm-worker.cjs"
+  - "apps/sim/lib/execution/isolated-vm-host.cjs"
+  - "apps/sim/lib/execution/isolated-vm-remote.ts"
   - "apps/sim/lib/execution/isolated-vm.ts"
   - "apps/sim/lib/execution/sandbox/**"
   - "apps/sim/sandboxes/**"
@@ -75,7 +77,31 @@ payload or `ivm.Reference` wrapper in the worker:
 - [ ] Does the worker read a new env var? Add it to the `buildWorkerEnv()`
       allowlist in `isolated-vm.ts` **and** to the allowlist regression test in
       `isolated-vm.test.ts` — the worker does not inherit the app environment,
-      so an un-allowlisted var is simply absent in the child.
+      so an un-allowlisted var is simply absent in the child. If the app owns
+      its value, put it in `appWorkerEnv()` and add it with its accepted shape
+      to `CLIENT_WORKER_ENV` in `isolated-vm-host.cjs`, so a remote worker
+      gets the same value.
+- [ ] Does the host send the worker a new message type? Add it to
+      `ALLOWED_INBOUND_TYPES` in `isolated-vm-host.cjs`, or remote workers
+      never receive it.
+
+## Where the worker runs
+
+The pool in `isolated-vm.ts` opens each worker either as a local child process
+or, when `IVM_REMOTE_WORKER_URL` is set (with `IVM_REMOTE_WORKER_TOKEN`; a URL
+without a valid token fails executions rather than falling back) and the
+`isolated-vm-local-workers` kill switch is off, as a WebSocket connection to
+`isolated-vm-host.cjs`, which runs the
+same worker in a dedicated service that holds no platform credentials. Every
+rule in this file applies in both modes.
+
+- `fetch()` and brokers always run in the app. The host only relays the
+  worker's messages, so it needs no network access of its own and never gets
+  a broker implementation.
+- A remote target never falls back to a local child process. An unreachable
+  host fails executions with a system error; keep it that way.
+- Changes to the worker message protocol must stay compatible with the
+  previous release: the host and its clients roll out independently.
 
 ## What the worker *may* hold
 

.cursor/rules/sim-sandbox.mdc

@@ -1,6 +1,6 @@
 ---
 description: "Isolated-vm sandbox worker security policy. Hard rules for anything that lives in the worker child process that runs user code."
-globs: ["apps/sim/lib/execution/isolated-vm-worker.cjs","apps/sim/lib/execution/isolated-vm.ts","apps/sim/lib/execution/sandbox/**","apps/sim/sandboxes/**"]
+globs: ["apps/sim/lib/execution/isolated-vm-worker.cjs","apps/sim/lib/execution/isolated-vm-host.cjs","apps/sim/lib/execution/isolated-vm-remote.ts","apps/sim/lib/execution/isolated-vm.ts","apps/sim/lib/execution/sandbox/**","apps/sim/sandboxes/**"]
 ---
 
 <!-- Generated from .claude/rules/sim-sandbox.md by `bun run skills:sync`. Edit the source, not this file. -->
@@ -73,7 +73,31 @@ payload or `ivm.Reference` wrapper in the worker:
 - [ ] Does the worker read a new env var? Add it to the `buildWorkerEnv()`
       allowlist in `isolated-vm.ts` **and** to the allowlist regression test in
       `isolated-vm.test.ts` — the worker does not inherit the app environment,
-      so an un-allowlisted var is simply absent in the child.
+      so an un-allowlisted var is simply absent in the child. If the app owns
+      its value, put it in `appWorkerEnv()` and add it with its accepted shape
+      to `CLIENT_WORKER_ENV` in `isolated-vm-host.cjs`, so a remote worker
+      gets the same value.
+- [ ] Does the host send the worker a new message type? Add it to
+      `ALLOWED_INBOUND_TYPES` in `isolated-vm-host.cjs`, or remote workers
+      never receive it.
+
+## Where the worker runs
+
+The pool in `isolated-vm.ts` opens each worker either as a local child process
+or, when `IVM_REMOTE_WORKER_URL` is set (with `IVM_REMOTE_WORKER_TOKEN`; a URL
+without a valid token fails executions rather than falling back) and the
+`isolated-vm-local-workers` kill switch is off, as a WebSocket connection to
+`isolated-vm-host.cjs`, which runs the
+same worker in a dedicated service that holds no platform credentials. Every
+rule in this file applies in both modes.
+
+- `fetch()` and brokers always run in the app. The host only relays the
+  worker's messages, so it needs no network access of its own and never gets
+  a broker implementation.
+- A remote target never falls back to a local child process. An unreachable
+  host fails executions with a system error; keep it that way.
+- Changes to the worker message protocol must stay compatible with the
+  previous release: the host and its clients roll out independently.
 
 ## What the worker *may* hold
 

apps/sim/lib/audio/extractor.test.ts

@@ -73,6 +73,7 @@ describe('audio FFmpeg execution', () => {
   it('keeps MIME-derived input filenames inside the temporary directory', async () => {
     await getAudioMetadata(Buffer.from('audio'), 'audio/../../../../escaped')
 
+    expect(calls[0].args.slice(0, 2)).toEqual(['-protocol_whitelist', 'file,pipe'])
     const inputFile = calls[0].args.at(-1)
     expect(inputFile).toBeDefined()
     expect(path.basename(inputFile as string)).toBe('input.dat')
@@ -97,6 +98,8 @@ describe('audio FFmpeg execution', () => {
     expect(conversion?.args).toEqual([
       '-y',
       '-nostdin',
+      '-protocol_whitelist',
+      'file,pipe',
       '-i',
       expect.stringMatching(/input\.mp4$/),
       '-f',

apps/sim/lib/audio/extractor.ts

@@ -10,7 +10,12 @@ import type {
 } from '@/lib/audio/types'
 import { assertKnownSizeWithinLimit } from '@/lib/core/utils/stream-limits'
 import { MAX_MEDIA_BYTES } from '@/lib/media/falai'
-import { FFMPEG_BASE_ARGS, resolveExecutable, runExecutable } from '@/lib/media/ffmpeg-process'
+import {
+  ffmpegArgs,
+  ffprobeJsonArgs,
+  resolveExecutable,
+  runExecutable,
+} from '@/lib/media/ffmpeg-process'
 
 const logger = createLogger('AudioExtractor')
 
@@ -59,7 +64,7 @@ async function withTempDir<T>(prefix: string, fn: (dir: string) => Promise<T>):
 
 async function runFfmpeg(args: string[], signal?: AbortSignal): Promise<void> {
   try {
-    await runExecutable(requireFfmpeg(), [...FFMPEG_BASE_ARGS, ...args], {
+    await runExecutable(requireFfmpeg(), ffmpegArgs(args), {
       maxOutputBytes: MAX_PROCESS_OUTPUT_BYTES,
       signal,
       timeoutMs: CONVERSION_TIMEOUT_MS,
@@ -195,15 +200,11 @@ async function getAudioMetadataFromFile(
 ): Promise<AudioMetadata> {
   let stdout: string
   try {
-    ;({ stdout } = await runExecutable(
-      requireFfprobe(),
-      ['-v', 'error', '-print_format', 'json', '-show_format', '-show_streams', filePath],
-      {
-        maxOutputBytes: MAX_PROCESS_OUTPUT_BYTES,
-        signal,
-        timeoutMs: PROBE_TIMEOUT_MS,
-      }
-    ))
+    ;({ stdout } = await runExecutable(requireFfprobe(), ffprobeJsonArgs(filePath), {
+      maxOutputBytes: MAX_PROCESS_OUTPUT_BYTES,
+      signal,
+      timeoutMs: PROBE_TIMEOUT_MS,
+    }))
   } catch (error) {
     signal?.throwIfAborted()
     const failure = error as NodeJS.ErrnoException & { killed?: boolean; stderr?: string }

apps/sim/lib/core/config/env.ts

@@ -456,6 +456,8 @@ export const env = createEnv({
     IVM_MAX_BROKER_ARGS_JSON_CHARS:        z.string().optional().default('262144'),  // Max JSON payload size for sandbox task broker args (isolate→host)
     IVM_MAX_BROKER_RESULT_JSON_CHARS:      z.string().optional().default('16777216'),// Max JSON payload size for sandbox task broker results (host→isolate)
     IVM_MAX_BROKERS_PER_EXECUTION:         z.string().optional().default('1000'),    // Max broker calls per sandbox task execution
+    IVM_REMOTE_WORKER_URL:                 z.string().optional(),                    // ws(s)://<host>/v1/worker of a dedicated worker host; workers run there unless the isolated-vm-local-workers kill switch is on
+    IVM_REMOTE_WORKER_TOKEN:               z.string().optional(),                    // Token the worker host expects (its IVM_HOST_TOKEN, >= 32 chars); required with IVM_REMOTE_WORKER_URL
 
     // Knowledge Base Processing Configuration - Shared across all processing methods
     KB_CONFIG_MAX_DURATION:                z.number().optional().default(600),     // Max processing duration in seconds (10 minutes)

apps/sim/lib/core/config/feature-flags.ts

@@ -121,6 +121,14 @@ const FEATURE_FLAGS = {
       'targeting. Off-AppConfig falls back to KNOWLEDGE_MEMBER_ACCESS.',
     fallback: 'KNOWLEDGE_MEMBER_ACCESS',
   },
+  'isolated-vm-local-workers': {
+    description:
+      'Kill switch: run JavaScript sandbox workers as local child processes even where ' +
+      'IVM_REMOTE_WORKER_URL names a dedicated worker host. Global on/off only; off by default, ' +
+      'so a configured host is used unless this is turned on. AppConfig-only: when AppConfig is ' +
+      'unreachable the host stays in use.',
+    fallback: null,
+  },
   'mothership-abandoned-run-takeover-disabled': {
     description:
       'Kill switch for Sim taking over Chat runs whose controller died (a deploy or a crash) and ' +