feat: add cpu-node sandbox benchmark with CI workflow

[kisernl]

Summary

Adds the CPU-node sandbox benchmark (cpu-node), a cross-provider CPU performance benchmark that runs a self-contained, stdlib-only Node.js workload inside a fresh sandbox and scores providers on wall-clock compute time.

The branch originally wired the benchmark into the legacy benchmarks/src/run.ts orchestrator, but that file was removed on master. This PR rebases the work and re-implements the runner integration as a @benchsdk/runner *.bench.ts entrypoint (benchmarks/sandbox/cpu-node.bench.ts), plus a legacy bridge that keeps the existing JSON/SVG results pipeline intact.

What the benchmark measures

The in-sandbox workload is benchmarks/scripts/cpu-node-workload.js. It is pure Node.js (no node_modules, no native bindings) and runs the same deterministic compute phases on every provider:

1. JSON round-trip cost — builds an AST of depth 7, stringifies it, parses it back, and repeats for 16 iterations.
2. SHA-256 throughput — hashes a 1 MiB random buffer, 64 blocks per iteration, for 8 iterations.
3. Regex + text walk — builds a 256 KiB pseudo-random "code" corpus, then runs four JavaScript-ish regex patterns over it 16 times.
4. Sum of primes — brute-force sums all primes up to 5,000,000, repeated 16 times.

The workload emits a single WorkloadResult JSON line with the total wall-clock time in milliseconds (value lower-is-better, unit ms).

Scoring

• Per-provider stats are computed from successful iterations in benchmarks/sandbox/cpu-node.ts:computeStats:
  • median, p95, p99, min, max
  • 2-sigma outlier trim
  • success rate (ok / total)
• scoreMetric converts the median into a 0–100 score against a ceiling of 45,000 ms: 100 * (1 - median / ceiling).
• compositeScore = scoreBeforeReliability * successRate, so a provider that fails most runs cannot score highly.
• Scores are clamped to [0, 100].

Architecture / files

• benchmarks/sandbox/cpu-node.ts — workload orchestrator (runCpuNodeBenchmark), scoring, stats, result writer, and stdout parser.
• benchmarks/scripts/cpu-node-workload.js — the script that actually executes inside the sandbox.
• benchmarks/scripts/cpu-node-stdout.js — helper that emits the WorkloadResult JSON line and installs crash handlers.
• benchmarks/sandbox/cpu-node.bench.ts — @benchsdk/runner entrypoint (defineBenchmarkConfig + defineTask).
• benchmarks/sandbox/cpu-node-legacy-results.ts — maps the runner's ParticipantRecords[] back into the existing CpuNodeBenchmarkResult[] shape and writes results/cpu_node/<YYYY-MM-DD>.json plus latest.json.
• benchmarks/sandbox/generate-cpu-node-svg.ts — renders cpu_node.svg from results/cpu_node/latest.json.
• benchmarks/scripts/smoke.ts — local smoke harness that runs the workload on the current machine (pnpm smoke:cpu-node).
• benchmarks/src/merge-results.ts — new mainHpc() merge path that groups per-provider latest.json artifacts by suite directory, deduplicates, and writes combined results/<suite>/<date>.json with schema-compliant version, timestamp, environment, and config metadata.
• .github/workflows/sandbox-cpu-node.yml — CI workflow.

CI workflow

• Triggers on pushes to master that touch the benchmark paths, plus workflow_dispatch with iterations, provider, and dry_run inputs.
• Runs on namespace-profile-default against a 28-provider matrix.
• Each provider job:
  • loads vault secrets for the provider
  • runs npx tsx packages/benchsdk-runner/dist/bin.js run benchmarks/sandbox/cpu-node.bench.ts --provider <p> --iterations <n> --run-key "$RUN_KEY" where RUN_KEY is ${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT} so all matrix jobs share one platform run and can be ranked against each other
  • uploads its results/ directory as an artifact
• The collect job:
  • downloads all results-* artifacts
  • runs merge-results.ts --input artifacts --mode benchmark
  • runs generate-cpu-node-svg to produce cpu_node.svg
  • ingests results to the platform (unless push or dry_run)
  • commits results/cpu_node/ and cpu_node.svg back to the branch

Removed dangling workflows

The PR also deletes the sibling sandbox-*.yml workflow files (sandbox-disk, sandbox-dns, sandbox-download, sandbox-latency, sandbox-memory, sandbox-network-localhost, sandbox-network-wan, sandbox-pgbench, sandbox-realworld, sandbox-system). These were registered on master but referenced the deleted benchmarks/src/run.ts and build-bundles.ts tooling and had no runnable benchmark code.

Running locally

# Run the smoke test locally (no provider credentials needed)
pnpm smoke:cpu-node

# Run the full benchmark against a provider (requires provider secrets + platform key)
pnpm bench:cpu-node --provider e2b --iterations 3

# Render the leaderboard SVG after results exist
pnpm generate-cpu-node-svg

Verification

• pnpm typecheck passes
• pnpm test:benchmarks passes
• pnpm smoke:cpu-node passes
• PR CI checks pass

Additional fix

benchmarks/scripts/load-vault-secrets.sh had a brittle add-mask loop that returned a non-zero exit code whenever a matched vault key had an empty value, killing the whole CI step under set -e. The script now skips empty keys/values instead of failing, which keeps provider jobs that are missing a credential from crashing the vault-loading step before the benchmark can skip them cleanly.

Link to Devin session: https://app.devin.ai/sessions/b6422a238bc0444b90e7af39407c6961
Requested by: @kisernl

---

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment, CI, and merge conflict monitoring

[open-cla]

Contributor License Agreement

All contributors are covered by a CLA.