Skip to content

Make bench CLI load TypeScript benchmarks - #346

Open
HeyGarrison wants to merge 4 commits into
masterfrom
devin/tensorlake-opencode-reliability-cli-fix
Open

Make bench CLI load TypeScript benchmarks#346
HeyGarrison wants to merge 4 commits into
masterfrom
devin/tensorlake-opencode-reliability-cli-fix

Conversation

@HeyGarrison

@HeyGarrison HeyGarrison commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

Summary

The bench CLI couldn't execute the new benchmarks/reliability/sandbox-opencode.bench.ts file because plain Node doesn't resolve .ts source or its .js.ts imports.

  • packages/benchsdk-runner/src/bin.ts: registers the tsx ESM loader at startup.
  • packages/benchsdk-runner/package.json: adds tsx as a runtime dependency and points the bench binary at the ESM build (./dist/bin.js) so the bin and any loaded TypeScript benchmark resolve the same @benchsdk/runner instance.
  • package.json and .github/workflows/reliability-tensorlake-opencode.yml: switch the reliability benchmark back to bench run ....

This removes the tsx packages/benchsdk-runner/dist/bin.js indirection. pnpm typecheck, pnpm --filter @benchsdk/runner test, and a dry bench run of the reliability benchmark all pass (the run fails as expected with a dummy TENSORLAKE_API_KEY, but it loads the module and reaches the Tensorlake provider step).

Link to Devin session: https://app.devin.ai/sessions/754c6cb646a146a0b6c883eae4b613b1
Requested by: @HeyGarrison


Open in Devin Review

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@open-cla

open-cla Bot commented Aug 20, 2026

Copy link
Copy Markdown

Contributor License Agreement

All contributors are covered by a CLA.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Open in Devin Review

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration devin-ai-integration Bot changed the title Fix Tensorlake OpenCode reliability benchmark CLI invocation Make bench CLI load TypeScript benchmarks Aug 20, 2026
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
devin-ai-integration[bot]

This comment was marked as resolved.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant