Leaderboard v1: Computer / Claude Opus 5

[c-suvamsa]

Summary

• Agent / Model: DevRev Computer / Claude Opus 5 (Anthropic).
• Source Harbor job URL: https://hub.harborframework.com/jobs/7b63a46c-1620-4fb0-b4c3-9e1a4f5454f0.
• Complete rerun from 2026-10-15 using enterprise-bench/tasks/v1: 10 attempts per task; 137/140 passed (97.9%). This supersedes the earlier Opus 5 result rather than replacing an individual trial.
• SDK-reported cost: $115.75.

PR checklist – leaderboard row entry

Provenance and access

• Source job is public and owned by the submitter; entry header and job URL match.
• Entry has all required metadata and metrics; 140 distinct trial IDs match n_trials.
• Accuracy, pass@k, token breakdown, and displayed SDK-reported cost reconcile with the recorded run.
• Agent, model, and provider are shown on the job.
• Every local trial has an ATIF trajectory, reward, and token usage; no trials were omitted.
• Reviewer to confirm logged-out replay of individual trajectories.

Run conditions and review

• Task digests for all 14 tasks match enterprise-bench main. The run used an absolute local task path rather than a registry version.
• Scored timeout multiplier is 1.0; no CPU/memory/storage overrides were recorded.
• Reviewer to confirm the agent had no benchmark-answer access and re-verify the submitted score.

Validation

• Main-branch leaderboard validator passed locally; repository validate and validate-leaderboard CI checks pass on the replacement commit.

Notes for reviewers

Model label was supplied by the run owner after execution; the saved Computer trace uses its generic gateway alias. This is a new full run with all original failures retained, not a selected retry.

[nimit2801]

Thanks for the submission. Nimit’s intake workflow has picked up this leaderboard entry for DevRev Computer / Claude Opus 5. Both validation checks pass; technical review is still required, not approval.

Verification points:

• Confirm public trajectory replay and reconcile scores, pass@k, tokens, and SDK cost with the source job.
• Verify the dataset version, original run configuration, and timeout/resource settings.
• Audit provenance and tool use, including absence of benchmark-answer access.

[nimit2801]

Nimit’s intake workflow has picked up the updated DevRev Computer / Claude Opus 5 leaderboard submission. Both validation checks pass; this is an acknowledgement, not approval.

Technical review still needs to:

• Verify public trajectory replay and reconcile the full rerun’s scores, pass@k, tokens, and SDK cost.
• Confirm task digests, timeout/resource settings, and model provenance given the generic gateway alias.
• Audit tool use and confirm no benchmark-answer access.