Leaderboard v1: Computer / Claude Opus 5.5

[c-suvamsa]

Summary

• Agent / Model: DevRev Computer / Claude Opus 5.5 (Anthropic).
• Source Harbor job URL: https://hub.harborframework.com/jobs/839c0f26-fdfd-455f-82ee-4bef9e7d5c9f.
• Complete rerun from 2026-10-15 using enterprise-bench/tasks/v1: 10 attempts per task; 135/140 passed (96.4%). This supersedes the earlier Opus 5.5 job rather than replacing an individual trial.
• SDK-reported cost: $106.24.

PR checklist – leaderboard row entry

Provenance and access

• Source job is public and owned by the submitter; entry header and job URL match.
• Entry has all required metadata and metrics; 140 distinct trial IDs match n_trials.
• Accuracy, pass@k, token breakdown, and displayed SDK-reported cost reconcile with the recorded run.
• Agent, model, and provider are shown on the job.
• Every local trial has an ATIF trajectory, reward, and token usage; no trials were omitted.
• Reviewer to confirm logged-out replay of individual trajectories.

Run conditions and review

• Task digests for all 14 tasks match enterprise-bench main. The run used an absolute local task path rather than a registry version.
• Scored timeout multiplier is 1.0; no CPU/memory/storage overrides were recorded.
• Reviewer to confirm the agent had no benchmark-answer access and re-verify the submitted score.

Validation

• Main-branch leaderboard validator passed locally; repository validate and validate-leaderboard CI checks pass on the replacement commit.

Notes for reviewers

Model label was supplied by the run owner after execution; the saved Computer trace uses its generic gateway alias. This is a new full run with all original failures retained, not a selected retry.

[nimit2801]

Thanks for the leaderboard submission for DevRev Computer / Claude Opus 5.5. Acknowledged through Nimit’s intake workflow; this is not approval. Both validation checks are passing. Technical review still needs to verify:

• Public trajectory replay and source-job/trial provenance.
• Dataset version, original run configuration, and no benchmark-answer access.
• Score/pass@k and token/cost reconciliation against the recorded trials.

[nimit2801]

Thanks for the updated DevRev Computer / Claude Opus 5.5 leaderboard submission. Acknowledged through Nimit’s intake workflow, not approval. Both validation checks pass. Technical review still needs to verify:

• Public trajectory replay and provenance for the replacement full run.
• Task version, run limits, and no benchmark-answer access.
• Score/pass@k, token usage, cost, and model-label evidence against the recorded trials.