Leaderboard v1: Claude Code / Claude Sonnet 5.5

[c-suvamsa]

Summary

• Agent / Model: Claude Code / Claude Sonnet 5.5 (Anthropic).
• Source Harbor job URL: https://hub.harborframework.com/jobs/e38f2267-a754-4246-8338-b6934678df58.
• Run: 2026-10-15; tasks/v1 via Daytona; 14 tasks × 10 attempts, concurrency 4; Harbor 0.24.0.
• Results: 92/140 rewards of 1.0 = 65.7% accuracy; 58,500,415 recorded input + output tokens; $24.89 recorded cost.

PR checklist – leaderboard row entry

Provenance and access

• Source job is public and owned by the submitter; entry header and URL match its job ID.
• Entry has required metadata and metrics; 140 distinct trial IDs match n_trials.
• Accuracy, pass@k, token breakdown, duration, and cost reconcile with the Harbor job and trial records.
• Agent, model, and provider appear on the job; the trajectories record the selected model route.
• All local trials have an ATIF trajectory and a reward/result; no failed or errored attempt was removed.
• Reviewer to confirm logged-out replay of every trajectory.

Run conditions and review

• Task content digests for all 14 tasks match enterprise-bench main. The job used a local tasks/v1 path, not a registry-pinned dataset version.
• Scored timeout multiplier is 1.0, setup/build multipliers are unchanged, and no CPU/memory/storage override was recorded.
• Reviewer to confirm benchmark-answer access boundaries and independently replay the score.

Validation

• Main-branch leaderboard validation passed locally.
• Repository validate and validate-leaderboard CI checks pass.

Notes for reviewers

One trial (sales-l2-b__eCx33ec) recorded AgentTimeoutError at the default 600-second agent limit after submitting a substantive answer. The verifier awarded that trial reward 1.0; its trajectory, usage, cost, and exception are retained, and the submitted score follows the job's recorded reward without excluding or substituting the trial. Please review that exception when assessing comparability.

[nimit2801]

Thanks for the Claude Code / Claude Sonnet 5.5 leaderboard submission. Acknowledged by Nimit’s intake workflow; this is not approval. Both validation checks currently pass. Technical review still needs to verify:

• Public trajectory access and score/usage reconciliation against the Harbor job.
• Task provenance and benchmark-answer access boundaries.
• Timeout comparability, including the disclosed rewarded AgentTimeoutError trial.