Leaderboard v1: Claude Code / Claude Opus 5

[c-suvamsa]

Summary

Adds one v1 leaderboard row for Claude Code / Claude Opus 5 (Anthropic), from the public Harbor job for run 2026-10-14__20-23-38.

• Agent / Model: Claude Code / Claude Opus 5 (Anthropic)
• Source Harbor job URL: https://hub.harborframework.com/jobs/4ae322ea-aab2-4b6d-b1ca-d27abdfd7152

Run summary: 116/140 trials passed = 82.9% accuracy. 14 tasks × 10 attempts = 140 trials, run at concurrency 4, on tasks/v1 via Daytona (DaytonaEnvironment) with v1 shared MCP (.daytona/v1-mcp.json). Model route: DevRev Bedrock proxy (bedrock/invoke/anthropic.claude-opus-5). Total 149,257,890 tokens; cost ~$169.11.

PR Checklist — Leaderboard Row Entry

Provenance & access

• Source Harbor job is PUBLIC (public URL opens unauthenticated; job name 2026-10-14__20-23-38 matches). Note: the results/trajectory table on the job page is client-side rendered, so a plain HTTP read could not fully render the per-trial trajectory viewer; reviewers should confirm trajectory replay in-browser.
• Submitter owns the job / job was uploaded via harbor upload
• Row entry YAML links to the public job URL (hub.harborframework.com/jobs/4ae322ea-aab2-4b6d-b1ca-d27abdfd7152)
• Job ID in the header comment matches the public job

Row file conforms to the template

• Conforms to leaderboard/row-template.yaml; no unknown keys
• Filename follows convention: 2026-10-14__claude-code__claude-opus-5.yaml
• All required metrics present: accuracy, display_accuracy, token breakdown (uncached_input, cached_input, output, total), avg_trial_duration_sec, pass_at_2/3/4/5/8/10, n_trials, total_cost_usd, display_total_cost_usd
• metadata block complete: agent_display_name, model_display_name, agent_org_display_name, model_org_display_name
• n_trials == number of trial_ids listed (140 == 140)
• trial_ids are unique — no duplicates within file or across existing main rows

Metrics integrity

• total_tokens reconciles: uncached (8,885,443) + cached (138,601,236) + output (1,771,211) = 149,257,890 = total_tokens
• display_accuracy ("82.9%") matches accuracy (82.86)
• display_total_cost_usd ("$169.11") matches total_cost_usd (169.11)
• status is display

Public job contents (each trial)

• Per-trial trajectory present in the local job (ATIF-format trajectories)
• Reward/result file per trial (reward 0.0/1.0; 116 passed, 24 failed)
• Token usage present: uncached input, cached, output
• Provider / model / agent surfaced on the job (claude-code / claude-opus-5 / Anthropic)

Run conditions (comparability)

• Timeout multipliers at default: timeout_multiplier=1.0, agent_timeout_multiplier=null, verifier_timeout_multiplier=null
• Setup/build timeout multipliers unchanged (null)
• No CPU / memory / storage overrides (Daytona defaults)
• Agent did not access benchmark site/repo or answers during the run
• Dataset = tasks/v1 (14 tasks); the entry is under leaderboard/entries/ on main.

Review

• Reviewer can independently re-verify: trajectories replay and reward matches submitted score

Validation

• Entry YAML validated against the main leaderboard schema and entries
• validate-leaderboard CI passes for this entry under leaderboard/entries/.

[nimit2801]

Thanks for the Claude Code / Claude Opus 5 v1 leaderboard submission. Acknowledged by Nimit's intake workflow; this is not approval. CI validate and validate-leaderboard are passing. Technical review still needs to verify:

• Public Harbor job access and trial provenance.
• Accuracy, pass@k, token and cost claims against the trial evidence.
• Default timeout/run conditions and trajectory integrity.