Leaderboard v1: Claude Code / Claude Opus 5.5

[c-suvamsa]

Summary

Adds one v1 leaderboard row for Claude Code / Claude Opus 5.5 (Anthropic), from the public Harbor job for run 2026-10-14__08-43-57.

• Agent / Model: Claude Code / Claude Opus 5.5 (Anthropic)
• Source Harbor job URL: https://hub.harborframework.com/jobs/55a40bce-1f7c-49fc-a270-83e770afe9a5

Run summary: 111/140 trials passed = 79.3% accuracy. 14 tasks × 10 attempts = 140 trials, run at concurrency 4, on tasks/v1 via Daytona (DaytonaEnvironment) with v1 shared MCP (.daytona/v1-mcp.json). Model route: DevRev Bedrock proxy (bedrock/invoke/anthropic.claude-opus-5-5). Total 82,566,380 tokens; cost ~$62.92.

PR Checklist — Leaderboard Row Entry

Provenance & access

• Source Harbor job is PUBLIC (public URL opens unauthenticated; job name 2026-10-14__08-43-57 matches). Note: the results/trajectory table on the job page is client-side rendered, so a plain HTTP read could not fully render the per-trial trajectory viewer; reviewers should confirm trajectory replay in-browser.
• Submitter owns the job / job was uploaded via harbor upload
• Row entry YAML links to the public job URL (hub.harborframework.com/jobs/55a40bce-1f7c-49fc-a270-83e770afe9a5)
• Job ID in the header comment matches the public job

Row file conforms to the template

• Conforms to leaderboard/row-template.yaml; no unknown keys
• Filename follows convention: 2026-10-14__claude-code__claude-opus-5-5.yaml
• All required metrics present: accuracy, display_accuracy, token breakdown (uncached_input, cached_input, output, total), avg_trial_duration_sec, pass_at_2/3/4/5/8/10, n_trials, total_cost_usd, display_total_cost_usd
• metadata block complete: agent_display_name, model_display_name, agent_org_display_name, model_org_display_name
• n_trials == number of trial_ids listed (140 == 140)
• trial_ids are unique — no duplicates within file or across existing main rows

Metrics integrity

• total_tokens reconciles: uncached (6,068,472) + cached (75,625,130) + output (872,778) = 82,566,380 = total_tokens
• display_accuracy ("79.3%") matches accuracy (79.29)
• display_total_cost_usd ("$62.92") matches total_cost_usd (62.92)
• status is display

Public job contents (each trial)

• Per-trial trajectory present in the local job (ATIF-format trajectories)
• Reward/result file per trial (reward 0.0/1.0; 111 passed, 29 failed)
• Token usage present: uncached input, cached, output
• Provider / model / agent surfaced on the job (claude-code / claude-opus-5-5 / Anthropic)

Run conditions (comparability)

• Timeout multipliers at default: timeout_multiplier=1.0, agent_timeout_multiplier=null, verifier_timeout_multiplier=null
• Setup/build timeout multipliers unchanged (null)
• No CPU / memory / storage overrides (Daytona defaults)
• Agent did not access benchmark site/repo or answers during the run
• Dataset = tasks/v1 (14 tasks); the entry is under leaderboard/entries/ on main.

Review

• Reviewer can independently re-verify: trajectories replay and reward matches submitted score

Validation

• Entry YAML validated against the main leaderboard schema and entries
• validate-leaderboard CI passes for this entry under leaderboard/entries/.

[nimit2801]

Thanks for the submission. Nimit’s intake workflow has picked up this v1 leaderboard entry for Claude Code / Claude Opus 5.5. Both CI checks are passing; this is an acknowledgement, not approval.

Review will need to verify:

• Public Harbor job access and trial provenance.
• Accuracy, pass@k, and token metrics against the trial artifacts.
• Default timeouts and real tool use under comparable run conditions.

The full leaderboard audit is still required.