Leaderboard v1: Computer / GPT-6 Astra

[c-suvamsa]

Summary

• Agent / Model: DevRev Computer / GPT-6 Astra (OpenAI).
• Source Harbor job URL: https://hub.harborframework.com/jobs/3dd7fbd2-a753-4ede-ad67-cca1eb4622c1.
• Complete rerun from 2026-10-15 using enterprise-bench/tasks/v1: 10 attempts per task; 130/140 passed (92.9%). This supersedes the earlier Astra job rather than replacing an individual failed trial.
• SDK-reported cost: $106.64.

PR checklist – leaderboard row entry

Provenance and access

• Source job is public and owned by the submitter; entry header and job URL match.
• Entry has all required metadata and metrics; 140 distinct trial IDs match n_trials.
• Accuracy, pass@k, token breakdown, and displayed SDK-reported cost reconcile with the recorded run.
• Agent, model, and provider are shown on the job.
• Every local trial has an ATIF trajectory, reward, and token usage; no trials were omitted.
• Reviewer to confirm logged-out replay of individual trajectories.

Run conditions and review

• Task digests for all 14 tasks match enterprise-bench main. The run used an absolute local task path rather than a registry version.
• Scored timeout multiplier is 1.0; no CPU/memory/storage overrides were recorded.
• Reviewer to confirm the agent had no benchmark-answer access and re-verify the submitted score.

Validation

• Main-branch leaderboard validator passed locally.
• Repository validate and validate-leaderboard CI checks pass on the replacement commit.

Notes for reviewers

Model label was supplied by the run owner after execution; the saved Computer trace uses its generic gateway alias. This is a new full run with all original failures retained, not a selected retry.

[nimit2801]

Thanks for the leaderboard submission for DevRev Computer / GPT-6 Astra. Acknowledged by Nimit’s PR intake workflow; this is not approval. Both validation checks are passing. Technical review still needs to verify:

• Harbor trial evidence against accuracy, pass@k, and token totals.
• Dataset version, original run configuration, and benchmark-answer isolation.
• Public trajectory replay, including the disclosed missing failed-trial trajectory and incomplete cost reporting.

[nimit2801]

Acknowledged the updated DevRev Computer / GPT-6 Astra leaderboard submission through Nimit’s PR intake workflow. This is not approval. Both validation checks pass; technical review still needs to verify:

• The replacement full-run Harbor evidence against accuracy, pass@k, token totals, and SDK-reported cost.
• Logged-out replay of every trial, including failures, and model provenance behind the generic gateway alias.
• Task version/digests, run limits, and benchmark-answer isolation.