Repository navigation
Add claude-sonnet-5-5 (34.65, provisional) and gemini-4-argon (32.61, provisional) community results - #3
Add claude-sonnet-5-5 (34.65, provisional) and gemini-4-argon (32.61, provisional) community results#3Wyyyb wants to merge 10 commits into
Conversation
Equal-weight mean 30.66/100 over 47 tasks. Trajectories on HF: https://huggingface.co/datasets/ubowang/search-swe-claude-sonnet-5-5 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Equal-weight mean 21.80/100 over 47 tasks. Trajectories on HF: https://huggingface.co/datasets/ubowang/search-swe-gemini-4-argon Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…fix completeness notes Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add 141 max-effort trials across DeepSeek / Claude Code, DeepSeek / Pi, and Kimi K3 / Claude Code, bringing the leaderboard to 10 settings and 470 trials. Pin downloads to the verified Hugging Face revision while preserving all 329 earlier selections, costs and trajectory payloads. Recover original coding durations, reconcile Pi native tail receipts, and apply the existing recorded-response fallback with official-rate cost estimates. Keep all DeepSeek harnesses in the table and show its best overall harness in Pareto charts. Align publication documentation. Validation: 87 Python tests; production and task build checks; full public package verification; Pareto and browser suites; anonymous download hash and CORS checks; all 482 leaderboard pages have valid local links. Step-5-preview, Sonnet, Argon and PR #3 remain on hold.
…ns, provisional means Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Thanks for uploading the additional evidence. It helped us identify where several of the zero scores came from. Search-SWE’s run_task.py has no --verify-only flag, but Harbor 0.22.0 has a built-in trial regrade command. Your evaluation agent can check harbor trial regrade --help and use this command with the original trial directory and the corresponding Search-SWE task directory. harbor trial regrade "$ORIGINAL_TRIAL_DIR" --task-path "$TASK_DIR" This runs the scoring pipeline, including the judge where required, against the saved original submission. It skips the model’s coding run. Use the original Harbor trial directory with its config.json, result.json, agent/ and artifacts/, including the files the verifier needs. The public HF trajectory and logs alone are insufficient. Please keep the task’s standard verifier configuration and document any changes needed to fix the scoring environment. Please regrade these 11 existing trials.
For each regrade, please share the original and new trial IDs, the final reward, and links to the scoring report and relevant verifier logs. A confirmed valid zero is fine. If the original submission or a required artifact is missing, please mark the item pending and say what’s missing. We’ll handle these three full reruns on our side. Please leave them out of your queue.
|
….66->34.65), cgroup-v1 pending items Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The 2026-11-19 PR comment asked for both the original and new trial IDs per regrade; the report previously listed only the new (regrade) trial. Add original_trial_id to every record in review-response-2026-11-19.json and a consolidated "Original -> new trial IDs (all 11 regrades)" table to the .md. No scores change; linkage only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Thanks for linking the original and regraded trial IDs. The cgroup v1/v2 fix (VectorSpaceLab/Search-SWE#55) and the Query Encoder calibration fix (VectorSpaceLab/Search-SWE#53) are both on main. Please use Search-SWE CLI 0.3.1 (https://pypi.org/project/search-swe/0.3.1/) with a checkout at bec6079 (VectorSpaceLab/Search-SWE@bec6079) to regrade these five saved trials.
For the DeepSeek integrity judge, you can reuse the OpenRouter key from the .env you used for the evaluation, provided it can access the configured DeepSeek model. Set VERIFIER_OPENAI_API_KEY to that existing key and VERIFIER_OPENAI_BASE_URL to https://openrouter.ai/api/v1. The CLI passes these to the verifier as OPENAI_API_KEY and OPENAI_BASE_URL, so you can use the evaluation .env directly without passing --verifier-env manually. search-swe regrade "$ORIGINAL_TRIAL_DIR" --repo "$SEARCH_SWE_CHECKOUT" --env-file "$EVAL_ENV_FILE" --dry-run Run the second command once the preview confirms the recorded inputs are present and shows Missing verifier settings: none. Please share the new trial IDs, final rewards, and links to the redacted regrade.json summaries. If a run fails, please share the specific error and leave that score pending. The three CPU reruns remain on hold on our side. |
search-swe regrade (verify-only) @ bec6079 / CLI 0.3.1 with the merged
cgroup v1/v2 fix (PR#55) and query-encoder calibration fix (PR#53):
- query-encoder-alignment: sonnet 0->52.50, argon 0->47.50
- constrained-molecular-search: sonnet 0->22.93, argon 0->18.57
- budgeted-candidate-reranking: sonnet 0->29.54
All status=completed, measurement_valid=true, integrity_judge=1.0 (cgroup
tasks). Provisional means: sonnet 34.65->36.89, argon 32.61->34.02.
Align the model tables/headlines with the authoritative bundle results.json
(sonnet per-task table regenerated), add submissions/review-response-2026-11-19b.
{json,md} with original->new trial IDs + redacted regrade.json links. The 3
full reruns remain maintainer-side.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Thanks again for the Argon submission. Could you check whether the original Antigravity CLI logs or Vertex AI records still contain token usage for the 46 selected Vertex AI trials? We’d like to show input, cache-read, and output tokens alongside the corresponding steps in the leaderboard’s trajectory viewer. In our local OpenRouter rerun, the native antigravity-cli.trajectory.jsonl recorded input_tokens, cache_read_tokens, and output_tokens on PLANNER_RESPONSE events. Your original logs may have the same fields. If you can recover the usage, please upload a redacted usage.json to the Argon Hugging Face dataset and share the updated revision. Please include:
For trials where only per-task totals are recoverable, please include them and label them as aggregate-only. They can fill the summary columns. Please flag any missing or partial coverage. Please use the saved runs for this. We’ll handle the usage for the OpenRouter Formula rerun on our side. |
Recovered input/cache-read/output tokens for all 46 selected Vertex AI
Argon trials from each trial's native antigravity-cli.trajectory.jsonl
(saved runs; no re-run) and uploaded to the Argon HF dataset (rev a3cd1787):
- usage.json: per-task totals + grand total + semantics + cost note
- usage_responses.jsonl: 6,373 per-response records, ATIF step_id-mapped
(6335/6335 PLANNER_RESPONSE = 100%; CHECKPOINT->null)
input_tokens excludes cache_read_tokens (separate); CHECKPOINT = compaction
calls (included, not cumulative). Vertex cost not recorded by the agy route.
Add submissions/review-response-2026-11-20.{json,md}; update the argon
submission's token-usage note.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two community evaluation results on all 47 Search-SWE tasks, with complete redacted trajectories + per-trial evidence on Hugging Face. Updated per the 2026-11-13, 11-18, 11-19 and 11-20 reviews (latest: comment 7032165652 →
submissions/review-response-2026-11-20.md).claude-sonnet-5-5 (Claude Code harness) — 36.89 / 100 (provisional; 30.66 → 34.65 → 36.89)
search-swe regrade@bec6079:query-encoder-alignment0→52.50 (PR#53 calibration fix),constrained-molecular-search0→22.93 andbudgeted-candidate-reranking0→29.54 (PR#55 cgroup v1/v2 fix; integrity judge 1.0).OPENAI_*env): chunqiu 55.89, legal-stance 48.26, quantity-aware 39.89, code-embedding 33.50, biomedical 10.00.e0758f61)gemini-4-argon (Antigravity CLI harness) — 34.02 / 100 (provisional; 21.80 → 32.61 → 34.02)
antigravity-cli.trajectory.jsonl→usage.json+usage_responses.jsonl(per-response input / cache-read / output for all 46 selected trials, ATIFstep_id-mapped at 100%; CHECKPOINT=compaction calls noted; cost not recorded by the agy route). Seesubmissions/review-response-2026-11-20.md.query-encoder-alignment0→47.50,constrained-molecular-search0→18.57 (argon agent-trajectory JSON repaired so the in-verifier integrity gate could parse).a3cd1787)Means are provisional only for the 3 maintainer-side full reruns, counted 0: argon
large-scale-formula-retrieval, sonnetrust-bm25-search, sonnetlarge-scale-formula-retrieval. Means over resolved tasks: sonnet 38.52 (45), argon 34.76 (46).Per-task regrade records (original→new trial IDs, final rewards, redacted
regrade.json) are insubmissions/review-response-2026-11-19b.jsonand each dataset'sevidence/<task>/regrade/; Argon token usage is insubmissions/review-response-2026-11-20.json+ the dataset'susage.json/usage_responses.jsonl.🤖 Generated with Claude Code