Skip to content

Add claude-sonnet-5-5 (34.65, provisional) and gemini-4-argon (32.61, provisional) community results - #3

Open
Wyyyb wants to merge 10 commits into
search-swe:mainfrom
Wyyyb:add-claude-sonnet-5-5
Open

Wyyyb wants to merge 10 commits into
search-swe:mainfrom
Wyyyb:add-claude-sonnet-5-5

Conversation

@Wyyyb

@Wyyyb Wyyyb commented Nov 10, 2026 •

Copy link
Copy Markdown

Two community evaluation results on all 47 Search-SWE tasks, with complete redacted trajectories + per-trial evidence on Hugging Face. Updated per the 2026-11-13, 11-18, 11-19 and 11-20 reviews (latest: comment 7032165652 → submissions/review-response-2026-11-20.md).

claude-sonnet-5-5 (Claude Code harness) — 36.89 / 100 (provisional; 30.66 → 34.65 → 36.89)

  • 2026-11-19 round 2 (comment 7013193625), search-swe regrade @ bec6079: query-encoder-alignment 0→52.50 (PR#53 calibration fix), constrained-molecular-search 0→22.93 and budgeted-candidate-reranking 0→29.54 (PR#55 cgroup v1/v2 fix; integrity judge 1.0).
  • 2026-11-19 round 1 recovered 5 integrity-judge false zeros (missing OPENAI_* env): chunqiu 55.89, legal-stance 48.26, quantity-aware 39.89, code-embedding 33.50, biomedical 10.00.
  • Trajectories + evidence: https://huggingface.co/datasets/ubowang/search-swe-claude-sonnet-5-5 (commit e0758f61)

gemini-4-argon (Antigravity CLI harness) — 34.02 / 100 (provisional; 21.80 → 32.61 → 34.02)

  • 2026-11-20 (comment 7032165652): token usage recovered from the native antigravity-cli.trajectory.jsonl → usage.json + usage_responses.jsonl (per-response input / cache-read / output for all 46 selected trials, ATIF step_id-mapped at 100%; CHECKPOINT=compaction calls noted; cost not recorded by the agy route). See submissions/review-response-2026-11-20.md.
  • 2026-11-19 round 2: query-encoder-alignment 0→47.50, constrained-molecular-search 0→18.57 (argon agent-trajectory JSON repaired so the in-verifier integrity gate could parse).
  • Trajectories + evidence + usage: https://huggingface.co/datasets/ubowang/search-swe-gemini-4-argon (commit a3cd1787)

Means are provisional only for the 3 maintainer-side full reruns, counted 0: argon large-scale-formula-retrieval, sonnet rust-bm25-search, sonnet large-scale-formula-retrieval. Means over resolved tasks: sonnet 38.52 (45), argon 34.76 (46).

Per-task regrade records (original→new trial IDs, final rewards, redacted regrade.json) are in submissions/review-response-2026-11-19b.json and each dataset's evidence/<task>/regrade/; Argon token usage is in submissions/review-response-2026-11-20.json + the dataset's usage.json/usage_responses.jsonl.

🤖 Generated with Claude Code

Wyyyb and others added 2 commits November 10, 2026 18:31
Equal-weight mean 30.66/100 over 47 tasks. Trajectories on HF:
https://huggingface.co/datasets/ubowang/search-swe-claude-sonnet-5-5

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Equal-weight mean 21.80/100 over 47 tasks. Trajectories on HF:
https://huggingface.co/datasets/ubowang/search-swe-gemini-4-argon

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Wyyyb Wyyyb changed the title Add claude-sonnet-5-5 (Claude Code harness) community result Add claude-sonnet-5-5 and gemini-4-argon community results Nov 11, 2026
Wyyyb and others added 2 commits November 13, 2026 20:42
…fix completeness notes

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Wyyyb Wyyyb changed the title Add claude-sonnet-5-5 and gemini-4-argon community results Add claude-sonnet-5-5 (30.66) and gemini-4-argon (32.61) community results Nov 13, 2026
hanhainebula added a commit that referenced this pull request Nov 16, 2026
Add 141 max-effort trials across DeepSeek / Claude Code, DeepSeek / Pi,
and Kimi K3 / Claude Code, bringing the leaderboard to 10 settings and
470 trials. Pin downloads to the verified Hugging Face revision while
preserving all 329 earlier selections, costs and trajectory payloads.

Recover original coding durations, reconcile Pi native tail receipts,
and apply the existing recorded-response fallback with official-rate
cost estimates. Keep all DeepSeek harnesses in the table and show its
best overall harness in Pareto charts. Align publication documentation.

Validation: 87 Python tests; production and task build checks; full public
package verification; Pareto and browser suites; anonymous download hash
and CORS checks; all 482 leaderboard pages have valid local links.
Step-5-preview, Sonnet, Argon and PR #3 remain on hold.
…ns, provisional means

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@hanhainebula

Copy link
Copy Markdown
Contributor

Thanks for uploading the additional evidence. It helped us identify where several of the zero scores came from. Search-SWE’s run_task.py has no --verify-only flag, but Harbor 0.22.0 has a built-in trial regrade command. Your evaluation agent can check harbor trial regrade --help and use this command with the original trial directory and the corresponding Search-SWE task directory.

harbor trial regrade "$ORIGINAL_TRIAL_DIR" --task-path "$TASK_DIR"

This runs the scoring pipeline, including the judge where required, against the saved original submission. It skips the model’s coding run. Use the original Harbor trial directory with its config.json, result.json, agent/ and artifacts/, including the files the verifier needs. The public HF trajectory and logs alone are insufficient. Please keep the task’s standard verifier configuration and document any changes needed to fix the scoring environment.

Please regrade these 11 existing trials.

Model Task What failed
Claude Sonnet 5.5 legal-stance-retrieval Integrity judge failed
Claude Sonnet 5.5 quantity-aware-retrieval Integrity judge failed
Claude Sonnet 5.5 biomedical-graph-retrieval Integrity judge failed
Claude Sonnet 5.5 chunqiu-temporal-retrieval Integrity judge failed
Claude Sonnet 5.5 icsi-meeting-memory Answer generation service failed during scoring
Claude Sonnet 5.5 code-embedding-finetuning Score finalization failed
Claude Sonnet 5.5 query-encoder-alignment Main evaluation failed its baseline calibration check
Claude Sonnet 5.5 budgeted-candidate-reranking Verifier could not read the cgroup memory limit
Claude Sonnet 5.5 constrained-molecular-search Verifier could not read the cgroup CPU limit
Gemini 4 Argon query-encoder-alignment Main evaluation failed its baseline calibration check
Gemini 4 Argon constrained-molecular-search Verifier could not read the cgroup CPU limit

For each regrade, please share the original and new trial IDs, the final reward, and links to the scoring report and relevant verifier logs. A confirmed valid zero is fine. If the original submission or a required artifact is missing, please mark the item pending and say what’s missing.

We’ll handle these three full reruns on our side. Please leave them out of your queue.

  • Gemini 4 Argon — large-scale-formula-retrieval
  • Claude Sonnet 5.5 — rust-bm25-search
  • Claude Sonnet 5.5 — large-scale-formula-retrieval

….66->34.65), cgroup-v1 pending items

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@Wyyyb Wyyyb changed the title Add claude-sonnet-5-5 (30.66) and gemini-4-argon (32.61) community results Add claude-sonnet-5-5 (34.65, provisional) and gemini-4-argon (32.61, provisional) community results Nov 19, 2026
Wyyyb and others added 2 commits November 19, 2026 06:16
The 2026-11-19 PR comment asked for both the original and new trial IDs
per regrade; the report previously listed only the new (regrade) trial.
Add original_trial_id to every record in review-response-2026-11-19.json
and a consolidated "Original -> new trial IDs (all 11 regrades)" table to
the .md. No scores change; linkage only.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@hanhainebula

hanhainebula commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Thanks for linking the original and regraded trial IDs. The cgroup v1/v2 fix (VectorSpaceLab/Search-SWE#55) and the Query Encoder calibration fix (VectorSpaceLab/Search-SWE#53) are both on main. Please use Search-SWE CLI 0.3.1 (https://pypi.org/project/search-swe/0.3.1/) with a checkout at bec6079 (VectorSpaceLab/Search-SWE@bec6079) to regrade these five saved trials.

Model Task Original trial
Sonnet 5.5 query-encoder-alignment query-encoder-alignment__7M4cETf
Argon query-encoder-alignment query-encoder-alignment__yUnmYuE
Sonnet 5.5 constrained-molecular-search constrained-molecular-search__yfaRDio
Argon constrained-molecular-search constrained-molecular-search__JeQEvXr
Sonnet 5.5 budgeted-candidate-reranking budgeted-candidate-reranking__6nmMaWv

For the DeepSeek integrity judge, you can reuse the OpenRouter key from the .env you used for the evaluation, provided it can access the configured DeepSeek model. Set VERIFIER_OPENAI_API_KEY to that existing key and VERIFIER_OPENAI_BASE_URL to https://openrouter.ai/api/v1. The CLI passes these to the verifier as OPENAI_API_KEY and OPENAI_BASE_URL, so you can use the evaluation .env directly without passing --verifier-env manually.

search-swe regrade "$ORIGINAL_TRIAL_DIR" --repo "$SEARCH_SWE_CHECKOUT" --env-file "$EVAL_ENV_FILE" --dry-run
search-swe regrade "$ORIGINAL_TRIAL_DIR" --repo "$SEARCH_SWE_CHECKOUT" --env-file "$EVAL_ENV_FILE"

Run the second command once the preview confirms the recorded inputs are present and shows Missing verifier settings: none. Please share the new trial IDs, final rewards, and links to the redacted regrade.json summaries. If a run fails, please share the specific error and leave that score pending. The three CPU reruns remain on hold on our side.

search-swe regrade (verify-only) @ bec6079 / CLI 0.3.1 with the merged
cgroup v1/v2 fix (PR#55) and query-encoder calibration fix (PR#53):
- query-encoder-alignment: sonnet 0->52.50, argon 0->47.50
- constrained-molecular-search: sonnet 0->22.93, argon 0->18.57
- budgeted-candidate-reranking: sonnet 0->29.54
All status=completed, measurement_valid=true, integrity_judge=1.0 (cgroup
tasks). Provisional means: sonnet 34.65->36.89, argon 32.61->34.02.

Align the model tables/headlines with the authoritative bundle results.json
(sonnet per-task table regenerated), add submissions/review-response-2026-11-19b.
{json,md} with original->new trial IDs + redacted regrade.json links. The 3
full reruns remain maintainer-side.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@hanhainebula

hanhainebula commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Thanks again for the Argon submission. Could you check whether the original Antigravity CLI logs or Vertex AI records still contain token usage for the 46 selected Vertex AI trials? We’d like to show input, cache-read, and output tokens alongside the corresponding steps in the leaderboard’s trajectory viewer.

In our local OpenRouter rerun, the native antigravity-cli.trajectory.jsonl recorded input_tokens, cache_read_tokens, and output_tokens on PLANNER_RESPONSE events. Your original logs may have the same fields.

If you can recover the usage, please upload a redacted usage.json to the Argon Hugging Face dataset and share the updated revision. Please include:

  • One record per model response with the task name, selected trial ID, and input, cache-read, and output token counts. Include the ATIF step_id where available, or the native event index and timestamp so we can match the response to the displayed trajectory.
  • A short note on how the counters work, including whether input tokens already include cache reads and how any cumulative checkpoint records were handled.
  • Recorded Vertex AI cost per trial, if available, with the currency.

For trials where only per-task totals are recoverable, please include them and label them as aggregate-only. They can fill the summary columns. Please flag any missing or partial coverage.

Please use the saved runs for this. We’ll handle the usage for the OpenRouter Formula rerun on our side.

Recovered input/cache-read/output tokens for all 46 selected Vertex AI
Argon trials from each trial's native antigravity-cli.trajectory.jsonl
(saved runs; no re-run) and uploaded to the Argon HF dataset (rev a3cd1787):
- usage.json: per-task totals + grand total + semantics + cost note
- usage_responses.jsonl: 6,373 per-response records, ATIF step_id-mapped
  (6335/6335 PLANNER_RESPONSE = 100%; CHECKPOINT->null)

input_tokens excludes cache_read_tokens (separate); CHECKPOINT = compaction
calls (included, not cumulative). Vertex cost not recorded by the agy route.
Add submissions/review-response-2026-11-20.{json,md}; update the argon
submission's token-usage note.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants