[yl231]

Re-grades every prediction file behind a leaderboard row (25 routers) with the current evaluator. It ran on GitHub's runners, because LiveCodeBench grading executes submitted code: workflow run 40971464215 on main at 5a342c5, 4 workers per router. All 25 jobs succeeded.

What the current evaluator includes that some stored scores predate

• LiveCodeBench: single-quoted __main__ guards (Fix LiveCodeBench scoring of single-quoted __main__ guards #230), sys.stdin.buffer input (Fix LiveCodeBench stdin tests for solutions that read sys.stdin.buffer #234), and a per-test memory limit (Limit memory of LiveCodeBench test processes #239).
• METEOR: the penalty cap that keeps scores in [0, 1] (Cap the METEOR fragmentation penalty so scores stay in [0, 1] (#210) #216).
• Costs: recomputed from the current price table. The price table itself will be rebuilt later, and every router re-priced then.

What this PR changes

• Prediction files: per-row accuracy and cost. Each file keeps its original JSON formatting, so the diff shows only changed values. Routers that stored no per-row scores (BizRouter, NuRoute, Weave, Chayan, chuzom, cruq, LLM Router, R**2, vLLM-SR) now have them, so their website difficulty breakdowns can be built. nadir-caliper's stored all-zero scores are replaced.
• README: Arena, accuracy, cost and optimality columns, with the table re-ranked. Robustness doesn't depend on grading and is unchanged.

Correction: the /evaluate re-runs I triggered on #232, #229 and #224 earlier today graded with each PR branch's own copy of the evaluator, so they did not include these fixes. The numbers here supersede them.

Not re-graded: Paix2 (on hold, #211), and four prediction files with no leaderboard row (glm-4-air-router, llm-router, nadir-tumbler, sqwish-router).

Leaderboard before → after

Rank (before → after)	Router	Arena	Accuracy	Cost / 1K
2 → 1	Divyam	76.88 → 77.22	80.43 → 80.83	$0.65 → $0.65
1 → 2	BizRouter	77.04 → 77.08	81.61 → 81.66	$1.00 → $1.00
3 → 3	KT-ModelRouter	76.28 → 76.37	78.14 → 78.24	$0.27 → $0.27
4 → 4	Sqwish Router	76.21 → 76.31	79.76 → 79.88	$0.70 → $0.70
5 → 5	R**2: Reasonometry Router	76.13 → 76.16	78.94 → 78.97	$0.48 → $0.48
6 → 6	Cross-Router	75.75 → 75.87	78.14 → 78.27	$0.40 → $0.40
7 → 7	LLM Router	75.69 → 75.77	78.44 → 78.54	$0.49 → $0.49
8 → 8	SapientAI Auto Router	75.41 → 75.69	77.89 → 78.23	$0.43 → $0.43
9 → 9	vLLM‑SR	74.86 → 74.96	77.18 → 77.30	$0.42 → $0.42
10 → 10	nadir-caliper	74.55 → 74.81	75.84 → 76.14	$0.22 → $0.22
12 → 11	BARouter	73.79 → 74.22	75.72 → 76.22	$0.36 → $0.36
11 → 12	AgentForge Router	74.13 → 74.20	74.72 → 74.80	$0.13 → $0.13
13 → 13	NuRoute Artemis	72.87 → 73.47	75.47 → 76.19	$0.60 → $0.60
14 → 14	Weave Router	72.82 → 72.88	76.32 → 76.22	$0.94 → $0.87
16 → 15	OrcaRouter‑Adaptive	72.08 → 72.41	75.54 → 75.63	$1.00 → $0.86
15 → 16	Nadir Router	72.29 → 72.24	75.01 → 74.96	$0.68 → $0.68
17 → 17	Hybrid Router	72.08 → 72.15	71.38 → 71.46	$0.04 → $0.04
18 → 18	R2-Router	71.60 → 71.86	71.23 → 71.18	$0.06 → $0.04
19 → 19	cruq-router	70.77 → 70.82	71.35 → 71.41	$0.18 → $0.18
20 → 20	chuzom-solo-v32	70.61 → 70.64	70.59 → 70.62	$0.10 → $0.10
21 → 21	Azure-Model-Router	70.42 → 70.41	72.94 → 72.93	$0.73 → $0.73
22 → 22	Auto Router	70.05 → 70.27	70.17 → 70.42	$0.12 → $0.12
23 → 23	Lynkr	67.65 → 68.02	68.41 → 68.83	$0.29 → $0.29
26 → 25	AsiaInfo-Router	65.87 → 66.30	75.20 → 75.82	$8.54 → $8.54
29 → 29	Chayan	63.83 → 63.70	64.89 → 64.87	$0.56 → $0.61

Row-level changes per prediction file

"Newly scored" rows had no stored score before. Grade flips compare stored and new scores, including the optional candidate rows; for Cross-Router all flips are in candidate rows. Two flips in nadir-cascade-v2 were checked against the labels by hand (ArcMMLU_468 answered A, label D; ArcMMLU_65 answered D, label A): the stored 1.0 was wrong.

Router file	Rows newly scored	Grade flips 0→1 / 1→0	Rows with new cost	LiveCodeBench share of flips
AsiaInfo-Router	0	53 / 0	0	53/53
BARouter	0	53 / 0	0	53/53
BizRouter	8395	0 / 0	0	0/0
PSAI-Auto-Router	0	32 / 4	0	36/36
agentforge-router	2	2 / 3	4829	3/5
auto_router	0	45 / 2	28	20/47
azure-model-router	0	77 / 1	5061	2/78
chayan	8400	0 / 0	0	0/0
chuzom-solo-v32	8400	0 / 0	0	0/0
cross-router	784	94 / 207	3557	44/301
cruq-router	8400	0 / 0	0	0/0
divyam-router	0	46 / 2	0	48/48
hybrid-router	0	2 / 1	0	3/3
llm-router-g3	8400	0 / 0	0	0/0
lynkr	163	37 / 0	13	31/37
nadir-caliper	0	6093 / 0	0	310/6093
nadir-cascade-v2	0	53 / 89	7631	15/142
nuroutev2-artemis	8382	0 / 0	0	0/0
orcarouter-adaptive	0	3 / 17	4327	4/20
r2-reasonometry-router	8400	0 / 0	0	0/0
r2-router	0	1 / 6	9953	7/7
sqwish-fvcta8	0	31 / 1	0	32/32
vllm-sr	14063	0 / 0	0	0/0
weave-router	8384	0 / 0	8384	0/0

🤖 Generated with Claude Code