Skip to content

Leaderboard: re-price every router at OpenRouter's 2026-10-31 prices - #262

Merged
yl231 merged 1 commit into
mainfrom
leaderboard/reprice-2026-10-31
Oct 31, 2026
Merged

yl231 merged 1 commit into
mainfrom
leaderboard/reprice-2026-10-31

Conversation

@yl231

@yl231 yl231 commented Oct 31, 2026

Copy link
Copy Markdown
Contributor

Re-prices every leaderboard router at OpenRouter's list prices of 2026-10-31, the price table now on main (#218, #250). Made with scripts/pricing/reprice.py --all --write (#261).

  • What changes: only cost values in the prediction files. Accuracy isn't re-graded and no submitted code ran.
  • README: the 23 routers with prediction files get new Arena, Cost/1K and optimality values, and the table is re-ranked. External baselines without prediction files (GPT‑5, CARROT, RouterBench, RouteLLM, …) keep their published numbers.

Changes (rows that moved)

Rank Router Arena Cost / 1K Opt.Sel Opt.Cost
1 Divyam 77.22 → 77.19 $0.65 → $0.66 6.56 → 17.06 17.35 → 19.96
4 → 3 Sqwish Router 76.31 → 76.34 $0.70 → $0.69 9.31 → 11.46 24.06 → 24.13
5 → 4 R**2: Reasonometry Router 76.16 $0.48 — —
3 → 5 KT-ModelRouter 76.37 → 76.08 $0.27 → $0.34 — —
7 → 6 LLM Router 75.77 $0.49 — —
8 → 7 SapientAI Auto Router 75.69 → 75.58 $0.43 → $0.47 5.93 → 5.78 19.86 → 19.04
6 → 8 Cross-Router 75.87 → 75.25 $0.40 → $0.59 17.24 → 45.73 33.64 → 35.46
10 → 9 nadir-caliper 74.81 → 74.63 $0.22 → $0.25 — —
9 → 10 vLLM‑SR 74.96 → 74.24 $0.42 → $0.67 16.81 → 20.40 21.17 → 29.18
11 BARouter 74.22 → 73.67 $0.36 → $0.54 64.62 → 66.16 67.53 → 46.55
12 NuRoute Artemis 73.47 → 73.63 $0.60 → $0.54 — —
14 → 13 OrcaRouter‑Adaptive 72.41 → 72.48 $0.86 → $0.83 — —
15 → 14 Nadir Router 72.24 → 72.25 $0.68 → $0.67 100.00 100.00
16 → 15 Hybrid Router 72.15 → 71.20 $0.04 → $0.13 89.87 → 16.67 94.19 → 63.09
18 → 16 chuzom-solo-v32 70.64 $0.10 — —
19 → 17 Azure-Model-Router 70.41 → 70.37 $0.73 → $0.75 — —
13 → 18 Weave Router 72.88 → 70.00 $0.87 → $3.21 — —
23 → 19 AsiaInfo-Router 66.30 → 70.00 $8.54 → $2.86 — —
17 → 20 cruq-router 70.82 → 69.89 $0.18 → $0.41 — —
20 → 21 Auto Router 70.27 → 69.80 $0.12 → $0.21 37.77 → 47.71 41.21 → 60.25
21 → 22 Lynkr 68.02 → 67.97 $0.29 → $0.31 11.29 → 13.48 17.03 → 19.93
22 → 23 MIRT‑BERT 66.89 $0.15 3.44 19.62

BizRouter (#2, 77.08) and Chayan are unchanged.

What drives it:

  • deepseek-v4-pro was entered at its launch discount ($0.435/$0.87) and OpenRouter now lists $1.60/$3.20. This moves Weave, vLLM‑SR and Cross-Router.
  • qwen3-235b-a22b-2507 output goes from $0.10 to $0.55. This moves Hybrid, cruq and Auto Router.
  • claude-opus-4-7 was charged an older Opus price ($15/$75) against $5/$25. This moves AsiaInfo.
  • gemma-4-31b-it goes from $0.08/$0.35 to $0.20/$0.40. This moves KT-ModelRouter.

Optimality columns move too. Opt.Sel and Opt.Cost compare each pick with the cheapest correct candidate, and the new prices change which candidate that is. Hybrid Router's Opt.Sel goes from 89.87 to 16.67 because qwen3-235b is no longer the cheapest. Opt.Acc doesn't depend on price.

Not included yet: DT-Router (#241) and Corealis (#235). Both routers make model calls of their own that aren't charged yet: DT-Router's LingDT drafts and judging calls, and Corealis's router model. Their rows come once they report those calls in router_usage (#251).

🤖 Generated with Claude Code

Ran scripts/pricing/reprice.py --all --write on main, whose price table
(#218, #250) now holds OpenRouter's list prices of 2026-10-31. Only cost
values change in the prediction files; accuracy is not re-graded. The
README rows of the 23 routers with prediction files are updated (Arena,
Cost/1K, optimality columns) and the table re-ranked. External baseline
rows keep their published numbers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@yl231
yl231 merged commit 44cc1ff into main Oct 31, 2026
10 checks passed
@yl231
yl231 deleted the leaderboard/reprice-2026-10-31 branch October 31, 2026 22:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant