[yl231]

A tool to re-price routers at the current price table without re-grading. Use it after a price sync, to keep the leaderboard at current prices, or on a new submission whose file was graded with an older table. Builds on #218, #250 and #251 (merged).

What it does

scripts/pricing/reprice.py re-costs every row the evaluator costed, from the token counts already in the prediction file:

• Same cost code as the evaluator: it uses compute_row_cost, which covers the answer plus the router's own model calls, billed at the model that actually answered when it has a price.
• Same metric code: it then recomputes the metrics with the evaluator's own router_metrics.
• Offline and safe: no answer is graded and no submitted code runs, so it can run anywhere in a few minutes.

python scripts/pricing/reprice.py ROUTER [ROUTER ...]   # report only, e.g. a new submission
python scripts/pricing/reprice.py --all                   # every router in the manifest
python scripts/pricing/reprice.py --all --write           # rewrite the files' costs and the README rows, re-rank
python scripts/pricing/reprice.py --all --report out.md   # save the before/after table

With --write:

• Each prediction file keeps its JSON formatting, so the diff shows only changed cost values.
• The README rows of the re-priced routers are updated and the table is re-ranked: Arena, Accuracy, Cost/1K, and the three optimality columns.
• README rows without a prediction file (external baselines) keep their numbers.

run.py: the metric computation moves out of compute_router_metrics into router_metrics(), which returns the metrics.json dict. compute_router_metrics logs and writes metrics.json as before.

Testing

Check	Result
tests/test_reprice.py (14): current prices, uncosted rows, served model, router calls, file formatting, README update and re-rank, router_metrics vs metrics.json	pass, and all 39 pricing tests pass
pre-commit, including mypy	pass
All 23 leaderboard routers, before re-pricing	Arena, cost and the optimality columns equal the README for every router
Re-priced with #218's table	only OrcaRouter (+0.08) and Hybrid Router (−0.02) move, as an independent check found
Re-priced with #250's synced table	matches the independent preview for every router, e.g. AsiaInfo 66.30 → 70.00, Weave 72.88 → 70.00
--write	prediction files change only on cost lines (88,086 lines, one for one); README: 22 rows updated and re-ranked

Note: re-pricing also moves the optimality columns. Opt.Sel and Opt.Cost compare a router's pick with the cheapest correct candidate, so new prices can change which candidate that is. With the synced prices, Hybrid Router's Opt.Sel goes from 89.87 to 16.67 (qwen3-235b is no longer the cheapest) and Cross-Router's from 17.24 to 45.73. Opt.Acc doesn't depend on price.

🤖 Generated with Claude Code