[yl231]

Introduces model price profiles as the single source of truth for prices, makes price lookup exact, and charges models with no price profile the highest price in the table instead of $0. Prices are not synced to the market here; that is the next PR.

Update (Oct 31): merged with main, one entry per model

• Merge: brings in main, including the 31 price entries added by submissions since September (BizRouter, NuRoute, Divyam, DT-Router, Corealis, …).

• One profile per model. Every name a price can be looked up by is now an id or alias of exactly one profile. That covers the model_cost.json keys, the earlier profiles, and every model name billed in a prediction file, whether routed or served (model_used).

  • 144 names become 106 profiles. 75 of them are keyed by the model's OpenRouter id; the other spellings (provider prefix, date, effort suffix such as @high, provider_model) are aliases.
  • With exact matching, every row of every leaderboard router finds a price. Before, deepseek/deepseek-reasoner, anthropic/claude-sonnet-4 and Lynkr's openai/gpt-oss-120b were priced only by the substring fallback.

• Prices kept, not synced. Each profile keeps the price the evaluator charges most of that model's rows today. Spellings that had another price are listed in the profile's note. 8 table entries change:

  Entry	Was	Now	Why
  claude-haiku-4-5	0.80 / 4.00	1.00 / 5.00	same model as claude-haiku-4.5
  qwen3-235b-a22b-instruct-2507	0.50 / 2.00	0.071 / 0.10	same model as qwen/qwen3-235b-a22b-2507 (OrcaRouter's Alibaba spelling)
  qwen/qwen3-30b-a3b-instruct-2507	0.08 / 0.33	0.15 / 0.60	joins the Alibaba spelling, which bills more rows
  glm-5.3-flash, z-ai/glm-5.3-flash	0.15 / 0.50	0.041 / 0.077	same model as z-ai/glm-5.3-flash@default
  deepseek-v3.2, openai_gpt-oss-120b, qwen_qwen3-235b-a22b-2507		profile price	unused duplicates

  Re-costing the leaderboard with this table moves two routers: OrcaRouter 72.41 → 72.49 and Hybrid Router 72.15 → 72.13. The README isn't re-priced in this PR.

• No OpenRouter id:

  • models retired from OpenRouter, kept at their last listed price: grok-4-1-fast-reasoning, claude-sonnet-4, claude-3-haiku;
  • DeepSeek's API name deepseek-reasoner;
  • models never listed: agnes-2.0-flash, GLM-4-9B-0414, DeepSeek-R1-0528-Qwen3-8B, glm-4-air;
  • unused older entries.

• README: submitters add new models to model_profiles.yaml and regenerate model_cost.json.

What this PR changes overall

• model_cost/model_profiles.yaml (new): one profile per model with its price (USD per 1M input/output tokens), aliases, as_of, source and note.
• scripts/pricing/build_model_cost.py (new): generates model_cost.json from the profiles, so the evaluator, the /evaluate bot and other tools keep reading the same file. --check fails if the JSON was edited by hand. It's wired in as a pre-commit hook, so CI runs it.
• Evaluator (evaluate_models.py): price lookup is exact id or alias only, with no substring fallback. A model with no price profile is charged the table's highest input and output price. It's warned about once per name and summarized at the end of the run.
• run.py: a row whose model name isn't in universal_model_names.py is now graded and charged the maximum price, instead of being dropped from the results.
• Submission check: a model without a price profile is a warning, with the max-price consequence spelled out, not a failure.

Testing

Check	Result
Unit tests (tests/test_pricing.py, 7)	pass
pre-commit (ruff, format, codespell, yaml/json, mypy, model-cost-in-sync)	pass
Every model name billed by a leaderboard router resolves by exact match	0 unpriced rows
Re-cost all leaderboard routers with this table vs main's evaluator	2 routers move, as above (≤ 0.08)
Original (Sep 29): re-price all 265,568 rows, old vs new code	identical

🤖 Generated with Claude Code