Skip to content

Pricing: sync model prices with OpenRouter (manual), applied with the 2026-10-31 list - #250

Merged
yl231 merged 4 commits into
mainfrom
pricing/openrouter-sync
Oct 31, 2026
Merged

yl231 merged 4 commits into
mainfrom
pricing/openrouter-sync

Conversation

@yl231

@yl231 yl231 commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Adds the manual price sync with OpenRouter, and applies it once with OpenRouter's list prices of 2026-10-31. Based on #218, which needs to merge first; then this PR is retargeted to main.

How the manual path works

  1. Sync prices: python scripts/pricing/sync_openrouter.py downloads OpenRouter's model list. Every profile in model_cost/model_profiles.yaml keyed by an OpenRouter id, or with an openrouter_id, takes OpenRouter's list price (never the :batch or :free tiers). The prices used are recorded in model_cost/price_snapshots/<date>.json, and model_cost.json is regenerated. --dry-run and --report print the changes without writing them.
  2. Re-price the leaderboard: a separate step, done with the re-pricing tool (next PR). Syncing alone changes no score.

Pinned prices: a profile with sync: false keeps its price. That applies to models retired from OpenRouter and to models it never listed.

This run (2026-10-31)

30 prices change. These move the leaderboard most:

Model Was OpenRouter Routers most affected
anthropic/claude-opus-4.7 15 / 75 5 / 25 AsiaInfo
deepseek/deepseek-v4-pro 0.435 / 0.87 (launch discount) 1.6 / 3.2 Weave, vLLM‑SR, Cross-Router
qwen/qwen3-235b-a22b-2507 0.071 / 0.10 0.09 / 0.55 Hybrid, cruq, Auto Router
deepseek/deepseek-v4-flash 0.14 / 0.28 0.091 / 0.182 several, cheaper
google/gemma-4-31b-it 0.08 / 0.35 0.2 / 0.4 KT-ModelRouter

The full list is in the commit and the snapshot file.

Pinned (sync: false):

  • grok-4-1-fast-reasoning, claude-sonnet-4 and claude-3-haiku are retired from OpenRouter and kept at their last listed price.
  • deepseek-reasoner is DeepSeek's API name, and the model behind it changed over time.
  • agnes-2.0-flash, GLM-4-9B-0414, DeepSeek-R1-0528-Qwen3-8B and glm-4-air were never listed on OpenRouter.

Unmatched: 23 older profiles that no leaderboard router uses match no OpenRouter model. The sync lists them each run and leaves their price.

Effect on the leaderboard once re-priced: Arena score, now → re-priced at these prices.

Router Arena
AsiaInfo 66.30 → 70.00
Weave 72.88 → 70.00
Hybrid Router 72.15 → 71.20
cruq-router 70.82 → 69.89
vLLM‑SR 74.96 → 74.24
Cross-Router 75.87 → 75.25
BARouter 74.22 → 73.67
Auto Router 70.27 → 69.80
KT-ModelRouter 76.37 → 76.08
others within ±0.2

Not in this PR

  • The daily or every-other-day automatic sync. It comes later, once this manual path is settled; this branch's earlier draft of the workflow is removed for now.
  • Local-model prices.

Testing

  • tests/test_price_sync.py (6) and tests/test_pricing.py (7) pass.
  • pre-commit passes, including the check that model_cost.json matches the profiles.
  • Re-costing every leaderboard router with the synced table gives the effect above, with no unpriced rows.

🤖 Generated with Claude Code

yl231 and others added 4 commits October 29, 2026 17:46
scripts/pricing/sync_openrouter.py sets each price profile that resolves to
an OpenRouter model (through openrouter_id, or an exact id/alias match) to
OpenRouter's current list price, updating as_of and source in place and
regenerating model_cost.json. Profiles with sync: false keep their price.
When a price changes, the run also writes model_cost/price_snapshots/<date>.json.

The Price sync workflow runs it daily and proposes the result as one pull
request (branch price-sync/auto, updated in place). It changes the price
table only; leaderboard costs change when the routers are re-priced.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The manual path comes first: a maintainer runs sync_openrouter.py and
re-prices the leaderboard as separate steps. The scheduled workflow will
be added once that path is settled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ran scripts/pricing/sync_openrouter.py against OpenRouter's model list of
2026-10-31. 30 profiles keyed by an OpenRouter id take its list price; the
prices used are recorded in model_cost/price_snapshots/2026-10-31.json.

Eight profiles are pinned with sync: false and keep their price: models
retired from OpenRouter (grok-4-1-fast-reasoning, claude-sonnet-4,
claude-3-haiku), DeepSeek's API name deepseek-reasoner, and models never
listed there (agnes-2.0-flash, GLM-4-9B-0414, DeepSeek-R1-0528-Qwen3-8B,
glm-4-air). 23 older profiles that no leaderboard router uses match no
OpenRouter model; the sync lists them and leaves their price.

The leaderboard is not re-priced here.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@yl231 yl231 changed the title Pricing: daily sync of model prices with OpenRouter Pricing: sync model prices with OpenRouter (manual), applied with the 2026-10-31 list Oct 31, 2026
@yl231
yl231 marked this pull request as ready for review October 31, 2026 22:07
@yl231
yl231 changed the base branch from pricing/model-profiles to main October 31, 2026 22:15
@yl231
yl231 merged commit e4d81a6 into main Oct 31, 2026
10 checks passed
@yl231
yl231 deleted the pricing/openrouter-sync branch October 31, 2026 22:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant