feat(pricing): add generic selector pricing and Responses cache writes#178
feat(pricing): add generic selector pricing and Responses cache writes#178Menci wants to merge 47 commits into
Conversation
…tesian grid Reworks GPT-5.6 long-context pricing so the two orthogonal selectors — the upstream-stamped service tier and the per-request input-length band — form an explicit (service tier × input length) grid rather than a pair of overlays that silently compose. Protocol / translation (kept from the prior attempt): - Add optional-compatible input_tokens_details.cache_write_tokens to the Responses usage type. - Responses usage extractor splits input into disjoint bare / cache-read / cache-write buckets and rejects a malformed negative bare input rather than clamping. - Responses -> Messages usage (streaming message_start + terminal result) maps cache writes onto cache_creation_input_tokens. Pricing model: - ModelPricing is a grid: base cell (default tier, base length), `tiers` for service-tier cells at base length, and `inputLengthTiers` bands keyed by `aboveInputTokens` (strictly-greater selection; 272000 stays short). Each band carries its own `tiers` for explicit (service tier × input length) cells. - resolveEffectivePricing resolves ONE cell. When both selectors are non-default it uses only the explicit combined cell; a missing combination returns null (unpriced) — no silent single-axis win, no auto-multiply. - selectInputLengthTier picks the band coordinate per request from the disjoint input total; recordTokenUsage derives it at the persistence boundary. - The whole request is charged at the selected cell's rates, not a marginal overage. input_above_tokens is a price selector, not a token bucket. Persistence: - usage rows carry an input_above_tokens coordinate (migration 0051, CHECK > 0, folded into the bucket unique key). SQL + memory repos, aggregation, and data-transfer import/export thread it through; import rejects 0 / negative / fractional coordinates. Tables: - Copilot GPT-5.6 sol/terra/luna: standard short/long grid (no priority lane), cache-write at 1.25x input. - Codex GPT-5.6 sol/terra/luna: full short/long x standard/priority grid, every cell with an explicit cache-write. priority-long cache-write is derived from the family-wide cache_write = 1.25x input ratio (OpenAI publishes no standalone priority-long cache-write rate). Values cross-checked against the OpenAI pricing archive, models.dev, LiteLLM, the Codex models.json, and caozhiyuan/copilot-api. Docs: fetching-models-pricing and backfill-model-pricing describe the grid, the selector-not-bucket semantics, and the new coordinate column.
|
Claude Code agent update: this PR is paused for a broader architecture comparison. The strict |
|
Claude Code agent architecture result after comparing LiteLLM, models.dev, Portkey, Helicone, Langfuse, OpenRouter, and Cloudflare's public contract:
The current nested The review also found that the new coordinate changes backup identity, so the export/import format will be bumped to version 9 rather than silently extending version 8. |
…93876eafc9e # Conflicts: # packages/gateway/src/control-plane/data-transfer/routes.ts # packages/gateway/src/repo/sql.ts
What
Add standard Responses cache-write accounting and a generic pricing-selector mechanism used for context-length and service-tier prices.
usage.input_tokens_details.cache_write_tokens, split disjoint ordinary/cache-read/cache-write usage with one shared invariant checker, and throw on malformed overlapping totalscache_creation_input_tokensin streaming and non-streaming translations@floway-dev/protocols/commonserviceTieropen-string equality andinputTokensgt/gtethreshold axespricingCell/modelPricing/basePricinghelpersCanonical selector identity is sorted-key JSON; threshold objects serialize as
{ "operator", "value" }, and the base coordinate is{}.Persistence and transfer
Migration
0052_usage_pricing_selector.sqlreplaces per-axis usage columns with one JSON-object-validatedpricing_selectorcolumn. Existing nonblank tiers migrate through SQLitejson_object; NULL/blank tiers become{}. SQL and memory repos canonicalize typed selector identity.Data-transfer format v10 exports the self-describing
pricingSelector; imports validate selectors and non-negative rates and reject historical versions.Current threshold prices
All thresholds reprice the whole request.
inputTokens > 272000long; Codex also has priority-short. Priority-long is intentionally absent/unpriced.>272000=10 / 1 / 45input/cache-read/output.>272000=5 / .5 / 22.5.>200000=4 / .4 / 18; Gemini 2.5 Pro remains base-only because the current Copilot catalog caps it below the threshold..3 / .06 / 1.2;>512000=.6 / .12 / 2.4.Authoring UI
The model editor iterates exported axis metadata. Equality axes render free text; threshold axes render operator + integer controls. Catalog-level validation errors render once above the cell list, while invalid in-progress cells remain safe to edit. Threshold clearing, row changes, and manual/auto mode switches are covered.
Custom upstream catalog pricing remains best-effort: malformed cost metadata is omitted without dropping the model.
Test plan
pnpm run typecheckpnpm run lintpnpm run test— 337 files / 4084 tests passgt/gteboundaries, highest-match projection