Skip to content

feat(pricing): add generic selector pricing and Responses cache writes#178

Open
Menci wants to merge 47 commits into
mainfrom
worktree-agent-a5434393876eafc9e
Open

feat(pricing): add generic selector pricing and Responses cache writes#178
Menci wants to merge 47 commits into
mainfrom
worktree-agent-a5434393876eafc9e

Conversation

@Menci

@Menci Menci commented Jul 11, 2026

Copy link
Copy Markdown
Owner

What

Add standard Responses cache-write accounting and a generic pricing-selector mechanism used for context-length and service-tier prices.

  • model usage.input_tokens_details.cache_write_tokens, split disjoint ordinary/cache-read/cache-write usage with one shared invariant checker, and throw on malformed overlapping totals
  • map Responses cache writes to Messages cache_creation_input_tokens in streaming and non-streaming translations
  • define pricing as flat explicit cells with a generic selector map
  • centralize selector axes, price-vector validation, canonicalization, catalog compilation, authoring helpers, runtime projection, and exact lookup in @floway-dev/protocols/common
  • ship serviceTier open-string equality and inputTokens gt/gte threshold axes
  • require threshold bands to be declared without equality coordinates before service-specific combinations can reference them
  • project the highest matching global threshold, then exact-match the canonical Cartesian coordinate
  • leave missing coordinates unpriced by design; there is no cross-cell fallback, precedence, inheritance, or multiplication
  • compile stable pricing catalogs once with a WeakMap; static provider tables eagerly validate through shared pricingCell / modelPricing / basePricing helpers

Canonical selector identity is sorted-key JSON; threshold objects serialize as { "operator", "value" }, and the base coordinate is {}.

Persistence and transfer

Migration 0052_usage_pricing_selector.sql replaces per-axis usage columns with one JSON-object-validated pricing_selector column. Existing nonblank tiers migrate through SQLite json_object; NULL/blank tiers become {}. SQL and memory repos canonicalize typed selector identity.

Data-transfer format v10 exports the self-describing pricingSelector; imports validate selectors and non-negative rates and reject historical versions.

Current threshold prices

All thresholds reprice the whole request.

  • GPT-5.6 Sol / Terra / Luna: standard short and inputTokens > 272000 long; Codex also has priority-short. Priority-long is intentionally absent/unpriced.
  • Copilot GPT-5.5: >272000 = 10 / 1 / 45 input/cache-read/output.
  • Copilot and Codex GPT-5.4: >272000 = 5 / .5 / 22.5.
  • Copilot Gemini 3.1 Pro Preview: >200000 = 4 / .4 / 18; Gemini 2.5 Pro remains base-only because the current Copilot catalog caps it below the threshold.
  • Ollama MiniMax M3: base .3 / .06 / 1.2; >512000 = .6 / .12 / 2.4.

Authoring UI

The model editor iterates exported axis metadata. Equality axes render free text; threshold axes render operator + integer controls. Catalog-level validation errors render once above the cell list, while invalid in-progress cells remain safe to edit. Threshold clearing, row changes, and manual/auto mode switches are covered.

Custom upstream catalog pricing remains best-effort: malformed cost metadata is omitted without dropping the model.

Test plan

  • pnpm run typecheck
  • pnpm run lint
  • pnpm run test — 337 files / 4084 tests pass
  • canonical identity, selector/rate validation, duplicate/conflicting coordinates
  • global threshold-band invariant, gt/gte boundaries, highest-match projection
  • exact-missing combinations and unknown runtime tiers remain unpriced
  • shared provider authoring helpers eagerly validate static catalogs
  • SQL/memory selector identity and 0052 blank/special-character migration
  • v10 export/import and historical-version rejection
  • generic Vue controls, threshold clear/invalid drafts, one-time error rendering, row/mode resync
  • shared Responses cache-write splitting and malformed-usage exceptions
  • malformed custom catalog cost omission

Menci added 4 commits July 12, 2026 05:28
…tesian grid

Reworks GPT-5.6 long-context pricing so the two orthogonal selectors — the
upstream-stamped service tier and the per-request input-length band — form an
explicit (service tier × input length) grid rather than a pair of overlays that
silently compose.

Protocol / translation (kept from the prior attempt):
- Add optional-compatible input_tokens_details.cache_write_tokens to the
  Responses usage type.
- Responses usage extractor splits input into disjoint bare / cache-read /
  cache-write buckets and rejects a malformed negative bare input rather than
  clamping.
- Responses -> Messages usage (streaming message_start + terminal result) maps
  cache writes onto cache_creation_input_tokens.

Pricing model:
- ModelPricing is a grid: base cell (default tier, base length), `tiers` for
  service-tier cells at base length, and `inputLengthTiers` bands keyed by
  `aboveInputTokens` (strictly-greater selection; 272000 stays short). Each band
  carries its own `tiers` for explicit (service tier × input length) cells.
- resolveEffectivePricing resolves ONE cell. When both selectors are non-default
  it uses only the explicit combined cell; a missing combination returns null
  (unpriced) — no silent single-axis win, no auto-multiply.
- selectInputLengthTier picks the band coordinate per request from the disjoint
  input total; recordTokenUsage derives it at the persistence boundary.
- The whole request is charged at the selected cell's rates, not a marginal
  overage. input_above_tokens is a price selector, not a token bucket.

Persistence:
- usage rows carry an input_above_tokens coordinate (migration 0051, CHECK > 0,
  folded into the bucket unique key). SQL + memory repos, aggregation, and
  data-transfer import/export thread it through; import rejects 0 / negative /
  fractional coordinates.

Tables:
- Copilot GPT-5.6 sol/terra/luna: standard short/long grid (no priority lane),
  cache-write at 1.25x input.
- Codex GPT-5.6 sol/terra/luna: full short/long x standard/priority grid, every
  cell with an explicit cache-write. priority-long cache-write is derived from
  the family-wide cache_write = 1.25x input ratio (OpenAI publishes no standalone
  priority-long cache-write rate). Values cross-checked against the OpenAI pricing
  archive, models.dev, LiteLLM, the Codex models.json, and caozhiyuan/copilot-api.

Docs: fetching-models-pricing and backfill-model-pricing describe the grid, the
selector-not-bucket semantics, and the new coordinate column.
@Menci

Menci commented Jul 11, 2026

Copy link
Copy Markdown
Owner Author

Claude Code agent update: this PR is paused for a broader architecture comparison. The strict >272k whole-request math and explicit priority×context cells are verified, but I am now comparing how LiteLLM, models.dev, Portkey, Helicone, OpenRouter, Cloudflare and other gateways represent context-length pricing alongside service tiers. I will not treat the current inputLengthTiers/input_above_tokens design as final until that multi-implementation review is complete.

@Menci Menci changed the title feat(pricing): model service tier and input length as a price grid feat(pricing): account for Responses cache writes and context price bands Jul 11, 2026
@Menci

Menci commented Jul 11, 2026

Copy link
Copy Markdown
Owner Author

Claude Code agent architecture result after comparing LiteLLM, models.dev, Portkey, Helicone, Langfuse, OpenRouter, and Cloudflare's public contract:

  • context length is a pricing selector independent of wire service_tier; it is not another token billing dimension and not a subtype of service tier
  • crossed thresholds select a whole-request price, not marginal overage
  • selector combinations are often sparse explicit cells; missing cells must not be auto-multiplied or silently inherit an unrelated axis
  • persistence should retain tier and input_above_tokens as separate bucket coordinates

The current nested inputLengthTiers[].tiers shape still privileges one axis. I am refactoring this PR to a symmetric flat list of explicit selector coordinates and complete price vectors. Empty selector is the base cell; exact missing combinations remain unpriced. The existing migration coordinates and cache-write accounting stay.

The review also found that the new coordinate changes backup identity, so the export/import format will be bumped to version 9 rather than silently extending version 8.

@Menci Menci changed the title feat(pricing): account for Responses cache writes and context price bands feat(pricing): add explicit selector cells and Responses cache writes Jul 11, 2026
@Menci Menci changed the title feat(pricing): add explicit selector cells and Responses cache writes feat(pricing): add generic selector pricing and Responses cache writes Jul 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant