Skip to content

fix(cache): deterministic Responses ids, stable session anchor, structural Anthropic breakpoints - #251

Merged
ltmoerdani merged 1 commit into
ltmoerdani:mainfrom
lecommander:feature/prompt-cache-stability
Oct 13, 2026
Merged

ltmoerdani merged 1 commit into
ltmoerdani:mainfrom
lecommander:feature/prompt-cache-stability

Conversation

@lecommander

Copy link
Copy Markdown
Contributor

Summary
Reduce provider prompt-cache misses caused by client-side prefix instability. Three independent fixes, each verified by the existing suite plus two new regression tests. No behavior change for cache-ineligible routes.

Changes

  • Responses item ids are now deterministic (P0a): responsesFunctionCallItemId maps non-fc ids to fc + sha256(call id, 32 hex) instead of randomUUID per request, fc ids pass through verbatim. Same history now replays byte-identically every turn. Previously the prefix cache broke at the first function_call item on every tool turn (same class as OpenHands SDK 2905).
  • Stable session anchor (P0b): the x-opencode-session fallback no longer hashes the first 3 messages (messages 1-2 shift on every history trim by design). It now hashes first message + sorted tool names + model id, memoized per conversation for its lifetime. Legacy conversationAnchor kept for compatibility.
  • Structural Anthropic breakpoints (P1): buildAnthropicMessages placed cache_control on the first 4 content blocks in emit order, so any added text/image part shifted all later marks and consumed the 4-slot budget gateways also inject into. Now at most 2 marks at structural boundaries: end of message 0 (static prefix) and end of message N-2 (tail checkpoint); only the last block of a marked message carries the mark.

Before / After

  • Before: Responses replay emitted a fresh fc id per request, session id re-hashed over shifting messages, Anthropic marks drifted with content growth.
  • After: replay payloads are deterministic, session header is constant across trims, Anthropic marks sit on stable boundaries. DeepSeek-class implicit routes get stability with zero new markers.

Verification

  • npx tsc -p ./ clean
  • Targeted: 24/24 pass (responsesRequest, session-header, headers)
  • Full npm test: 497/497 pass
  • Live gate to run after install: 30-turn run, prefix hash and session constant, cacheHit at 98 percent plus after turn 2, exactly one dip across a trim event.

Notes

  • Explicit markers remain gated to Anthropic-native and GPT-5.6-family routes by follow-up work; DeepSeek/Qwen/Kimi/GLM/MiniMax/MiMo/Muse stay marker-free (implicit only).
  • No keepalive, no previous_response_id chaining, no 1h TTL stamp in this PR.
  • Draft status: asking for direction on breakpoint policy and per-route gating before packaging a vsix.

@ltmoerdani
ltmoerdani marked this pull request as ready for review October 13, 2026 07:05
@ltmoerdani

Copy link
Copy Markdown
Owner

@lecommander Tested this locally against latest main, all green: tsc clean, 497/497 tests pass, full lint gate passed. Merge state is clean and the Responses identity work from #246 doesn't conflict with the deterministic fc id change, which was my main concern going in.

Nice one on separating the item id namespacing from the call_id correlation, that was the part I was most worried about breaking.

Two small notes, neither blocking:

  1. responsesFunctionCallItemId("") returns the same hash for any empty input. If a history ever contains two tool calls with empty/identical ids, they'd now map to the same fc_ id where randomUUID used to guarantee uniqueness, and the API could reject it as a duplicate. Providers always issue real ids so this is very unlikely, but a tiny guard (append a deterministic counter on duplicate inputs) would close it. Fine as a follow-up if you'd rather keep this PR tight.

  2. The new tests cover the Responses side. The session anchor memo and the Anthropic 4-marks-to-2 change have no direct unit tests yet. Existing suite passes so I'm comfortable, but a trim-stability test for stableConversationAnchorKey would be a good follow-up.

On your open question about breakpoint policy: I'm good with implicit-only for now. DeepSeek/Qwen/Kimi/GLM etc. staying marker-free matches how they actually cache, and explicit markers for Anthropic-native + GPT-5.6 can land as follow-up work with its own live verification.

I'll mark this ready for review and approve so we can get it in. Plan is a regular merge commit to keep your commit history intact.

@ltmoerdani
ltmoerdani merged commit 0ad1b49 into ltmoerdani:main Oct 13, 2026
2 checks passed
ltmoerdani added a commit that referenced this pull request Oct 13, 2026
- CHANGELOG: date 0.7.9 + prompt-cache stability entry (PR #251 by @lecommander)
- package.json/lock: version 0.7.8 -> 0.7.9
- README: add prompt-cache stability bullet to Smart Routing & Reliability
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants