Volatile context causes repeated prompt cache writes

[sentry-junior]

Summary

Short and multi-turn conversations incurred large prompt-cache write costs even when the visible request and reply were only about 10 tokens. A verified production call spent $0.3186 of $0.3543 model cost on cache writes, so cache creation accounted for about 90% of that call's model cost.

Impact

• Simple conversations paid to create thousands of cache tokens that had little chance of reuse.
• Across conversation turns, the reusable durable history prefix was disrupted by transient context. This caused repeated cache writes as history grew.
• The cumulative prompt-cache work could grow approximately quadratically with the number of similar-sized turns. The static system and tool prefix still received cache hits, and tool-loop calls within one Turn retained an exact prefix.

Root cause

Junior assembled the first model request for each Turn in this order:

system + durable conversation history + runtime-turn-context + current request

runtime-turn-context included the actor, configuration, runtime facts, installed skill metadata, active MCP catalogs, and tool guidance. Junior treated this as transient bootstrap data and removed it from durable conversation history before the next Turn.

The pi-ai Anthropic adapter defaults prompt caching to short retention. It adds cache_control to the last user message to cache conversation history. For Junior's request shape, that marker followed both runtime-turn-context and the current request. The provider therefore treated the full suffix through the current request as cacheable.

On the next Turn, Junior rebuilt a new transient context block instead of replaying the prior block. The new request no longer matched the previously cached prompt after the stable system prefix. The provider read the stable prefix from cache, then wrote the changed suffix again. This was an interaction between Junior's prompt layout and Pi's automatic marker placement; Junior did not directly issue cache-write operations.

Verified evidence

On September 11, 2026, Sentry span a2095ac46e2c07f5 in project 4510944073809921 reported:

• model: openai/gpt-6-astra using the anthropic-messages adapter
• total input tokens: 12,012
• cache-read input tokens: 8,281
• cache-write input tokens: 3,728
• ordinary uncached input tokens: about 3
• output tokens: 10

The recorded cost breakdown for the example conversation was:

• model total: $0.3543
• input: $0.0001
• output: $0.0022
• cache read: $0.0334
• cache write: $0.3186

Repository inspection confirmed:

• packages/junior/src/chat/agent/prompt.ts appended bootstrap context before the current request.
• packages/junior/src/chat/pi/transcript.ts removed runtime-turn-context before durable history reuse.
• Pi's Anthropic adapter placed cache_control on the last user message.

Reproduction

1. Start a new Junior conversation with a short request.
2. Inspect the main gen_ai.chat span for an Anthropic-format model.
3. Observe a stable cache read for the system/tool prefix and a several-thousand-token cache write for the transient suffix.
4. Send another short request in the same conversation.
5. Observe another large cache write because the prior transient context was removed and a new context block was inserted before the request.

Resolution in progress

PR #1838 moves the mostly stable capability catalog into the system-prefix area and moves the first-call cache boundary before volatile runtime context and the current request. Later tool-loop calls keep Pi's normal durable-history caching behavior.

Validation

The PR adds focused coverage for marker placement and capability prompt layout. Unit and component tests, type checking, lint, dependency checks, and the full GitHub Actions suite pass on commit 635cd86136efaa2072ccd2f19f52c4ef0fa5dbec.

Remaining verification

After deployment, compare cache-read and cache-write token counts for short one-turn and multi-turn conversations. Cache writes should exclude volatile runtime context and the current request on the first model call of each Turn.

via David Cramer.

--

View Junior Session [Sentry]