Provider-agnostic prompt caching in Strands (Bedrock Converse vs Mantle vs OpenAI/Gemini)

[philmerrell]

Summary

Prompt caching is currently a single boolean (caching_enabled) that only affects the Bedrock Converse path. Every other provider/model surface we support — Bedrock Mantle (gpt-5.x Responses, grok, gemma), direct OpenAI, Gemini — silently ignores it, even though those endpoints do support caching via their own mechanisms. As we broaden the model catalog beyond Converse-Claude, we need a provider-agnostic caching strategy rather than a Bedrock-only flag, or we leave cost savings on the table and report costs inconsistently.

This issue asks us to design that abstraction, not just patch one path.

Current state

ModelConfig.caching_enabled is threaded end-to-end (routes → service → base_agent → ModelConfig), but it is only consumed in one place:

• to_bedrock_config() — sets cache_config=CacheConfig(strategy="auto") on BedrockModel, which injects cachePoint blocks (Converse). ✅
• to_mantle_config() — ignores caching_enabled entirely. ❌
• to_openai_config() — ignores it. ❌
• to_gemini_config() — ignores it. ❌

(backend/src/agents/main_agent/core/model_config.py, applied per-branch in agent_factory.py.)

Meanwhile the mechanism differs per surface:

Surface	Strands class	Caching mechanism	Wired today?	Usage surfaced by Strands
Bedrock Converse (Claude/Nova)	BedrockModel	CacheConfig(strategy="auto") → cachePoint blocks (auto placement on system/tools/last-user)	✅	read and write tokens (cacheReadInputTokens / cacheWriteInputTokens)
Bedrock Mantle — gpt-5.x (Responses)	OpenAIResponsesModel	Explicit opt-in: prompt_cache_key + prompt_cache_retention request params (in_memory for 5.4, 24h for 5.5+). Not automatic, not cache_control.	❌	read only (cached_tokens → cacheReadInputTokens); no write tokens mapped
Bedrock Mantle — grok / gemma (Chat Completions)	OpenAIModel	prompt_cache_key (Chat Completions); retention behavior TBD	❌	read only
Direct OpenAI	OpenAIModel	Automatic prefix caching (≥~1024 tok) + optional prompt_cache_key	❌	read only
Gemini	GeminiModel	Implicit + explicit context caching (CachedContent API) — entirely different model	❌	TBD

Two consequences:

1. Cost left on the table. With no prompt_cache_key emitted, Mantle caching is opt-in and never triggers, so gpt-5.x/grok/gemma conversations pay full input price every turn.
2. Cost/badge undercounting. Our cost calculator already models cache_read_cost and cache_write_cost (apis/app_api/costs/), but Strands' OpenAI classes only map cacheReadInputTokens — cache-write tokens Mantle bills for are invisible on non-Converse paths. So even the read discount that could appear won't be balanced by the write cost.

Key SDK facts (strands-agents 1.47.0, verified against vendored source)

• CacheConfig / cachePoint injection lives only in strands/models/bedrock.py. OpenAIModel / OpenAIResponsesModel expose no cache_config param (confirmed in source and API docs).
• The OpenAI/Responses classes do spread params straight into the API call — openai_responses.py:540 (**self.config.get("params", {})) → responses.create(**request) at :314. So prompt_cache_key / prompt_cache_retention can be injected via params without forking Strands.
• Both OpenAI classes already read back cached_tokens → cacheReadInputTokens (openai.py:601, openai_responses.py:854), but neither maps a cache-write field.

Design question (the wider lens)

How should prompt caching be modeled so that:

1. One user/admin-facing intent (caching_enabled, or something richer) maps to the right mechanism per provider automatically — Converse CacheConfig, Mantle/OpenAI prompt_cache_key(+retention), Gemini CachedContent — instead of a Bedrock-only no-op elsewhere.
2. Per-model caching metadata (does it support caching at all? what retention values are legal? min prefix length?) lives next to the other per-model facts we already carry (mantle_api_mode, mantle_region).
3. Cache-key granularity is correct — prompt_cache_key is a routing/affinity hint; the hit is still prefix-based, so the key must be stable across a conversation/assistant (derived from system prompt + tool set), never per-request-random.
4. Usage accounting is symmetric — read and write cache tokens flow into the cost/context badge on every provider, or we explicitly document where a provider can't report writes.

Proposed direction (for discussion, not final)

• Introduce a small caching-strategy resolver keyed on provider/model that each to_*_config() delegates to, rather than special-casing Converse. Output is provider-native (a CacheConfig for Bedrock; {prompt_cache_key, prompt_cache_retention} merged into params for Mantle/OpenAI; a CachedContent handle or no-op for Gemini).
• Add per-model caching fields (e.g. cache_supported, cache_retention) to the model catalog alongside mantle_api_mode.
• Derive prompt_cache_key deterministically from the stable prefix (assistant id + system prompt hash + tool-set hash).
• Backfill a cacheWriteInputTokens mapping on the OpenAI/Responses usage path (local patch and/or upstream Strands issue) so the badge stops undercounting.
• Keep the existing caching_enabled kill-switch semantics (default-on with opt-out) but make it provider-aware.

Acceptance criteria

• caching_enabled (or its successor) produces caching on all supported provider paths, or is explicitly, per-model documented as unsupported.
• gpt-5.x-via-Mantle conversations show non-zero cacheReadInputTokens on repeat turns.
• Cost badge reflects both cache read and cache write for every provider that reports them.
• Per-model caching capability + retention lives in the catalog, not hardcoded.
• No Strands fork required for the Mantle/OpenAI path (inject via params); any upstream gap (write-token mapping) filed separately.

Out of scope / open questions

• Gemini CachedContent (explicit cache lifecycle) may warrant its own follow-up — it's a create/delete resource, not a per-request flag.
• Whether to expose caching controls to users/admins per-assistant vs. a global platform default.
• Chat Completions (grok/gemma) retention semantics need confirmation before enabling.

References

• backend/src/agents/main_agent/core/model_config.py (to_bedrock_config / to_mantle_config)
• backend/src/apis/shared/models/mantle.py (build_mantle_model)
• backend/src/apis/app_api/costs/ (cost calculator already models cache read + write)
• OpenAI models on Amazon Bedrock — caching
• Strands OpenAI model API
• Bedrock prompt caching
• Upstream context: strands-agents/sdk-python #1508 (PromptCachingHook), #807 (granular system-prompt cache points)