Skip to content

fix[models]: support cache_system_prompt parameter for model configu… - #41

Open
rishu685 wants to merge 1 commit into
extra-org:mainfrom
rishu685:feature/issue-31-prompt-caching
Open

fix[models]: support cache_system_prompt parameter for model configu…#41
rishu685 wants to merge 1 commit into
extra-org:mainfrom
rishu685:feature/issue-31-prompt-caching

Conversation

@rishu685

Copy link
Copy Markdown
Contributor

This PR addresses #31 by introducing a new cache_system_prompt field to the model configuration (ModelConfig). This field defaults to True and controls whether the system prompt and tool definitions are cached on supported providers (specifically Anthropic).

Scope & Implementation Details

  1. Spec & Dataclass: Added cache_system_prompt: bool = True to the BaseModelConfig dataclass in src/agent_engine/core/spec.py.
  2. YAML Validation & Parser:
    • Updated _validate_model in src/agent_engine/parsers/yaml/parser.py to ensure cache_system_prompt is validated as a boolean.
    • Updated _build_model and _build_base_model to correctly parse cache_system_prompt from the YAML definitions, defaulting to True.
  3. Model Factory:
    • Added cache_system_prompt parameter to build_chat_model in src/agent_engine/models/factory.py.
    • Updated _build_anthropic_model to conditionally pass cache_control={"type": "ephemeral"} parameter via init_chat_model() kwargs when cache_system_prompt is True.
  4. Engine Configuration:
    • Updated _model_factory_kwargs in src/agent_engine/engine/langgraph/engine.py to forward cache_system_prompt from node configurations to the model factory.
  5. Formatting & Quality Fixes:
    • Fixed some pre-existing long lines (E501) and sorted imports in nodes.py and engine.py.

Verification

  • Added YAML configuration parser tests under tests/parsers/test_cache_system_prompt.py.
  • Added model factory test coverage under tests/models/test_factory.py.
  • Verified that the full test suite (make check) passes cleanly:
    • ruff check src tests: Passing
    • mypy src tests: Passing
    • agentctl validate examples/enterprise-knowledge-assistant/agents.yaml: Passing
    • pytest: Passing (541 tests passed)

@AmitAvital1

Copy link
Copy Markdown
Collaborator

I understand this commit is specifically for anthropic, what about other provider? we will do on other commits? or you think its harder?
what about enforce it only for anthropic?

@rishu685

rishu685 commented Jul 24, 2026

Copy link
Copy Markdown
Contributor Author

See what i think is prompt caching is handled differently across providers:

--Anthropic is the only one requiring this explicit client-side flag (cache_control) here.
--OpenAI caches automatically behind the scenes (for >1024 tokens) without any parameter so i don't think we should enforce it here.
--Gemini and Bedrock don't support simple pass-through caching flags at initialization.
Because of this, the parameter is ignored for other providers. We chose not to fail validation for non-Anthropic models so developers don't have to keep adding or removing this line when switching providers in their YAML.

Let me know if you'd prefer us to strictly enforce this flag only for Anthropic, or make the change for others!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants