docs(adaptive): document opt-in tool result caching

crates/adaptive/README.md

@@ -19,7 +19,7 @@ SPDX-License-Identifier: Apache-2.0
 `nemo-relay-adaptive` is the Rust companion crate for adaptive NeMo Relay
 runtime behavior. Use it with `nemo-relay` when an agent runtime should learn
 from observed executions, inject runtime hints, persist adaptive state, or
-cache repeated LLM responses.
+cache repeated LLM responses and classified tool results.
 
 Adaptive behavior is installed through the same plugin system used by the core
 runtime, so applications can enable it without changing their orchestration

docs/configure-plugins/adaptive/about.mdx

@@ -22,7 +22,7 @@ The plugin can coordinate:
 - Adaptive hints injected into outgoing model requests.
 - Tool-parallelism observation or scheduling behavior.
 - Adaptive Cache Governor (ACG) prompt-cache planning.
-- Opt-in response caching for repeated LLM calls.
+- Opt-in response caching for repeated LLM calls and explicitly classified tool results.
 - Component-local validation policy.
 
 ## Use Adaptive When
@@ -52,8 +52,9 @@ If instrumentation is not in place yet, start with
   cache planning accomplishes.
 - [Adaptive Hints](/configure-plugins/adaptive/adaptive-hints) explains request hint injection and how
   downstream model paths can consume the hints.
-- [Response Cache](/configure-plugins/adaptive/response-cache) explains the opt-in LLM response cache:
-  turning it on, what gets cached, and how savings are reported.
+- [Response Cache](/configure-plugins/adaptive/response-cache) explains the opt-in LLM response
+  and tool-result cache: turning it on, what gets cached, and how savings are
+  reported.
 
 State, telemetry, tool parallelism, and policy are whole-plugin configuration
 areas. They are documented on [Adaptive Configuration](/configure-plugins/adaptive/configuration) rather

docs/configure-plugins/adaptive/configuration.mdx

@@ -33,7 +33,7 @@ The top-level adaptive object contains:
 | `adaptive_hints` | Request hint-injection behavior. |
 | `tool_parallelism` | Tool scheduling observation or scheduling behavior. |
 | `acg` | Adaptive Cache Governor prompt-cache planning. |
-| `response_cache` | Opt-in LLM response cache for repeated managed calls. Requires a non-empty trust-domain `namespace`. |
+| `response_cache` | Opt-in cache for repeated LLM responses and classified tool results. |
 | `policy` | Adaptive-local handling for unknown fields and unsupported values. |
 
 Dedicated pages cover [Adaptive Cache Governor (ACG)](/configure-plugins/adaptive/acg),

docs/configure-plugins/adaptive/response-cache.mdx

@@ -1,27 +1,28 @@
 ---
 title: "Response Cache"
-description: "Configure exact-match response caching for managed LLM calls."
+description: "Configure exact-match response caching for managed LLM calls and tool results."
 position: 5
 ---
 {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
 SPDX-License-Identifier: Apache-2.0 */}
 
 
-Use the response cache when the same LLM request is made more than once and the
-repeat should be served from a store instead of calling the provider again. An
-eligible request that matches an unexpired cache entry can be served from the
-store without calling the provider. Buffered hits preserve the stored response
-shape and usage fields; streaming hits replay an equivalent provider-native
-stream.
+Use the response cache when a repeatable managed LLM request or explicitly
+classified tool call should be served from a store instead of running live.
+An eligible request or tool call that matches an unexpired cache entry can be
+served without calling the provider or tool. Buffered LLM hits preserve the
+stored response shape and usage fields; streaming hits replay an equivalent
+provider-native stream.
 
 The cache is an optional `response_cache` section of the
 [Adaptive plugin](/configure-plugins/adaptive/configuration), not a standalone plugin
-kind. It is off until the section is present, applies to
+kind. It is off until the section is present. Its LLM surface applies to
 [managed LLM calls](/instrument-applications/instrument-llm-call) without
-changing the execution API. By default, only requests with an explicit numeric
-`temperature = 0` are eligible; set `cache_nondeterministic = true` to opt
-sampled requests into caching. Runtime backend errors fail open to a normal
-live call, while invalid configuration is rejected during validation.
+changing the execution API; its tool-result surface is separately opt-in. By
+default, only requests with an explicit numeric `temperature = 0` are eligible;
+set `cache_nondeterministic = true` to opt sampled requests into caching.
+Runtime backend errors fail open to a normal live call, while invalid
+configuration is rejected during validation.
 
 `namespace` is required and defines one trusted cache-sharing domain. Do not
 use one namespace across mutually untrusted tenants or upstreams.
@@ -292,9 +293,9 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
 
 </Tabs>
 
-## What Gets Cached
+## LLM Responses
 
-Only complete, replayable answers are stored:
+Only complete, replayable LLM answers are stored:
 
 - A response with a non-null `error` or a `status` such as `failed`,
   `cancelled`, `incomplete`, or `in_progress` is never stored.
@@ -360,28 +361,110 @@ normalization, under the default `key_strategy = "exact_request"`:
 - Requests containing integers outside the exactly representable RFC 8785
   range (less than `-2^53` or greater than `2^53`) bypass the cache.
 
+## Tool-Result Cache
+
+The same `response_cache` section can also cache results from
+[managed tool calls](/instrument-applications/instrument-tool-call). This is a
+separate, opt-in surface: it shares the configured store and namespace with
+the LLM cache, but tool keys carry a distinct surface tag and cannot collide
+with LLM keys.
+
+Caching a tool call suppresses the real call. Cache only tools that are
+read-only and stable for their TTL; do not cache a tool merely because it is
+idempotent. A cache hit skips even an idempotent write. Classify each tool
+explicitly before enabling the tool surface:
+
+```toml
+[components.config.response_cache.tools]
+enabled = true
+cache_errors = false # default: do not store conventional in-band error results
+
+[components.config.response_cache.tools.classes.read_only]
+cacheable = true
+ttl_seconds = 300
+arg_skip = ["trace_id"]
+members = ["docs_*", "unit_convert"]
+
+[components.config.response_cache.tools.overrides.docs_search]
+tool_version = "v1" # identifies the deployed tool contract
+```
+
+An exact class member wins. Otherwise, Relay selects the most-specific matching
+`*` pattern; an exact override wins over a wildcard override. Configuration
+validation rejects overlapping wildcard class patterns or wildcard overrides
+that disagree on `cacheable`, so a broad deny policy cannot silently become a
+cacheable result. Unmatched tools use `tools.default`, which must remain
+uncacheable; classify every cacheable tool through a named class or override.
+
+### Tool Keys and Identity
+
+A tool key includes the namespace, tool name, optional `tool_version`,
+effective arguments, `arg_skip` policy, and `cache_errors` policy. It has no
+automatic tenant, scope, caller-identity, or request-header partition. In
+particular, `header_allowlist` applies only to LLM keys.
+
+Do not cache a tool whose result depends on tenant identity, caller identity,
+the active scope, permissions, ambient state, or another input absent from its
+arguments. If a cacheable tool needs a caller or tenant partition, add a trusted
+discriminator to the real arguments in a tool request interceptor before the
+cache runs. A scope-local execution interceptor is not a cache partition.
+
+`arg_skip` removes only top-level argument keys before keying. Skip only fields
+that cannot change the result, such as tracing metadata. The normalized
+`arg_skip` policy itself is part of key identity, so changing it starts a
+separate keyspace instead of reusing values created under a different policy.
+
+### Tool Errors and Middleware Order
+
+Tool callbacks that return an actual execution error are never stored. By
+default, Relay also does not store a JSON object with a non-null `error` field
+or `isError = true`; these are conventional in-band error signals rather than a
+universal tool-result schema. Set `tools.cache_errors = true` only when such
+results are stable and safe to reuse. With the default, a sampled refresh that
+returns one of these error-shaped values leaves a previously stored successful
+result in place.
+
+Tool conditional-execution guardrails and tool request interceptors run before
+the cache derives its key. Sanitize guardrails affect emitted observability
+payloads only; they do not change the real arguments, result, or cache key.
+`tools.priority` controls the tool execution interceptor: lower priorities run
+outermost. A hit returns before later execution interceptors and the managed
+callback. For example, the default tool cache priority (`150`) runs after a
+NeMo Guardrails execution interceptor at its default priority (`100`), so that
+interceptor runs on a hit. Give a rail a strictly lower priority than the cache
+if it must run on both hits and misses. In that arrangement, a miss is written
+before the outer rail transforms its result, so choose the ordering and
+stored-result semantics deliberately.
+
 ## Observability
 
 Every cache decision emits a `response_cache` mark with
 `data.status` set to one of:
 
 | Status | Meaning |
 |---|---|
-| `hit` | Served the stored answer; the provider was skipped. |
+| `hit` | Served the stored value; the provider or tool callback was skipped. |
 | `miss` | No entry was served; the call ran live. After an ordinary lookup miss, Relay attempts to store a cacheable result. |
 | `bypass` | The request is not cacheable, or the `bypass_rate` sampler chose to run live. |
 
 Mark attributes use `nemo_relay.response_cache.*`: `backend`, `surface`,
 `key_hash` (the `sha256:…` fingerprint), `ttl_ms`, and `age_ms` as applicable;
-`saved_tokens` and `saved_cost_usd` appear on hits when they can be derived. A
-`reason` appears on bypasses and store-error misses (for example `sampled`,
-`stateful_store`, `store_error`, or `stream_no_codec`). Cache marks never
-include prompts, answers, or credentials.
+`saved_tokens` and `saved_cost_usd` appear on LLM hits when they can be
+derived, while `saved_invocations` appears on tool hits. Tool marks use
+`surface = "tool"`; unclassified, uncacheable tools pass through without a
+cache mark. A `reason` appears on bypasses and store-error misses (for example
+`sampled`, `stateful_store`, `store_error`, or `stream_no_codec`). Cache marks
+never include prompts, answers, or credentials, but treat `key_hash` as
+sensitive telemetry: a hash can still reveal information when an observer can
+guess the keyed input.
 
 `nemo-relay doctor` reports the cache state: `not configured` when the section
 is absent, `configured but disabled (adaptive plugin disabled)` when the
 adaptive component is off, `on; backend '<kind>' reachable` when healthy, and
-a failure when the config is invalid or the backend is unreachable.
+a failure when the config is invalid or the backend is unreachable. When the
+tool surface is configured, `Response cache (tools)` reports `configured but
+disabled` when its switch is off. When it is on, the line reports the number of
+cacheable classes and cacheable overrides, plus the default policy.
 
 ## Fields
 
@@ -399,18 +482,33 @@ a failure when the config is invalid or the backend is unreachable.
 | `backend.config.url` | — | Redis connection URL. Required for the `redis` backend. |
 | `backend.config.key_prefix` | `"nemo-relay:llm-cache:"` | Prefix for keys in Redis. |
 
+### Tool Cache Fields
+
+The following fields configure the separately opt-in tool-result surface:
+
+| Field | Default | Notes |
+|---|---|---|
+| `tools.enabled` | `false` | Master switch. Classes and overrides are validated even when the surface is disabled. |
+| `tools.priority` | `150` | Tool execution-intercept priority. Lower values run earlier and outermost. |
+| `tools.cache_errors` | `false` | Store conventional in-band error objects only when `true`; callback errors are never stored. |
+| `tools.default` | uncacheable | Policy for tools that match no class. Keep it uncacheable; classify every cacheable tool through a named class or override. |
+| `tools.classes.<name>` | — | Named policy with `cacheable`, optional `ttl_seconds` and `bypass_rate`, top-level `arg_skip`, and `members` containing exact names or `*` patterns. Omitted TTL and bypass rate inherit the response-cache values. |
+| `tools.overrides.<name>` | — | Per-tool refinement after class resolution. `cacheable`, `ttl_seconds`, `bypass_rate`, and `tool_version` override when supplied; `arg_skip` replaces the class list, including when set to `[]`. |
+
 For a gateway that uses Switchyard, `switchyard.priority` must be lower than
 `response_cache.priority`. To derive keys before ACG rewrites requests, set
 `response_cache.priority` lower than `acg.priority`. With all three components,
 priorities of `0`, `40`, and `50`, respectively, satisfy both orderings.
 
 <Warning>
-Cached responses are stored unredacted. PII sanitize guardrails rewrite emitted
-telemetry, never payloads, so the store holds full response bodies. Cache
-entries can also store provider and model diagnostics plus the key fingerprint;
-they do not store full request bodies or headers. A shared Redis backend must be
-trusted and access-controlled. Use a separate configuration and namespace for
-each mutually untrusted tenant or upstream domain.
+Cached LLM responses and tool results are stored unredacted. PII sanitize
+guardrails rewrite emitted telemetry, never payloads, so the store holds full
+result bodies. Cache entries can also store provider and model diagnostics plus
+the key fingerprint; they do not store full LLM request bodies or headers. A
+shared Redis backend must be trusted and access-controlled. Use a separate
+configuration and namespace for each mutually untrusted tenant or upstream
+domain. `backend.config.max_bytes` limits only the in-memory backend; configure
+Redis capacity and eviction in Redis itself.
 </Warning>
 
 ## Common Validation Failures
@@ -426,3 +524,22 @@ each mutually untrusted tenant or upstream domain.
   unavailable because Relay was built without the `redis-backend` feature.
 - Gateway Switchyard priority is equal to or greater than
   `response_cache.priority`.
+- A tool policy sets `ttl_seconds = 0` or a `bypass_rate` outside `[0.0, 1.0]`,
+  or the same member name or pattern appears in two classes.
+- Overlapping wildcard class patterns or wildcard overrides disagree on
+  `cacheable`. A cacheable catch-all `*` member or override also warns; use
+  named classes to limit caching to explicitly read-only tools.
+
+Tool-cache diagnostics use the following codes. The catch-all diagnostics are
+warnings; the others are errors.
+
+| Diagnostic | Condition |
+|---|---|
+| `response_cache.tool_default_members` | `tools.default.members` is non-empty. The default bucket is not a matcher. |
+| `response_cache.tool_multiple_classes` | The same exact member or pattern appears in more than one named class. |
+| `response_cache.tool_invalid_ttl` | A class, default policy, or override sets `ttl_seconds = 0`. |
+| `response_cache.tool_invalid_bypass_rate` | A class, default policy, or override sets `bypass_rate` outside `[0.0, 1.0]`. |
+| `response_cache.tool_catch_all_member` | A cacheable class uses a catch-all `*` member. |
+| `response_cache.tool_catch_all_override` | A cacheable `*` override applies to every tool. |
+| `response_cache.tool_conflicting_classes` | Overlapping wildcard members in different classes have different `cacheable` values. |
+| `response_cache.tool_conflicting_overrides` | Overlapping wildcard overrides have different `cacheable` declarations. An omitted value inherits policy, so it cannot safely overlap an explicit value. |

docs/reference/migration-guides.mdx

@@ -13,6 +13,12 @@ intervening release in sequence.
 
 ## Upgrade to NeMo Relay 0.8
 
+### Update Exhaustive Cache Configuration Literals
+
+Rust code that constructs `ResponseCacheConfig` with an exhaustive struct
+literal must add `tools: None`. Prefer `..ResponseCacheConfig::default()` when
+the literal should remain compatible with new optional cache surfaces.
+
 ### Move Hermes Agent to Its Native Relay Integration
 
 NeMo Relay 0.8 removes Hermes Agent from the Relay CLI. The `nemo-relay hermes`
@@ -44,8 +50,8 @@ Move settings that should apply to your account into these files:
   `XDG_CONFIG_HOME` is not set
 
 Use `/etc/nemo-relay/config.toml` and `/etc/nemo-relay/plugins.toml` on Unix,
-or `%ProgramData%\nemo-relay\config.toml` and
-`%ProgramData%\nemo-relay\plugins.toml` on Windows, for system policy. System
+or `%ProgramData%\\nemo-relay\\config.toml` and
+`%ProgramData%\\nemo-relay\\plugins.toml` on Windows, for system policy. System
 configuration has higher precedence than the selected user or explicit
 configuration.