Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions content/develop/ai/context-engine/context-gateway/concepts.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,11 +85,11 @@ In `search` mode the agent calls `search_tools` and receives complete tool defin

A **tool policy** sets optional behavior for one tool of one Upstream. Both behaviors need a Gateway database, selected with the Gateway's `databaseId`.

Each Gateway also turns the result cache and result handles on or off for all of its tools with its `resultCache` and `resultHandles` settings, `enabled` or `disabled`. A Gateway created with a `databaseId` starts with both enabled, and a Gateway without one has both disabled. A feature serves a Gateway only when the Gateway's setting is `enabled` and the deployment turns the feature on.
Each Gateway also turns the result cache and result handles on or off for all of its tools with two settings, `enabled` or `disabled`: `cache` for the result cache and `compression` for result handles. A Gateway created with a `databaseId` starts with both enabled, and a Gateway without one has both disabled. A feature serves a Gateway only when the Gateway's setting is `enabled` and the deployment turns the feature on.

### Result cache

The result cache serves a repeated call from the Gateway database instead of calling the Upstream again. Each tool opts in with a `resultCache` policy and a time to live (TTL).
The result cache serves a repeated call from the Gateway database instead of calling the Upstream again. Each tool opts in with a `cache` policy and a time to live (TTL).

A tool can be cached unless its definition says `readOnlyHint: false` or `destructiveHint: true`. An absent hint does not block caching, and no annotation turns caching on. An entry is served until its TTL expires or the Upstream's revision changes.

Expand All @@ -102,7 +102,7 @@ All callers with access to a Gateway share its cache entries.

A result handle holds a large tool result in the Gateway database. When result handles are on for a Gateway, every eligible result over 32 kibibytes (KiB) becomes a handle. The agent receives a small stub with the handle and reads the result in pages, queries, counts, or aggregates.

A tool opts out with a `resultHandle` policy of `capture: never`, or sets its own handle lifetime with `ttlSeconds`. An agent can also hold a result of any size with `run_tool_as_handle`.
A tool opts out with a `compression` policy of `capture: never`, or sets its own handle lifetime with `ttlSeconds`. An agent can also hold a result of any size with `run_tool_as_handle`.

A handle is bound to the caller that created it, and it survives a rotation of that caller's agent key. It ends when the agent releases it or its TTL expires. Handles are bounded by their TTL and the Gateway database's memory.

Expand All @@ -116,7 +116,7 @@ A handle is bound to the caller that created it, and it survives a rotation of t
| Namespace | The prefix that makes an Upstream's tool names unique within a Gateway. |
| Catalog | The stored list of an Upstream's tools. |
| Tool policy | Per-tool settings for the result cache and result handles. |
| `resultCache`, `resultHandles` | Per-Gateway settings that turn the result cache and result handles on or off. |
| `cache`, `compression` | Per-Gateway settings that turn the result cache and result handles on or off. |
| Gateway database | The Redis database that holds cached results and result handles. |

## Next steps
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ The [API reference](/content/develop/ai/context-engine/context-gateway/api-refer
| Upstream namespace | 1 to 63 lowercase letters and digits, with single hyphens between them |
| Gateway `databaseId` | 1 to 32 letters, digits, and hyphens |
| `search_tools` query | 1 to 200 characters. `limit` 1 to 100, default 5. |
| Result cache time to live (TTL) (`resultCache.ttlSeconds`) | 1 to 86400 seconds |
| Result cache time to live (TTL) (`cache.ttlSeconds`) | 1 to 86400 seconds |
| Result handle threshold | Results over 32 kibibytes (KiB) become handles when the deployment turns handles on. |
| Largest result a handle holds | 8 mebibytes (MiB). A larger result comes back whole. |
| Result handle lifetime | 15 minutes by default, at most 24 hours. A tool policy can set 1 to 86400 seconds. |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ Create a Gateway and save its `id` and `mcpEndpoint`. The Gateway serves Model C

*Example set `iris_gateway_manage`, step `gateway-create`, in Python (Iris), cURL (Iris), and CLI (Iris).*

To use the result cache or result handles, select a Gateway database with `databaseId`. Your administrator names the available databases. A Gateway created with a `databaseId` has `resultCache` and `resultHandles` enabled unless you send them as `disabled`. Sending `enabled` without a `databaseId` returns `400`.
To use the result cache or result handles, select a Gateway database with `databaseId`. Your administrator names the available databases. A Gateway created with a `databaseId` has `cache` and `compression` enabled unless you send them as `disabled`. Sending `enabled` without a `databaseId` returns `400`.

## List and inspect Gateways and Upstreams

Expand All @@ -42,22 +42,22 @@ List the deployment's Gateways and a Gateway's Upstreams, and read one by ID. Ev

## Change a Gateway

Update a Gateway's `toolExposureMode` to `full` or `search`, its display name, its `databaseId`, or its `resultCache` and `resultHandles` settings. Agents see a new mode on their next request. See [Tool exposure modes](/content/develop/ai/context-engine/context-gateway/concepts.md#tool-exposure-modes).
Update a Gateway's `toolExposureMode` to `full` or `search`, its display name, its `databaseId`, or its `cache` and `compression` settings. Agents see a new mode on their next request. See [Tool exposure modes](/content/develop/ai/context-engine/context-gateway/concepts.md#tool-exposure-modes).

*Example step `gateway-update-mode`.*

### Turn the result cache and result handles on or off

Set `resultCache` or `resultHandles` to `enabled` or `disabled` on a Gateway update. Every Gateway response returns both settings. A Gateway's setting takes effect only when the deployment also turns the feature on. See [Result cache and result handles](/content/operate/iris/context-gateway/self-managed/configuration.md#result-cache-and-result-handles).
Set `cache` or `compression` to `enabled` or `disabled` on a Gateway update. Every Gateway response returns both settings. A Gateway's setting takes effect only when the deployment also turns the feature on. See [Result cache and result handles](/content/operate/iris/context-gateway/self-managed/configuration.md#result-cache-and-result-handles).

```json
{"revision": "<revision>", "resultCache": "enabled", "resultHandles": "disabled"}
{"revision": "<revision>", "cache": "enabled", "compression": "disabled"}
```

| Change | What to send |
| --- | --- |
| Remove the Gateway database | `"databaseId": null` with `"resultCache": "disabled"` and `"resultHandles": "disabled"` in the same request. Without them, the update returns `400`. |
| Add a Gateway database to a Gateway that had none | `databaseId` with `"resultCache": "enabled"`, `"resultHandles": "enabled"`, or both. Adding a database alone keeps both settings as they were, which is `disabled` for a Gateway created without one. |
| Remove the Gateway database | `"databaseId": null` with `"cache": "disabled"` and `"compression": "disabled"` in the same request. Without them, the update returns `400`. |
| Add a Gateway database to a Gateway that had none | `databaseId` with `"cache": "enabled"`, `"compression": "enabled"`, or both. Adding a database alone keeps both settings as they were, which is `disabled` for a Gateway created without one. |
| Stop caching or handles for a Gateway | `"disabled"` for that setting. The Gateway stops reading and writing cache entries, or stops creating handles and removes the handle tools from `tools/list`. Stored entries and handles stay until their TTL. |

*Example step `gateway-update-features`.*
Expand Down Expand Up @@ -97,15 +97,15 @@ Set a tool's policies with `toolPolicies` on an Upstream update. The result cach

| Policy | Member | Effect |
| --- | --- | --- |
| `resultCache` | `ttlSeconds`, 1 to 86400 | Caches the tool's results for that long. |
| `resultHandle` | `capture: never` | Always returns the tool's results whole, also from `run_tool_as_handle`. |
| `resultHandle` | `ttlSeconds`, 1 to 86400 | Keeps the tool's handles for that long instead of the deployment's default. |
| `cache` | `ttlSeconds`, 1 to 86400 | Caches the tool's results for that long. |
Comment thread
nkanu17 marked this conversation as resolved.
Resolved
| `compression` | `capture: never` | Always returns the tool's results whole, also from `run_tool_as_handle`. |
| `compression` | `ttlSeconds`, 1 to 86400 | Keeps the tool's handles for that long instead of the deployment's default. |

*Example step `gateway-tool-policy`.*

- Cache only tools whose data changes more slowly than the time to live (TTL). All callers with access to the Gateway share its cache entries.
- A tool whose definition says `readOnlyHint: false` or `destructiveHint: true` is never cached.
- A new or changed `resultCache` policy returns `400` while the Gateway's `resultCache` is `disabled`, including on a Gateway without a `databaseId`. A stored policy sent back unchanged is kept.
- A new or changed `cache` policy returns `400` while the Gateway's `cache` setting is `disabled`, including on a Gateway without a `databaseId`. A stored policy sent back unchanged is kept.
- When you move an Upstream to a new endpoint, the response lists any policies it removed in `droppedToolPolicies`.

## Rotate an Upstream credential
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ export OPERATIONS_ID="$(jq -er '.id' operations.json)"
jq '{id, mcpEndpoint, toolExposureMode}' front-desk.json operations.json
```

The `search` mode Gateway has a `databaseId`, so it starts with `resultCache` and `resultHandles` enabled. Result handles also need `result_handles.enabled` on the Data Plane. See [Result cache and result handles](/content/operate/iris/context-gateway/self-managed/configuration.md#result-cache-and-result-handles).
The `search` mode Gateway has a `databaseId`, so it starts with `cache` and `compression` enabled. Result handles also need `result_handles.enabled` on the Data Plane. See [Result cache and result handles](/content/operate/iris/context-gateway/self-managed/configuration.md#result-cache-and-result-handles).

<!-- TODO: Run every step against a deployed chart before review. -->

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,7 @@ dataplane:
| `result_handles.max_capture_bytes` | `8388608` | Results above this size come back whole. |
| `result_handles.preview_items`, `preview_bytes` | `3`, `2048` | The size of a handle stub's preview. |

These settings turn each feature on for the deployment. Each Gateway then has its own `resultCache` and `resultHandles` settings, and a feature serves a Gateway only when both the deployment and the Gateway turn it on. Both need the Gateway's `databaseId` to name a registered database.
These settings turn each feature on for the deployment. Each Gateway then has its own `cache` and `compression` settings, and a feature serves a Gateway only when both the deployment and the Gateway turn it on. Both need the Gateway's `databaseId` to name a registered database.

The result cache still needs a policy for each tool. Result handles apply to every eligible result on a Gateway that has them on, unless a tool's policy opts out. See [Turn the result cache and result handles on or off](/content/develop/ai/context-engine/context-gateway/manage-gateways.md#turn-the-result-cache-and-result-handles-on-or-off).

Expand Down
Loading
Loading