Skip to content

Commit 361aa3f

Browse files
committed
Make tool-set churn visible between turns
The tools array is the head of the provider's cached prompt prefix, so any change to it re-prefills the whole request. Whether that actually happens in real sessions is unmeasured: it shows up only as a billing and latency spike a turn later, with nothing tying it back to a mount. Log a digest when the set changes. Hashed, not verbatim — MCP tool descriptions are arbitrary-length server-supplied text and do not belong in the log stream. This replaces an earlier attempt that sorted the array by name. That was wrong: advertisedTools already orders deterministically, so sorting added no stability, and an alphabetical insert can land at index 0 and invalidate more of the prefix than appending does. A gate run carrying it measured cache-hit rate down 3-8 points across all four eval tiers. CL-7868
1 parent d9ec82c commit 361aa3f

1 file changed

Lines changed: 24 additions & 0 deletions

File tree

‎src/agent/director.ts‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@ import {
22
DefaultDirector,
33
type ExtendedInferenceOptions,
44
} from "@intx/inference";
5+
import { createHash } from "node:crypto";
56
import { getLogger } from "@intx/log";
67
import type {
78
ReactorDirector,
@@ -48,6 +49,25 @@ import {
4849

4950
const logger = getLogger([LOG_NAMESPACE_ROOT, "agent", "director"]);
5051

52+
// The serialized `tools` array is the head of the provider's cached prompt
53+
// prefix, ahead of the system prompt. Measured on OpenCode Go Responses, a warm
54+
// session holds 99.3% cached and ANY change to that array — a mount, a
55+
// description edit, or a pure reorder of an unchanged set — drops the next turn
56+
// to 2-4%. Appending at the end is not cheaper than prepending: 4.5% versus
57+
// 2.1%, both full misses.
58+
//
59+
// `advertisedTools` (src/agent/tool-search.ts) already keeps this array
60+
// deterministic, so the array should only ever change when a genuine discovery
61+
// grows it. This digest is here to prove that, because prefix churn is
62+
// otherwise invisible — it shows up only as a billing and latency spike a turn
63+
// later. Hashed rather than logged verbatim: MCP tool descriptions are
64+
// arbitrary-length, server-supplied text and do not belong in the log stream.
65+
// See CL-7868.
66+
function toolSetDigest(tools: readonly ToolDefinition[]): string {
67+
const shape = tools.map((t) => `${t.name}:${t.description ?? ""}`).join("|");
68+
return createHash("sha256").update(shape).digest("hex").slice(0, 12);
69+
}
70+
5171
function isInternalRecoveryAbort(
5272
event: Extract<ReactorInboundEvent, { type: "inference.error" }>,
5373
): boolean {
@@ -508,7 +528,11 @@ class ChatDirectorImpl extends DefaultDirector {
508528
}
509529

510530
updateToolDefinitions(toolDefinitions: ToolDefinition[]): void {
531+
const before = toolSetDigest(this._toolDefinitions);
532+
const after = toolSetDigest(toolDefinitions);
511533
this._toolDefinitions = toolDefinitions;
534+
if (before === after) return;
535+
logger.debug`tool-set-changed count=${String(this._toolDefinitions.length)} before=${before} after=${after}`;
512536
}
513537

514538
getTasks(): Task[] {

0 commit comments

Comments
 (0)