Skip to content

Commit 77a7cf5

Browse files
feat(client)!: capture conversation content by default (#213)
## Summary `SDKOptions.capture_content` now defaults to `true`. Online handler spans and offline evaluation spans (from #170, now merged) use that same gate. An explicit `false` on `configure`, a handler factory, or a runner-written span still strips prompts, output, tool payloads, and judge explanations, including on offline evaluation spans. The `$ld:ai:offline-evals:*` events still carry no transcript. Marked breaking (`!`): a process that never calls `configure` now sends conversation text to its collector. Without the marker, `bump-patch-for-minor-pre-major` would ship this as a patch release. Also corrects `TELEMETRY-CONTRACT.md` section 4a. It said `gen_ai.evaluation.explanation` is never emitted, but the judge path writes it whenever the judge handler captures content. With capture on by default, the explanation is now on judge spans by default. Rebased onto `main` after #170 was squash-merged, so the diff is only the default change. ## Test plan - [x] `uv run pytest`: 3644 passed, 18 skipped - [x] `uv run mypy packages/*/src`: 107 source files, no issues - [x] `uv run ruff check .` and `uv run ruff format --check .`: clean - [x] `make api-report-check`: 14 API reports are current <!-- CURSOR_SUMMARY --> --- > [!NOTE] > <sup>[Cursor Bugbot](https://cursor.com/bugbot) is generating a summary for commit 435e4cc. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> <!-- CURSOR_AGENT_PR_BODY_BEGIN --> ## Summary `SDKOptions.capture_content` now defaults to `true`. Online handler spans and offline evaluation spans (from #170, now merged) use that same gate. An explicit `false` on `configure`, a handler factory, or a runner-written span still strips prompts, output, tool payloads, and judge explanations, including on offline evaluation spans. The `$ld:ai:offline-evals:*` events still carry no transcript. Marked breaking (`!`): a process that never calls `configure` now sends conversation text to its collector. Without the marker, `bump-patch-for-minor-pre-major` would ship this as a patch release. Also corrects `TELEMETRY-CONTRACT.md` section 4a. It said `gen_ai.evaluation.explanation` is never emitted, but the judge path writes it whenever the judge handler captures content. With capture on by default, the explanation is now on judge spans by default. Rebased onto `main` after #170 was squash-merged, so the diff is only the default change. ## Decision Opt-out, for online and offline. In [this thread](https://launchdarkly.slack.com/archives/C0B9D3510TG/p1792783332999349?thread_ts=1792695162.107739&cid=C0B9D3510TG), Malachi said the current AI terms cover moving the SDK updates forward and to set this up as opt-out rather than opt-in. Ben then asked to flip the default to on. ## Test plan - [x] `uv run pytest`: 3644 passed, 18 skipped - [x] `uv run pytest packages/client/tests` after the content-module docstring fix: 1904 passed - [x] `uv run mypy packages/*/src`: 107 source files, no issues - [x] `uv run ruff check .` and `uv run ruff format --check .`: clean - [x] `make api-report-check`: 14 API reports are current <!-- CURSOR_AGENT_PR_BODY_END --> <div><a href="https://cursor.com/agents/bc-4232ef78-2eeb-54ed-ae75-6629724917b3?cursor_ref=pr_footer&cursor_cta=open_in_web"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/open-in-web-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/open-in-web-light.png"><img alt="Open in Web" width="114" height="28" src="https://cursor.com/assets/images/open-in-web-dark.png"></picture></a>&nbsp;<a href="https://cursor.com/background-agent?bcId=bc-4232ef78-2eeb-54ed-ae75-6629724917b3&cursor_ref=pr_footer&cursor_cta=open_in_cursor"><picture><source media="(prefers-color-scheme: dark)" srcset="https://cursor.com/assets/images/open-in-cursor-dark.png"><source media="(prefers-color-scheme: light)" srcset="https://cursor.com/assets/images/open-in-cursor-light.png"><img alt="Open in Cursor" width="131" height="28" src="https://cursor.com/assets/images/open-in-cursor-dark.png"></picture></a>&nbsp;</div> --------- Co-authored-by: Cursor Agent <cursoragent@cursor.com>
1 parent 882d099 commit 77a7cf5

36 files changed

Lines changed: 380 additions & 163 deletions

File tree

‎AGENTS.md‎

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -156,7 +156,7 @@ The client manages a singleton connection to LaunchDarkly and the associated tel
156156
|---|---|
157157
| `init_client(options?)` | Auto-discovers and initializes `launchdarkly-server-sdk` (optional dep, loaded via `importlib`). Optional — the first AI API call triggers lazy init when `LD_SDK_KEY` is set. Accepts optional overrides for SDK key, base URIs, service name, environment, and OTLP endpoint. Returns `Awaitable[LDClientInterface]`. On every successful path, including the already-initialized path, flushes `$ld:ai:sdk:info` for any LaunchDarkly AI packages that have not yet reported. Does not take behavior defaults. |
158158
| `init_client(client=...)` | **BYOC overload** — accepts a pre-initialized `LDClientInterface`. Stores it directly without calling the SDK. Flushes pending `$ld:ai:sdk:info` events. |
159-
| `configure(*, options)` | Sets process-wide behavior defaults on the lifecycle singleton. `options` is a frozen `SDKOptions` (`capture_content: bool = False`). Keyword-only. Handlers read it at invocation. An explicit factory or wrapper `capture_content` overrides it. Replaceable until the first AI invocation; an identical value is allowed after. Not read from request data. |
159+
| `configure(*, options)` | Sets process-wide behavior defaults on the lifecycle singleton. `options` is a frozen `SDKOptions` (`capture_content: bool = True`). Keyword-only. Handlers read it at invocation. An explicit factory or wrapper `capture_content` overrides it, and an explicit `False` strips content everywhere, including offline evaluation spans. Replaceable until the first AI invocation; an identical value is allowed after. Not read from request data. |
160160
| `get_client()` | Returns the initialized `LDClientInterface`. Throws if initialization has not completed. |
161161
| `shutdown()` | Flushes all pending events and telemetry, then closes the client. Must be awaited before the process exits. Clears sdk-info reporting so a later client reports again. Restores the built-in SDK defaults. |
162162
| `register_ai_sdk_package(name, version)` | Records a LaunchDarkly AI package identity. Handler and convenience packages call this at import time. |
@@ -571,14 +571,15 @@ beside the input count, so its handlers add it in. OpenAI and LangChain already
571571
input, so theirs pass the figure through. Centralising that rule would double-count for two
572572
providers out of three. `SpanUsage` is the type that means the folding is already done.
573573

574-
#### Content is off by default
574+
#### Content is on by default
575575

576576
Every factory takes `capture_content: bool | None = None`. `None` reads `configure()` at
577-
invocation, which is off until a process sets `SDKOptions(capture_content=True)`. An explicit
578-
`True` or `False` overrides that default. Guard at the call site as well as inside the helper: the
579-
helper's guard makes a forgotten call site harmless, and the call site's guard avoids serialising
580-
JSON that would then be discarded, once per turn, in a loop. Resolve the bool when the handler
581-
runs, not when it is constructed.
577+
invocation. The built-in default is on until a process sets `SDKOptions(capture_content=False)`.
578+
An explicit `True` or `False` overrides that default, and an explicit `False` strips content
579+
everywhere, including offline evaluation spans. Guard at the call site as well as inside the
580+
helper: the helper's guard makes a forgotten call site harmless, and the call site's guard avoids
581+
serialising JSON that would then be discarded, once per turn, in a loop. Resolve the bool when the
582+
handler runs, not when it is constructed.
582583

583584
#### Finish reasons have three mechanisms, not one
584585

‎README.md‎

Lines changed: 9 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -542,20 +542,23 @@ This matters most on Anthropic, which reports cache reads and writes beside the
542542
than inside it. A turn that reads 19,971 tokens from cache and writes 3,580 more reports an input of
543543
3 from the provider; the span reports 23,554.
544544

545-
### Conversation content is off by default
545+
### Conversation content is on by default
546546

547-
Prompts, model output, tool arguments and tool results are personal data, so a span carries only
548-
metadata unless you ask for more: models, token counts, timings, tool names.
547+
Prompts, model output, tool arguments and tool results are personal data. The built-in default
548+
sends that text. A span carries only metadata — models, token counts, timings, tool names — when
549+
capture is turned off.
549550

550-
Pass `capture_content=True` to a handler factory to include the conversation, or turn it on for every handler from process startup. See [Process-wide defaults](packages/client/README.md#process-wide-defaults).
551+
Pass `capture_content=False` to a handler factory to leave the conversation off that handler, or
552+
turn it off for every handler from process startup. An explicit `False` wins everywhere, including
553+
on offline evaluation spans. See [Process-wide defaults](packages/client/README.md#process-wide-defaults).
551554

552555
```python
553556
from launchdarkly_ai_claude_messages import create_claude_messages_handler
554557

555-
handler = create_claude_messages_handler(capture_content=True)
558+
handler = create_claude_messages_handler(capture_content=False)
556559
```
557560

558-
Turning this on sends the text of every request and response to whatever collector the SDK points
561+
Leaving this on sends the text of every request and response to whatever collector the SDK points
559562
at.
560563

561564
### Streaming

‎TELEMETRY-CONTRACT.md‎

Lines changed: 14 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,7 @@ able to tell from the trace which path ran.
9898
| `launchdarkly.stream.abandoned` | `True`, only when abandoned | `end_span_once` |
9999
| `gen_ai.evaluation.name` | judge config key, judge roots only | `with_judge_evaluation`, see section 4a |
100100
| `gen_ai.evaluation.score.value` | numeric score, judge roots only | `with_judge_evaluation`, see section 4a |
101+
| `gen_ai.evaluation.explanation` | judge reasoning, judge roots only, gated by section 7 | `with_judge_evaluation`, see section 4a |
101102

102103
The root also carries one span event, `feature_flag`, with these event attributes:
103104

@@ -226,17 +227,18 @@ SDK writes a `gen_ai.evaluation.result` span event on that `invoke_agent` span:
226227
|---|---|
227228
| `gen_ai.evaluation.name` | judge config key |
228229
| `gen_ai.evaluation.score.value` | numeric score, only when the judge returned a finite number |
230+
| `gen_ai.evaluation.explanation` | the judge's reasoning, gated. See section 7 |
229231

230232
The same keys are mirrored as span attributes, so section 2 lists them too.
231233
`gen_ai.evaluation.score.label` is not invented.
232234
The existing `track(evaluationMetricKey)` call is unchanged and still feeds AI Config Monitoring —
233235
a judge that returns a non-numeric score emits no evaluation event but still tracks the metric.
234236

235-
`gen_ai.evaluation.explanation` is deliberately **not** emitted. The judge's reasoning is
236-
model-generated prose about the user's conversation — content, under section 7 — and content
237-
attributes require `captureContent` / `capture_content`, a handler-factory option this layer does
238-
not receive. The reasoning is still returned to the caller in `judgeResults` / `judge_results`;
239-
only the telemetry copy is withheld. Exporting it needs its own opt-in.
237+
`gen_ai.evaluation.explanation` is content under section 7. The judge's reasoning is
238+
model-generated prose about the user's conversation, so it follows the judge handler's own
239+
`capture_content` decision. It is written by default, and an explicit `False` on the judge's
240+
handler or the process default withholds it. The reasoning is always returned to the caller in
241+
`judgeResults` / `judge_results`; only the telemetry copy is gated.
240242

241243
---
242244

@@ -397,12 +399,13 @@ error for an abandoned stream, so two dashboards would describe the same run dif
397399
Every handler factory takes `capture_content: bool | None = None`.
398400

399401
`None` reads the process-wide default at invocation (`configure(options=SDKOptions(...))`), which
400-
is off. `True` or `False` overrides that default for the handler, and an explicit `False` wins
401-
over a configured `True`. A convenience wrapper's `capture_content` is the per-call override: when
402-
the caller passes it, that value is what the handler is built with; when they omit it, the wrapper
403-
passes `None` and the process default applies. Conversation content is personal data. A run emits
404-
only metadata, meaning models, token counts, timings and tool names, until capture is turned on.
405-
Turning it on sends the text of every request and response to whatever collector the SDK points at.
402+
is on. `True` or `False` overrides that default for the handler, and an explicit `False` wins
403+
over a configured `True`, including on offline evaluation spans. A convenience wrapper's
404+
`capture_content` is the per-call override: when the caller passes it, that value is what the
405+
handler is built with; when they omit it, the wrapper passes `None` and the process default
406+
applies. Conversation content is personal data. The default sends the text of every request and
407+
response to whatever collector the SDK points at. An explicit `False` emits only metadata, meaning
408+
models, token counts, timings and tool names.
406409

407410
Guard twice, on purpose. The helper returns without writing when capture is off, which makes a
408411
forgotten call site harmless. The call site also checks before building the argument, which avoids

‎examples/capture_content.py‎

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,13 @@
11
"""
2-
Example: enable conversation content for every handler at process startup.
3-
4-
The handlers in the global registry are built with no ``capture_content``
5-
argument. They read the process default when they run. ``configure`` is
6-
separate from ``init_client``: ``main.py`` initializes the client, and this
7-
example calls ``configure`` once before the first AI invocation. An explicit
8-
``True`` or ``False`` on a factory would override this default.
2+
Example: pin conversation content on for every handler at process startup.
3+
4+
The built-in default is already on. The handlers in the global registry are
5+
built with no ``capture_content`` argument and read that default when they
6+
run. ``configure`` is separate from ``init_client``: ``main.py`` initializes
7+
the client, and this example calls ``configure`` once before the first AI
8+
invocation so the choice is visible at startup. Pass
9+
``SDKOptions(capture_content=False)`` to opt out. An explicit ``False`` on a
10+
factory removes the transcript everywhere, including offline evaluation spans.
911
1012
Usage (via main.py):
1113
python main.py capture-content <flag-key> "<user input>"

‎examples/conversation.py‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -9,10 +9,10 @@
99
the same id: root, ``chat``, ``execute_tool``, and the judge's own ``invoke_agent``.
1010
2. Each turn shows a score badge, sourced from the ``gen_ai.evaluation.result`` span event on
1111
the judge span.
12-
3. No judge reasoning anywhere in the telemetry. The score and the judge's config key are
13-
exported; the explanation is not, because it is model prose about the user's conversation
14-
and content attributes require ``capture_content``. The reasoning IS printed below, straight
15-
from ``judge_results`` — that is the caller's copy, and it is unaffected.
12+
3. The judge explanation is on the span, because capture is on by default. The score and
13+
the judge's config key are exported either way. An explicit ``False`` removes the
14+
explanation from telemetry. The reasoning IS printed below, straight from
15+
``judge_results`` — that is the caller's copy, and it is unaffected.
1616
1717
The flag key must point at an AI Config with a ``judge_configuration``, otherwise there are no
1818
judge turns to look at.

‎packages/claude-agents/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@ asyncio.run(main())
4040

4141
| Option | Default | Description |
4242
|---|---|---|
43-
| `capture_content` | `None` | Put prompts, model output, and tool arguments and results on the emitted spans. `None` follows `configure()`, which is off by default. |
43+
| `capture_content` | `None` | Put prompts, model output, and tool arguments and results on the emitted spans. `None` follows `configure()`, which is on by default. An explicit `False` leaves them off. |
4444
| `cwd` | the process's working directory | Working directory for the Claude Code process the Agent SDK starts. A `str` or path-like. |
4545
| `setting_sources` | `[]` (none) | Which Claude Code settings sources to load: `"user"` (`~/.claude/settings.json`), `"project"` (`.claude/settings.json`, and `CLAUDE.md`), `"local"` (`.claude/settings.local.json`). |
4646

‎packages/claude-agents/agents.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
> **Span shape.** This package emits `invoke_agent` → `chat {model}` → `execute_tool {name}`, with
44
> tool spans as siblings of `chat`. Span construction lives in `spans.py` beside the handler.
5-
> Conversation content is off unless the caller passes `capture_content=True`, or the process sets it. See [Process-wide defaults](../client/README.md#process-wide-defaults).
5+
> Conversation content is on unless the caller passes `capture_content=False`, or the process turns it off. See [Process-wide defaults](../client/README.md#process-wide-defaults).
66
> `TELEMETRY-CONTRACT.md` at the repo root is the authority; read it before changing span code.
77
88

‎packages/claude-agents/src/launchdarkly_ai_claude_agents/handler.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -739,8 +739,8 @@ def create_claude_agents_handler(
739739
740740
Set *capture_content* to put prompts, model output, tool arguments and tool results on the
741741
emitted spans. ``None`` (the default) reads the process-wide default at invocation,
742-
which is off until ``configure`` sets it. ``True`` or ``False`` overrides that default
743-
for this handler. See TELEMETRY-CONTRACT.md section 7.
742+
which is on until ``configure`` turns it off. ``True`` or ``False`` overrides that default
743+
for this handler. An explicit ``False`` emits only metadata. See TELEMETRY-CONTRACT.md section 7.
744744
745745
*cwd* and *setting_sources* configure the Claude Code process the Agent SDK starts, not the
746746
model, so they come from the application and never from a config. Each one given is passed to

‎packages/claude-agents/tests/test_handler.py‎

Lines changed: 28 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -862,7 +862,25 @@ async def test_cache_tokens_folded_into_input(
862862

863863

864864
class TestContentCapture:
865-
async def test_emits_no_content_at_all_by_default(
865+
async def test_emits_content_by_default(
866+
self, monkeypatch: pytest.MonkeyPatch
867+
) -> None:
868+
monkeypatch.setattr(
869+
handler_mod,
870+
"query",
871+
_fake_query(
872+
[
873+
assistant_message(content=[TextBlock(text="hi")]),
874+
result_message("hi"),
875+
]
876+
),
877+
)
878+
await create_claude_agents_handler()(BASE_CONFIG, "q")
879+
[chat] = named("chat ")
880+
assert "gen_ai.input.messages" in chat.attributes
881+
assert "gen_ai.output.messages" in chat.attributes
882+
883+
async def test_emits_no_content_when_capture_content_is_false(
866884
self, monkeypatch: pytest.MonkeyPatch
867885
) -> None:
868886
monkeypatch.setattr(
@@ -872,22 +890,22 @@ async def test_emits_no_content_at_all_by_default(
872890
[init_message(tools=["Read"]), assistant_message(), result_message()]
873891
),
874892
)
875-
await create_claude_agents_handler()(TOOL_CONFIG, "q")
893+
await create_claude_agents_handler(capture_content=False)(TOOL_CONFIG, "q")
876894
for span in (root(), *named("chat ")):
877895
assert "gen_ai.input.messages" not in span.attributes
878896
assert "gen_ai.output.messages" not in span.attributes
879897
assert "gen_ai.system_instructions" not in span.attributes
880898
assert "gen_ai.tool.definitions" not in span.attributes
881899

882-
async def test_root_carries_no_content_by_default_either(
900+
async def test_root_carries_no_output_when_capture_is_false(
883901
self, monkeypatch: pytest.MonkeyPatch
884902
) -> None:
885903
monkeypatch.setattr(
886904
handler_mod,
887905
"query",
888906
_fake_query([assistant_message(), result_message("hi")]),
889907
)
890-
await create_claude_agents_handler()(BASE_CONFIG, "q")
908+
await create_claude_agents_handler(capture_content=False)(BASE_CONFIG, "q")
891909
assert "gen_ai.output.messages" not in root().attributes
892910

893911
async def test_tool_call_arguments_and_result_gated_by_capture(
@@ -925,7 +943,7 @@ async def _query(**kwargs: Any) -> AsyncIterator[Any]:
925943
yield result_message("done")
926944

927945
monkeypatch.setattr(handler_mod, "query", _query)
928-
await create_claude_agents_handler()(
946+
await create_claude_agents_handler(capture_content=False)(
929947
TOOL_CONFIG, "q", {"search": lambda _: "r"}
930948
)
931949
[tool_span] = named("execute_tool ")
@@ -1610,7 +1628,11 @@ async def test_ld_span_attributes_land_on_root_only(
16101628
assert attrs["launchdarkly.config.key"] == "cfg"
16111629
assert attrs["launchdarkly.variation.key"] == "var"
16121630
assert attrs["launchdarkly.run.id"] == "run-1"
1613-
assert [e.name for e in root().events] == ["feature_flag"]
1631+
assert [
1632+
event.name
1633+
for event in root().events
1634+
if not event.name.startswith("gen_ai.content")
1635+
] == ["feature_flag"]
16141636

16151637

16161638
class TestModelParametersForwarding:

‎packages/claude-messages/agents.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
> **Span shape.** This package emits `invoke_agent` → `chat {model}` → `execute_tool {name}`, with
44
> tool spans as siblings of `chat`. Span construction lives in `spans.py` beside the handler.
5-
> Conversation content is off unless the caller passes `capture_content=True`, or the process sets it. See [Process-wide defaults](../client/README.md#process-wide-defaults).
5+
> Conversation content is on unless the caller passes `capture_content=False`, or the process turns it off. See [Process-wide defaults](../client/README.md#process-wide-defaults).
66
> `TELEMETRY-CONTRACT.md` at the repo root is the authority; read it before changing span code.
77
88

0 commit comments

Comments
 (0)