diff --git a/AGENTS.md b/AGENTS.md index 7cd59fbf..dac8348d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -140,10 +140,11 @@ The client manages a singleton connection to LaunchDarkly and the associated tel | Export | Description | |---|---| -| `init_client(options?)` | Auto-discovers and initializes `launchdarkly-server-sdk` (optional dep, loaded via `importlib`). Optional — the first AI API call triggers lazy init when `LD_SDK_KEY` is set. Accepts optional overrides for SDK key, base URIs, service name, environment, and OTLP endpoint. Returns `Awaitable[LDClientInterface]`. On every successful path, including the already-initialized path, flushes `$ld:ai:sdk:info` for any LaunchDarkly AI packages that have not yet reported. | +| `init_client(options?)` | Auto-discovers and initializes `launchdarkly-server-sdk` (optional dep, loaded via `importlib`). Optional — the first AI API call triggers lazy init when `LD_SDK_KEY` is set. Accepts optional overrides for SDK key, base URIs, service name, environment, and OTLP endpoint. Returns `Awaitable[LDClientInterface]`. On every successful path, including the already-initialized path, flushes `$ld:ai:sdk:info` for any LaunchDarkly AI packages that have not yet reported. Does not take behavior defaults. | | `init_client(client=...)` | **BYOC overload** — accepts a pre-initialized `LDClientInterface`. Stores it directly without calling the SDK. Flushes pending `$ld:ai:sdk:info` events. | +| `configure(*, options)` | Sets process-wide behavior defaults on the lifecycle singleton. `options` is a frozen `SDKOptions` (`capture_content: bool = False`). Keyword-only. Handlers read it at invocation. An explicit factory or wrapper `capture_content` overrides it. Replaceable until the first AI invocation; an identical value is allowed after. Not read from request data. | | `get_client()` | Returns the initialized `LDClientInterface`. Throws if initialization has not completed. | -| `shutdown()` | Flushes all pending events and telemetry, then closes the client. Must be awaited before the process exits. Clears sdk-info reporting so a later client reports again. | +| `shutdown()` | Flushes all pending events and telemetry, then closes the client. Must be awaited before the process exits. Clears sdk-info reporting so a later client reports again. Restores the built-in SDK defaults. | | `register_ai_sdk_package(name, version)` | Records a LaunchDarkly AI package identity. Handler and convenience packages call this at import time. | ### Core Data Types @@ -554,9 +555,12 @@ providers out of three. `SpanUsage` is the type that means the folding is alread #### Content is off by default -Every factory takes `capture_content: bool = False`. Guard at the -call site as well as inside the helper: the helper's guard makes a forgotten call site harmless, and -the call site's guard avoids serialising JSON that would then be discarded, once per turn, in a loop. +Every factory takes `capture_content: bool | None = None`. `None` reads `configure()` at +invocation, which is off until a process sets `SDKOptions(capture_content=True)`. An explicit +`True` or `False` overrides that default. Guard at the call site as well as inside the helper: the +helper's guard makes a forgotten call site harmless, and the call site's guard avoids serialising +JSON that would then be discarded, once per turn, in a loop. Resolve the bool when the handler +runs, not when it is constructed. #### Finish reasons have three mechanisms, not one diff --git a/README.md b/README.md index 0e693665..a50f0103 100644 --- a/README.md +++ b/README.md @@ -519,7 +519,7 @@ than inside it. A turn that reads 19,971 tokens from cache and writes 3,580 more Prompts, model output, tool arguments and tool results are personal data, so a span carries only metadata unless you ask for more: models, token counts, timings, tool names. -Pass `capture_content=True` to a handler factory to include the conversation: +Pass `capture_content=True` to a handler factory to include the conversation, or turn it on for every handler from process startup. See [Process-wide defaults](packages/client/README.md#process-wide-defaults). ```python from launchdarkly_ai_claude_messages import create_claude_messages_handler diff --git a/TELEMETRY-CONTRACT.md b/TELEMETRY-CONTRACT.md index ef183049..2b5157d0 100644 --- a/TELEMETRY-CONTRACT.md +++ b/TELEMETRY-CONTRACT.md @@ -394,11 +394,15 @@ error for an abandoned stream, so two dashboards would describe the same run dif ## 7. Content capture -Every handler factory takes `capture_content: bool = False`. - -The default is off. Conversation content is personal data. A run emits only metadata, meaning -models, token counts, timings and tool names, until a caller asks for more. Turning it on sends the -text of every request and response to whatever collector the SDK points at. +Every handler factory takes `capture_content: bool | None = None`. + +`None` reads the process-wide default at invocation (`configure(options=SDKOptions(...))`), which +is off. `True` or `False` overrides that default for the handler, and an explicit `False` wins +over a configured `True`. A convenience wrapper's `capture_content` is the per-call override: when +the caller passes it, that value is what the handler is built with; when they omit it, the wrapper +passes `None` and the process default applies. Conversation content is personal data. A run emits +only metadata, meaning models, token counts, timings and tool names, until capture is turned on. +Turning it on sends the text of every request and response to whatever collector the SDK points at. Guard twice, on purpose. The helper returns without writing when capture is off, which makes a forgotten call site harmless. The call site also checks before building the argument, which avoids diff --git a/packages/claude-agents/agents.md b/packages/claude-agents/agents.md index 8ff60679..bce87584 100644 --- a/packages/claude-agents/agents.md +++ b/packages/claude-agents/agents.md @@ -2,7 +2,7 @@ > **Span shape.** This package emits `invoke_agent` → `chat {model}` → `execute_tool {name}`, with > tool spans as siblings of `chat`. Span construction lives in `spans.py` beside the handler. -> Conversation content is off unless the caller passes `capture_content=True`. +> Conversation content is off unless the caller passes `capture_content=True`, or the process sets it. See [Process-wide defaults](../client/README.md#process-wide-defaults). > `TELEMETRY-CONTRACT.md` at the repo root is the authority; read it before changing span code. diff --git a/packages/claude-agents/pyproject.toml b/packages/claude-agents/pyproject.toml index 8b2cb547..0c1f71c1 100644 --- a/packages/claude-agents/pyproject.toml +++ b/packages/claude-agents/pyproject.toml @@ -3,7 +3,7 @@ name = "launchdarkly-ai-claude-agents" version = "0.2.5" requires-python = ">=3.12" dependencies = [ - "launchdarkly-ai-server", + "launchdarkly-ai-server>=0.3.1", "opentelemetry-api>=1.25", "anthropic>=0.40", "claude-agent-sdk>=0.2", diff --git a/packages/claude-agents/src/launchdarkly_ai_claude_agents/handler.py b/packages/claude-agents/src/launchdarkly_ai_claude_agents/handler.py index eb7665e1..5256e5ea 100644 --- a/packages/claude-agents/src/launchdarkly_ai_claude_agents/handler.py +++ b/packages/claude-agents/src/launchdarkly_ai_claude_agents/handler.py @@ -48,6 +48,7 @@ set_output_content_attributes, set_tool_call_content_attributes, ) +from launchdarkly_ai_server.lifecycle import effective_capture_content from ._version import PACKAGE_NAME, __version__ from .spans import ( @@ -504,18 +505,25 @@ def _build_query_options( # --------------------------------------------------------------------------- -def create_claude_agents_handler(*, capture_content: bool = False) -> ProviderHandler: +def create_claude_agents_handler( + *, capture_content: bool | None = None +) -> ProviderHandler: """Creates a ``ProviderHandler`` for Anthropic's Claude via the claude-agent-sdk. Set *capture_content* to put prompts, model output, tool arguments and tool results on the - emitted spans. It defaults to off. See TELEMETRY-CONTRACT.md section 7. + emitted spans. ``None`` (the default) reads the process-wide default at invocation, + which is off until ``configure`` sets it. ``True`` or ``False`` overrides that default + for this handler. See TELEMETRY-CONTRACT.md section 7. """ report_usage("claude-agents.createClaudeAgentsHandler", PACKAGE_NAME, __version__) return _create_claude_agents_handler(capture_content=capture_content) -def _create_claude_agents_handler(*, capture_content: bool = False) -> ProviderHandler: +def _create_claude_agents_handler( + *, capture_content: bool | None = None +) -> ProviderHandler: """Non-reporting :func:`create_claude_agents_handler`, used by this package's wrappers.""" + content_override = capture_content async def _call_impl( config: AiConfigRep, @@ -524,6 +532,8 @@ async def _call_impl( variables: dict[str, Any] | None = None, history: list[dict[str, Any]] | None = None, ) -> dict[str, Any]: + # None follows configure(); an explicit bool overrides it. Read when the call runs. + capture_content = effective_capture_content(content_override) th = tool_handlers or {} vs = variables or {} @@ -699,14 +709,14 @@ def _stream_impl( tool_handlers or {}, variables or {}, history, - capture_content=capture_content, + capture_content=effective_capture_content(content_override), ) return _create_handler( ("Anthropic", "agent"), _call_impl, # type: ignore[arg-type] _stream_impl, # type: ignore[arg-type] - capture_content=capture_content, + capture_content=content_override, ) @@ -896,7 +906,7 @@ def claude_agents( """Convenience wrapper: creates a handler and calls config(...).invoke().""" report_usage("claude-agents.claudeAgents", PACKAGE_NAME, __version__) variables = kwargs.pop("variables", None) - capture_content = kwargs.pop("capture_content", False) + capture_content = kwargs.pop("capture_content", None) return _config( key=config_key, handler=_create_claude_agents_handler(capture_content=capture_content), diff --git a/packages/claude-agents/tests/test_handler.py b/packages/claude-agents/tests/test_handler.py index 83aef4de..765367d9 100644 --- a/packages/claude-agents/tests/test_handler.py +++ b/packages/claude-agents/tests/test_handler.py @@ -903,6 +903,31 @@ async def test_content_present_when_enabled( assert "gen_ai.output.messages" in chat.attributes assert "gen_ai.system_instructions" in root().attributes + async def test_process_default_is_read_on_the_streaming_path( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + from launchdarkly_ai_server import SDKOptions, configure + from launchdarkly_ai_server.lifecycle import _reset_for_testing + + _reset_for_testing() + try: + configure(options=SDKOptions(capture_content=True)) + monkeypatch.setattr( + handler_mod, + "query", + _fake_query( + [stream_event("Hi"), assistant_message(), result_message("Hi")] + ), + ) + await _collect( + await create_claude_agents_handler().stream(BASE_CONFIG, "q", {}, {}) + ) + assert any( + "gen_ai.input.messages" in (span.attributes or {}) for span in spans() + ) + finally: + _reset_for_testing() + # --------------------------------------------------------------------------- # Errors, and the failure path — section 6 diff --git a/packages/claude-messages/agents.md b/packages/claude-messages/agents.md index 95705470..6ca5d3c5 100644 --- a/packages/claude-messages/agents.md +++ b/packages/claude-messages/agents.md @@ -2,7 +2,7 @@ > **Span shape.** This package emits `invoke_agent` → `chat {model}` → `execute_tool {name}`, with > tool spans as siblings of `chat`. Span construction lives in `spans.py` beside the handler. -> Conversation content is off unless the caller passes `capture_content=True`. +> Conversation content is off unless the caller passes `capture_content=True`, or the process sets it. See [Process-wide defaults](../client/README.md#process-wide-defaults). > `TELEMETRY-CONTRACT.md` at the repo root is the authority; read it before changing span code. diff --git a/packages/claude-messages/pyproject.toml b/packages/claude-messages/pyproject.toml index 9ae07831..b599231e 100644 --- a/packages/claude-messages/pyproject.toml +++ b/packages/claude-messages/pyproject.toml @@ -3,7 +3,7 @@ name = "launchdarkly-ai-claude-messages" version = "0.2.4" requires-python = ">=3.12" dependencies = [ - "launchdarkly-ai-server", + "launchdarkly-ai-server>=0.3.1", "opentelemetry-api>=1.25", "anthropic>=0.40", ] diff --git a/packages/claude-messages/src/launchdarkly_ai_claude_messages/handler.py b/packages/claude-messages/src/launchdarkly_ai_claude_messages/handler.py index 9d01c14f..391ca5df 100644 --- a/packages/claude-messages/src/launchdarkly_ai_claude_messages/handler.py +++ b/packages/claude-messages/src/launchdarkly_ai_claude_messages/handler.py @@ -25,6 +25,7 @@ set_tool_call_content_attributes, to_semconv_finish_reason, ) +from launchdarkly_ai_server.lifecycle import effective_capture_content from ._version import PACKAGE_NAME, __version__ from .spans import ( @@ -351,14 +352,18 @@ def _is_coroutine(fn: Any) -> bool: _MAX_STEPS = 10 -def create_claude_messages_handler(*, capture_content: bool = False) -> ProviderHandler: +def create_claude_messages_handler( + *, capture_content: bool | None = None +) -> ProviderHandler: """Creates a ``ProviderHandler`` for Anthropic Claude (messages API). Requires ``anthropic`` to be installed as a peer dependency. Set *capture_content* to put prompts, model output, tool arguments and tool results on the - emitted spans. It defaults to off. Conversation content is PII, so a run emits only metadata, - meaning models, token counts, timings and tool names, until a caller asks for more. Turning this + emitted spans. ``None`` (the default) reads the process-wide default at invocation, + which is off until ``configure`` sets it. ``True`` or ``False`` overrides that default + for this handler. Conversation content is PII, so a run emits only metadata, meaning + models, token counts, timings and tool names, until capture is turned on. Turning this on sends the text of every request and response to whatever collector the SDK is pointed at. """ report_usage( @@ -368,9 +373,10 @@ def create_claude_messages_handler(*, capture_content: bool = False) -> Provider def _create_claude_messages_handler( - *, capture_content: bool = False + *, capture_content: bool | None = None ) -> ProviderHandler: """Non-reporting :func:`create_claude_messages_handler`, used by this package's wrappers.""" + content_override = capture_content import importlib anthropic_mod = importlib.import_module("anthropic") @@ -383,6 +389,8 @@ async def _call_impl( variables: dict[str, Any] | None = None, history: list[dict[str, Any]] | None = None, ) -> dict[str, Any]: + # None follows configure(); an explicit bool overrides it. Read when the call runs. + capture_content = effective_capture_content(content_override) th = tool_handlers or {} vs = variables or {} @@ -466,14 +474,14 @@ def _stream_impl( tool_handlers or {}, variables or {}, history, - capture_content=capture_content, + capture_content=effective_capture_content(content_override), ) return _create_handler( ("Anthropic", "messages"), _call_impl, # type: ignore[arg-type] _stream_impl, # type: ignore[arg-type] - capture_content=capture_content, + capture_content=content_override, ) @@ -722,7 +730,7 @@ def claude_messages( # asking for content on spans got a TypeError instead of content. report_usage("claude-messages.claudeMessages", PACKAGE_NAME, __version__) variables = kwargs.pop("variables", None) - capture_content = kwargs.pop("capture_content", False) + capture_content = kwargs.pop("capture_content", None) return _config( key=config_key, handler=_create_claude_messages_handler(capture_content=capture_content), diff --git a/packages/claude-messages/tests/test_handler.py b/packages/claude-messages/tests/test_handler.py index 0a0353d9..e107a7a5 100644 --- a/packages/claude-messages/tests/test_handler.py +++ b/packages/claude-messages/tests/test_handler.py @@ -1653,6 +1653,28 @@ async def test_emits_no_content_by_default_on_the_streaming_path( assert [k for k in span.attributes if k.startswith("gen_ai.prompt")] == [] assert [n for n, _ in span.events if n.startswith("gen_ai.content")] == [] + async def test_process_default_is_read_on_the_streaming_path( + self, mock_anthropic: MagicMock + ) -> None: + from launchdarkly_ai_server import SDKOptions, configure + from launchdarkly_ai_server.lifecycle import _reset_for_testing + + _reset_for_testing() + try: + configure(options=SDKOptions(capture_content=True)) + self._patch_stream(mock_anthropic, ["hi"]) + ctx, rec = _recording() + from launchdarkly_ai_claude_messages import create_claude_messages_handler + + with ctx: + async for _ in await create_claude_messages_handler().stream( + CONFIG, "q" + ): + pass + assert any("gen_ai.input.messages" in span.attributes for span in rec.spans) + finally: + _reset_for_testing() + # --------------------------------------------------------------------------- # History parameter @@ -1824,8 +1846,11 @@ def test_capture_content_reaches_the_factory(self) -> None: # And it must not have been forwarded to config(), which does not accept it. assert "capture_content" not in seen["config_kwargs"] - def test_defaults_to_off(self) -> None: - assert self._run()["capture_content"] is False + def test_omitted_capture_content_defers_to_the_process_default(self) -> None: + assert self._run()["capture_content"] is None + + def test_explicit_false_reaches_the_factory(self) -> None: + assert self._run(capture_content=False)["capture_content"] is False class TestToolSpanNeverLeaks: diff --git a/packages/client/README.md b/packages/client/README.md index 73ba687d..e117c518 100644 --- a/packages/client/README.md +++ b/packages/client/README.md @@ -211,10 +211,11 @@ asyncio.run(main()) | Export | Description | |---|---| +| `configure(options=...)` | Set process-wide behavior defaults. Keyword-only. Separate from `init_client`. | | `init_client(options?)` | Auto-discover and initialize `launchdarkly-server-sdk`. Optional — the first AI API call triggers lazy init when `LD_SDK_KEY` is set. Returns `Awaitable[LDClientInterface]`. | | `init_client(client=...)` | **BYOC overload** — accept a pre-initialized `LDClientInterface`. Skips SDK auto-discovery. | | `get_client()` | Return the initialized `LDClientInterface`. Raises if `init_client` has not completed. | -| `shutdown()` | Flush all events and telemetry, then close the client. Await before process exit. | +| `shutdown()` | Flush all events and telemetry, then close the client. Await before process exit. Restores the built-in SDK defaults. | | `inspect_config(key, context)` | Read an AI Config variation without invoking the model. Never raises. Returns `{"enabled", "config", "meta"}`. | `init_client` reads these option keys. Each overrides the environment variable of the same @@ -231,6 +232,22 @@ warning and ignored. | `serviceName` | `LD_SERVICE_NAME` | OTel `service.name` resource attribute | | `environment` | `LD_ENVIRONMENT` | `deployment.environment` resource attribute | +### Process-wide defaults + +`configure` sets behavior for the process. It does not initialize the LaunchDarkly client; call `init_client` for that. Handlers built without an explicit `capture_content` read this value when they run, so one startup call covers every handler. An explicit `True` or `False` on a handler factory or convenience wrapper overrides it. Call `configure` from process startup, not from request data. It can be replaced until the first AI invocation; after that only the same options are accepted. `shutdown()` restores the built-in default, which is off. + +```python +from launchdarkly_ai_server import SDKOptions, configure, init_client, shutdown + +async def main(): + configure(options=SDKOptions(capture_content=True)) + await init_client({"sdkKey": "sdk-...", "serviceName": "my-service"}) + # create_*_handler() with no capture_content argument follows the default above. + await shutdown() +``` + +`SDKOptions` is the extension point for further process-wide defaults. + ### Give the evaluation tools Pass `tools` to `run()` as a list of `EvalTool`. Construct one to define a tool in code. Await `evals.tools.get()` to use a tool that already exists in your project's AI library, which reads the tool and pins its version at that point. One list can hold both kinds. diff --git a/packages/client/agents.md b/packages/client/agents.md index 1ca5ae3a..c5190059 100644 --- a/packages/client/agents.md +++ b/packages/client/agents.md @@ -867,6 +867,7 @@ When `enabled` is `False`, `config` is always `None`. When `enabled` is `True` b ## Lifecycle Invariants +- **Process defaults.** `configure(options=SDKOptions(capture_content=True))` sets behavior defaults for the process. Call it once at startup, not from request data, and not via `init_client`. Handlers omit `capture_content` to follow it. `shutdown()` restores the built-in defaults. - **Lazy initialization.** Importing the package does not initialize the LD client. The first API call that needs LaunchDarkly calls `init_client()` internally when `LD_SDK_KEY` is set. - **Explicit initialization — SDK path.** `await init_client(options?)` dynamically imports `launchdarkly-server-sdk` at runtime (optional peer dep). If the package is not installed it raises with a clear message. - **Explicit initialization — BYOC path.** `await init_client(client)` accepts any pre-initialized object that satisfies `LDClientInterface` — this is the path for custom or edge environments whose SDK has different init semantics. diff --git a/packages/client/src/launchdarkly_ai_server/__init__.py b/packages/client/src/launchdarkly_ai_server/__init__.py index c0117b80..8726e950 100644 --- a/packages/client/src/launchdarkly_ai_server/__init__.py +++ b/packages/client/src/launchdarkly_ai_server/__init__.py @@ -52,6 +52,7 @@ ) from .judges import build_judge_tasks, run_judge, run_judges from .lifecycle import ( + configure, extract_variation, get_client, init_client, @@ -97,6 +98,7 @@ ProviderGraphResponse, ProviderHandler, ProviderResponse, + SDKOptions, StreamChunkEvent, StreamDoneEvent, StreamEvent, @@ -148,6 +150,7 @@ "HandlerResult", "HandlerStreamEvent", "InitClientOptions", + "SDKOptions", "JudgeResult", "JudgeRunResult", "JudgeTask", @@ -237,6 +240,7 @@ "execute_and_track", "execute_and_stream", # lifecycle + "configure", "init_client", "get_client", "shutdown", diff --git a/packages/client/src/launchdarkly_ai_server/content.py b/packages/client/src/launchdarkly_ai_server/content.py index 99eaf1c9..1c51bf34 100644 --- a/packages/client/src/launchdarkly_ai_server/content.py +++ b/packages/client/src/launchdarkly_ai_server/content.py @@ -7,10 +7,11 @@ canonical shape, so a ``gen_ai.content.prompt`` event, which is what these handlers emitted before, is read by nothing on the LaunchDarkly side. -**Off by default.** Everything here is conversation content, which is PII. A handler opts in with -``capture_content=True``; every function below takes that decision as its ``capture`` argument and -returns without writing when it is false. Passing the flag rather than reading ambient state keeps -the gate visible at each call site. +**Off by default.** Everything here is conversation content, which is PII. Handlers resolve +``capture_content`` to a bool before calling in: an explicit ``True`` or ``False``, otherwise the +process default from ``configure``, otherwise off. Every function below takes that decision as its +``capture`` argument and returns without writing when it is false. Passing the flag rather than +reading ambient state keeps the gate visible at each call site. Handlers also test the same flag themselves before building the argument, so the check appears twice on purpose: the guard here is the safety net that makes a forgotten call site harmless, and diff --git a/packages/client/src/launchdarkly_ai_server/lifecycle.py b/packages/client/src/launchdarkly_ai_server/lifecycle.py index 0cb5e6a3..5d4bbadc 100644 --- a/packages/client/src/launchdarkly_ai_server/lifecycle.py +++ b/packages/client/src/launchdarkly_ai_server/lifecycle.py @@ -9,7 +9,7 @@ from . import skills from .sdk_info import flush_ai_sdk_info, reset_ai_sdk_info from .sdk_usage import flush_sdk_usage, report_usage, reset_sdk_usage -from .types import InitClientOptions +from .types import InitClientOptions, SDKOptions from .utils import model_stamps_from_meta logger = logging.getLogger(__name__) @@ -43,6 +43,56 @@ def _env(name: str) -> str | None: # is once-guarded, so when another library registered first ours is refused and # the global stays theirs. Only the owner may release it on shutdown. _owns_otel_globals: bool = False +# Process-wide behavior defaults. Separate from ``_client``: ``init_client`` only +# builds the LaunchDarkly client and telemetry, and ``configure`` can run before +# that exists. ``_ai_invoked`` latches on the first handler call so a later +# ``configure`` cannot change what an in-flight process already started using. +_sdk_options: SDKOptions = SDKOptions() +_ai_invoked: bool = False + + +def configure(*, options: SDKOptions) -> None: + """Sets process-wide SDK behavior defaults. + + Call once from process startup, before the first AI invocation, with an + :class:`SDKOptions` value the process owns. Do not build it from request + data. Handlers constructed without an explicit ``capture_content`` read the + configured value when they run, including handlers built before this call. + + A later call may replace the options until the first AI invocation. After + that, only an identical ``options`` value is accepted. ``shutdown()`` + restores the built-in defaults and allows configuration again. + + ``init_client`` is unchanged: it only initializes the LaunchDarkly client + and telemetry. + """ + global _sdk_options + if not isinstance(options, SDKOptions): + raise TypeError("configure() options must be an SDKOptions") + if _ai_invoked and options != _sdk_options: + raise RuntimeError( + "SDK options cannot change after the first AI invocation. " + "shutdown() restores the built-in defaults." + ) + _sdk_options = options + + +def effective_capture_content(override: bool | None) -> bool: + """Resolves content capture for one invocation. + + An explicit ``True`` or ``False`` — from a convenience wrapper, then from + the handler factory — wins. ``None`` reads the configured default, which + itself defaults to ``False``. + """ + if override is not None: + return override + return _sdk_options.capture_content + + +def note_ai_invocation() -> None: + """Latches the process so ``configure`` can no longer change options.""" + global _ai_invoked + _ai_invoked = True def get_client() -> Any: @@ -318,13 +368,13 @@ async def shutdown() -> None: even if the client was never initialized or already shut down. Also clears any experimental feature state, such as a configured skill - store. + store, and restores process-wide SDK defaults from ``configure``. When telemetry was running, this also releases the process-global tracer provider so a later ``init_client`` can install its own — see ``_release_otel_globals``. """ - global _client, _tracer_provider, _owns_otel_globals + global _client, _tracer_provider, _owns_otel_globals, _sdk_options, _ai_invoked local_client = _client local_provider = _tracer_provider @@ -332,10 +382,14 @@ async def shutdown() -> None: _clear_experimental_state() - # Null the singleton before any awaits so a second call is a no-op + # Null the singleton before any awaits so a second call is a no-op. + # Options reset even when no client was initialized: configure() does not + # require one, and shutdown is what puts the process back to built-in defaults. _client = None _tracer_provider = None _owns_otel_globals = False + _sdk_options = SDKOptions() + _ai_invoked = False reset_ai_sdk_info() reset_sdk_usage() @@ -373,11 +427,13 @@ def _set_client_for_testing(c: Any) -> None: def _reset_for_testing() -> None: """Test helper — clear all singleton state.""" - global _client, _tracer_provider, _owns_otel_globals + global _client, _tracer_provider, _owns_otel_globals, _sdk_options, _ai_invoked owned_globals = _owns_otel_globals _client = None _tracer_provider = None _owns_otel_globals = False + _sdk_options = SDKOptions() + _ai_invoked = False # Directly, not through _clear_experimental_state: a failed reset should # fail the test rather than leak state into the next one. skills._clear_state() diff --git a/packages/client/src/launchdarkly_ai_server/types.py b/packages/client/src/launchdarkly_ai_server/types.py index 44c876fc..1a7f6819 100644 --- a/packages/client/src/launchdarkly_ai_server/types.py +++ b/packages/client/src/launchdarkly_ai_server/types.py @@ -151,27 +151,43 @@ class ProviderHandler: - ``__call__`` — blocking invocation - ``stream`` — optional async-generator streaming (may be ``None``) - ``provides_for`` — ``(provider_name, mode)`` tuple or ``None`` - - ``capture_content`` — whether this handler was built with content capture on + - ``capture_content`` — the effective content-capture decision for this handler ``capture_content`` is declared here so the client core can apply the handler's own content decision to content it writes on the handler's behalf — notably the judge's reasoning — - without reaching into the factory's closure. + without reaching into the factory's closure. ``None`` at construction reads the process + default from ``configure`` when the value is used. An explicit ``True`` or ``False`` is an + override. The property returns that effective bool, so judge and span paths share one gate. """ provides_for: tuple[str, Literal["agent", "messages"]] | None - capture_content: bool def __init__( self, fn: _HandlerFn, provides_for: tuple[str, Literal["agent", "messages"]] | None = None, stream_fn: _StreamFn | None = None, - capture_content: bool = False, + capture_content: bool | None = None, ) -> None: self._fn = fn self.provides_for = provides_for self._stream_fn = stream_fn - self.capture_content = capture_content + self._capture_content_override = capture_content + + @property + def capture_content(self) -> bool: + """Effective content capture: an explicit override, else the process default. + + Assigning ``True`` or ``False`` sets this handler's override, the same as + passing it to the factory. It does not change the process default. + """ + from .lifecycle import effective_capture_content + + return effective_capture_content(self._capture_content_override) + + @capture_content.setter + def capture_content(self, value: bool) -> None: + self._capture_content_override = value async def __call__( self, @@ -181,6 +197,9 @@ async def __call__( variables: dict[str, Any] | None = None, history: list[dict[str, Any]] | None = None, ) -> HandlerResult: + from .lifecycle import note_ai_invocation + + note_ai_invocation() return await self._fn(config, user_input, tool_handlers, variables, history) async def stream( @@ -193,6 +212,9 @@ async def stream( ) -> AsyncGenerator[dict[str, Any], None]: if self._stream_fn is None: raise NotImplementedError("This handler does not support streaming") + from .lifecycle import note_ai_invocation + + note_ai_invocation() return self._stream_fn(config, user_input, tool_handlers, variables, history) @property @@ -577,6 +599,26 @@ def errors(self) -> list[ReconcileAction]: RoutedModelArgs = dict[str, Any] GraphOptions = dict[str, Any] GraphArgs = dict[str, Any] + + +@dataclass(frozen=True) +class SDKOptions: + """Process-wide behavior defaults, applied by :func:`configure`. + + ``capture_content`` is the default for writing prompts, model output, tool + arguments, tool results, and judge explanations. It is a default, not a + ceiling: an explicit ``True`` or ``False`` on a handler factory or a + convenience wrapper overrides it. Further process-wide defaults are added + as fields on this object. + """ + + capture_content: bool = False + + def __post_init__(self) -> None: + if not isinstance(self.capture_content, bool): + raise TypeError("SDKOptions.capture_content must be a bool") + + InitClientOptions = dict[str, Any] # --------------------------------------------------------------------------- diff --git a/packages/client/src/launchdarkly_ai_server/utils.py b/packages/client/src/launchdarkly_ai_server/utils.py index 7c656abc..9ae5ca57 100644 --- a/packages/client/src/launchdarkly_ai_server/utils.py +++ b/packages/client/src/launchdarkly_ai_server/utils.py @@ -23,11 +23,15 @@ def create_handler( provides_for: tuple[str, Literal["agent", "messages"]], fn: _HandlerFn, stream_fn: _StreamFn | None = None, - capture_content: bool = False, + capture_content: bool | None = None, ) -> ProviderHandler: """ Wraps a plain async callable in a :class:`ProviderHandler` with the given ``provides_for`` metadata and optional streaming implementation. + + ``capture_content`` is this handler's override. ``None`` reads the process + default from :func:`configure` when the value is used. ``True`` or ``False`` + overrides that default. """ report_usage("client.createHandler") return _create_handler(provides_for, fn, stream_fn, capture_content) @@ -37,7 +41,7 @@ def _create_handler( provides_for: tuple[str, Literal["agent", "messages"]], fn: _HandlerFn, stream_fn: _StreamFn | None = None, - capture_content: bool = False, + capture_content: bool | None = None, ) -> ProviderHandler: """Non-reporting :func:`create_handler`, used by the package factories.""" return ProviderHandler( diff --git a/packages/client/tests/test_configure.py b/packages/client/tests/test_configure.py new file mode 100644 index 00000000..53f0fe5b --- /dev/null +++ b/packages/client/tests/test_configure.py @@ -0,0 +1,248 @@ +"""Process-wide ``configure`` defaults. + +``init_client`` stays the LaunchDarkly client and telemetry initializer. +``SDKOptions.capture_content`` is a default, not a ceiling: an explicit handler +override wins, including an explicit ``False`` over a configured ``True``. +""" + +from __future__ import annotations + +from dataclasses import FrozenInstanceError +from typing import Any +from unittest.mock import AsyncMock, MagicMock, patch + +import pytest + +import launchdarkly_ai_server.lifecycle as lifecycle_module +from launchdarkly_ai_server import ( + SDKOptions, + configure, + init_client, + shutdown, +) +from launchdarkly_ai_server.lifecycle import _reset_for_testing +from launchdarkly_ai_server.types import ProviderHandler + + +@pytest.fixture(autouse=True) +def reset_singleton() -> None: + _reset_for_testing() + yield + _reset_for_testing() + + +async def _fn( + config: Any, + user_input: str | None = None, + tool_handlers: dict[str, Any] | None = None, + variables: dict[str, Any] | None = None, + history: list[dict[str, Any]] | None = None, +) -> dict[str, Any]: + return {"output": "ok", "usage": {}} + + +def _handler(capture_content: bool | None = None) -> ProviderHandler: + return ProviderHandler(_fn, capture_content=capture_content) + + +def test_builtin_default_is_off() -> None: + assert _handler().capture_content is False + + +def test_configure_is_read_after_the_handler_is_built() -> None: + handler = _handler() + configure(options=SDKOptions(capture_content=True)) + assert handler.capture_content is True + + +def test_configure_can_be_replaced_before_an_invocation() -> None: + configure(options=SDKOptions(capture_content=True)) + configure(options=SDKOptions(capture_content=False)) + assert _handler().capture_content is False + + +def test_explicit_false_overrides_a_configured_true() -> None: + configure(options=SDKOptions(capture_content=True)) + assert _handler(False).capture_content is False + assert _handler(True).capture_content is True + assert _handler(None).capture_content is True + + +def test_assigning_capture_content_sets_the_override() -> None: + configure(options=SDKOptions(capture_content=True)) + handler = _handler() + handler.capture_content = False + assert handler.capture_content is False + handler.capture_content = True + assert handler.capture_content is True + + +def test_configure_is_keyword_only() -> None: + with pytest.raises(TypeError): + configure(SDKOptions()) # type: ignore[misc] + + +def test_configure_rejects_a_non_options_value() -> None: + with pytest.raises(TypeError, match="SDKOptions"): + configure(options={"capture_content": True}) # type: ignore[arg-type] + + +def test_sdk_options_rejects_a_non_bool() -> None: + with pytest.raises(TypeError, match="bool"): + SDKOptions(capture_content=1) # type: ignore[arg-type] + + +def test_sdk_options_is_frozen() -> None: + options = SDKOptions() + with pytest.raises(FrozenInstanceError): + options.capture_content = True # type: ignore[misc] + + +async def test_a_changed_configuration_is_rejected_after_invocation() -> None: + await _handler()({}) + with pytest.raises(RuntimeError, match="cannot change"): + configure(options=SDKOptions(capture_content=True)) + + +async def test_an_identical_configuration_is_allowed_after_invocation() -> None: + configure(options=SDKOptions(capture_content=True)) + await _handler()({}) + configure(options=SDKOptions(capture_content=True)) + assert _handler().capture_content is True + + +async def test_streaming_latches_invocation_before_the_body_runs() -> None: + async def _stream(*_args: Any, **_kwargs: Any) -> Any: + yield {"type": "done", "output": "", "usage": {}} + + handler = ProviderHandler(_fn, stream_fn=_stream) + gen = await handler.stream({}) + await gen.aclose() + with pytest.raises(RuntimeError, match="cannot change"): + configure(options=SDKOptions(capture_content=True)) + + +async def test_a_missing_stream_does_not_latch() -> None: + with pytest.raises(NotImplementedError): + await _handler().stream({}) + configure(options=SDKOptions(capture_content=True)) + assert _handler().capture_content is True + + +async def test_shutdown_restores_the_builtin_default() -> None: + configure(options=SDKOptions(capture_content=True)) + await _handler()({}) + await shutdown() + assert _handler().capture_content is False + configure(options=SDKOptions(capture_content=True)) + assert _handler().capture_content is True + + +async def test_run_judges_writes_the_explanation_from_the_configured_default() -> None: + """The judge explanation follows the handler's effective capture flag. + + Goes through ``run_judges``, not a copy of the ``if capture_content`` + expression, and reads ``gen_ai.evaluation.explanation`` off the span. + """ + import random + + from opentelemetry.sdk.trace import TracerProvider + from opentelemetry.sdk.trace.export import SimpleSpanProcessor + from opentelemetry.sdk.trace.export.in_memory_span_exporter import ( + InMemorySpanExporter, + ) + + from launchdarkly_ai_server import run_judges + from launchdarkly_ai_server.conversation import ConversationIdSpanProcessor + + exporter = InMemorySpanExporter() + provider = TracerProvider() + provider.add_span_processor(ConversationIdSpanProcessor()) + provider.add_span_processor(SimpleSpanProcessor(exporter)) + tracer = provider.get_tracer("@launchdarkly/ai-server") + + async def fn( + config: Any, + user_input: str | None = None, + tool_handlers: dict[str, Any] | None = None, + variables: dict[str, Any] | None = None, + history: list[dict[str, Any]] | None = None, + ) -> dict[str, Any]: + span = tracer.start_span("invoke_agent") + span.end() + return { + "output": '{"score": 0.9, "reasoning": "good"}', + "usage": {"input_tokens": 1, "output_tokens": 1}, + } + + client = MagicMock() + client.track = MagicMock() + client.variation = AsyncMock( + return_value={ + "model": {"name": "gpt-4"}, + "provider": {"name": "TestProvider"}, + "instructions": "judge", + "_ldMeta": { + "enabled": True, + "variationKey": "j1", + "version": 1, + "mode": "messages", + }, + } + ) + lifecycle_module._set_client_for_testing(client) + configure(options=SDKOptions(capture_content=True)) + parent = { + "model": {"name": "gpt-4"}, + "provider": {"name": "TestProvider"}, + "instructions": "hi", + "judgeConfiguration": {"judges": [{"key": "judge-1", "samplingRate": 1.0}]}, + } + + def explanation() -> str | None: + spans = [ + span + for span in exporter.get_finished_spans() + if span.name == "invoke_agent" + ] + assert spans, "judge invoke_agent span was not exported" + value = (spans[-1].attributes or {}).get("gen_ai.evaluation.explanation") + return value if isinstance(value, str) else None + + with patch.object(random, "random", return_value=0.0): + await run_judges( + config=parent, + user_context={"kind": "user", "key": "u1"}, + handler=ProviderHandler(fn, provides_for=("TestProvider", "messages")), + user_input="q", + llm_response="r", + base_track_data={"runId": "x"}, + ) + assert explanation() == "good" + + exporter.clear() + with patch.object(random, "random", return_value=0.0): + await run_judges( + config=parent, + user_context={"kind": "user", "key": "u1"}, + handler=ProviderHandler( + fn, provides_for=("TestProvider", "messages"), capture_content=False + ), + user_input="q", + llm_response="r", + base_track_data={"runId": "x"}, + ) + assert explanation() is None + + +async def test_init_client_does_not_carry_behavior_defaults( + caplog: pytest.LogCaptureFixture, +) -> None: + stub = MagicMock() + with ( + patch.object(lifecycle_module, "_setup_telemetry", return_value=None), + caplog.at_level("WARNING", logger="launchdarkly_ai_server.lifecycle"), + ): + await init_client({"capture_content": True}, stub) + assert _handler().capture_content is False + assert any("capture_content" in record.getMessage() for record in caplog.records) diff --git a/packages/client/tests/test_public_surface.py b/packages/client/tests/test_public_surface.py index 043e0f3e..06bcbf4e 100644 --- a/packages/client/tests/test_public_surface.py +++ b/packages/client/tests/test_public_surface.py @@ -35,6 +35,7 @@ "HandlerResult", "HandlerStreamEvent", "InitClientOptions", + "SDKOptions", "InputTokenDetails", "Judge", "JudgeResult", @@ -71,6 +72,7 @@ "any_multimodal", "build_judge_tasks", "compose", + "configure", "compose_history", "config", "content_to_text", diff --git a/packages/langchain-agents/agents.md b/packages/langchain-agents/agents.md index e771dc13..2e4187ec 100644 --- a/packages/langchain-agents/agents.md +++ b/packages/langchain-agents/agents.md @@ -2,7 +2,7 @@ > **Span shape.** This package emits `invoke_agent` → `chat {model}` → `execute_tool {name}`, with > tool spans as siblings of `chat`. Span construction lives in `spans.py` beside the handler. -> Conversation content is off unless the caller passes `capture_content=True`. +> Conversation content is off unless the caller passes `capture_content=True`, or the process sets it. See [Process-wide defaults](../client/README.md#process-wide-defaults). > `TELEMETRY-CONTRACT.md` at the repo root is the authority; read it before changing span code. diff --git a/packages/langchain-agents/pyproject.toml b/packages/langchain-agents/pyproject.toml index db0f74ab..f0bef6e7 100644 --- a/packages/langchain-agents/pyproject.toml +++ b/packages/langchain-agents/pyproject.toml @@ -3,7 +3,7 @@ name = "launchdarkly-ai-langchain-agents" version = "0.2.5" requires-python = ">=3.12" dependencies = [ - "launchdarkly-ai-server", + "launchdarkly-ai-server>=0.3.1", "opentelemetry-api>=1.25", "langchain-core>=0.3", "langgraph>=0.2", diff --git a/packages/langchain-agents/src/launchdarkly_ai_langchain_agents/handler.py b/packages/langchain-agents/src/launchdarkly_ai_langchain_agents/handler.py index fde27d23..9bdab685 100644 --- a/packages/langchain-agents/src/launchdarkly_ai_langchain_agents/handler.py +++ b/packages/langchain-agents/src/launchdarkly_ai_langchain_agents/handler.py @@ -31,6 +31,7 @@ set_input_content_attributes, set_output_content_attributes, ) +from launchdarkly_ai_server.lifecycle import effective_capture_content from ._version import PACKAGE_NAME, __version__ from .messages import to_lang_chain_messages @@ -255,7 +256,7 @@ def _run_usage_from_messages(messages: list[Any]) -> Any: def create_langchain_agents_handler( - llm: Any = None, *, capture_content: bool = False + llm: Any = None, *, capture_content: bool | None = None ) -> ProviderHandler: """Creates a ``ProviderHandler`` for LangChain via ``create_react_agent``. @@ -263,8 +264,10 @@ def create_langchain_agents_handler( called after flag evaluation so ``model.parameters`` can be applied unchanged. Set *capture_content* to put prompts, model output, tool arguments and tool results on the - emitted spans. It defaults to off. Conversation content is PII, so a run emits only metadata, - meaning models, token counts, timings and tool names, until a caller asks for more. + emitted spans. ``None`` (the default) reads the process-wide default at invocation, + which is off until ``configure`` sets it. ``True`` or ``False`` overrides that default + for this handler. Conversation content is PII, so a run emits only metadata, meaning + models, token counts, timings and tool names, until capture is turned on. """ report_usage( "langchain-agents.createLangChainAgentsHandler", PACKAGE_NAME, __version__ @@ -273,9 +276,10 @@ def create_langchain_agents_handler( def _create_langchain_agents_handler( - llm: Any = None, *, capture_content: bool = False + llm: Any = None, *, capture_content: bool | None = None ) -> ProviderHandler: """Non-reporting :func:`create_langchain_agents_handler`, used by this package's wrappers.""" + content_override = capture_content async def _call_impl( config: AiConfigRep, @@ -284,6 +288,8 @@ async def _call_impl( variables: dict[str, Any] | None = None, history: list[dict[str, Any]] | None = None, ) -> dict[str, Any]: + # None follows configure(); an explicit bool overrides it. Read when the call runs. + capture_content = effective_capture_content(content_override) import importlib th = tool_handlers or {} @@ -420,14 +426,14 @@ def _stream_impl( tool_handlers or {}, variables or {}, history, - capture_content=capture_content, + capture_content=effective_capture_content(content_override), ) return _create_handler( ("*", "agent"), _call_impl, # type: ignore[arg-type] _stream_impl, # type: ignore[arg-type] - capture_content=capture_content, + capture_content=content_override, ) @@ -583,7 +589,7 @@ def langchain_agents( # asking for content on spans got a TypeError instead of content. report_usage("langchain-agents.langchainAgents", PACKAGE_NAME, __version__) variables = kwargs.pop("variables", None) - capture_content = kwargs.pop("capture_content", False) + capture_content = kwargs.pop("capture_content", None) return _config( key=config_key, handler=_create_langchain_agents_handler(capture_content=capture_content), diff --git a/packages/langchain-agents/tests/test_handler.py b/packages/langchain-agents/tests/test_handler.py index 4571a79b..15d59ef2 100644 --- a/packages/langchain-agents/tests/test_handler.py +++ b/packages/langchain-agents/tests/test_handler.py @@ -1573,6 +1573,25 @@ async def test_emits_no_content_by_default_on_the_streaming_path(self) -> None: assert [k for k in span.attributes if k.startswith("gen_ai.prompt")] == [] assert [n for n, _ in span.events if n.startswith("gen_ai.content")] == [] + @pytest.mark.asyncio + async def test_process_default_is_read_on_the_streaming_path(self) -> None: + from launchdarkly_ai_server import SDKOptions, configure + from launchdarkly_ai_server.lifecycle import _reset_for_testing + + _reset_for_testing() + try: + configure(options=SDKOptions(capture_content=True)) + ctx, rec = _recording() + llm = _FakeToolModel(replies=[_ai_message("hi")]) + with ctx: + async for _ in await create_langchain_agents_handler(llm).stream( + BASE_CONFIG, "q", {}, {} + ): + pass + assert any("gen_ai.input.messages" in span.attributes for span in rec.spans) + finally: + _reset_for_testing() + # --------------------------------------------------------------------------- # §1.9 Output format @@ -1849,8 +1868,11 @@ def test_capture_content_reaches_the_factory(self) -> None: # And it must not have been forwarded to config(), which does not accept it. assert "capture_content" not in seen["config_kwargs"] - def test_defaults_to_off(self) -> None: - assert self._run()["capture_content"] is False + def test_omitted_capture_content_defers_to_the_process_default(self) -> None: + assert self._run()["capture_content"] is None + + def test_explicit_false_reaches_the_factory(self) -> None: + assert self._run(capture_content=False)["capture_content"] is False class TestCallbackSpansNeverLeak: diff --git a/packages/langchain-messages/agents.md b/packages/langchain-messages/agents.md index 6361a8d3..6c889d95 100644 --- a/packages/langchain-messages/agents.md +++ b/packages/langchain-messages/agents.md @@ -2,7 +2,7 @@ > **Span shape.** This package emits `invoke_agent` → `chat {model}` → `execute_tool {name}`, with > tool spans as siblings of `chat`. Span construction lives in `spans.py` beside the handler. -> Conversation content is off unless the caller passes `capture_content=True`. +> Conversation content is off unless the caller passes `capture_content=True`, or the process sets it. See [Process-wide defaults](../client/README.md#process-wide-defaults). > `TELEMETRY-CONTRACT.md` at the repo root is the authority; read it before changing span code. diff --git a/packages/langchain-messages/pyproject.toml b/packages/langchain-messages/pyproject.toml index 90985088..198e7959 100644 --- a/packages/langchain-messages/pyproject.toml +++ b/packages/langchain-messages/pyproject.toml @@ -3,7 +3,7 @@ name = "launchdarkly-ai-langchain-messages" version = "0.2.4" requires-python = ">=3.12" dependencies = [ - "launchdarkly-ai-server", + "launchdarkly-ai-server>=0.3.1", "opentelemetry-api>=1.25", "langchain-core>=0.3", ] diff --git a/packages/langchain-messages/src/launchdarkly_ai_langchain_messages/handler.py b/packages/langchain-messages/src/launchdarkly_ai_langchain_messages/handler.py index d41876ba..95390dfa 100644 --- a/packages/langchain-messages/src/launchdarkly_ai_langchain_messages/handler.py +++ b/packages/langchain-messages/src/launchdarkly_ai_langchain_messages/handler.py @@ -32,6 +32,7 @@ set_output_content_attributes, set_tool_call_content_attributes, ) +from launchdarkly_ai_server.lifecycle import effective_capture_content from ._version import PACKAGE_NAME, __version__ from .spans import ( @@ -371,7 +372,7 @@ async def _run_structured_turn( def create_langchain_messages_handler( - llm: Any = None, *, capture_content: bool = False + llm: Any = None, *, capture_content: bool | None = None ) -> ProviderHandler: """ Creates a ``ProviderHandler`` for LangChain (chat models). @@ -382,17 +383,20 @@ def create_langchain_messages_handler( config's model name and parameters. Set *capture_content* to put prompts, model output, tool arguments and tool results on the - emitted spans. It defaults to off. Conversation content is PII, so a run emits only metadata, - meaning models, token counts, timings and tool names, until a caller asks for more. + emitted spans. ``None`` (the default) reads the process-wide default at invocation, + which is off until ``configure`` sets it. ``True`` or ``False`` overrides that default + for this handler. Conversation content is PII, so a run emits only metadata, meaning + models, token counts, timings and tool names, until capture is turned on. """ report_usage("langchain-messages.createLangChainHandler", PACKAGE_NAME, __version__) return _create_langchain_messages_handler(llm, capture_content=capture_content) def _create_langchain_messages_handler( - llm: Any = None, *, capture_content: bool = False + llm: Any = None, *, capture_content: bool | None = None ) -> ProviderHandler: """Non-reporting :func:`create_langchain_messages_handler`, used by this package's wrappers.""" + content_override = capture_content async def _call_impl( config: AiConfigRep, @@ -401,6 +405,8 @@ async def _call_impl( variables: dict[str, Any] | None = None, history: list[dict[str, Any]] | None = None, ) -> dict[str, Any]: + # None follows configure(); an explicit bool overrides it. Read when the call runs. + capture_content = effective_capture_content(content_override) import importlib th = tool_handlers or {} @@ -700,14 +706,14 @@ def _stream_impl( tool_handlers or {}, variables or {}, history, - capture_content=capture_content, + capture_content=effective_capture_content(content_override), ) return _create_handler( ("*", "messages"), _call_impl, # type: ignore[arg-type] _stream_impl, # type: ignore[arg-type] - capture_content=capture_content, + capture_content=content_override, ) @@ -1024,7 +1030,7 @@ def langchain_messages( # asking for content on spans got a TypeError instead of content. report_usage("langchain-messages.langchainMessages", PACKAGE_NAME, __version__) variables = kwargs.pop("variables", None) - capture_content = kwargs.pop("capture_content", False) + capture_content = kwargs.pop("capture_content", None) return _config( key=config_key, handler=_create_langchain_messages_handler( diff --git a/packages/langchain-messages/tests/test_handler.py b/packages/langchain-messages/tests/test_handler.py index 0b337c71..32768dff 100644 --- a/packages/langchain-messages/tests/test_handler.py +++ b/packages/langchain-messages/tests/test_handler.py @@ -1377,6 +1377,28 @@ async def test_emits_no_content_by_default_on_the_streaming_path(self) -> None: assert [k for k in span.attributes if k.startswith("gen_ai.prompt")] == [] assert [n for n, _ in span.events if n.startswith("gen_ai.content")] == [] + async def test_process_default_is_read_on_the_streaming_path(self) -> None: + from launchdarkly_ai_server import SDKOptions, configure + from launchdarkly_ai_server.lifecycle import _reset_for_testing + + _reset_for_testing() + try: + configure(options=SDKOptions(capture_content=True)) + ctx, rec = _recording() + from launchdarkly_ai_langchain_messages import ( + create_langchain_messages_handler, + ) + + llm = self._make_streaming_llm(["hi"]) + with ctx: + async for _ in await create_langchain_messages_handler(llm=llm).stream( + CONFIG, "q" + ): + pass + assert any("gen_ai.input.messages" in span.attributes for span in rec.spans) + finally: + _reset_for_testing() + # --------------------------------------------------------------------------- # §1.10 MAX_STEPS cap @@ -1655,8 +1677,11 @@ def test_capture_content_reaches_the_factory(self) -> None: # And it must not have been forwarded to config(), which does not accept it. assert "capture_content" not in seen["config_kwargs"] - def test_defaults_to_off(self) -> None: - assert self._run()["capture_content"] is False + def test_omitted_capture_content_defers_to_the_process_default(self) -> None: + assert self._run()["capture_content"] is None + + def test_explicit_false_reaches_the_factory(self) -> None: + assert self._run(capture_content=False)["capture_content"] is False class TestChatSpanAndTeardownNeverLeak: diff --git a/packages/openai-agents/agents.md b/packages/openai-agents/agents.md index 414dcc9f..3393c4a4 100644 --- a/packages/openai-agents/agents.md +++ b/packages/openai-agents/agents.md @@ -2,7 +2,7 @@ > **Span shape.** This package emits `invoke_agent` → `chat {model}` → `execute_tool {name}`, with > tool spans as siblings of `chat`. Span construction lives in `spans.py` beside the handler. -> Conversation content is off unless the caller passes `capture_content=True`. +> Conversation content is off unless the caller passes `capture_content=True`, or the process sets it. See [Process-wide defaults](../client/README.md#process-wide-defaults). > `TELEMETRY-CONTRACT.md` at the repo root is the authority; read it before changing span code. diff --git a/packages/openai-agents/pyproject.toml b/packages/openai-agents/pyproject.toml index 94af1952..24cf57f0 100644 --- a/packages/openai-agents/pyproject.toml +++ b/packages/openai-agents/pyproject.toml @@ -3,7 +3,7 @@ name = "launchdarkly-ai-openai-agents" version = "0.2.5" requires-python = ">=3.12" dependencies = [ - "launchdarkly-ai-server", + "launchdarkly-ai-server>=0.3.1", "opentelemetry-api>=1.25", "openai>=1.0", "openai-agents>=0.0.1", diff --git a/packages/openai-agents/src/launchdarkly_ai_openai_agents/handler.py b/packages/openai-agents/src/launchdarkly_ai_openai_agents/handler.py index 355aaab9..d840e019 100644 --- a/packages/openai-agents/src/launchdarkly_ai_openai_agents/handler.py +++ b/packages/openai-agents/src/launchdarkly_ai_openai_agents/handler.py @@ -45,6 +45,7 @@ set_tool_call_content_attributes, text_message, ) +from launchdarkly_ai_server.lifecycle import effective_capture_content from ._version import PACKAGE_NAME, __version__ from .spans import ( @@ -459,19 +460,26 @@ def _usage_from_error(error: BaseException) -> SpanUsage | None: return to_span_usage(usage) -def create_openai_agent_handler(*, capture_content: bool = False) -> ProviderHandler: +def create_openai_agent_handler( + *, capture_content: bool | None = None +) -> ProviderHandler: """Creates a ``ProviderHandler`` for OpenAI via the openai-agents SDK. Set *capture_content* to put prompts, model output, tool arguments and tool results on the - emitted spans. It defaults to off. Conversation content is PII, so a run emits only metadata, - meaning models, token counts, timings and tool names, until a caller asks for more. + emitted spans. ``None`` (the default) reads the process-wide default at invocation, + which is off until ``configure`` sets it. ``True`` or ``False`` overrides that default + for this handler. Conversation content is PII, so a run emits only metadata, meaning + models, token counts, timings and tool names, until capture is turned on. """ report_usage("openai-agents.createOpenAIAgentHandler", PACKAGE_NAME, __version__) return _create_openai_agent_handler(capture_content=capture_content) -def _create_openai_agent_handler(*, capture_content: bool = False) -> ProviderHandler: +def _create_openai_agent_handler( + *, capture_content: bool | None = None +) -> ProviderHandler: """Non-reporting :func:`create_openai_agent_handler`, used by this package's wrappers.""" + content_override = capture_content async def _call_impl( config: AiConfigRep, @@ -480,6 +488,8 @@ async def _call_impl( variables: dict[str, Any] | None = None, history: list[dict[str, Any]] | None = None, ) -> dict[str, Any]: + # None follows configure(); an explicit bool overrides it. Read when the call runs. + capture_content = effective_capture_content(content_override) import importlib agents_mod = importlib.import_module("agents") @@ -565,14 +575,14 @@ def _stream_impl( tool_handlers or {}, variables or {}, history, - capture_content=capture_content, + capture_content=effective_capture_content(content_override), ) return _create_handler( ("OpenAI", "agent"), _call_impl, # type: ignore[arg-type] _stream_impl, # type: ignore[arg-type] - capture_content=capture_content, + capture_content=content_override, ) @@ -722,7 +732,7 @@ def openai_agents( # asking for content on spans got a TypeError instead of content. report_usage("openai-agents.openaiAgents", PACKAGE_NAME, __version__) variables = kwargs.pop("variables", None) - capture_content = kwargs.pop("capture_content", False) + capture_content = kwargs.pop("capture_content", None) return _config( key=config_key, handler=_create_openai_agent_handler(capture_content=capture_content), diff --git a/packages/openai-agents/tests/test_handler.py b/packages/openai-agents/tests/test_handler.py index fe0d4af2..e9eb7f40 100644 --- a/packages/openai-agents/tests/test_handler.py +++ b/packages/openai-agents/tests/test_handler.py @@ -998,6 +998,68 @@ async def test_puts_prompt_and_completion_on_spans_when_enabled(self) -> None: prompt_written = json.dumps(rec.root.attributes) assert "my question" in prompt_written + async def test_process_default_is_read_when_the_handler_runs(self) -> None: + from launchdarkly_ai_server import SDKOptions, configure + from launchdarkly_ai_server.lifecycle import _reset_for_testing + + _reset_for_testing() + try: + # Built before configure, with no capture flag. Vega's startup call + # has to reach this handler when it later runs. + handler = create_openai_agent_handler() + configure(options=SDKOptions(capture_content=True)) + ctx, rec = _recording() + agents_mod = _fake_agents_module( + run=_make_run([{"output": _text_output("answer")}]) + ) + with ctx, _patched_agents(agents_mod): + await handler(CONFIG, "my question", {}, {}) + assert "my question" in json.dumps(rec.root.attributes) + finally: + _reset_for_testing() + + async def test_explicit_false_overrides_a_process_default_of_on(self) -> None: + from launchdarkly_ai_server import SDKOptions, configure + from launchdarkly_ai_server.lifecycle import _reset_for_testing + + _reset_for_testing() + try: + configure(options=SDKOptions(capture_content=True)) + ctx, rec = _recording() + agents_mod = _fake_agents_module( + run=_make_run([{"output": _text_output("answer")}]) + ) + with ctx, _patched_agents(agents_mod): + await create_openai_agent_handler(capture_content=False)( + CONFIG, "my question", {}, {} + ) + for span in rec.spans: + assert "gen_ai.input.messages" not in span.attributes + assert "my question" not in json.dumps(span.attributes) + finally: + _reset_for_testing() + + async def test_process_default_is_read_on_the_streaming_path(self) -> None: + from launchdarkly_ai_server import SDKOptions, configure + from launchdarkly_ai_server.lifecycle import _reset_for_testing + + _reset_for_testing() + try: + configure(options=SDKOptions(capture_content=True)) + turns = [{"deltas": ["hi"], "output": _text_output("hi")}] + ctx, rec = _recording() + agents_mod = _fake_agents_module( + run_streamed=_make_run_streamed(turns, "hi") + ) + with ctx, _patched_agents(agents_mod): + async for _ in await create_openai_agent_handler().stream( + CONFIG, "stream question", {}, {} + ): + pass + assert "stream question" in json.dumps(rec.root.attributes) + finally: + _reset_for_testing() + async def test_tool_call_arguments_and_result_gated(self) -> None: turns = [ { @@ -1700,7 +1762,7 @@ def _factory(*, capture_content: bool = False) -> Any: # And it must not have been forwarded to config(), which does not accept it. assert "capture_content" not in fake_config.call_args.kwargs - async def test_defaults_to_off(self) -> None: + async def test_omitted_capture_content_defers_to_the_process_default(self) -> None: import launchdarkly_ai_openai_agents.handler as handler_mod seen: dict[str, Any] = {} @@ -1717,6 +1779,25 @@ def _factory(*, capture_content: bool = False) -> Any: ): handler_mod.openai_agents("k", "q", {}) + assert seen["capture_content"] is None + + async def test_explicit_false_reaches_the_factory(self) -> None: + import launchdarkly_ai_openai_agents.handler as handler_mod + + seen: dict[str, Any] = {} + + def _factory(*, capture_content: bool | None = None) -> Any: + seen["capture_content"] = capture_content + return MagicMock() + + fake_config = MagicMock() + fake_config.return_value.invoke = MagicMock(return_value="ok") + with ( + patch.object(handler_mod, "_create_openai_agent_handler", _factory), + patch.object(handler_mod, "_config", fake_config), + ): + handler_mod.openai_agents("k", "q", {}, capture_content=False) + assert seen["capture_content"] is False diff --git a/packages/openai-messages/agents.md b/packages/openai-messages/agents.md index 5f29cae4..3057b39c 100644 --- a/packages/openai-messages/agents.md +++ b/packages/openai-messages/agents.md @@ -2,7 +2,7 @@ > **Span shape.** This package emits `invoke_agent` → `chat {model}` → `execute_tool {name}`, with > tool spans as siblings of `chat`. Span construction lives in `spans.py` beside the handler. -> Conversation content is off unless the caller passes `capture_content=True`. +> Conversation content is off unless the caller passes `capture_content=True`, or the process sets it. See [Process-wide defaults](../client/README.md#process-wide-defaults). > `TELEMETRY-CONTRACT.md` at the repo root is the authority; read it before changing span code. diff --git a/packages/openai-messages/pyproject.toml b/packages/openai-messages/pyproject.toml index 4a3b91df..a0a3824b 100644 --- a/packages/openai-messages/pyproject.toml +++ b/packages/openai-messages/pyproject.toml @@ -3,7 +3,7 @@ name = "launchdarkly-ai-openai-messages" version = "0.2.4" requires-python = ">=3.12" dependencies = [ - "launchdarkly-ai-server", + "launchdarkly-ai-server>=0.3.1", "opentelemetry-api>=1.25", "openai>=1.0", ] diff --git a/packages/openai-messages/src/launchdarkly_ai_openai_messages/handler.py b/packages/openai-messages/src/launchdarkly_ai_openai_messages/handler.py index 6afc2b49..9d59a2d6 100644 --- a/packages/openai-messages/src/launchdarkly_ai_openai_messages/handler.py +++ b/packages/openai-messages/src/launchdarkly_ai_openai_messages/handler.py @@ -27,6 +27,7 @@ set_output_content_attributes, set_tool_call_content_attributes, ) +from launchdarkly_ai_server.lifecycle import effective_capture_content from ._version import PACKAGE_NAME, __version__ from .spans import ( @@ -212,23 +213,28 @@ async def _run_model_turn( return response -def create_openai_messages_handler(*, capture_content: bool = False) -> ProviderHandler: +def create_openai_messages_handler( + *, capture_content: bool | None = None +) -> ProviderHandler: """ Creates a ``ProviderHandler`` for OpenAI (responses API). Requires ``openai`` to be installed as a peer dependency. Set *capture_content* to put prompts, model output, tool arguments and tool results on the - emitted spans. It defaults to off. Conversation content is PII, so a run emits only metadata, - meaning models, token counts, timings and tool names, until a caller asks for more. + emitted spans. ``None`` (the default) reads the process-wide default at invocation, + which is off until ``configure`` sets it. ``True`` or ``False`` overrides that default + for this handler. Conversation content is PII, so a run emits only metadata, meaning + models, token counts, timings and tool names, until capture is turned on. """ report_usage("openai-messages.createOpenAIHandler", PACKAGE_NAME, __version__) return _create_openai_messages_handler(capture_content=capture_content) def _create_openai_messages_handler( - *, capture_content: bool = False + *, capture_content: bool | None = None ) -> ProviderHandler: """Non-reporting :func:`create_openai_messages_handler`, used by this package's wrappers.""" + content_override = capture_content import importlib openai_mod = importlib.import_module("openai") @@ -241,6 +247,8 @@ async def _call_impl( variables: dict[str, Any] | None = None, history: list[dict[str, Any]] | None = None, ) -> dict[str, Any]: + # None follows configure(); an explicit bool overrides it. Read when the call runs. + capture_content = effective_capture_content(content_override) th = tool_handlers or {} vs = variables or {} @@ -414,14 +422,14 @@ def _stream_impl( tool_handlers or {}, variables or {}, history, - capture_content=capture_content, + capture_content=effective_capture_content(content_override), ) return _create_handler( ("OpenAI", "messages"), _call_impl, # type: ignore[arg-type] _stream_impl, # type: ignore[arg-type] - capture_content=capture_content, + capture_content=content_override, ) @@ -653,7 +661,7 @@ def openai_messages( # asking for content on spans got a TypeError instead of content. report_usage("openai-messages.openaiMessages", PACKAGE_NAME, __version__) variables = kwargs.pop("variables", None) - capture_content = kwargs.pop("capture_content", False) + capture_content = kwargs.pop("capture_content", None) return _config( key=config_key, handler=_create_openai_messages_handler(capture_content=capture_content), diff --git a/packages/openai-messages/tests/test_handler.py b/packages/openai-messages/tests/test_handler.py index 35da2f9d..564765a7 100644 --- a/packages/openai-messages/tests/test_handler.py +++ b/packages/openai-messages/tests/test_handler.py @@ -1699,6 +1699,28 @@ async def test_emits_no_content_by_default_on_the_streaming_path( assert [k for k in span.attributes if k.startswith("gen_ai.prompt")] == [] assert [n for n, _ in span.events if n.startswith("gen_ai.content")] == [] + async def test_process_default_is_read_on_the_streaming_path( + self, mock_openai: MagicMock + ) -> None: + from launchdarkly_ai_server import SDKOptions, configure + from launchdarkly_ai_server.lifecycle import _reset_for_testing + + _reset_for_testing() + try: + configure(options=SDKOptions(capture_content=True)) + self._patch_stream(mock_openai, ["hi"]) + ctx, rec = _recording() + from launchdarkly_ai_openai_messages import create_openai_messages_handler + + with ctx: + async for _ in await create_openai_messages_handler().stream( + CONFIG, "q" + ): + pass + assert any("gen_ai.input.messages" in span.attributes for span in rec.spans) + finally: + _reset_for_testing() + # --------------------------------------------------------------------------- # History parameter @@ -1879,8 +1901,11 @@ def test_capture_content_reaches_the_factory(self) -> None: # And it must not have been forwarded to config(), which does not accept it. assert "capture_content" not in seen["config_kwargs"] - def test_defaults_to_off(self) -> None: - assert self._run()["capture_content"] is False + def test_omitted_capture_content_defers_to_the_process_default(self) -> None: + assert self._run()["capture_content"] is None + + def test_explicit_false_reaches_the_factory(self) -> None: + assert self._run(capture_content=False)["capture_content"] is False class TestChatSpanNeverLeaks: