Repository navigation
Add Arize Phoenix / OpenInference tracing for agents and tools - #306
Merged
Merged
Conversation
New atomic_agents.observability package with AtomicAgentsInstrumentor. It emits an OpenInference AGENT span per AtomicAgent run/run_async/run_stream/ run_async_stream and a TOOL span per BaseTool run/run_async/arun, so Arize Phoenix and other OpenTelemetry backends show the agent and tool structure around the LLM spans the provider instrumentors already produce. Nothing in atomic_agents imports the module, and the new `observability` extra carries the OpenInference/OpenTelemetry dependencies. The `openinference_instrumentor` entry point lets phoenix.otel.register(auto_instrument=True) pick it up automatically, which also means existing auto_instrument users start exporting agent and tool payloads on upgrade (OPENINFERENCE_HIDE_INPUTS/OUTPUTS redact them). The entry-point module imports no third-party code and refuses versions below the floor with a warning, so a Phoenix environment never fails to load it. Verified: 45 new tests (streams, early close, cancellation, async, re-entrancy guard for tools that delegate run -> arun / to_thread / a context-copying thread pool, mixin, generated and late-defined tools, TraceConfig redaction, using_session/using_metadata, entry point, floor version refusal, restore on uninstrument) pass on the current and floor versions of the OpenInference packages; the wheel carries the extra and the entry point. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Eight forge tools run their async implementation through ThreadPoolExecutor.submit(asyncio.run, ...). A pool thread starts with an empty contextvars context, so the current OpenTelemetry span and the tracing re-entrancy guard were lost inside the pool: with the instrumentor installed each sync run produced a second, orphaned TOOL span. Submitting through contextvars.copy_context().run keeps the caller's context. Verified: the eight tools' suites pass (214 tests; the one bocha failure needs a real BOCHA_API_KEY and fails identically before this change), and a new instrumentor test asserts one TOOL span in the caller's trace for this pattern. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Planner agent -> calculator tool -> answer agent, wrapped in one CHAIN span and a session, traced into Arize Phoenix through phoenix.otel.register(auto_instrument=True). Listed in the README and the docs examples index. Verified by importing the module with a dummy key; the full run needs an OpenAI key and a Phoenix server. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
New observability guide (what spans you get, Phoenix auto setup, manual OTel setup, grouping calls under a parent span, sessions, privacy controls, service and async notes, limitations), linked from the guides index and the logging guide's OpenTelemetry section. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…agation Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
Opt-in OpenInference tracing so Atomic Agents apps show up properly in Arize Phoenix (and any other OpenTelemetry backend). Until now you could install
openinference-instrumentation-openaiand get a flat list of chat completions. This PR adds the framework layer on top:AtomicAgent.run/run_async/run_stream/run_async_stream, namedClass[InputSchema, OutputSchema].run, with input/output as JSON plusatomic_agents.*attributes (model, mode, schema names, history size entering the turn, context provider names, SHA-256 of the static prompt sections)BaseToolrun/run_async/arun, including MCP-generated tools, withtool.name,tool.description,tool.parametersand I/OPhoenix picks it up with zero code through the
openinference_instrumentorentry point:Manual setup for any OTLP backend:
AtomicAgentsInstrumentor().instrument(tracer_provider=..., config=TraceConfig(...)).Scope
atomic_agents.observability(new, nothing in core imports it) + 45 testsobservabilityextra inpyproject.tomland the entry point; dev group carries the deps so CI runs the testsThreadPoolExecutor.submitnow goes throughcontextvars.copy_context().run. Without that the pool thread dropped the current span and the re-entrancy guard, which gave a second orphaned TOOL span per callatomic-examples/phoenix-tracing: planner agent -> calculator tool -> answer agent under one CHAIN spandocs/guides/observability.mdplus index/README/logging-guide linksThings worth knowing
register(auto_instrument=True)with atomic-agents installed starts exporting agent and tool payloads after upgrading.OPENINFERENCE_HIDE_INPUTS/OPENINFERENCE_HIDE_OUTPUTSredact them. The guide says this.openinference-instrumentationbelow 0.1.38 with a warning, so a Phoenix environment can never crash on load.atomic_agents.stream.closed_early=trueon early break or cancellation). The system prompt text stays on the LLM span; the agent span carries a hash of the static sections so you can tell prompt versions apart even with inputs hidden.runwithout callingsuper()are not traced (documented).Verification
uv run pytest --cov=atomic_agents atomic-agents: 453 passed, 3 skipped; the new module is at 97% coverageopeninference-instrumentation==0.1.38,openinference-semantic-conventions==0.1.20)Provides-Extra: observabilityand the[openinference_instrumentor]group are presentBOCHA_API_KEY(fails identically on main)The plan and the implementation each went through three independent reviews (OpenInference maintainer angle, Atomic Agents maintainer angle, adversarial user/test angle); everything they flagged as blocking or should-fix is in.
🤖 Generated with Claude Code