Skip to content

Add Arize Phoenix / OpenInference tracing for agents and tools - #306

Merged
Eigenwise merged 5 commits into
mainfrom
feature/phoenix-openinference-tracing
Oct 15, 2026
Merged

Eigenwise merged 5 commits into
mainfrom
feature/phoenix-openinference-tracing

Conversation

@Eigenwise

Copy link
Copy Markdown
Owner

What this adds

Opt-in OpenInference tracing so Atomic Agents apps show up properly in Arize Phoenix (and any other OpenTelemetry backend). Until now you could install openinference-instrumentation-openai and get a flat list of chat completions. This PR adds the framework layer on top:

  • an AGENT span per AtomicAgent.run / run_async / run_stream / run_async_stream, named Class[InputSchema, OutputSchema].run, with input/output as JSON plus atomic_agents.* attributes (model, mode, schema names, history size entering the turn, context provider names, SHA-256 of the static prompt sections)
  • a TOOL span per BaseTool run / run_async / arun, including MCP-generated tools, with tool.name, tool.description, tool.parameters and I/O
  • LLM spans still come from the provider instrumentor and nest under the agent span

Phoenix picks it up with zero code through the openinference_instrumentor entry point:

from phoenix.otel import register
register(project_name="my-app", auto_instrument=True)

Manual setup for any OTLP backend: AtomicAgentsInstrumentor().instrument(tracer_provider=..., config=TraceConfig(...)).

Scope

  • atomic_agents.observability (new, nothing in core imports it) + 45 tests
  • observability extra in pyproject.toml and the entry point; dev group carries the deps so CI runs the tests
  • 8 forge tools: ThreadPoolExecutor.submit now goes through contextvars.copy_context().run. Without that the pool thread dropped the current span and the re-entrancy guard, which gave a second orphaned TOOL span per call
  • atomic-examples/phoenix-tracing: planner agent -> calculator tool -> answer agent under one CHAIN span
  • docs/guides/observability.md plus index/README/logging-guide links
  • codebase map refresh

Things worth knowing

  • The entry point is on the core package, so anyone already running register(auto_instrument=True) with atomic-agents installed starts exporting agent and tool payloads after upgrading. OPENINFERENCE_HIDE_INPUTS / OPENINFERENCE_HIDE_OUTPUTS redact them. The guide says this.
  • The entry-point module imports only stdlib and refuses openinference-instrumentation below 0.1.38 with a warning, so a Phoenix environment can never crash on load.
  • Streams record the final response (or the last partial with atomic_agents.stream.closed_early=true on early break or cancellation). The system prompt text stays on the LLM span; the agent span carries a hash of the static sections so you can tell prompt versions apart even with inputs hidden.
  • Agent subclasses that override run without calling super() are not traced (documented).

Verification

  • uv run pytest --cov=atomic_agents atomic-agents: 453 passed, 3 skipped; the new module is at 97% coverage
  • same observability tests pass at the floor versions (openinference-instrumentation==0.1.38, openinference-semantic-conventions==0.1.20)
  • black / flake8 clean across all four dirs; every new function has cyclomatic complexity 4 or lower
  • wheel metadata checked: Provides-Extra: observability and the [openinference_instrumentor] group are present
  • the 8 touched forge tools' suites: 214 passed, 1 pre-existing failure that needs a real BOCHA_API_KEY (fails identically on main)
  • the example was import-checked with a dummy key; a full run needs an OpenAI key and a Phoenix server, I did not run that

The plan and the implementation each went through three independent reviews (OpenInference maintainer angle, Atomic Agents maintainer angle, adversarial user/test angle); everything they flagged as blocking or should-fix is in.

🤖 Generated with Claude Code

Eigenwise and others added 5 commits October 15, 2026 12:14
New atomic_agents.observability package with AtomicAgentsInstrumentor. It
emits an OpenInference AGENT span per AtomicAgent run/run_async/run_stream/
run_async_stream and a TOOL span per BaseTool run/run_async/arun, so Arize
Phoenix and other OpenTelemetry backends show the agent and tool structure
around the LLM spans the provider instrumentors already produce.

Nothing in atomic_agents imports the module, and the new `observability`
extra carries the OpenInference/OpenTelemetry dependencies. The
`openinference_instrumentor` entry point lets
phoenix.otel.register(auto_instrument=True) pick it up automatically, which
also means existing auto_instrument users start exporting agent and tool
payloads on upgrade (OPENINFERENCE_HIDE_INPUTS/OUTPUTS redact them). The
entry-point module imports no third-party code and refuses versions below
the floor with a warning, so a Phoenix environment never fails to load it.

Verified: 45 new tests (streams, early close, cancellation, async,
re-entrancy guard for tools that delegate run -> arun / to_thread / a
context-copying thread pool, mixin, generated and late-defined tools,
TraceConfig redaction, using_session/using_metadata, entry point, floor
version refusal, restore on uninstrument) pass on the current and floor
versions of the OpenInference packages; the wheel carries the extra and
the entry point.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Eight forge tools run their async implementation through
ThreadPoolExecutor.submit(asyncio.run, ...). A pool thread starts with an
empty contextvars context, so the current OpenTelemetry span and the
tracing re-entrancy guard were lost inside the pool: with the instrumentor
installed each sync run produced a second, orphaned TOOL span. Submitting
through contextvars.copy_context().run keeps the caller's context.

Verified: the eight tools' suites pass (214 tests; the one bocha failure
needs a real BOCHA_API_KEY and fails identically before this change), and
a new instrumentor test asserts one TOOL span in the caller's trace for
this pattern.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Planner agent -> calculator tool -> answer agent, wrapped in one CHAIN span
and a session, traced into Arize Phoenix through
phoenix.otel.register(auto_instrument=True). Listed in the README and the
docs examples index. Verified by importing the module with a dummy key;
the full run needs an OpenAI key and a Phoenix server.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
New observability guide (what spans you get, Phoenix auto setup, manual OTel
setup, grouping calls under a parent span, sessions, privacy controls,
service and async notes, limitations), linked from the guides index and the
logging guide's OpenTelemetry section.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…agation

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@Eigenwise
Eigenwise merged commit cdc369d into main Oct 15, 2026
1 check passed
@Eigenwise
Eigenwise deleted the feature/phoenix-openinference-tracing branch October 15, 2026 10:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant