Skip to content

OTELExporter retains already-ended spans in _span_info until TTL (memory growth with tracing_mode="otel") #119

Open

Description

Summary

With Client(tracing_mode="otel"), OTELExporter._span_info keeps every run whose post already carries an end_time until OTEL_SPAN_TTL_SECONDS (default 3600s). Memory grows with throughput x TTL, and each retained entry is an ended span holding its attributes (prompts, messages, outputs).

Cause

  1. combine_serialized_queue_operations merges a patch into its post when both are in the same batch, so a fast run reaches the exporter as a single post that already has end_time and no patch follows.
  2. _create_span_for_run sees end_time, ends the span and returns it.
  3. export_batch stores it unconditionally:
span = self._create_span_for_run(op, run_info, otel_context_map.get(op.id))
if span:
    self._span_info.set(op.id, span)
  1. Entries are only popped in _update_span_for_run (a patch with end_time) or by the TTL cleanup. Since no patch will come, the ended span stays until the TTL.

Evidence

Instrumented export_batch and _cleanup_stale_spans locally in a LangGraph app under load, with OTEL_SPAN_TTL_SECONDS=120:

  • Spans stored from post ops: 20,528
  • of which the post already had end_time: 9,853
  • of which the span was already ended (is_recording() == False) when stored: 9,853 (the same spans)
  • Spans evicted by the TTL: 1,519, all with is_recording() == False; none were root or still recording
  • Spans still held at the end: 8,716, growing without bound (about 2,500 spans / 91 traces, then 3,700 / 122 traces, 20 seconds apart)

The other half of the stored spans were normal in-progress posts that a later patch popped. Retained spans were mostly fast LangChain/LangGraph internal runs (_router, RunnableLambda, ChatPromptTemplate, RunnableSequence, ...).

Expected

An already-ended span should not be retained for the full TTL. Later children still need the parent's context, so storing only a NonRecordingSpan(span.get_span_context()) for spans that are no longer recording would keep parent linking without holding the payloads (or skip storing and rebuild the parent context from the deterministic ids).

Environment

  • langsmith 0.10.10 (the same code is on current main)
  • opentelemetry-sdk 1.44.0
  • Python 3.14

Activity

  1. visy-ani commented on Nov 22, 2026

    @visy-ani

    hey, can i take this? i'd drop the span from _span_info as soon as it ends (or on export) instead of waiting for the TTL sweep, and add a test that the dict doesn't grow across many finished spans with tracing_mode='otel'.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions