You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
OTELExporter retains already-ended spans in _span_info until TTL (memory growth with tracing_mode="otel") #119
With Client(tracing_mode="otel"), OTELExporter._span_info keeps every run whose post already carries an end_time until OTEL_SPAN_TTL_SECONDS (default 3600s). Memory grows with throughput x TTL, and each retained entry is an ended span holding its attributes (prompts, messages, outputs).
Cause
combine_serialized_queue_operations merges a patch into its post when both are in the same batch, so a fast run reaches the exporter as a single post that already has end_time and no patch follows.
_create_span_for_run sees end_time, ends the span and returns it.
Entries are only popped in _update_span_for_run (a patch with end_time) or by the TTL cleanup. Since no patch will come, the ended span stays until the TTL.
Evidence
Instrumented export_batch and _cleanup_stale_spans locally in a LangGraph app under load, with OTEL_SPAN_TTL_SECONDS=120:
Spans stored from post ops: 20,528
of which the post already had end_time: 9,853
of which the span was already ended (is_recording() == False) when stored: 9,853 (the same spans)
Spans evicted by the TTL: 1,519, all with is_recording() == False; none were root or still recording
Spans still held at the end: 8,716, growing without bound (about 2,500 spans / 91 traces, then 3,700 / 122 traces, 20 seconds apart)
The other half of the stored spans were normal in-progress posts that a later patch popped. Retained spans were mostly fast LangChain/LangGraph internal runs (_router, RunnableLambda, ChatPromptTemplate, RunnableSequence, ...).
Expected
An already-ended span should not be retained for the full TTL. Later children still need the parent's context, so storing only a NonRecordingSpan(span.get_span_context()) for spans that are no longer recording would keep parent linking without holding the payloads (or skip storing and rebuild the parent context from the deterministic ids).
Environment
langsmith 0.10.10 (the same code is on current main)
hey, can i take this? i'd drop the span from _span_info as soon as it ends (or on export) instead of waiting for the TTL sweep, and add a test that the dict doesn't grow across many finished spans with tracing_mode='otel'.
Activity
visy-ani commentedon Nov 22, 2026
hey, can i take this? i'd drop the span from _span_info as soon as it ends (or on export) instead of waiting for the TTL sweep, and add a test that the dict doesn't grow across many finished spans with tracing_mode='otel'.