Repository navigation
feat(telemetry): trace propagation, GenAI semconv, run outcomes - #156
Merged
Merged
Conversation
- Install TraceContext + Baggage propagators when tracing is enabled so agent
runs join the caller's distributed trace; demo extracts inbound headers.
- Export stays off (no-op handles, Providers.Enabled=false) unless an endpoint
is configured; honor OTEL_SDK_DISABLED and OTEL_{TRACES,METRICS}_EXPORTER=none.
- Support http/protobuf via Config.Protocol / OTEL_EXPORTER_OTLP_[SIGNAL_]PROTOCOL.
- Run span carries gen_ai.operation.name=invoke_agent, gen_ai.conversation.id and gen_ai.agent.name (new agent.WithAgentName; builder sets it from YAML). - OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental switches to "invoke_agent <name>" span names and gen_ai.provider.name; the default keeps agent.run and gen_ai.system metric labels so dashboards do not break. - otelllm precomputes span attrs and metric attribute sets once per provider.
One series per conversation made metric cardinality unbounded and contradicted the documented label set; the session key stays on the iteration span.
Worker loops were built without a tracer, so a sub-agent produced no run span and its iterations went untraced. New agent.WithTracerFromContext adopts the tracer of the recording span on ctx; call_sub_agent and call_sql_agent use it and name the worker run span.
In-process OTLP receivers assert the exported tree: inbound traceparent joined, sub-agent run nested under execute_tool, schema names, metric shape. Plus a builder test that the YAML name reaches gen_ai.agent.name.
- Run span ends with gopheragent.run.outcome (closed enum) and an Error status
on failure; a user cancel stays Unset.
- New gopheragent.agent.runs{outcome,agent} and gopheragent.agent.cost{agent}.
- hitl.await_approval span isolates time blocked on a human.
- WithTracerFromContext becomes WithTelemetryFromContext: workers also adopt
the parent's meter, so sub-agent runs and cost are counted.
- call_sub_agent labels worker runs "sub_agent": the persona is model-generated and would make the runs/cost metric labels unbounded. - approval_pending runs keep an Unset span status (a pause, not a failure). - hitl.await_approval records "cancelled" when the run stops mid-wait.
- spanattr.With(ctx, kvs...) carries tenant/user attrs in-process; a span processor in otelsetup stamps them on every span, sub-agents included. - Config.BaggageKeys allowlists inbound W3C Baggage members to copy. - Instrumentation attrs win over ctx attrs; nothing lands on metrics.
- OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT or WithContentCapture records gen_ai.input/output.messages, system_instructions, and tool arguments/results as semconv JSON. Off by default. - Per-field truncation before marshaling keeps JSON valid; input keeps the newest messages within a budget; media bytes are never recorded.
…capture feat(telemetry): request-scoped span attributes and opt-in content capture
…ation-outcomes # Conflicts: # README.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Fixes gaps in the OpenTelemetry layer: agent runs didn't join a caller's trace, sub-agents were untraced, run spans never showed failures, and the iteration metric had one series per session. Also aligns span naming with the GenAI semantic conventions.
Changes
otelsetup: installs W3C TraceContext + Baggage propagators; supports OTLP over HTTP (http/protobuf); honorsOTEL_SDK_DISABLEDandOTEL_{TRACES,METRICS}_EXPORTER=none.Setupno longer defaults tolocalhost:4317. Export is off (no-op handles,Providers.Enabled=false) until an endpoint is configured.invoke_agentattributes (gen_ai.agent.name,gen_ai.conversation.id);otelllmemitsgen_ai.provider.name.OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimentalswitches toinvoke_agent <name>span names; the default keeps existing names and metric labels.gopheragent.run.outcomeand an Error status on failure (cancel and approval-pending stay Unset). Newgopheragent.agent.runsandgopheragent.agent.costmetrics;hitl.await_approvalspan for human wait time.agent.WithTelemetryFromContext/agent.WithAgentName;call_sub_agentandcall_sql_agentnow trace and meter their worker runs under the parent.otelllmprecomputes attribute sets (~21% faster, 37→25 allocs per call).Testing
otel/opentelemetry-collectorand the demo with an inboundtraceparent.Checklist
gofmt -l .prints nothingmake lintis cleanmake buildpassesmake testandmake test-racepassCHANGELOG.mdupdated if this PR is cut as a release tag (at tag time)pkg/telemetry/changes approved by the maintainer