feat(golang): implement /otel_drop_in_extract_and_make_distant_call endpoint by genesor · Pull Request #7264 · DataDog/system-tests

genesor · 2026-07-06T13:13:38Z

Motivation

Enable Test_ExtractBehavior_Restart_Otel for Go weblogs. This test covers the case where a trace is initiated via the OTel propagation API rather than the auto-instrumented HTTP path, and verifies that span links are correctly created when DD_TRACE_PROPAGATION_BEHAVIOR_EXTRACT=restart.

The endpoint was already implemented for Java, .NET, and Ruby but was missing for Go. Tracked as missing_feature (OTel extraction endpoint not implemented) in manifests/golang.yml.

Changes

Add /otel_drop_in_extract_and_make_distant_call endpoint to all 5 Go weblogs (net-http, chi, echo, gin, net-http-orchestrion)
- Extracts incoming trace context via OTel propagation API
- Starts an OTel span named otel_extract_distant_call using the dd-trace-go OTel bridge
- Injects both DD headers and OTel baggage into the outgoing request
- Returns JSON with url, status_code, request_headers, response_headers
Add OTel imports (ddotel, go.opentelemetry.io/otel, go.opentelemetry.io/otel/propagation) to chi, echo, and gin which previously had none
Update manifests/golang.yml: Test_ExtractBehavior_Restart_Otel: missing_feature → v2.10.0-dev

Reviewer checklist

Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
A docker base image is modified?
- the relevant build-XXX-image label is present
A scenario is added, removed or renamed?
- Get a review from R&P team

github-actions · 2026-07-06T14:23:11Z

CODEOWNERS have been resolved as:

manifests/golang.yml                                                    @DataDog/dd-trace-go-guild
utils/build/docker/golang/app/chi/main.go                               @DataDog/dd-trace-go-guild @DataDog/system-tests-core
utils/build/docker/golang/app/echo/main.go                              @DataDog/dd-trace-go-guild @DataDog/system-tests-core
utils/build/docker/golang/app/gin/main.go                               @DataDog/dd-trace-go-guild @DataDog/system-tests-core
utils/build/docker/golang/app/net-http-orchestrion/main.go              @DataDog/dd-trace-go-guild @DataDog/system-tests-core
utils/build/docker/golang/app/net-http/main.go                          @DataDog/dd-trace-go-guild @DataDog/system-tests-core

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3940445a17

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-07-06T15:27:52Z

+
+		otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}))
+		propagator := otel.GetTextMapPropagator()
+		ctx := propagator.Extract(r.Context(), propagation.HeaderCarrier(r.Header))


Use a clean context for the OTel extraction

In the Go weblogs this handler runs under the existing Datadog server middleware, and r.Context() already contains the auto-instrumented inbound server span (the existing /make_distant_call code reads it from the same context). Passing that context into OTel extraction means the dd-trace-go OTel bridge will parent otel_extract_distant_call to the auto server span instead of making it the restarted root span with the incoming context as a span link, so the newly enabled Test_ExtractBehavior_Restart_Otel fails its parentID is None and span-link assertions. Use a clean/background context for the OTel extraction in all added Go handlers.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-06T15:27:52Z

+			return
+		}
+
+		otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}))


Honor Datadog extraction order in the OTel endpoint

The conflicting-header cases in Test_ExtractBehavior_Restart_Otel send both Datadog and W3C headers while the scenario configures extraction as datadog,tracecontext,baggage, and the assertions expect the restart span link to come from the Datadog context. This line replaces the OTel propagator with only TraceContext and Baggage, so the OTel extraction ignores x-datadog-* and cannot produce the expected Datadog restart link when the traceparent differs. The same W3C-only propagator was added to each Go weblog handler.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-06T15:27:52Z

+		propagator := otel.GetTextMapPropagator()
+		ctx := propagator.Extract(r.Context(), propagation.HeaderCarrier(r.Header))
+
+		p := ddotel.NewTracerProvider()


Avoid restarting the tracer per request

When this endpoint is hit, ddotel.NewTracerProvider() calls tracer.Start, which replaces any running Datadog tracer; here that happens in the middle of an already instrumented HTTP request, after the server middleware created its span. That can drop or fragment the in-flight trace and races with any concurrent request using the global tracer, making the newly enabled OTel restart tests unreliable across all added Go handlers. Initialize the provider once at startup or reuse an existing provider instead of recreating it inside the handler.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-07-06T15:27:52Z

+		p := ddotel.NewTracerProvider()
+		otel.SetTracerProvider(p)
+		otelTracer := p.Tracer("")
+		ctx, span := otelTracer.Start(ctx, "otel_extract_distant_call")


Tag the OTel span with the request id

Test_ExtractBehavior_Restart_Otel retrieves spans with interfaces.agent.get_spans_list(self.r), which filters to spans carrying the request id from the request's User-Agent tag. This manually-created OTel span is started without any HTTP/user-agent attributes, so it won't be returned by that filter; the test will inspect the auto-instrumented HTTP server span instead of otel_extract_distant_call and fail the OTel-specific parent/span-link assertions. Add the request user-agent/header tag to the OTel span in each new Go handler.

Useful? React with 👍 / 👎.

…t why

datadog-datadog-prod-us1 · 2026-07-06T16:18:41Z

Tests

✨ Fix all issues with BitsAI

⚠️ Warnings

🚦 6 Pipeline jobs failed

Testing the test | System Tests (golang, dev) / End-to-end #17 / chi 17

🧪 2 Tests failed

tests.test_library_conf.Test_ExtractBehavior_Restart_Otel.test_multiple_tracecontexts_with_overrides[chi] from system_tests_suite

AssertionError: assert &#39;1&#39; != &#39;1&#39;

self = &lt;tests.test_library_conf.Test_ExtractBehavior_Restart_Otel object at 0x7fbe545715e0&gt;

    def test_multiple_tracecontexts_with_overrides(self):
        interfaces.library.assert_trace_exists(self.r)
        spans = interfaces.agent.get_spans_list(self.r)
    
        span = self._get_otel_span(spans)
        assert span.get(&#34;traceID&#34;) != &#34;1&#34;
...

tests.test_library_conf.Test_ExtractBehavior_Restart_Otel.test_single_tracecontext[chi] from system_tests_suite

AssertionError: assert &#39;1&#39; != &#39;1&#39;

self = &lt;tests.test_library_conf.Test_ExtractBehavior_Restart_Otel object at 0x7fbe54571be0&gt;

    def test_single_tracecontext(self):
        interfaces.library.assert_trace_exists(self.r)
        spans = interfaces.agent.get_spans_list(self.r)
    
        span = self._get_otel_span(spans)
        assert span.get(&#34;traceID&#34;) != &#34;1&#34;
...

Testing the test | System Tests (golang, dev) / End-to-end #17 / net-http 17

🧪 2 Tests failed

tests.test_library_conf.Test_ExtractBehavior_Restart_Otel.test_multiple_tracecontexts_with_overrides[net-http] from system_tests_suite

AssertionError: assert &#39;1&#39; != &#39;1&#39;

self = &lt;tests.test_library_conf.Test_ExtractBehavior_Restart_Otel object at 0x7fc6a43aec00&gt;

    def test_multiple_tracecontexts_with_overrides(self):
        interfaces.library.assert_trace_exists(self.r)
        spans = interfaces.agent.get_spans_list(self.r)
    
        span = self._get_otel_span(spans)
        assert span.get(&#34;traceID&#34;) != &#34;1&#34;
...

tests.test_library_conf.Test_ExtractBehavior_Restart_Otel.test_single_tracecontext[net-http] from system_tests_suite

AssertionError: assert &#39;1&#39; != &#39;1&#39;

self = &lt;tests.test_library_conf.Test_ExtractBehavior_Restart_Otel object at 0x7fc6a43ae7e0&gt;

    def test_single_tracecontext(self):
        interfaces.library.assert_trace_exists(self.r)
        spans = interfaces.agent.get_spans_list(self.r)
    
        span = self._get_otel_span(spans)
        assert span.get(&#34;traceID&#34;) != &#34;1&#34;
...

Testing the test | System Tests (golang, dev) / End-to-end #20 / echo 20

🧪 2 Tests failed

tests.test_library_conf.Test_ExtractBehavior_Restart_Otel.test_multiple_tracecontexts_with_overrides[echo] from system_tests_suite

AssertionError: assert &#39;1&#39; != &#39;1&#39;

self = &lt;tests.test_library_conf.Test_ExtractBehavior_Restart_Otel object at 0x7f245848dbe0&gt;

    def test_multiple_tracecontexts_with_overrides(self):
        interfaces.library.assert_trace_exists(self.r)
        spans = interfaces.agent.get_spans_list(self.r)
    
        span = self._get_otel_span(spans)
        assert span.get(&#34;traceID&#34;) != &#34;1&#34;
...

tests.test_library_conf.Test_ExtractBehavior_Restart_Otel.test_single_tracecontext[echo] from system_tests_suite

AssertionError: assert &#39;1&#39; != &#39;1&#39;

self = &lt;tests.test_library_conf.Test_ExtractBehavior_Restart_Otel object at 0x7f245848dc70&gt;

    def test_single_tracecontext(self):
        interfaces.library.assert_trace_exists(self.r)
        spans = interfaces.agent.get_spans_list(self.r)
    
        span = self._get_otel_span(spans)
        assert span.get(&#34;traceID&#34;) != &#34;1&#34;
...

View all 6 failed jobs.

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 367978d | Docs | Datadog PR Page | Give us feedback!}

feat(golang): add OTel extract-and-make-distant-call endpoint

3940445

genesor force-pushed the ben.db/feat-golang-otel-extract-restart branch from 066cfd4 to 3940445 Compare July 6, 2026 14:22

genesor marked this pull request as ready for review July 6, 2026 15:20

genesor requested review from a team as code owners July 6, 2026 15:20

genesor requested a review from MilanGarnier July 6, 2026 15:20

chatgpt-codex-connector Bot reviewed Jul 6, 2026

View reviewed changes

fix(golang): use context.Background() for OTel extraction and documen…

367978d

…t why

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(golang): implement /otel_drop_in_extract_and_make_distant_call endpoint#7264

feat(golang): implement /otel_drop_in_extract_and_make_distant_call endpoint#7264
genesor wants to merge 2 commits into
mainfrom
ben.db/feat-golang-otel-extract-restart

genesor commented Jul 6, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 6, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Jul 6, 2026

Uh oh!

chatgpt-codex-connector Bot Jul 6, 2026

Uh oh!

chatgpt-codex-connector Bot Jul 6, 2026

Uh oh!

chatgpt-codex-connector Bot Jul 6, 2026

Uh oh!

datadog-datadog-prod-us1 Bot commented Jul 6, 2026 •

edited by datadog-prod-us1-6 Bot

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

genesor commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Reviewer checklist

Uh oh!

github-actions Bot commented Jul 6, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jul 6, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 6, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 6, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jul 6, 2026

Choose a reason for hiding this comment

Uh oh!

datadog-datadog-prod-us1 Bot commented Jul 6, 2026 • edited by datadog-prod-us1-6 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ Warnings

ℹ️ Info

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

genesor commented Jul 6, 2026 •

edited

Loading

datadog-datadog-prod-us1 Bot commented Jul 6, 2026 •

edited by datadog-prod-us1-6 Bot

Loading