Skip to content

feat(golang): implement /otel_drop_in_extract_and_make_distant_call endpoint #7264

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
genesor wants to merge 2 commits into
base: main
Choose a base branch
from
Open

feat(golang): implement /otel_drop_in_extract_and_make_distant_call endpoint #7264

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3940445
feat(golang): add OTel extract-and-make-distant-call endpoint
genesor Jul 6, 2026
367978d
fix(golang): use context.Background() for OTel extraction and documen…
genesor Jul 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion manifests/golang.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1441,7 +1441,7 @@ manifest:
tests/test_library_conf.py::Test_ExtractBehavior_Default: v2.10.0-dev
tests/test_library_conf.py::Test_ExtractBehavior_Ignore: v2.10.0-dev
tests/test_library_conf.py::Test_ExtractBehavior_Restart: v2.10.0-dev
tests/test_library_conf.py::Test_ExtractBehavior_Restart_Otel: missing_feature (OTel extraction endpoint not implemented)
tests/test_library_conf.py::Test_ExtractBehavior_Restart_Otel: v2.10.0-dev
tests/test_library_conf.py::Test_ExtractBehavior_Restart_With_Extract_First: v2.10.0-dev
tests/test_library_conf.py::Test_HeaderTags: v1.53.0
tests/test_library_conf.py::Test_HeaderTags_Colon_Leading: v1.53.0
Expand Down
60 changes: 60 additions & 0 deletions utils/build/docker/golang/app/chi/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,12 @@ import (
httptrace "github.com/DataDog/dd-trace-go/contrib/net/http/v2"
dd_logrus "github.com/DataDog/dd-trace-go/contrib/sirupsen/logrus/v2"
"github.com/DataDog/dd-trace-go/v2/appsec"
ddotel "github.com/DataDog/dd-trace-go/v2/ddtrace/opentelemetry"
_ "github.com/DataDog/dd-trace-go/v2/ddtrace/opentelemetry/metric"
"github.com/DataDog/dd-trace-go/v2/ddtrace/tracer"
"github.com/DataDog/dd-trace-go/v2/profiler"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/propagation"
)

func main() {
Expand Down Expand Up @@ -195,6 +198,63 @@ func main() {
w.Write(jsonResponse)
})

mux.HandleFunc("/otel_drop_in_extract_and_make_distant_call", func(w http.ResponseWriter, r *http.Request) {
url := r.URL.Query().Get("url")
if url == "" {
w.Write([]byte("OK"))
return
}

// Extract the incoming trace context via the OTel propagation API and start a new span.
// We intentionally use context.Background() instead of r.Context(): r.Context() already
// carries the server span created by the HTTP middleware, which would cause the dd-trace-go
// OTel bridge to parent otel_extract_distant_call to that span rather than creating a
// fresh root with a span link (the expected restart behavior).
otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}))

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 6, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Honor Datadog extraction order in the OTel endpoint

The conflicting-header cases in Test_ExtractBehavior_Restart_Otel send both Datadog and W3C headers while the scenario configures extraction as datadog,tracecontext,baggage, and the assertions expect the restart span link to come from the Datadog context. This line replaces the OTel propagator with only TraceContext and Baggage, so the OTel extraction ignores x-datadog-* and cannot produce the expected Datadog restart link when the traceparent differs. The same W3C-only propagator was added to each Go weblog handler.

Useful? React with 👍 / 👎.

propagator := otel.GetTextMapPropagator()
ctx := propagator.Extract(context.Background(), propagation.HeaderCarrier(r.Header))

p := ddotel.NewTracerProvider()

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 6, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Avoid restarting the tracer per request

When this endpoint is hit, ddotel.NewTracerProvider() calls tracer.Start, which replaces any running Datadog tracer; here that happens in the middle of an already instrumented HTTP request, after the server middleware created its span. That can drop or fragment the in-flight trace and races with any concurrent request using the global tracer, making the newly enabled OTel restart tests unreliable across all added Go handlers. Initialize the provider once at startup or reuse an existing provider instead of recreating it inside the handler.

Useful? React with 👍 / 👎.

otel.SetTracerProvider(p)
otelTracer := p.Tracer("")
ctx, span := otelTracer.Start(ctx, "otel_extract_distant_call")

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 6, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Tag the OTel span with the request id

Test_ExtractBehavior_Restart_Otel retrieves spans with interfaces.agent.get_spans_list(self.r), which filters to spans carrying the request id from the request's User-Agent tag. This manually-created OTel span is started without any HTTP/user-agent attributes, so it won't be returned by that filter; the test will inspect the auto-instrumented HTTP server span instead of otel_extract_distant_call and fail the OTel-specific parent/span-link assertions. Add the request user-agent/header tag to the OTel span in each new Go handler.

Useful? React with 👍 / 👎.

defer span.End()

req, _ := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if ddSpan, ok := tracer.SpanFromContext(ctx); ok {
tracer.Inject(ddSpan.Context(), tracer.HTTPHeadersCarrier(req.Header))
}
propagator.Inject(ctx, propagation.HeaderCarrier(req.Header))

res, err := http.DefaultClient.Do(req)
if err != nil {
logrus.Fatalln("client.Do", err)
}
defer res.Body.Close()

requestHeaders := make(map[string]string, len(req.Header))
for key, values := range req.Header {
requestHeaders[strings.ToLower(key)] = strings.Join(values, ",")
}

responseHeaders := make(map[string]string, len(res.Header))
for key, values := range res.Header {
responseHeaders[key] = strings.Join(values, ",")
}

jsonResponse, err := json.Marshal(struct {
URL string `json:"url"`
StatusCode int `json:"status_code"`
RequestHeaders map[string]string `json:"request_headers"`
ResponseHeaders map[string]string `json:"response_headers"`
}{URL: url, StatusCode: res.StatusCode, RequestHeaders: requestHeaders, ResponseHeaders: responseHeaders})
if err != nil {
logrus.Fatalln(err)
}
w.Header().Set("Content-Type", "application/json")
w.Write(jsonResponse)
})

mux.HandleFunc("/headers/", headers)
mux.HandleFunc("/headers", headers)

Expand Down
54 changes: 54 additions & 0 deletions utils/build/docker/golang/app/echo/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,12 @@ import (
httptrace "github.com/DataDog/dd-trace-go/contrib/net/http/v2"
dd_logrus "github.com/DataDog/dd-trace-go/contrib/sirupsen/logrus/v2"
"github.com/DataDog/dd-trace-go/v2/appsec"
ddotel "github.com/DataDog/dd-trace-go/v2/ddtrace/opentelemetry"
_ "github.com/DataDog/dd-trace-go/v2/ddtrace/opentelemetry/metric"
"github.com/DataDog/dd-trace-go/v2/ddtrace/tracer"
"github.com/DataDog/dd-trace-go/v2/profiler"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/propagation"
)

func main() {
Expand Down Expand Up @@ -205,6 +208,57 @@ func main() {
}{URL: url, StatusCode: res.StatusCode, RequestHeaders: requestHeaders, ResponseHeaders: responseHeaders})
})

r.Any("/otel_drop_in_extract_and_make_distant_call", func(c echo.Context) error {
url := c.Request().URL.Query().Get("url")
if url == "" {
return c.String(200, "OK")
}

// Extract the incoming trace context via the OTel propagation API and start a new span.
// We intentionally use context.Background() instead of c.Request().Context(): that context
// already carries the server span created by the HTTP middleware, which would cause the
// dd-trace-go OTel bridge to parent otel_extract_distant_call to that span rather than
// creating a fresh root with a span link (the expected restart behavior).
otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}))
propagator := otel.GetTextMapPropagator()
ctx := propagator.Extract(context.Background(), propagation.HeaderCarrier(c.Request().Header))

p := ddotel.NewTracerProvider()
otel.SetTracerProvider(p)
otelTracer := p.Tracer("")
ctx, span := otelTracer.Start(ctx, "otel_extract_distant_call")
defer span.End()

req, _ := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if ddSpan, ok := tracer.SpanFromContext(ctx); ok {
tracer.Inject(ddSpan.Context(), tracer.HTTPHeadersCarrier(req.Header))
}
propagator.Inject(ctx, propagation.HeaderCarrier(req.Header))

res, err := http.DefaultClient.Do(req)
if err != nil {
logrus.Fatalln("client.Do", err)
}
defer res.Body.Close()

requestHeaders := make(map[string]string, len(req.Header))
for key, values := range req.Header {
requestHeaders[strings.ToLower(key)] = strings.Join(values, ",")
}

responseHeaders := make(map[string]string, len(res.Header))
for key, values := range res.Header {
responseHeaders[key] = strings.Join(values, ",")
}

return c.JSON(200, struct {
URL string `json:"url"`
StatusCode int `json:"status_code"`
RequestHeaders map[string]string `json:"request_headers"`
ResponseHeaders map[string]string `json:"response_headers"`
}{URL: url, StatusCode: res.StatusCode, RequestHeaders: requestHeaders, ResponseHeaders: responseHeaders})
})

r.Any("/headers/", headers)
r.Any("/headers", headers)

Expand Down
55 changes: 55 additions & 0 deletions utils/build/docker/golang/app/gin/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,12 @@ import (
httptrace "github.com/DataDog/dd-trace-go/contrib/net/http/v2"
dd_logrus "github.com/DataDog/dd-trace-go/contrib/sirupsen/logrus/v2"
"github.com/DataDog/dd-trace-go/v2/appsec"
ddotel "github.com/DataDog/dd-trace-go/v2/ddtrace/opentelemetry"
_ "github.com/DataDog/dd-trace-go/v2/ddtrace/opentelemetry/metric"
"github.com/DataDog/dd-trace-go/v2/ddtrace/tracer"
"github.com/DataDog/dd-trace-go/v2/profiler"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/propagation"
)

func main() {
Expand Down Expand Up @@ -196,6 +199,58 @@ func main() {
}{URL: url, StatusCode: res.StatusCode, RequestHeaders: requestHeaders, ResponseHeaders: responseHeaders})
})

r.Any("/otel_drop_in_extract_and_make_distant_call", func(ctx *gin.Context) {
url := ctx.Request.URL.Query().Get("url")
if url == "" {
ctx.Writer.Write([]byte("OK"))
return
}

// Extract the incoming trace context via the OTel propagation API and start a new span.
// We intentionally use context.Background() instead of ctx.Request.Context(): that context
// already carries the server span created by the HTTP middleware, which would cause the
// dd-trace-go OTel bridge to parent otel_extract_distant_call to that span rather than
// creating a fresh root with a span link (the expected restart behavior).
otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}))
propagator := otel.GetTextMapPropagator()
gctx := propagator.Extract(context.Background(), propagation.HeaderCarrier(ctx.Request.Header))

p := ddotel.NewTracerProvider()
otel.SetTracerProvider(p)
otelTracer := p.Tracer("")
gctx, span := otelTracer.Start(gctx, "otel_extract_distant_call")
defer span.End()

req, _ := http.NewRequestWithContext(gctx, http.MethodGet, url, nil)
if ddSpan, ok := tracer.SpanFromContext(gctx); ok {
tracer.Inject(ddSpan.Context(), tracer.HTTPHeadersCarrier(req.Header))
}
propagator.Inject(gctx, propagation.HeaderCarrier(req.Header))

res, err := http.DefaultClient.Do(req)
if err != nil {
logrus.Fatalln("client.Do", err)
}
defer res.Body.Close()

requestHeaders := make(map[string]string, len(req.Header))
for key, values := range req.Header {
requestHeaders[strings.ToLower(key)] = strings.Join(values, ",")
}

responseHeaders := make(map[string]string, len(res.Header))
for key, values := range res.Header {
responseHeaders[key] = strings.Join(values, ",")
}

ctx.JSON(200, struct {
URL string `json:"url"`
StatusCode int `json:"status_code"`
RequestHeaders map[string]string `json:"request_headers"`
ResponseHeaders map[string]string `json:"response_headers"`
}{URL: url, StatusCode: res.StatusCode, RequestHeaders: requestHeaders, ResponseHeaders: responseHeaders})
})

r.Any("/headers/", headers)
r.Any("/headers", headers)

Expand Down
57 changes: 57 additions & 0 deletions utils/build/docker/golang/app/net-http-orchestrion/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -188,6 +188,63 @@ func main() {
w.Write(jsonResponse)
})

mux.HandleFunc("/otel_drop_in_extract_and_make_distant_call", func(w http.ResponseWriter, r *http.Request) {
url := r.URL.Query().Get("url")
if url == "" {
w.Write([]byte("OK"))
return
}

// Extract the incoming trace context via the OTel propagation API and start a new span.
// We intentionally use context.Background() instead of r.Context(): r.Context() already
// carries the server span created by the HTTP middleware, which would cause the dd-trace-go
// OTel bridge to parent otel_extract_distant_call to that span rather than creating a
// fresh root with a span link (the expected restart behavior).
otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}))
propagator := otel.GetTextMapPropagator()
ctx := propagator.Extract(context.Background(), propagation.HeaderCarrier(r.Header))

p := opentelemetry.NewTracerProvider()
otel.SetTracerProvider(p)
otelTracer := p.Tracer("")
ctx, span := otelTracer.Start(ctx, "otel_extract_distant_call")
defer span.End()

req, _ := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if ddSpan, ok := tracer.SpanFromContext(ctx); ok {
tracer.Inject(ddSpan.Context(), tracer.HTTPHeadersCarrier(req.Header))
}
propagator.Inject(ctx, propagation.HeaderCarrier(req.Header))

res, err := http.DefaultClient.Do(req)
if err != nil {
log.Fatalln("client.Do", err)
}
defer res.Body.Close()

requestHeaders := make(map[string]string, len(req.Header))
for key, values := range req.Header {
requestHeaders[strings.ToLower(key)] = strings.Join(values, ",")
}

responseHeaders := make(map[string]string, len(res.Header))
for key, values := range res.Header {
responseHeaders[key] = strings.Join(values, ",")
}

jsonResponse, err := json.Marshal(struct {
URL string `json:"url"`
StatusCode int `json:"status_code"`
RequestHeaders map[string]string `json:"request_headers"`
ResponseHeaders map[string]string `json:"response_headers"`
}{URL: url, StatusCode: res.StatusCode, RequestHeaders: requestHeaders, ResponseHeaders: responseHeaders})
if err != nil {
log.Fatalln(err)
}
w.Header().Set("Content-Type", "application/json")
w.Write(jsonResponse)
})

mux.HandleFunc("/headers", headers)
mux.HandleFunc("/headers/", headers)

Expand Down
57 changes: 57 additions & 0 deletions utils/build/docker/golang/app/net-http/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -687,6 +687,63 @@ func main() {
w.Write(jsonData)
})

mux.HandleFunc("/otel_drop_in_extract_and_make_distant_call", func(w http.ResponseWriter, r *http.Request) {
url := r.URL.Query().Get("url")
if url == "" {
w.Write([]byte("OK"))
return
}

// Extract the incoming trace context via the OTel propagation API and start a new span.
// We intentionally use context.Background() instead of r.Context(): r.Context() already
// carries the server span created by the HTTP middleware, which would cause the dd-trace-go
// OTel bridge to parent otel_extract_distant_call to that span rather than creating a
// fresh root with a span link (the expected restart behavior).
otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}))
propagator := otel.GetTextMapPropagator()
ctx := propagator.Extract(context.Background(), propagation.HeaderCarrier(r.Header))

p := ddotel.NewTracerProvider()
otel.SetTracerProvider(p)
otelTracer := p.Tracer("")
ctx, span := otelTracer.Start(ctx, "otel_extract_distant_call")
defer span.End()

req, _ := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if ddSpan, ok := tracer.SpanFromContext(ctx); ok {
tracer.Inject(ddSpan.Context(), tracer.HTTPHeadersCarrier(req.Header))
}
propagator.Inject(ctx, propagation.HeaderCarrier(req.Header))

res, err := http.DefaultClient.Do(req)
if err != nil {
logrus.Fatalln("client.Do", err)
}
defer res.Body.Close()

requestHeaders := make(map[string]string, len(req.Header))
for key, values := range req.Header {
requestHeaders[strings.ToLower(key)] = strings.Join(values, ",")
}

responseHeaders := make(map[string]string, len(res.Header))
for key, values := range res.Header {
responseHeaders[key] = strings.Join(values, ",")
}

jsonResponse, err := json.Marshal(struct {
URL string `json:"url"`
StatusCode int `json:"status_code"`
RequestHeaders map[string]string `json:"request_headers"`
ResponseHeaders map[string]string `json:"response_headers"`
}{URL: url, StatusCode: res.StatusCode, RequestHeaders: requestHeaders, ResponseHeaders: responseHeaders})
if err != nil {
logrus.Fatalln(err)
}
w.Header().Set("Content-Type", "application/json")
w.Write(jsonResponse)
})

mux.HandleFunc("/session/new", func(w http.ResponseWriter, r *http.Request) {
sessionID := strconv.Itoa(rand.Int())
w.Header().Add("Set-Cookie", "session="+sessionID+"; Path=/; Max-Age=3600; Secure; HttpOnly")
Expand Down
Loading