Repository navigation
feat(openfeature): add agentless feature flag configuration source - #19331
Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 30 commits intoAug 7, 2026
Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 30 commits into
gh-worker-dd-mergequeue-cf854d[bot] merged 30 commits into
Conversation
Codeowners resolved as |
Circular import analysis
|
Contributor
🎉 All green!🧪 All tests passed 🔄 Datadog auto-retried 1 job - 1 passed on retry 🔗 Commit SHA: 774fa3a | Docs | Datadog PR Page | Give us feedback! |
pavlokhrebto
changed the title
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
BenchmarksBenchmark execution time: 2026-08-07 09:45:51 Comparing candidate commit 774fa3a in PR branch Found 0 performance improvements and 4 performance regressions! Performance is the same for 606 metrics, 10 unstable metrics. scenario:iastaspects-swapcase_aspect
scenario:iastaspectsospath-ospathbasename_aspect
scenario:span-start
scenario:tracer-small
|
pavlokhrebto
marked this pull request as ready for review
July 28, 2026 13:50
pavlokhrebto
requested review from
Kyle-Verhoog,
emmettbutler,
mabdinur,
sameerank and
vjfridge
July 28, 2026 13:50
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 128172f5a9
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
leoromanovsky
left a comment
leoromanovsky
left a comment
Contributor
There was a problem hiding this comment.
Two things to review or augment please.
Blocker: provider readiness is not canonical.
Python returns from initialize() before the first UFC configuration arrives. Its own test confirms that the SDK can emit PROVIDER_READY while the provider remains NOT_READY. The first evaluation can therefore return the default with PROVIDER_NOT_READY.
Node.js waits for configuration through setProviderAndWait(). Java completes an initial poll and waits for configuration during provider initialization.
References: Python provider
(https://github.com/DataDog/dd-trace-py/blob/b5cb1213cf2f8fa18294d3ecf7d56444885606c0/ddtrace/internal/openfeature/_provider.py#L222-L286), Python
readiness test (https://github.com/DataDog/dd-trace-py/blob/b5cb1213cf2f8fa18294d3ecf7d56444885606c0/tests/openfeature/test_provider_status.py#L143-L165),
Node.js contract
(https://github.com/DataDog/dd-trace-js/blob/d678882c997c1c0f5b7f6f779cff9995bc84f806/integration-tests/openfeature/app/configuration-source-evaluation.js#L34-L41),
Java provider
(https://github.com/DataDog/dd-trace-java/blob/f54355526dabf515d0c8884f74a6e01ced61c1ff/products/feature-flagging/feature-flagging-api/src/main/java/datadog/trace/api/openfeature/Provider.java#L109-L133).
Should fix: shutdown does not cancel an active HTTP request.
[Python ](https://github.com/DataDog/dd-trace-py/blob/b5cb1213cf2f8fa18294d3ecf7d56444885606c0/tests/openfeature/test_provider_status.py#L143-L165)interrupts retry waits, but shutdown() joins the poller without cancelling the current connection. Shutdown can block until the configured request
timeout. Node.js aborts the request. Java cancels the active call.
Blocker: final L3 validation is stale.
The documented 29 passed result is useful, but it predates the current head and later retry changes. The current head is also 19 commits behind Python main. Latest main includes a periodic-thread crash fix (#19445), which directly affects this new poller.
Merge current main, then rerun all 29 tests with [system-tests #7411](https://github.com/DataDog/system-tests/pull/7411) (https://github.com/DataDog/system-tests/pull/7411) against the exact final head.
Contributor
Author
|
@leoromanovsky made the fixes:
Two details worth calling out, because they're deliberate: The wait does not raise on expiry. A previous revision (#18613 reverted it) waited and then raised The default stays 10s rather than Java/Node's 30s. This is a Python-specific constraint, not drift. Gunicorn's default worker timeout is also 30s, so a 30s blocking initialize() at import time sits right on the boundary and gets the worker SIGKILLed with Side effects of the change: the initialization_timeout constructor argument was accepted and silently discarded, and the initialization_timeout_ms setting was dead — both are wired up now, with the constructor argument taking precedence. The release note for #18613 still claimed "the provider now waits for configuration before returning," which was false on main; it's accurate again and now documents the timeout behavior. New tests cover: config arriving mid-wait → READY; timeout → NOT_READY and no raise; late delivery still promotes; timeout read from the env var; constructor override. If you'd rather standardize on a single cross-SDK number, I'm happy to — but I'd argue it should be 10s for everyone rather than 30s for Python, and that's a guild decision rather than something to settle in this PR.
836b9d7. _stop_service now sets the stop flag first, then half-closes the socket of the poll in flight (socket.shutdown(SHUT_RDWR)) before joining. close() alone wouldn't do it — the worker is already parked inside a syscall on that descriptor — so the half-close is what makes the pending recv return immediately. The worker still owns cleanup: _request closes the connection in its finally block and reports the resulting error as a failed poll, which preserves last-known-good. One window is genuinely uncancellable and documented in the code: http.client connects lazily inside request(), so a stop landing before the socket exists has nothing to half-close and the worker can block in connect() for up to the request timeout. Everything past connect — the long pole, since a poll spends its time reading the UFC body — cancels at once. Claiming the connection slot and re-reading the stop flag happen under one lock, so a poll can't start after shutdown has already looked for something to cancel. Seven new tests cover the cancel path, the no-op paths, and the ordering guarantee. |
leoromanovsky
approved these changes
Aug 6, 2026
leoromanovsky
left a comment
leoromanovsky
left a comment
Contributor
There was a problem hiding this comment.
Functional result:
- Startup timeout now raises the SDK’s canonical ProviderNotReadyError.
- OpenFeature 0.8 reports provider status ERROR.
- OpenFeature 0.10 set_provider_and_wait() raises the error.
- OpenFeature 0.10 set_provider() returns immediately. Background initialization then reports ERROR.
- A later valid configuration changes the provider to READY.
- The implementation emits one recovery event.
Validation:
- OpenFeature 0.8 full suite: 625 passed, 2 skipped.
- OpenFeature 0.10 focused suite: 18 passed.
- L3 at system-tests commit ef1cb447c9: 29 passed in 168.20s.
Contributor
|
/merge |
|
View all feedbacks in Devflow UI.
It will be processed automatically as soon as GitHub reports it as mergeable. View in MergeQueue UI.
The expected merge time in
Tests failed on this commit 23f1a14: What to do next?
|
Contributor
Author
|
/code blockers |
|
View all feedbacks in Devflow UI.
Checking merge blockers for #19331...
No merge blockers detected. |
gh-worker-dd-mergequeue-cf854d
Bot
deleted the
pavlo.khrebto/FFL-2699/agentless-ff-configs
branch
August 7, 2026 10:35
gh-worker-dd-mergequeue-cf854d Bot
pushed a commit
that referenced
this pull request
Aug 12, 2026
#19622) ## Description Follow-up to the agentless Feature Flagging configuration source (#19331, FFL-2699). `OpenFeatureConfig` is instantiated at module scope, so envier raising on a bad cast surfaced as an `ImportError` for `ddtrace.openfeature` — a single malformed value took the whole application down at startup instead of degrading one setting. All four numeric FFE settings now use a lenient parser that logs a warning and falls back to the documented default, matching dd-trace-java's `ConfigProvider` behaviour. An `AIDEV-NOTE` records why. ## Testing Three cases added to `tests/openfeature/test_source_selection.py`: each integer setting falls back to its default given `"0.2"`, and the float setting falls back given a non-numeric value. Full openfeature suite on py3.13: 629 passed, 2 skipped. The fallback tests use `monkeypatch.setenv` rather than `OpenFeatureConfig(source=...)` — `DDConfig` ranks `source` *below* the real environment, and `tests/openfeature/conftest.py` pins `DD_EXPERIMENTAL_FLAGGING_PROVIDER_INITIALIZATION_TIMEOUT_MS=0` for all tests in that directory. ## Risks Low. The lenient parsers only widen what is accepted — no previously working value changes meaning, and no default changes. ## Additional Notes No ticket of its own; follow-up to FFL-2699. An earlier revision of this PR also changed the agentless request timeout default from 5s to 2s to match the Feature Flagging RFC. That was dropped: the agreed default is 5s. Co-authored-by: pavlo.khrebto <pavlo.khrebto@datadoghq.com>
This was referenced Aug 13, 2026
gh-worker-dd-mergequeue-cf854d Bot
pushed a commit
that referenced
this pull request
Aug 21, 2026
…9809) ## Description ### Problem `PromptManager._fetch_from_ff()` uses the now-deprecated `DD_EXPERIMENTAL_FLAGGING_PROVIDER_ENABLED` value as a separate activation gate and then enables Remote Configuration unconditionally. That became incorrect when #19331 introduced `agentless`, `remote_config`, and `disabled` as the authoritative resolved sources, defaulting to `agentless`. `PromptManager` was never updated to follow that resolved decision. Consequently, stable `agentless` and `remote_config` configurations can silently use `/resolve`. Prompt retrieval still works, but **the local Feature Flag exposure needed for experiment attribution is lost**. The required lifecycle is: Resolved source | Required action -- | -- `disabled` | Neither helper; fall back to /resolve `agentless` | `_ensure_ffe_provider()` only `remote_config` | `_ensure_ffe_rc()` and `_ensure_ffe_provider()` Just as a quick reminder: * `_ensure_ffe_provider()` is essential as `PromptManager` evaluates against the private `datadog-llmobs-prompts` OpenFeature domain; Without it, the SDK uses its no-op provider or potentially an unrelated application provider. * `_ensure_ffe_rc()` is necessary only for `remote_config`, as removing would effectively make local RC prompt evaluation supported only under `ddtrace-run`, an unintended compatibility break. ### What changes Prompt-backed Feature Flags now honor the existing resolved Feature Flagging configuration source. - `disabled` retains the existing HTTP `/resolve` fallback. - `remote_config` lazily enables the existing Remote Configuration path. - `agentless` and `remote_config` register and use the existing `DataDogProvider` for local evaluation. ## Testing Added a regression test ## Risks Low. The change only replaces the obsolete Prompt Manager activation gate with the already-resolved Feature Flagging source and preserves the HTTP fallback. ## Additional Notes I also had to add move resolver/constants to `openfeature.py` since the [detect_layering_violation](https://mosaic.us1.ddbuild.io/change-request/3539408316382087817?owner=datadog&repository=dd-trace-py&taskId=gitlab&taskExecutionId=1972050526) would otherwise fail as I'm using them in LLMObs's code and i couldn't find a way to elegantly work around that without adding code there. It's simply moving code around. Co-authored-by: alexandre.choura <alexandre.choura@datadoghq.com>
pavlokhrebto
added a commit
to DataDog/dd-trace-dotnet
that referenced
this pull request
Aug 27, 2026
…endpoint derivation (#9040) ## Summary of changes Adds the foundational configuration layer for agentless Feature Flags delivery: new configuration keys, a `FeatureFlagsSettings` class with source resolution and validation, and an `AgentlessEndpoint` struct for deriving the CDN URL from the Datadog site or a custom base URL. This is PR 1 of a stacked PR series implementing agentless Feature Flags configuration delivery (FFL-2703), porting functionality from [dd-trace-py#19331](DataDog/dd-trace-py#19331) and [dd-trace-java#11892](DataDog/dd-trace-java#11892). ## Reason for change The .NET tracer currently delivers Feature Flags configuration exclusively through the Datadog Agent's Remote Configuration. To support agentless (CDN-backed) delivery, we need a configuration layer that: - Selects between `agentless` (new default), `remote_config`, and `disabled` sources - Derives the managed CDN endpoint from `DD_SITE` or accepts a custom `DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_BASE_URL` - Validates poll interval and request timeout settings - Grandfathers existing `DD_EXPERIMENTAL_FLAGGING_PROVIDER_ENABLED` users onto Remote Configuration ## Implementation details - **`FeatureFlagsSource` enum** — `Disabled`, `Agentless`, `RemoteConfig` - **`FeatureFlagsSettings`** — reads all new env vars, resolves the source with the cross-SDK precedence contract (kill switch → explicit source → fail-closed → legacy grandfathering → default agentless), validates poll interval (1–3600s) and request timeout (>0s) - **`AgentlessEndpoint`** — derives `https://ufc-server.ff-cdn.<site>/api/v2/feature-flagging/config/rules-based/server` from the site, appends `dd_env` when configured, accepts custom HTTP/HTTPS URLs, and never echoes URLs in error messages (credentials safety) - **New config keys** in `supported-configurations.yaml` + generated `ConfigurationKeys.FeatureFlags.g.cs`: - `DD_FEATURE_FLAGS_ENABLED` (default `true`, supersedes `DD_EXPERIMENTAL_FLAGGING_PROVIDER_ENABLED`) - `DD_FEATURE_FLAGS_CONFIGURATION_SOURCE` (default `agentless`) - `DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_BASE_URL` - `DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_POLL_INTERVAL_SECONDS` (default 30, type `int`) - `DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_REQUEST_TIMEOUT_SECONDS` (default 5, type `int`) - `DD_EXPERIMENTAL_FLAGGING_PROVIDER_INITIALIZATION_TIMEOUT_MS` (default 10000, implementation B) - **`[Obsolete]`** on `FlaggingProviderEnabled` with pragma suppression in `FeatureFlagsSettings` This PR does not wire anything to `FeatureFlagsModule` yet — that happens in a later PR in the stack. ## Test coverage - `FeatureFlagsSettingsTests` — 43 test cases covering source resolution (kill switch, explicit source, legacy grandfathering, `offline` sentinel, invalid values), blank/whitespace handling, casing normalization, defaults, poll interval validation, request timeout validation, initialization timeout, and base URL reading - `AgentlessEndpointTests` — 18 test cases covering managed endpoint derivation (site lowercasing, staging, govcloud), `dd_env` query parameter (addition, null, escaping), custom endpoint path handling (origin-only gets canonical path, custom path used verbatim), HTTP acceptance for custom endpoints, invalid URL rejection, empty/whitespace site rejection, and credentials-in-error safety ## Other details <!-- Fixes #{issue} --> Stacked PRs: - PR 1 (this): Configuration keys, settings, and endpoint derivation - PR 2: UFC parser and agentless poller - PR 3: Module wiring and OpenFeature activation - PR 4: Manual API activation <!--⚠️ Note: Where possible, please obtain 2 approvals prior to merging. Unless CODEOWNERS specifies otherwise, for external teams it is typically best to have one review from a team member, and one review from apm-dotnet. Trivial changes do not require 2 reviews. MergeQueue is NOT enabled in this repository. If you have write access to the repo, the PR has 1-2 approvals (see above), and all of the required checks have passed, you can use the Squash and Merge button to merge the PR. If you don't have write access, or you need help, reach out in the #apm-dotnet channel in Slack. --> System tests: [DataDog/system-tests#7496](DataDog/system-tests#7496)
pavlokhrebto
added a commit
to DataDog/dd-trace-dotnet
that referenced
this pull request
Aug 27, 2026
## Summary of changes Adds the agentless delivery mechanism: a `UfcConfigurationParser` that validates the JSON:API envelope returned by the CDN endpoint, and an `AgentlessConfigurationSource` HTTP poller with retries, ETag caching, gzip decompression, and shutdown safety. This is PR 2 of a stacked PR series implementing agentless Feature Flags configuration delivery (FFL-2703), porting functionality from [dd-trace-py#19331](DataDog/dd-trace-py#19331) and [dd-trace-java#11892](DataDog/dd-trace-java#11892). Stacked on [PR 1 #9040](#9040). ## Reason for change PR 1 added the configuration layer (settings, endpoint derivation). This PR adds the actual HTTP delivery mechanism that polls the CDN endpoint for UFC flag configuration. It is self-contained: the poller and parser are fully testable in isolation and are not yet wired to the `FeatureFlagsModule` (that happens in PR 3). ## Implementation details - **`UfcConfigurationParser`** — validates the JSON:API Universal Flag Configuration response envelope, extracts `data.attributes` into `ServerConfiguration`. Rejects raw UFC documents (without the JSON:API wrapper) so all delivery sources agree on one wire format. Returns descriptive errors; never throws. - **`AgentlessConfigurationSource`** — HTTP poller with: - Fixed-delay polling (no overlap) - In-tick retries (max 3) for 408/429/5xx/network failures, with jittered backoff derived from the poll interval (`clamp(P/6, 2s, 10s)` then `clamp(P/3, 5s, 30s)`) - ETag / `If-None-Match` conditional polling; ETag advances only after parse + apply succeed - Gzip decompression (neither transport decompresses automatically) - Self-tracing suppression (`x-datadog-tracing-enabled: false` header) - API key sent only to managed (Datadog-derived) endpoints, never custom ones - Warn-once per failure category (authentication, http, request) to prevent log flooding - Shutdown safety: `CancellationTokenSource` cancels the loop; shutdown checks after request and before `ApplyAsync` prevent use-after-dispose - `Create()` factory returns `null` when endpoint can'"'t be built or API key is missing - Idempotent `Start()` via `Interlocked.CompareExchange` ## Test coverage - **`UfcConfigurationParserTests`** (16 tests) — valid envelope parsing, malformed JSON rejection, invalid envelope rejection (wrong type, missing data, raw UFC), invalid attributes rejection (missing format/createdAt/environment/flags, wrong types), flag parsing - **`AgentlessConfigurationSourceTests`** (12 tests) — 200 success, ETag forwarding, 304/401/malformed handling, disposal before/after request, retry on 500 then success, max retries on 500, no retry on 400, gzip response, network error, idempotent start ## Other details <!-- Fixes #{issue} --> Stacked PRs: - PR 1 [#9040](#9040): Configuration keys, settings, and endpoint derivation - PR 2 (this): UFC parser and agentless HTTP poller - PR 3: Module wiring and OpenFeature activation - PR 4: Manual API activation <!--⚠️ Note: Where possible, please obtain 2 approvals prior to merging. Unless CODEOWNERS specifies otherwise, for external teams it is typically best to have one review from a team member, and one review from apm-dotnet. Trivial changes do not require 2 reviews. MergeQueue is NOT enabled in this repository. If you have write access to the repo, the PR has 1-2 approvals (see above), and all of the required checks have passed, you can use the Squash and Merge button to merge the PR. If you don't have write access, or you need help, reach out in the #apm-dotnet channel in Slack. --> System tests: [DataDog/system-tests#7496](DataDog/system-tests#7496)
pavlokhrebto
added a commit
to DataDog/dd-trace-dotnet
that referenced
this pull request
Sep 14, 2026
…InitializeAsync (#9044) ## Summary of changes Wires the agentless poller and RC subscription into `FeatureFlagsModule` with source-agnostic activation, and adds `InitializeAsync` through the OpenFeature provider lifecycle so delivery starts when application code initializes the provider. This is PR 3 of a stacked PR series implementing agentless Feature Flags configuration delivery (FFL-2703), porting functionality from [dd-trace-py#19331](DataDog/dd-trace-py#19331) and [dd-trace-java#11892](DataDog/dd-trace-java#11892). Stacked on [PR 1 #9040](#9040) and [PR 2 #9042](#9042). ## Reason for change PR 1 added the configuration layer and PR 2 added the poller and parser. This PR connects them: the module now selects the delivery source at construction time, starts polling only when the provider is initialized, and waits for the first configuration before returning from `InitializeAsync`. ## Implementation details - **`TracerSettings`** — replaces `IsFlaggingProviderEnabled` with `FeatureFlagsSettings FeatureFlags` property - **`FeatureFlagsModule`** — source-agnostic constructor: - RC subscription only when `Source == RemoteConfig` (gated, no longer always-on) - `Create()` checks `settings.FeatureFlags.Enabled` - `Activate()` starts the agentless poller or marks RC intent, with `_deliveryStarted` tracking - `InitializeAsync()` calls `Activate()`, waits for `FirstConfigReceived` with timeout, skips the wait if no delivery started - `ApplyConfiguration()` — public method used by both agentless and RC paths, signals `FirstConfigReceived` - `Dispose()` now also disposes the agentless source and unsubscribes RC - **`OpenFeatureSdkInitializeAsyncIntegration`** (new CallTarget) — intercepts `FeatureFlagsSdk.InitializeAsync` stub and calls `FeatureFlagsModule.InitializeAsync` - **`DatadogProvider.InitializeAsync`** — override that delegates to `FeatureFlagsSdk.InitializeAsync` - **`FeatureFlagsSdk.InitializeAsync`** — no-op stub in the OpenFeature package - **`supported_calltargets.g.json`** — registers the new integration - **`SpanEnrichmentIntegrationTests`** — suppresses `[Obsolete]` warning on `FlaggingProviderEnabled` ## Test coverage - **`FeatureFlagsModuleTests`** (9 tests) — RC config update/reset, agentless source doesn't subscribe to RC, RC source subscribes and advertises capability, disabled returns null, `InitializeAsync` returns immediately when config already applied, returns when config arrives while waiting, returns on timeout without throwing, returns on cancellation without throwing, returns immediately when agentless source cannot start (no API key) ## Other details <!-- Fixes #{issue} --> Stacked PRs: - PR 1 [#9040](#9040): Configuration keys, settings, and endpoint derivation - PR 2 [#9042](#9042): UFC parser and agentless HTTP poller - PR 3 (this): Module wiring and OpenFeature activation - PR 4: Manual API activation <!--⚠️ Note: Where possible, please obtain 2 approvals prior to merging. Unless CODEOWNERS specifies otherwise, for external teams it is typically best to have one review from a team member, and one review from apm-dotnet. Trivial changes do not require 2 reviews. MergeQueue is NOT enabled in this repository. If you have write access to the repo, the PR has 1-2 approvals (see above), and all of the required checks have passed, you can use the Squash and Merge button to merge the PR. If you don't have write access, or you need help, reach out in the #apm-dotnet channel in Slack. --> System tests: [DataDog/system-tests#7496](DataDog/system-tests#7496)
korniltsev-grafanista
added a commit
to grafana/pyroscope-dotnet
that referenced
this pull request
Oct 29, 2026
* [Version Bump] 3.53.0 (#9104)
The following files were found to be modified (as expected)
- [x] docs/CHANGELOG.md
- [x] .azure-pipelines/ultimate-pipeline.yml
- [x]
profiler/src/ProfilerEngine/Datadog.Profiler.Native.Linux/CMakeLists.txt
- [x]
profiler/src/ProfilerEngine/Datadog.Profiler.Native.Windows/Resource.rc
- [x]
profiler/src/ProfilerEngine/Datadog.Profiler.Native/dd_profiler_version.h
- [x]
profiler/src/ProfilerEngine/Datadog.Linux.ApiWrapper/CMakeLists.txt
- [x] profiler/src/ProfilerEngine/ProductVersion.props
- [x] shared/src/Datadog.Trace.ClrProfiler.Native/CMakeLists.txt
- [x] shared/src/Datadog.Trace.ClrProfiler.Native/Resource.rc
- [x] shared/src/msi-installer/WindowsInstaller.wixproj
- [x] shared/src/native-src/version.h
- [x] tracer/build/artifacts/dd-dotnet.sh
- [x] tracer/build/_build/Build.cs
- [x]
tracer/samples/AutomaticTraceIdInjection/MicrosoftExtensionsExample/MicrosoftExtensionsExample.csproj
- [x]
tracer/samples/AutomaticTraceIdInjection/Log4NetExample/Log4NetExample.csproj
- [x]
tracer/samples/AutomaticTraceIdInjection/NLog40Example/NLog40Example.csproj
- [x]
tracer/samples/AutomaticTraceIdInjection/NLog45Example/NLog45Example.csproj
- [x]
tracer/samples/AutomaticTraceIdInjection/NLog46Example/NLog46Example.csproj
- [x]
tracer/samples/AutomaticTraceIdInjection/SerilogExample/SerilogExample.csproj
- [x] tracer/samples/ConsoleApp/Alpine3.10.dockerfile
- [x] tracer/samples/ConsoleApp/Alpine3.9.dockerfile
- [x] tracer/samples/ConsoleApp/Debian.dockerfile
- [x] tracer/samples/OpenTelemetry/Debian.dockerfile
- [x] tracer/samples/WindowsContainer/Dockerfile
- [x] tracer/src/Datadog.Trace.ClrProfiler.Managed.Loader/Startup.cs
- [x] tracer/src/Datadog.Tracer.Native/CMakeLists.txt
- [x] tracer/src/Datadog.Tracer.Native/dd_profiler_constants.h
- [x] tracer/src/Datadog.Tracer.Native/Resource.rc
- [x] tracer/src/Directory.Build.props
- [x] tracer/src/Datadog.Trace/TracerConstants.cs
@DataDog/apm-dotnet
Co-authored-by: dd-octo-sts[bot] <200755185+dd-octo-sts[bot]@users.noreply.github.com>
* Fix broken unit tests in `GcMemoryLoadCalculatorTests` (#9101)
## Summary of changes
Fixes the new unit tests that only fail on master
## Reason for change
The behaviour of `Environment.SetEnvironmentVariable(.., "")` is
different on < .NET 9, which causes the new unit tests introduced in
#9505 to break only on master, where we run .NET 7 and .NET 8. This
fixes those tests
## Implementation details
These tests cases don't work on < .NET 9, because they _don't_ set a
`""` variable, they instead delete the variable:
```c#
[InlineData(PlatformKeys.DotNetGCHighMemPercent, "")]
```
In .NET 9+ these tests work, as they _do_ set the variable with the `""`
value
## Test coverage
Added a couple of extra cases just to cover other edge cases while I was
there.
I'll make sure to do a full TFM run to be sure it's fixed
## Other details
Currently causing failures on master (not PRs) so high urgency
* [Test Package Versions Bump] Updating package versions (#9100)
Updates the package versions for integration tests. See below for a
summary of the changes.
## Package Version Updates
| Package | Integration | Previous | New | Published |
|---------|-------------|----------|-----|-----------|
| AWSSDK.Core | AwsSdk | 4.0.101 | 4.0.102 | 2026-08-20 |
| AWSSDK.DynamoDBv2 | AwsDynamoDb | 4.0.103.1 | 4.0.103.4 | 2026-08-20 |
| AWSSDK.Kinesis | AwsKinesis | 4.0.100.8 | 4.0.100.10 | 2026-08-20 |
| AWSSDK.SQS | AwsSqs | 4.0.100.8 | 4.0.100.10 | 2026-08-20 |
| AWSSDK.SimpleNotificationService | AwsSns | 4.0.100.8 | 4.0.100.10 |
2026-08-20 |
| AWSSDK.EventBridge | AwsEventBridge | 4.0.100.8 | 4.0.100.10 |
2026-08-20 |
| AWSSDK.S3 | AwsS3 | 4.0.102.1 | 4.0.102.3 | 2026-08-20 |
| AWSSDK.StepFunctions | AwsStepFunctions | 4.0.100.8 | 4.0.100.10 |
2026-08-20 |
| GraphQL | GraphQL7 | 8.8.4 | 8.8.5 | 2026-08-18 |
| Google.Protobuf | Protobuf | 3.35.1 | 3.36.0 | 2026-08-20 |
| ServiceStack.Redis | ServiceStackRedis | 10.1.2 | 10.1.4 | 2026-08-18
|
## New Major Versions Available
These have a new major outside our supported range. Review whether to
widen the
integration's `MaximumVersion` and the definition's
`MaxVersionExclusive`/`SpecificVersions`,
or add a new split entry, to adopt.
| Package | Integration | Current cap | Latest available |
|---------|-------------|-------------|------------------|
| xunit.v3 | XUnitRetriesV3 | 3.2.2 | 4.0.0 |
| xunit.v3 | XUnitV3 | 3.2.2 | 4.0.0 |
## Package Version Cooldown Report
The following versions were published less than **2 days** ago and have
been ignored.
These require manual review before inclusion.
| Package | Integration | Version Kept | Ignored Version | Published |
Age (days) |
|---------|-------------|--------------|-----------------|-----------|------------|
| AWSSDK.Kinesis | AwsKinesis | 4.0.100.10 | 4.0.101 | 2026-08-21 | 1 |
| StackExchange.Redis | StackExchangeRedis | 3.1.13 | 3.1.31 |
2026-08-21 | 1 |
| log4net | log4net | 3.3.2 | 3.4.0 | 2026-08-21 | 1 |
| OpenTelemetry.Api | OpenTelemetry | 1.17.0 | 1.18.0 | 2026-08-21 | 1 |
Co-authored-by: dd-octo-sts[bot] <200755185+dd-octo-sts[bot]@users.noreply.github.com>
* Move secrets to be protected environment secrets (#8865)
## Summary of changes
- Don't pass secrets down, instead obtain directly from environment
- Only expose secret on steps that need it (Nuke step)
## Reason for change
We're moving `AZURE_DEVOPS_TOKEN` from being a repo secret to a
protected environment secret. This does some of the associated cleanup.
## Implementation details
- **Don't try to pass the secrets in to the worfklow**. AFAICT, this
technically wasn't working as we expected 😅 The "top level"
`create_normal_draft_release` workflow was trying to access and pass in
the variables, but as it wasn't running in an `environment`, this was
passing an empty string. However, this didn't break anything, because
`{{ secrets.AZURE_DEVOPS_TOKEN }}` was transparently accessing the
environment secret _anyway_. So we can just stop _trying_ to pass them
in now.
- `create-system-test-docker-base-images.yml` needs to be in the
"release" env, because it needs the azure devops token to grab the build
results from CI (avoiding rate limits)
Additionally, I duplicated the repo-secret `AZURE_DEVOPS_TOKEN` into the
existing `publish-debug-symbols-env` environment, so when we merge this,
everything should "just work".
Finally, we can make the `publish-debug-symbols-env`, "protected" so it
only runs on release branches.
## Test coverage
Nope, we can't really test any of this 🙁 Hence why I want to babysit it
the first time 😅
## Other details
I'd like to change `publish-debug-symbols-env` to be called
`release-env` (or something similar), but that means changing a whole
bunch of `sts` token stuff, across multiple repos, and it doesn't seem
worth the hassle 😅
* [Feature Flags] Send the split serial id on exposure events (#9088)
## Summary of changes
Add an optional `serial_id` field to the feature-flag exposure event,
and send it when the split carries a serial id.
## Reason for change
The exposures intake uses the serial id to find the holdout that an
allocation comes from. The UFC compiler rewrites a holdout into a usual
allocation before the tracer receives the flag configuration, so the
exposure event holds no other link back to the holdout.
The tracer already reads the serial id and uses it for APM span
enrichment. This change also puts it on the exposure event.
## Implementation details
| File | Change |
|---|---|
| `Exposure/Model/ExposureEvent.cs` | Add the optional `SerialId`
property. It goes on the wire as a top-level `serial_id`. |
| `FeatureFlagsEvaluator.cs` | Give the serial id from the split to the
exposure dispatch. |
| `Exposure/ExposureCache.cs` | Compare the serial id in the
deduplication cache. |
| `Exposure/ExposureApi.cs` | Make `SerializerSettings` internal, so
that a test can assert the wire shape. |
`ResolveVariant` gives the serial id to `DispatchExposure` directly. It
does not read the value from the flag metadata, because `ResolveVariant`
writes that metadata entry only when span enrichment is on.
The serializer includes nulls, so `ExposureEvent` gives `SerialId` a
`ShouldSerializeSerialId` method. An event without a serial id omits the
key instead of sending `null`. A `JsonProperty` named argument cannot do
this, because `AttributeTests` forbids an attribute argument whose type
is not in the core assembly, and `NullValueHandling` is an enum in
`Datadog.Trace`. `ProbeException.ShouldSerializeStackTrace` solves the
same problem the same way.
The tracer does not validate the value. The flag configuration layer
validates it.
## Test coverage
Nine new test cases in three classes:
- `ExposureEventTests` — the wire shape: `serial_id` for `340132`,
`serial_id` for `0`, and no key when the value is absent.
- `FeatureFlagsEvaluatorTests` — the dispatched event, for a split with
a serial id and for a split without one.
- `ExposureCacheTests` — deduplication when the serial id appears,
changes, disappears, and when it is `0`.
A serial id of `0` is a usual value. Serial ids start at `0` for each
organization, so the first allocation of every organization has one. The
tests pin `0` as present, not as absent.
```
dotnet test tracer/test/Datadog.Trace.Tests/Datadog.Trace.Tests.csproj -f net10.0 \
--filter "FullyQualifiedName~Datadog.Trace.Tests.FeatureFlags"
```
378 tests pass.
## Other details
The deduplication cache now compares the serial id. A subject therefore
sends one more exposure when its serial id appears or changes while the
allocation key and the variant key stay the same. This is intended: a
configuration refresh can add or change a serial id without changing
either key, and the intake must receive the new value.
Part of the work to send the serial id from every Datadog SDK.
`DataDog/dd-trace-py#19753` and `DataDog/dd-trace-go#5214` make the same
change.
*This description was generated by Claude.*
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Avoid unobserved tasks for synchronous delegate exceptions (#9071)
## Summary of changes
Prevent delegate instrumentation from creating an orphaned faulted task
when a Task-returning delegate throws synchronously.
## Reason for change
[Handler exceptions in AWS Lambda resurface as UnobservedTaskException
after upgrading 3.45.0 -> 3.51.1
#9069 ](https://github.com/DataDog/dd-trace-dotnet/issues/9069)
AWS Lambda executes customer handlers through a generated synchronous
action. For async handlers, [AWS calls
GetAwaiter().GetResult()](https://github.com/aws/aws-lambda-dotnet/blob/91df4deb1e05d9b4042b8837ed81700e4db7224e/Libraries/src/Amazon.Lambda.RuntimeSupport/Bootstrap/InvokeDelegateBuilder.cs#L220-L243),
which can throw synchronously when the customer’s task is faulted. AWS
invokes this action [before creating the outer task with
Task.FromResult](https://github.com/aws/aws-lambda-dotnet/blob/91df4deb1e05d9b4042b8837ed81700e4db7224e/Libraries/src/Amazon.Lambda.RuntimeSupport/Bootstrap/HandlerWrapper.cs#L70-L80).
In this case, [the wrapper propagates the synchronous
exception](https://github.com/DataDog/dd-trace-dotnet/blob/ba875da17863b62331b9e4e0ec9e1a4e53ab9176/tracer/src/Datadog.Trace/Util/Delegates/DelegateInstrumentation.cs#L884-L904),
but the finally block also [creates an asynchronous continuation with a
null original
task](https://github.com/DataDog/dd-trace-dotnet/blob/ba875da17863b62331b9e4e0ec9e1a4e53ab9176/tracer/src/Datadog.Trace/Util/Delegates/DelegateInstrumentation.cs#L311-L322).
The continuation rethrows the same exception, becoming a faulted task
that cannot be returned because the original synchronous exception is
already leaving the wrapper.
That task is never observed and eventually raises
`TaskScheduler.UnobservedTaskException` during garbage collection.
The PR that likely triggered this error is the one adding support for
`Amazon.Lambda.RuntimeSupport` 2.x
https://github.com/DataDog/dd-trace-dotnet/pull/8797
## Implementation details
Only rethrow an exception from the continuation when the wrapped
delegate actually returned a task.
## Test coverage
Added a non-parallel regression test. The test fails against the
previous implementation and passes with this change.
* [Propagators] Generalize tracestate parsing (#9000)
## Summary of changes
- Use `RealOnlySpan` in W3C Tracestate parsing
## Reason for change
Isolate reusable tracestate parsing parts to later add OpenTelemetry
consistent-probability-sampling feature #8983.
There will be more overhead in the next PR since it requires parsing a
new tracestate member, and some of it is already there since this PR is
making the parsing less specific
---------
Co-authored-by: Zach Montoya <zach.montoya@datadoghq.com>
* Split system-test docker image creation to work in both protected and unprotected environments (#9094)
## Summary of changes
Update system-test docker image creation to work both on master and on
PRs
## Reason for change
#8865 moves our secrets into protected environments, which means they
can no longer be accessed from PRs. Unfortunately, the docker-image
artifact creation can be enabled on PRs by adding the
`docker_image_artifacts` label, so that PR breaks that workflow. This PR
re-enables it, with the caveat that it is subject to stricter rate
limits (which shouldn't be an issue in practice, given this is rarely
used, and the only workflow that needs this approach)
## Implementation details
- Stacked on #8865
- Extract a helper for making anonymous connections to Azure DevOps
- Move the majority of the existing workflow into the composite action
instead
- Create two different workflows
- One runs on master/hotfix/release branches, uses the protected
environment and the github secret
- The other runs on branches and uses the anonymous workflow
## Test coverage
~This is the test 🤞~
As this depends on new workflows, I don't think we can test it till
_after_ it's merged, so :yolo:
## Other details
* Don't need to download and sign when pushing a tag (#9105)
## Summary of changes
Fix benign failures during release
## Reason for change
When a release is published, we push a tag, which triggers a gitlab
build. This triggers _Some_ workflows (OCI/lib-injection images) but we
_don't_ need to build or publish. Currently we are trying to download
from an Azure DevOps pipeline that never exists, so ultimately we fail.
## Implementation details
Don't try to download/sign nuget packages on release (tag)
## Test coverage
Can't test this until the next release, but it's hopefully pretty simple
## Other details
Copied the syntax from the `publish-artifacts` stage
* Migrate remaining integration tests to CombinatorialOrPairwiseData (#8528)
## Summary of changes
Migrates (what I think are) the remaining integration tests to use
CombinatorialOrPairwiseData with the goal of speeding up PR CI runs with
the hope of not missing regressions
## Reason for change
As we have many tests our CI takes time running them, this aims to
reduce some pressure as if a PR meets the criteria for running in
Pairwise mode we will swap from combinatorial to pairwise for all of
these integrations tests.
This will hopefully reduce times for runs that _likely_ won't modify /
impact these tests.
## Implementation details
One commit per file with the following pattern(s):
- Drop `GetEnabledConfig()` and the matching `[MemberData(...)]`
- Replace with `[CombinatorialOrPairwiseData]`
- Add Attributes : `[PackageVersionData(nameof(PackageVersions.X))]` and
`[MetadataSchemaVersionData]`
Note that some tests needed a few more changes.
## Test coverage
Less!
Same though on non-PR runs or on PR runs that end up on the thorough
testing.
## Other details
<!-- Fixes #{issue} -->
I have a planned follow up to reduce the V1 Schema tests that are run in
general (the V0/V1 split causes most tests to be doubled for no real
benefit as the move to the V1 schema has largely been replaced)
<!-- ⚠️ Note:
Where possible, please obtain 2 approvals prior to merging. Unless
CODEOWNERS specifies otherwise, for external teams it is typically best
to have one review from a team member, and one review from apm-dotnet.
Trivial changes do not require 2 reviews.
MergeQueue is NOT enabled in this repository. If you have write access
to the repo, the PR has 1-2 approvals (see above), and all of the
required checks have passed, you can use the Squash and Merge button to
merge the PR. If you don't have write access, or you need help, reach
out in the #apm-dotnet channel in Slack.
-->
* Mark DSM TransportsWorkCorrectly as flaky (#9116)
## Summary of changes
TransportsWorkCorrectly flaked, marking it as flaky.
## Reason for change
Green CI progress.
## Implementation details
Marked it as flaky
## Test coverage
## Other details
<!-- Fixes #{issue} -->
Did not investigate, marking as flaky - if it flakes with this I say we
skip it and raise it to be reworked.
#incident-57303
<!-- ⚠️ Note:
Where possible, please obtain 2 approvals prior to merging. Unless
CODEOWNERS specifies otherwise, for external teams it is typically best
to have one review from a team member, and one review from apm-dotnet.
Trivial changes do not require 2 reviews.
MergeQueue is NOT enabled in this repository. If you have write access
to the repo, the PR has 1-2 approvals (see above), and all of the
required checks have passed, you can use the Squash and Merge button to
merge the PR. If you don't have write access, or you need help, reach
out in the #apm-dotnet channel in Slack.
-->
* Mark DSM ContextPropagation as flaky (#9117)
## Summary of changes
Mark
Datadog.Trace.ClrProfiler.IntegrationTests.DataStreamsMonitoringManualApiTest.ContextPropagation
as flaky
## Reason for change
Flaked in CI, Green CI, trying to make more progress in getting a green
CI
## Implementation details
Marked as flaky, didn't investigate further.
## Test coverage
Same-ish
## Other details
<!-- Fixes #{issue} -->
#incident-57303
<!-- ⚠️ Note:
Where possible, please obtain 2 approvals prior to merging. Unless
CODEOWNERS specifies otherwise, for external teams it is typically best
to have one review from a team member, and one review from apm-dotnet.
Trivial changes do not require 2 reviews.
MergeQueue is NOT enabled in this repository. If you have write access
to the repo, the PR has 1-2 approvals (see above), and all of the
required checks have passed, you can use the Squash and Merge button to
merge the PR. If you don't have write access, or you need help, reach
out in the #apm-dotnet channel in Slack.
-->
* [CI Visibility] Fix coverage rewrites with shared framework dependencies (#9102)
## Summary of changes
- resolve coverage rewrite dependencies from runtimeconfig-declared .NET
shared frameworks
- stage and validate rewritten DLL/PDB files before replacing the
originals
- restore the original assembly and symbols if publication fails
## Reason for change
The coverage collector could not resolve assemblies that were available
only through an installed shared framework, such as
`Microsoft.Extensions.Logging.Abstractions` from
`Microsoft.AspNetCore.App`. If Mono.Cecil encountered that dependency
while writing metadata, it had already truncated the assembly under test
and left a damaged DLL on disk.
## Implementation details
- discover `framework`, `frameworks`, and `includedFrameworks` entries
from sibling `*.runtimeconfig.json` files
- derive the installed shared-framework root from
`System.Private.CoreLib` and select the highest compatible patch version
using full semantic-version ordering
- cache immutable shared-framework search directories per output
directory while keeping Cecil assemblies resolver-owned
- write DLL/PDB output to unique sibling transaction files with
non-assembly extensions, validate both by reopening them, then publish
with `File.Replace` backups and rollback
- restore both backups after partial publication failures, including
when the target assembly is missing or already replaced
## Test coverage
- `CoverageResolverTests` on net10.0 and net8.0
- `CoverageRewriteTests` on net10.0
- `Datadog.Trace.Coverage.collector` project build
- net10.0 three-project reproduction with `Microsoft.AspNetCore.App` and
an optional `LogLevel` parameter
## Other details
Windows and Linux CI provide the remaining platform coverage for
replacement and locking semantics.
Fixes #9023
* [CI Visibility] Refactor CODEOWNERS parsing and source ownership resolution (#9099)
## Summary of changes
- Refactor CODEOWNERS support into the dedicated
`Datadog.Trace.Ci.CodeOwnership` namespace.
- Separate provider-independent matching from GitHub and GitLab rule
evaluation.
- Separate CODEOWNERS discovery, repository path resolution, ownership
matching, and CI session integration.
- Fix CODEOWNERS matching to follow GitHub and GitLab semantics.
- Normalize compiler-recorded source paths so CI Visibility can resolve
repository files and their owners.
- Resolve CODEOWNERS once per test session and reuse a bounded
source-path cache across test suites.
## Reason for change
CI Visibility uses CODEOWNERS metadata to determine test ownership. The
previous parser mixed GitHub and GitLab behavior and diverged from both
providers in several cases, which could assign incorrect owners.
Compiler-recorded source paths can also contain CI-agent-specific
relative prefixes. Valid repository files could therefore appear outside
the CODEOWNERS root and lose their ownership metadata.
The implementation grew while fixing these correctness issues, so the
final structure now gives parsing, provider behavior, file discovery,
path normalization, and session lifetime separate responsibilities.
## Architecture
- `CodeOwners.cs` contains parsing entry points and the shared bounded
glob matcher.
- `CodeOwners.GitHub.cs` implements GitHub owner validation, rule
parsing, and global last-match precedence.
- `CodeOwners.GitLab.cs` implements sections, default owners, scoped
exclusions, duplicate-pattern handling, and independent section
evaluation.
- `CodeOwnersFileLocator.cs` finds the applicable file and selects the
dialect from repository metadata, CI provider metadata, or an
unambiguous provider-specific location.
- `RepositorySourcePathResolver.cs` converts compiler paths into safe
repository-relative paths.
- `CodeOwnersResolver.cs` owns the session-level CODEOWNERS instance and
bounded source-path cache.
- `SourceOwnership.cs` carries the resolved repository path, matched
owners, and pre-serialized CI tag.
- `CIEnvironmentValues` now only initializes and exposes the
session-level resolver.
## Matching and path behavior
- GitHub uses the last matching rule globally.
- GitLab evaluates sections independently, applies section defaults, and
keeps exclusions scoped to their section.
- Rooted and unrooted patterns, directory patterns, wildcards,
globstars, escaped characters, comments, case sensitivity, and Windows
separators follow provider-specific behavior.
- The glob matcher has bounded pattern size and match steps so
pathological rules cannot block the instrumented process.
- Unresolved relative compiler paths are anchored using the longest
existing suffix under the CODEOWNERS root.
- Absolute paths outside the repository, URIs, bare filenames, interior
navigation segments, and non-existent suffixes are not re-anchored.
## Allocation and lifetime considerations
- The CODEOWNERS file is discovered and parsed once when the test
session is initialized.
- Test suites reuse the same resolver and source-path decisions.
- The source-path cache disables itself after its configured limit
instead of retaining paths without bounds.
- Repository suffix inspection uses span-based segment enumeration and
allocates only the selected normalized suffix.
- GitLab allocates a set only when owners from multiple matching
sections must be combined.
- Parser-only section names, defaults, pattern keys, and rule lists are
released after parsing.
- The serialized `test.codeowners` value is created once per cached
ownership result.
## Test coverage
- Added specification-focused tests for GitHub and GitLab matching and
owner parsing.
- Added tests against this repository CODEOWNERS file and real-world
GitHub and GitLab examples.
- Added provider and file-location detection coverage for GitHub,
GitLab, self-managed GitLab, Jenkins, and Azure Pipelines inputs.
- Extended source-path fallback coverage for foreign relative prefixes,
repository-root files, duplicate separators, URIs, absolute paths,
navigation segments, and paths outside the repository.
- Added lifecycle and allocation-focused coverage for session-level
discovery, cached resolutions, parser-state release, and bounded
matching behavior.
## Other details
No public API or dependency changes.
* Expose the `TrailerSize` of an `ISpanBufferSerializer` (#9115)
## Summary of changes
Expose the `TrailerSize` the same way we do `HeaderSize`
## Reason for change
The `SpanBuffer` implementation can use this to ensure that the buffer
is sized appropriately. Without it, there's an edge case that requires
doubling the size of the buffer _after_ the traces have been written,
when we're about to flush
## Implementation details
- Add `ISpanBufferSerializer.TrailerSize`
- Implement everywhere (`0` for messagepack, JSON has constant value)
- Update `SpanBuffer` to take `TrailerSize` into account during resizes
## Test coverage
Meh, covered by the next PR
## Other details
Preparatory work for #9107
* Reduce locking in `AgentWriter` to avoid contention (#9107)
## Summary of changes
Update `AgentWriter` to reduce chance of locked buffers
## Reason for change
In #9007 (and related PRs) we updated the `AgentWriter` flush loop to
avoid simultaneous flushes. The intention was to avoid the "both buffers
locked" failure scenario we have seen in CI.
However, there were still several edge cases that could lead to the
serialization thread seeing _both_ buffers as locked _despite_ the fact
we only ever locked one at a time (during flushing).
The serialization thread can "chase" the lock from one buffer to the
other and
see `Locked` twice even though the two buffers _were never locked
simultaneously_. #9007 prevented _simultaneous_ locking, but it doesn't
prevent this scenario.
What's worse, we kind of cause it ourselves, because of how we call
`WakeFlushLoop` from the serialization thread. For example
1. Flush loop is running, and is locking the front buffer (currently
active buffer)
2. Serialization thread tries to write, but finds the buffer locked, so
it calls `SwapBuffers()` and `WakeFlushLoop`
3. Flush Loop finishes, doesn't sleep because of the trigger, and so
immediately locks the back buffer (the currently active buffer, again)
4. Serialization thread tries to write, finds the buffer locked again,
and drops the trace (two buffers locked)
There are various other factors that also make this worse:
- In `SpanBuffer.TryWrite` we call `Monitor.TryEnter` with _zero_
timeout. Even if the flush loop is locking for a tiny time (e.g. if
`TraceCount == 0`) the write will fail. In other words, we don't
_actually_ have to be flushing to report `Locked`
- `FlushBuffer` locks the _active_ buffer _at least_ once every second
(more when there's heavy load/contention/forced flushing).
There's also a latent bug where a locked-but-not-full buffer can end up
skipped and sitting in the background and not being flushed until
shutdown. I'll address that in a follow up PR
## Implementation details
The key fix I implemented in the end is to _not_ lock the buffer for the
whole duration of the flush. Instead, we use a third buffer that we
"swap in", so that we can reset the buffer, and keep the contention
period to a minimum.
_Technically_, this _should_ mean that we can now _block_ on
`SpanBuffer.TryWrite`, instead of trying to aquire the lock, and bailing
if we can't. Given that the critical section during a flush is now just
"write the header + trailer, then reset the buffer", this _should_ be
tiny. However, I was too much of a coward to go this route, and instead
opted for `Monitor.TryEnter` with a 100ms timeout. We _shouldn't_ hit it
(though I'm interested to see if we do in CI). And if we do, we decide
the best way to handle it, but I just wanted to avoid the chance of
stalling indefinitely, with the queue building up, and would rather lose
traces in that case.
One thing to note is that #9115 is required so that we don't
accidentally try to do allocation while holding the lock on the buffer,
as that could increase the duration we hold it for.
## Test coverage
Mostly just unit tests, as this is an internal refactoring. Hopefully it
reduces our instances of double-lock in CI
## Other details
Depends on https://github.com/DataDog/dd-trace-dotnet/pull/9115
* Fix lock-contention causing trace to be "stranded" (#9120)
## Summary of changes
- Always try to flush both buffers
- Skip the flush entirely if the buffer is empty
- Don't log drop stats twice
## Reason for change
Spotted some minor bugs while working on #9107. Notably
- If a buffer was _locked_ (not full), then the serialization thread
swaps the buffers, and triggers a flush. However, as the buffer is not
full, the flush thread would _not_ flush it automatically, so the trace
sits in the back buffer unflushed, until we swap buffers again
- We were trying to log the "drop" stats _twice_ per flush, in cases
where we flush both buffers. Not strictly a bug, but unnecessary.
- We were locking the buffer and swapping the array, even if there were
no traces in it
- Not strictly a bug, and this had much more potential to cause genuine
drops prior to #9107
- Still seems like a worthwhile fix to me
## Implementation details
- Always flush both buffers, starting with the back buffer _regardless_
of whether it is full or not.
- Log the drop stats in `FlushBuffers` instead of the child
`FlushBuffer` calls
- Do a soft check on whether the buffer has stats before taking the lock
- Worse case, we get a `false` because the serialization thread is
actively writing into it, but there's naturally a race here anyway, and
we'll catch it next time around.
## Test coverage
Added a `[TestingOnly]` testing seam so that we can explicitly emulate
the scenario where the buffer was locked when we tried to write to it.
Allowed red-green testing to prove the edge case was there.
## Other details
Stacked on
- https://github.com/DataDog/dd-trace-dotnet/pull/9115
- https://github.com/DataDog/dd-trace-dotnet/pull/9107
* Consolidate and fix bugs in GitLab "download from Azure" scripts (#9103)
## Summary of changes
- Consolidates the 3 "wait for build and download artifacts from Azure"
scripts into one helper
- Fixes several minor bugs in the download logic
## Reason for change
As part of #9083, @NachoEchevarria flagged some potential improvements
to the scripts, for example a cancelled build today would cause
flakiness for the download scripts. We now also have 3 copies of
essentially the same code, and AI spotted several other bugs, all fixed
here
## Implementation details
- Extracted shared logic into a helper bash script
- Arguably we could/should make this a Nuke target given the bash
complexity, but given this all goes away completely when we move to
GitLab fully, I think the current approach is fine. It's largely
copy+paste with some generalisation.
- Fixed missing diagnostics when the loop timed out.
- Fixed an unnecessary 100s wait on each _successful_ run
- Fixed a cancelled build causing the jobs to sit waiting for artifacts
that will never run, before eventually timing out.
- Added retries for cancelled/failed builds, looking for "better"
candidate builds
- Can happen if there are multiple builds running against the same
commit (e.g. manual builds)
- Only checks a given build once, so bounded
## Test coverage
This is the test
## Other details
Note that the `.gitlab/download-serverless-artifacts.sh` and
`.gitlab/download-single-step-artifacts.sh` have different paths they
take when running against tags (during release).
`.gitlab/download-nuget-packages-to-sign.sh` does not have this path, as
it does not need to run on tags (there may be an edge case there where
we _are_ running - need to check)
* [Test Package Versions Bump] Updating package versions (#9124)
Updates the package versions for integration tests. See below for a
summary of the changes.
## Package Version Updates
| Package | Integration | Previous | New | Published |
|---------|-------------|----------|-----|-----------|
| AWSSDK.Kinesis | AwsKinesis | 4.0.100.10 | 4.0.101 | 2026-08-21 |
| StackExchange.Redis | StackExchangeRedis | 3.1.13 | 3.1.31 |
2026-08-21 |
| log4net | log4net | 3.3.2 | 3.4.0 | 2026-08-21 |
| OpenTelemetry.Api | OpenTelemetry | 1.17.0 | 1.18.0 | 2026-08-21 |
## New Major Versions Available
These have a new major outside our supported range. Review whether to
widen the
integration's `MaximumVersion` and the definition's
`MaxVersionExclusive`/`SpecificVersions`,
or add a new split entry, to adopt.
| Package | Integration | Current cap | Latest available |
|---------|-------------|-------------|------------------|
| xunit.v3 | XUnitRetriesV3 | 3.2.2 | 4.0.0 |
| xunit.v3 | XUnitV3 | 3.2.2 | 4.0.0 |
## Package Version Cooldown Report
The following versions were published less than **2 days** ago and have
been ignored.
These require manual review before inclusion.
| Package | Integration | Version Kept | Ignored Version | Published |
Age (days) |
|---------|-------------|--------------|-----------------|-----------|------------|
| AWSSDK.Core | AwsSdk | 4.0.102 | 4.0.102.1 | 2026-08-24 | 1 |
| AWSSDK.DynamoDBv2 | AwsDynamoDb | 4.0.103.4 | 4.0.103.5 | 2026-08-24 |
1 |
| AWSSDK.Kinesis | AwsKinesis | 4.0.101 | 4.0.101.1 | 2026-08-24 | 1 |
| AWSSDK.SQS | AwsSqs | 4.0.100.10 | 4.0.100.11 | 2026-08-24 | 1 |
| AWSSDK.SimpleNotificationService | AwsSns | 4.0.100.10 | 4.0.100.11 |
2026-08-24 | 1 |
| AWSSDK.EventBridge | AwsEventBridge | 4.0.100.10 | 4.0.100.11 |
2026-08-24 | 1 |
| AWSSDK.S3 | AwsS3 | 4.0.102.3 | 4.0.102.4 | 2026-08-24 | 1 |
| AWSSDK.StepFunctions | AwsStepFunctions | 4.0.100.10 | 4.0.100.11 |
2026-08-24 | 1 |
Co-authored-by: dd-octo-sts[bot] <200755185+dd-octo-sts[bot]@users.noreply.github.com>
* Bump test logger to latest (to try to fix codeowners) (#9128)
## Summary of changes
Bump the version of the testlogger
## Reason for change
It includes the latest codeowners code
## Implementation details
Bump the numbers
## Test coverage
This is the test
## Other details
Currently we're not reporting owners correctly
* Add `new Random()` to the banned API list, to encourage using those APIs (#9129)
## Summary of changes
Add `Random` to the banned API list
## Reason for change
We have `ThreadSafeRandom` which should generally be preferred
## Implementation details
- Added `Random()` constructors to the banned API lists
- I _didn't_ add `Random.Shared` (.NET 6+ only) seeing as
`ThreadSafeRandom.Shared` just delegates to this directly, so if you're
writing .NET 6+ code, using `Random.Shared` is fine.
## Test coverage
This is the test
## Other details
Spotted as part of reviewing #9042
* Try to make it so that CODEOWNERs triggers tests (#9092)
## Summary of changes
Tests rely on CODEOWNERS so if we change it we should at least run
normal integration tests.
## Reason for change
I broke tests in https://github.com/DataDog/dd-trace-dotnet/pull/9084 as
they didn't run
## Implementation details
Just went through and expanded `.github/`
Granted that does mean that _new_ files we add there (for some reason)
will need to be added, but all that happens is CI will just run if that
is forgotten so I think that is a reasonable tradeoff.
## Test coverage
😃
## Other details
<!-- Fixes #{issue} -->
<!-- ⚠️ Note:
Where possible, please obtain 2 approvals prior to merging. Unless
CODEOWNERS specifies otherwise, for external teams it is typically best
to have one review from a team member, and one review from apm-dotnet.
Trivial changes do not require 2 reviews.
MergeQueue is NOT enabled in this repository. If you have write access
to the repo, the PR has 1-2 approvals (see above), and all of the
required checks have passed, you can use the Squash and Merge button to
merge the PR. If you don't have write access, or you need help, reach
out in the #apm-dotnet channel in Slack.
-->
* [CI Visibility] Keep CODEOWNERS matches inside repository (#9127)
## Summary of changes
- Resolve CODEOWNERS from the Git repository root instead of accepting a
nested file first.
- Do not assign CODEOWNERS to source paths that cannot be resolved
inside the repository.
- Resolve absolute or relative PDB source paths recorded in a different
build environment.
- Preserve the source-file fallback tag when a compiler path cannot be
resolved.
## Reason for change
The CODEOWNERS implementation added in #9099 could select a nested
CODEOWNERS file before reaching the repository root. It could also
evaluate unresolved source text against repository rules, producing
incorrect owners.
PDB paths may come from a different machine or operating system than the
test runtime. Requiring those original paths to be valid locally drops
legitimate CODEOWNERS metadata, such as Windows build paths consumed by
Linux tests.
## Implementation details
- Find the nearest Git root before selecting a CODEOWNERS location.
- Fall back to the explicit workspace or source root when Git metadata
is unavailable.
- Re-anchor compiler paths by the longest existing suffix under the
repository root, including Windows, Unix, and file URI paths.
- Evaluate rules only after the source path has been resolved inside the
CODEOWNERS repository root.
- Reject non-file URIs, embedded rooted paths after parent traversal,
and suffixes that do not exist in the repository.
## Test coverage
- CodeOwnersFallbackTests: 34 passed.
- Added coverage for nested CODEOWNERS files, unresolved or external
paths, and cross-OS PDB paths.
- Includes the Azure layout from the CI regression: D:/a/_work/1/s/....
## Other details
- Companion logger port:
https://github.com/tonyredondo/datadog-test-logger/pull/37
- No public API or dependency changes.
* feat(feature-flags): add agentless configuration keys, settings, and endpoint derivation (#9040)
## Summary of changes
Adds the foundational configuration layer for agentless Feature Flags
delivery: new configuration keys, a `FeatureFlagsSettings` class with
source resolution and validation, and an `AgentlessEndpoint` struct for
deriving the CDN URL from the Datadog site or a custom base URL.
This is PR 1 of a stacked PR series implementing agentless Feature Flags
configuration delivery (FFL-2703), porting functionality from
[dd-trace-py#19331](https://github.com/DataDog/dd-trace-py/pull/19331)
and
[dd-trace-java#11892](https://github.com/DataDog/dd-trace-java/pull/11892).
## Reason for change
The .NET tracer currently delivers Feature Flags configuration
exclusively through the Datadog Agent's Remote Configuration. To support
agentless (CDN-backed) delivery, we need a configuration layer that:
- Selects between `agentless` (new default), `remote_config`, and
`disabled` sources
- Derives the managed CDN endpoint from `DD_SITE` or accepts a custom
`DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_BASE_URL`
- Validates poll interval and request timeout settings
- Grandfathers existing `DD_EXPERIMENTAL_FLAGGING_PROVIDER_ENABLED`
users onto Remote Configuration
## Implementation details
- **`FeatureFlagsSource` enum** — `Disabled`, `Agentless`,
`RemoteConfig`
- **`FeatureFlagsSettings`** — reads all new env vars, resolves the
source with the cross-SDK precedence contract (kill switch → explicit
source → fail-closed → legacy grandfathering → default agentless),
validates poll interval (1–3600s) and request timeout (>0s)
- **`AgentlessEndpoint`** — derives
`https://ufc-server.ff-cdn.<site>/api/v2/feature-flagging/config/rules-based/server`
from the site, appends `dd_env` when configured, accepts custom
HTTP/HTTPS URLs, and never echoes URLs in error messages (credentials
safety)
- **New config keys** in `supported-configurations.yaml` + generated
`ConfigurationKeys.FeatureFlags.g.cs`:
- `DD_FEATURE_FLAGS_ENABLED` (default `true`, supersedes
`DD_EXPERIMENTAL_FLAGGING_PROVIDER_ENABLED`)
- `DD_FEATURE_FLAGS_CONFIGURATION_SOURCE` (default `agentless`)
- `DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_BASE_URL`
-
`DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_POLL_INTERVAL_SECONDS`
(default 30, type `int`)
-
`DD_FEATURE_FLAGS_CONFIGURATION_SOURCE_AGENTLESS_REQUEST_TIMEOUT_SECONDS`
(default 5, type `int`)
- `DD_EXPERIMENTAL_FLAGGING_PROVIDER_INITIALIZATION_TIMEOUT_MS` (default
10000, implementation B)
- **`[Obsolete]`** on `FlaggingProviderEnabled` with pragma suppression
in `FeatureFlagsSettings`
This PR does not wire anything to `FeatureFlagsModule` yet — that
happens in a later PR in the stack.
## Test coverage
- `FeatureFlagsSettingsTests` — 43 test cases covering source resolution
(kill switch, explicit source, legacy grandfathering, `offline`
sentinel, invalid values), blank/whitespace handling, casing
normalization, defaults, poll interval validation, request timeout
validation, initialization timeout, and base URL reading
- `AgentlessEndpointTests` — 18 test cases covering managed endpoint
derivation (site lowercasing, staging, govcloud), `dd_env` query
parameter (addition, null, escaping), custom endpoint path handling
(origin-only gets canonical path, custom path used verbatim), HTTP
acceptance for custom endpoints, invalid URL rejection, empty/whitespace
site rejection, and credentials-in-error safety
## Other details
<!-- Fixes #{issue} -->
Stacked PRs:
- PR 1 (this): Configuration keys, settings, and endpoint derivation
- PR 2: UFC parser and agentless poller
- PR 3: Module wiring and OpenFeature activation
- PR 4: Manual API activation
<!-- ⚠️ Note:
Where possible, please obtain 2 approvals prior to merging. Unless
CODEOWNERS specifies otherwise, for external teams it is typically best
to have one review from a team member, and one review from apm-dotnet.
Trivial changes do not require 2 reviews.
MergeQueue is NOT enabled in this repository. If you have write access
to the repo, the PR has 1-2 approvals (see above), and all of the
required checks have passed, you can use the Squash and Merge button to
merge the PR. If you don't have write access, or you need help, reach
out in the #apm-dotnet channel in Slack.
-->
System tests:
[DataDog/system-tests#7496](https://github.com/DataDog/system-tests/pull/7496)
* Bump SLO for throughput tests (#9131)
## Summary of changes
Bumps the SLO for throughput tests
## Reason for change
They're occasionally touching the threshold
## Implementation details
Told the AI to bump them. It did what it did
## Test coverage
Meh, if this passes, it'll do
## Other details
Fixes https://datadoghq.atlassian.net/browse/SRTEE-7
* [Smoke Test Docker Image Bump] Updating docker image tags (#9134)
Updates the docker images used for smoke tests.
Co-authored-by: dd-octo-sts[bot] <200755185+dd-octo-sts[bot]@users.noreply.github.com>
* [Profiler] Fix crashes at shutdown (#9050)
* Add many timeouts and retries to build stages in Azure DevOps (#9141)
## Summary of changes
Add explicit timeouts + retries for stages that didn't have them
## Reason for change
We [recently had a
build](https://dev.azure.com/datadoghq/a51c4863-3eb4-4c5d-878a-58b41a049e4e/_build/results?buildId=207980)
where the unit tests stage failed because the _build_ part timed out.
The build was absolutely fine, there was just _Something_ that caused
the nuke call to hang, which eventually hit the stage timeout, and
caused everything to failure. With these extra timeouts, the build steps
should hit their "local" timeouts, and get retried, allowing the stage
to ultimately pass.
This only adds new retries to "build" stages, not to "test" stages.
In some cases, also extended the stage timeout to accommodate the new
retries - it's better for a job to take a long time than for it to fail
completely.
## Implementation details
Asked the API to grab the stats for each job in each stage, focus on the
"build" jobs, and suggest new timeouts. The timeouts are intentionally
conservative (a slow, no-retry complete is almost always preferable to a
retry that forces re-doing work).
## Test coverage
This is the test, but also, we'll see how it goes
## Other details
#incident-57303
<details><summary>The stats the AI produced. Obvs, take with a pinch of
salt, but they </summary>
<p>
| Stage | Step / command | Samples | Mean | P95 | Max | Applied timeout
| Applied retry |
| ---------------------------------- |
---------------------------------------------------------------------------------------------------------------------------------------------
| ------: | ----: | ----: | ------: | --------------: | ----------------
|
| build_windows_tracer | Build tracer home | 25 | 13.50 | 14.32 | 14.61
| 30 | 1 |
| build_windows_profiler | Build profiler home | 25 | 8.33 | 9.15 | 9.38
| 20 | 1 |
| build_linux_tracer | Clean CompileManagedLoader (docker) | 50 | 0.48 |
0.55 | 0.60 | 10 | 1 |
| build_linux_tracer | BuildManagedTracerHome ExtractDebugInfoLinux
ValidateNativeTracerGlibcCompatibility (docker) | 50 | 5.16 | 5.51 |
5.52 | 20 | 1 |
| build_linux_tracer_r2r | Clean CompileManagedLoader (docker) | 25 |
0.49 | 0.77 | 0.82 | 10 | 1 |
| build_linux_tracer_r2r | BuildManagedTracerHomeR2R
ExtractDebugInfoLinux (docker) | 25 | 5.68 | 6.15 | 6.17 | 20 | 1 |
| build_linux_universal | Clean BuildNativeLoader BuildNativeWrapper
ExtractDebugInfoLinux ValidateNativeLoaderSnapshotTestsLinux (docker) |
25 | 2.29 | 2.54 | 2.64 | **10** | 1 |
| build_arm64_tracer | Clean CompileManagedLoader BuildNativeTracerHome
BuildManagedTracerHome ExtractDebugInfoLinux
ValidateNativeTracerGlibcCompatibility (docker) | 50 | 8.62 | 9.59 |
9.80 | 20 | 1 |
| build_arm64_tracer_r2r | same, R2R variant (docker) | 25 | 9.39 | 9.97
| 10.41 | 20 | 1 |
| build_arm64_profiler | Clean BuildProfilerHome ExtractDebugInfoLinux
ValidateNativeProfilerGlibcCompatibility (docker) | 50 | 5.49 | 6.21 |
6.93 | 15 | 1 |
| build_arm64_universal | Clean BuildNativeLoader BuildNativeWrapper ...
(docker) | 25 | 1.47 | 1.69 | 1.70 | **10** | 1 |
| build_macos | Build native-tracer | 25 | 10.16 | 14.48 | 16.39 | 35 |
1 |
| build_macos | Build managed tracer + native loader | 25 | 12.72 |
19.46 | 20.76 | **30** | 1 |
| package_windows | Build MSI, Tracer home, FleetInstaller, and
TelemetryForwarder | 25 | 1.38 | 1.62 | 1.66 | **10** | 1 |
| build_dd_dotnet_windows | Build dd-dotnet | 25 | 1.30 | 1.54 | 1.70 |
10 | 1 |
| build_dd_dotnet_linux | Clean BuildDdDotnet --framework net8.0
(docker) | 50 | 1.26 | 1.37 | 1.59 | 10 | 1 |
| build_dd_dotnet_linux_arm64 | Clean BuildDdDotnet --framework TFM
(docker) | 50 | 0.95 | 1.06 | 1.08 | 10 | 1 |
| debug_builds | Clean CompileManagedLoader -buildConfiguration Debug
(docker) | 50 | 0.48 | 0.60 | 0.62 | 10 | 1 |
| debug_builds | BuildNativeTracerHome -buildConfiguration Debug
(docker) | 50 | 3.75 | 4.13 | 4.51 | 15 | 1 |
| debug_builds | BuildManagedTracerHome ExtractDebugInfoLinux
-buildConfiguration Debug (docker) | 50 | 5.18 | 5.60 | 6.21 | 20 | 1 |
| debug_builds | Build tracer home in debug | 25 | 6.27 | 7.00 | 7.05 |
30 | 1 |
| debug_builds | Clean Restore CreateRequiredDirectories
CompileManagedSrc BuildRunnerTool -buildConfiguration Debug (docker) |
25 | 4.97 | 5.57 | 5.77 | 30 | 1 |
| debug_builds | Build tracer home | 25 | 8.87 | 13.42 | 15.97 | 30 | 1
|
| unit_tests_windows | Build unit tests | 225 | 6.59 | 7.02 | 7.29 |
**15** | 3 (pre-existing) |
| unit_tests_macos | Build unit tests | 200 | 9.42 | 14.13 | 17.77 | 35
| 3 (pre-existing) |
| unit_tests_linux | BuildManagedUnitTests --framework TFM
--code-coverage-enabled BOOL (docker) | 400 | 5.92 | 6.26 | 57.72 ⚠ | 15
| 3 (pre-existing) |
| unit_tests_arm64 | same (docker) | 300 | 5.49 | 6.16 | 6.69 | 15 | 3
(pre-existing) |
| build_samples | Build sample projects | 275 | 5.19 | 7.52 | 8.11 |
**15** | 1 |
| build_samples_macos | Build sample projects | 25 | 10.43 | 18.17 |
20.43 | **40** | 1 |
| integration_tests_windows | Build integration tests | 900 | 2.64 |
3.36 | 4.32 | 10 | 3 (pre-existing) |
| integration_tests_windows_debugger | Build integration tests | 1600 |
1.53 | 1.92 | 2.91 | 10 | 3 (pre-existing) |
| integration_tests_windows_iis | Build Integration Tests | 100 | 2.36 |
2.94 | 3.44 | 10 | 3 (pre-existing) |
| msi_integration_tests_windows | BuildAspNetIntegrationTests | 50 |
2.04 | 2.66 | 3.14 | 10 | 3 (pre-existing) |
| msi_integration_tests_windows | BuildIntegrationTests | 25 | 1.65 |
2.02 | 2.05 | 10 | 3 (pre-existing) |
| integration_tests_linux | BuildIntegrationTests CompileTrimmingSamples
... (docker) | 1600 | 2.74 | 3.10 | 5.52 | 15 | 3 (pre-existing) |
| integration_tests_linux | BuildIntegrationTests --framework TFM
--SampleName Lambda (docker) | 250 | 1.40 | 2.07 | 2.43 | 15 | 3
(pre-existing) |
| integration_tests_arm64 | BuildIntegrationTests CompileTrimmingSamples
... (docker) | 300 | 2.34 | 2.60 | 3.63 | 15 | 3 (pre-existing) |
| integration_tests_arm64 | BuildIntegrationTests --framework TFM
--IncludeTestsRequiringDocker BOOL --SampleName (docker) | 300 | 1.61 |
1.73 | 2.07 | 15 | 3 (pre-existing) |
| integration_tests_linux_debugger | BuildDebuggerIntegrationTests ...
(docker) | 800 | 1.80 | 2.14 | 2.70 | 15 | 3 (pre-existing) |
| integration_tests_arm64_debugger | BuildDebuggerIntegrationTests ...
(docker) | 600 | 1.50 | 1.67 | 2.07 | 15 | 1 (was missing) |
| profiler_integration_tests_linux | BuildProfilerSamples (docker) | 50
| 1.24 | 1.54 | 1.80 | 10 | 3 (pre-existing) |
| profiler_integration_tests_arm64 | BuildProfilerSamples (docker) | 50
| 0.87 | 0.95 | 0.97 | 10 | 3 (pre-existing) |
| asan_profiler_tests | BuildProfilerAsanTest -Framework net7.0 (docker)
| 25 | 18.00 | 19.05 | 19.11 | **30** | 1 (was missing) |
| asan_profiler_tests | BuildProfilerSampleForSanitiserTests (docker) |
25 | 0.52 | 0.61 | 0.65 | **10** | 1 (was missing) |
| asan_arm64_profiler_tests | BuildProfilerAsanTest (docker) | 25 |
10.21 | 11.02 | 11.38 | **30** | 1 (was missing) |
| asan_arm64_profiler_tests | BuildProfilerSampleForSanitiserTests
(docker) | 25 | 0.25 | 0.31 | 0.34 | **10** | 1 (was missing) |
| ubsan_profiler_tests | BuildProfilerUbsanTest (docker) | 25 | 18.10 |
19.20 | 19.49 | **30** | 1 (was missing) |
| ubsan_profiler_tests | BuildProfilerSampleForSanitiserTests (docker) |
25 | 0.51 | 0.62 | 0.65 | **10** | 1 (was missing) |
| ubsan_arm64_profiler_tests | BuildProfilerUbsanTest (docker) | 25 |
10.39 | 11.27 | 11.81 | **30** | 1 (was missing) |
| ubsan_arm64_profiler_tests | BuildProfilerSampleForSanitiserTests
(docker) | 25 | 0.25 | 0.31 | 0.32 | **10** | 1 (was missing) |
| dotnet_tool | Build tools and pack NuGet packages (Bundle,
AzureFunctions, Benchmark) | 25 | 8.64 | 9.26 | 9.92 | 20 | 1 (was
missing) |
</p>
</details>
* [CI Visibility] Stabilize CODEOWNERS concurrency test (#9143)
## Summary of changes
- Run the CODEOWNERS concurrency check on four dedicated threads.
- Start all workers together while keeping the existing timeout and
ownership assertions.
## Reason for change
The test used `Task.Run` and then blocked its xUnit worker with
`Task.WaitAll`. On a busy CI agent, the shared ThreadPool could not
schedule all 64 tasks before the timeout, so the test reported a
CODEOWNERS deadlock even though matching had completed normally. This
failed in [Azure build
208058](https://dev.azure.com/datadoghq/dd-trace-dotnet/_build/results?buildId=208058).
## Implementation details
`TaskCreationOptions.LongRunning` gives each of the four workers a
dedicated thread, and a start signal makes the matching calls
concurrent. The test no longer depends on ThreadPool capacity shared
with the rest of the test suite.
## Test coverage
- `CodeOwnersSpecTests`: 92 passed.
- The affected test passed 10 consecutive runs (approximately 27-30 ms
each).
## Other details
Test-only change; no production behavior or public API changes.
#incident-57303
* Move CI uploads to new subscription (#9142)
## Summary of changes
Upload/download build artifacts to new Azure subscription
## Reason for change
We are moving uploads to a new location for technical reasons. We'll
need to update all downstream consumers to pull from the new location.
We already have some build dependencies stored here, so we're just
expanding that for now (until we move to gitlab)
## Implementation details
- Dual-upload to the old and new location, until we can update all the
consumers to pull from the new place.
- Extracted common code to reusable action for now
- We can inline this again once we have all the consumers of the old
location migrated
## Test coverage
Did a run
[here](https://dev.azure.com/datadoghq/dd-trace-dotnet/_build/results?buildId=208065&view=results)
that forces the dual upload location, verified it exists
## Other details
AFAICT, this covers everything using the "old" subscription in this
repo. Next step is to update consumers to pull from the new location
instead
* feat(openfeature): support arbitrary semver parts (#9122)
## Summary
- refresh FFE fixtures from 2c0c8d55f665fe4a847fc0ded3abf59b7e2896fc
- allow any number of numeric SemVer core components
- cover six-part and longer core version comparisons
## Testing
- `dotnet test
tracer/test/Datadog.Trace.Tests/Datadog.Trace.Tests.csproj --no-restore
--framework net10.0 --filter FullyQualifiedName~SemanticVersionTests
--logger "console;verbosity=minimal"`
- `dotnet test
tracer/test/Datadog.Trace.Tests/Datadog.Trace.Tests.csproj --no-restore
--framework net10.0 --filter
FullyQualifiedName~FeatureFlagsEvaluatorTests.BundledTest --logger
"console;verbosity=minimal"`
* feat(feature-flags): add UFC parser and agentless HTTP poller (#9042)
## Summary of changes
Adds the agentless delivery mechanism: a `UfcConfigurationParser` that
validates the JSON:API envelope returned by the CDN endpoint, and an
`AgentlessConfigurationSource` HTTP poller with retries, ETag caching,
gzip decompression, and shutdown safety.
This is PR 2 of a stacked PR series implementing agentless Feature Flags
configuration delivery (FFL-2703), porting functionality from
[dd-trace-py#19331](https://github.com/DataDog/dd-trace-py/pull/19331)
and
[dd-trace-java#11892](https://github.com/DataDog/dd-trace-java/pull/11892).
Stacked on [PR 1
#9040](https://github.com/DataDog/dd-trace-dotnet/pull/9040).
## Reason for change
PR 1 added the configuration layer (settings, endpoint derivation). This
PR adds the actual HTTP delivery mechanism that polls the CDN endpoint
for UFC flag configuration. It is self-contained: the poller and parser
are fully testable in isolation and are not yet wired to the
`FeatureFlagsModule` (that happens in PR 3).
## Implementation details
- **`UfcConfigurationParser`** — validates the JSON:API Universal Flag
Configuration response envelope, extracts `data.attributes` into
`ServerConfiguration`. Rejects raw UFC documents (without the JSON:API
wrapper) so all delivery sources agree on one wire format. Returns
descriptive errors; never throws.
- **`AgentlessConfigurationSource`** — HTTP poller with:
- Fixed-delay polling (no overlap)
- In-tick retries (max 3) for 408/429/5xx/network failures, with
jittered backoff derived from the poll interval (`clamp(P/6, 2s, 10s)`
then `clamp(P/3, 5s, 30s)`)
- ETag / `If-None-Match` conditional polling; ETag advances only after
parse + apply succeed
- Gzip decompression (neither transport decompresses automatically)
- Self-tracing suppression (`x-datadog-tracing-enabled: false` header)
- API key sent only to managed (Datadog-derived) endpoints, never custom
ones
- Warn-once per failure category (authentication, http, request) to
prevent log flooding
- Shutdown safety: `CancellationTokenSource` cancels the loop; shutdown
checks after request and before `ApplyAsync` prevent use-after-dispose
- `Create()` factory returns `null` when endpoint can'"'t be built or
API key is missing
- Idempotent `Start()` via `Interlocked.CompareExchange`
## Test coverage
- **`UfcConfigurationParserTests`** (16 tests) — valid envelope parsing,
malformed JSON rejection, invalid envelope rejection (wrong type,
missing data, raw UFC), invalid attributes rejection (missing
format/createdAt/environment/flags, wrong types), flag parsing
- **`AgentlessConfigurationSourceTests`** (12 tests) — 200 success, ETag
forwarding, 304/401/malformed handling, disposal before/after request,
retry on 500 then success, max retries on 500, no retry on 400, gzip
response, network error, idempotent start
## Other details
<!-- Fixes #{issue} -->
Stacked PRs:
- PR 1 [#9040](https://github.com/DataDog/dd-trace-dotnet/pull/9040):
Configuration keys, settings, and endpoint derivation
- PR 2 (this): UFC parser and agentless HTTP poller
- PR 3: Module wiring and OpenFeature activation
- PR 4: Manual API activation
<!-- ⚠️ Note:
Where possible, please obtain 2 approvals prior to merging. Unless
CODEOWNERS specifies otherwise, for external teams it is typically best
to have one review from a team member, and one review from apm-dotnet.
Trivial changes do not require 2 reviews.
MergeQueue is NOT enabled in this repository. If you have write access
to the repo, the PR has 1-2 approvals (see above), and all of the
required checks have passed, you can use the Squash and Merge button to
merge the PR. If you don't have write access, or you need help, reach
out in the #apm-dotnet channel in Slack.
-->
System tests:
[DataDog/system-tests#7496](https://github.com/DataDog/system-tests/pull/7496)
* [Profiler] Fix flacky tests (#9126)
## Summary of changes
- retry on ARM64 when possible truncated call stacks
- don't assume all events for class histograms are received compared to
reference chains
## Reason for change
Flack tests
## Implementation details
- add retry mechanism on ARM64 + invalid frame
- remove to strict comparison between reference chain and class
histogram
## Test coverage
Corresponding tests have been updated
## Other details
<!-- Fixes #{issue} -->
#incident-57303
<!-- ⚠️ Note:
Where possible, please obtain 2 approvals prior to merging. Unless
CODEOWNERS specifies otherwise, for external teams it is typically best
to have one review from a team member, and one review from apm-dotnet.
Trivial changes do not require 2 reviews.
MergeQueue is NOT enabled in this repository. If you have write access
to the repo, the PR has 1-2 approvals (see above), and all of the
required checks have passed, you can use the Squash and Merge button to
merge the PR. If you don't have write access, or you need help, reach
out in the #apm-dotnet channel in Slack.
-->
* [Profiler] Fix start/stop crash at shutdown (#9037)
* [Profiler] ManagedCodeCache updates synchronously (#8840)
* [CI Visibility] Resolve CODEOWNERS from matching local checkout (#9148)
## Summary of changes
- Use the current local Git checkout as a CODEOWNERS fallback when the
CI workspace paths are not available in the test process.
- Trust that fallback only when the local Git commit exactly matches the
commit reported by CI.
## Reason for change
Distributed CI jobs can build and run tests on different machines and
filesystems. Our benchmark job reports a Linux CI workspace and a GitLab
mirror URL, while BenchmarkDotNet executes in a Windows checkout cloned
from GitHub. The resolver therefore cannot reach the reported workspace
and selects the GitLab CODEOWNERS layout instead of the
.github/CODEOWNERS file present in the worker checkout. The benchmark
source paths are valid, but their test events receive no owners.
## Implementation details
- Keep the GitInfo instance that is already collected during CI
environment initialization.
- Compare its commit with the CI commit once when the test session
starts.
- When they match, pass the local repository root and URL to CODEOWNERS
discovery.
- Search the local Git root only after the CI source and workspace roots
fail.
- Use the local repository URL to select the matching CODEOWNERS dialect
and location.
This adds no extra Git command and no per-test allocation or filesystem
scan. CODEOWNERS is still located and loaded once per test session.
## Test coverage
- Added a regression test for a GitLab mirror with an unavailable CI
workspace, a GitHub worker checkout, and a relocated Windows PDB source
path.
- Added a guard test proving that a local checkout at a different commit
is not trusted.
- All 179 CODEOWNERS tests pass on .NET 6 and .NET 8.
## Other details
No public API or dependency changes.
* [CI Visibility] Bump Datadog test packages to 0.0.59 (#9151)
## Summary of changes
- Bump `DatadogTestLogger` from `0.0.56` to `0.0.59`.
- Bump `DatadogTestCollector` from `0.0.56` to `0.0.59`.
## Reason for change
Consume the latest test logger release, including the CODEOWNERS
local-checkout fixes from
[datadog-test-logger#40](https://github.com/tonyredondo/datadog-test-logger/pull/40).
## Implementation details
- Update the shared tracer test package references.
- Update the profiler integration test logger reference.
## Test coverage
- Restored `Datadog.Trace.Tests` successfully and verified both packages
resolve to `0.0.59`.
- Restored `Datadog.Profiler.IntegrationTests` successfully and verified
`DatadogTestLogger` resolves to `0.0.59`.
## Other details
Dependency-only change.
* [Profiler] Fix bug in FrameStore (#9140)
* [Profiler] Fix flaky tests caused by failing`timer_create` (#8988)
* Add SDK capabilities and LP as owners in addition to tracing-dotnet (#9130)
## Summary of changes
Adds SDK Capabilities and Language Platform as owners for paths
previously owned solely by `tracing-dotnet`. Retains `tracing-dotnet` as
the review team.
## Reason for change
Green CI and other initiatives are trying to better define "owning"
teams and "reviewing" teams, this is a step toward that.
`tracing-dotnet` is intended to be a "reviewing" team.
More specific teams are intended to be "owning" teams.
We don't want this to be _super_ disruptive at the moment so trying to
implement this bit by bit, so for this case we are going from _pure_
`tracing-dotnet` to be `apm-sdk-capabilities-dotnet` &&
`apm-lang-platform-dotnet` && `tracing-dotnet`.
**Functionally**, there isn't a major change at the moment with this.
## Implementation details
Took lines that were ONLY owned by `tracing-dotnet` to be expanded into
the three.
## Test coverage
Well this may impact tests which should run now that
https://github.com/DataDog/dd-trace-dotnet/pull/9092 is merged 😄
## Other details
<!-- Fixes #{issue} -->
<!-- ⚠️ Note:
Where possible, please obtain 2 approvals prior to merging. Unless
CODEOWNERS specifies otherwise, for external teams it is typically best
to have one review from a team member, and one review from apm-dotnet.
Trivial changes do not require 2 reviews.
MergeQueue is NOT enabled in this repository. If you have write access
to the repo, the PR has 1-2 approvals (see above), and all of the
required checks have passed, you can use the Squash and Merge button to
merge the PR. If you don't have write access, or you need help, reach
out in the #apm-dotnet channel in Slack.
-->
* [Smoke Test Docker Image Bump] Updating docker image tags (#9153)
Updates the docker images used for smoke tests.
Co-authored-by: dd-octo-sts[bot] <200755185+dd-octo-sts[bot]@users.noreply.github.com>
* Increase profiler integration test timeout (#9158)
## Summary of changes
Increases profiler integration test timeout
## Reason for change
They timed out (but they all passed 🎉) but because they timed out they
failed 😢.
## Implementation details
60 -> 90
## Test coverage
nope
## Other details
<!-- Fixes #{issue} -->
#incident-57303
<!-- ⚠️ Note:
Where possible, please obtain 2 approvals prior to merging. Unless
CODEOWNERS specifies otherwise, for external teams it is typically best
to have one review from a team member, and one review from apm-dotnet.
Trivial changes do not require 2 reviews.
MergeQueue is NOT enabled in this repository. If you have write access
to the repo, the PR has 1-2 approvals (see above), and all of the
required checks have passed, you can use the Squash and Merge button to
merge the PR. If you don't have write access, or you need help, reach
out in the #apm-dotnet channel in Slack.
-->
Co-authored-by: Andrew Lock <andrew.lock@datadoghq.com>
* [Test Package Versions Bump] Updating package versions (#9152)
Updates the package versions for integration tests. See below for a
summary of the changes.
## Package Version Updates
| Package | Integration | Previous | New | Published |
|---------|-------------|----------|-----|-----------|
| Quartz | Quartz | 3.19.1 | 3.20.0 | 2026-08-27 |
| AWSSDK.Core | AwsSdk | 4.0.102 | 4.0.102.1 | 2026-08-24 |
| AWSSDK.DynamoDBv2 | AwsDynamoDb | 4.0.103.4 | 4.0.103.5 | 2026-08-24 |
| AWSSDK.Kinesis | AwsKinesis | 4.0.101 | 4.0.101.1 | 2026-08-24 |
| AWSSDK.SQS | AwsSqs | …
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds agentless delivery for the Feature Flagging & Experimentation (FFE) OpenFeature
provider: Universal Flag Configuration (UFC) is loaded directly from Datadog over HTTPS
with no Datadog Agent. Agentless becomes the default source; the existing Agent Remote
Configuration path remains available as explicit opt-in. Ports dd-trace-js #9397 and the
follow-ups #9481 (custom endpoints) / #9482 (grandfathering, fail-closed), adapted to
Python provider-lifecycle conventions.
Key pieces:
DD_FEATURE_FLAGS_ENABLED(stable kill switch, defaulttrue),DD_FEATURE_FLAGS_CONFIGURATION_SOURCE(agentlessdefault |remote_config;offlinereserved), and agentless tunables
..._AGENTLESS_BASE_URL,..._AGENTLESS_POLL_INTERVAL_SECONDS(30, capped at 1h),..._AGENTLESS_REQUEST_TIMEOUT_SECONDS(5).offline→ legacy grandfathering (
DD_EXPERIMENTAL_FLAGGING_PROVIDER_ENABLED) → default agentless.AgentlessConfigurationSource(PeriodicService): fixed-delay polling, in-tickretry/backoff (408/429/5xx/network, jittered), per-request timeout, gzip, strict JSON:API
validation, ETag/
304, last-known-good preserved on every failure, self-tracing suppressed.(only when the OpenFeature provider is registered), so there is no default-on CDN polling.
RC is subscribed only when
remote_configis the resolved source. Both sources feed thesame apply path.
hit the CDN in lockstep.
JIRA: FFL-2699
Testing
dd_env), JSON:API accept/reject,gzip, poller behavior (200/304/401/malformed/retry/last-known-good/ETag/headers/no-trace),
source-resolution matrix (kill switch, grandfathering, fail-closed), agentless factory
(API-key handling), fork jitter.
initialize()/shutdown()start/stop the poller; kill switch starts no source.tests/openfeaturesuite passes (613). style/typing/spelling/registry/suitespec pass.System tests
Ran the shared configuration-source contract against this branch, with the suite activated for
Python via DataDog/system-tests#7411 and
binaries/python-load-from-localpointing at it:Covers default lazy agentless delivery, explicit and grandfathered Remote Configuration, the kill
switch, invalid and reserved-
offlinefail-closed behavior, custom-endpoint authentication,recovery and ETag handling, timeout/retry, and non-overlapping polling.
No regressions in the rest of the FFE parametric directory:
30 passed, 12 xfailed, 29 xpassed(
test_dynamic_evaluation.py24 passed,test_span_enrichment.py6 passed + 12 pre-existingxfails).
The suite is run sequentially; with parallel workers two cases intermittently error with test-agent
port-routing noise unrelated to the library.
Risks
DD_FEATURE_FLAGS_ENABLEDdefaultstrue, but the agentless poller only activates whenthe OpenFeature provider is registered — users who don't use FFE are unaffected and there is
no default-on CDN traffic.
DD_EXPERIMENTAL_FLAGGING_PROVIDER_ENABLEDnow only feeds source grandfathering. Existing userswho set it keep Remote Configuration delivery and reporting; agentless users get reporting too.
planned follow-up.
Additional Notes