[Feature Flags] Add privacy-protected flagevaluation EVP events through the Agent by vjfridge · Pull Request #9351 · DataDog/dd-trace-dotnet

vjfridge · 2026-10-01T21:01:15Z

Pre-merge blocker: no-Agent behavior

#9352 and #9353 are merged and incorporated into this branch. #9235 remains open; revisit no-Agent flagevaluation delivery when that transport work lands.
The related shared/exposure transport work does not yet wire up no-Agent flagevaluation emission. Before merging this PR, decide and validate transport reuse or an explicit no-Agent gating policy. The current implementation remains Agent-only and can build/queue events that cannot be delivered without an Agent. Review discussion.

Summary of changes

Adds aggregated flagevaluation track EVP events for successful and unsuccessful Datadog OpenFeature evaluations, protecting targeting keys and context by default without changing flag results.

Agent-only delivery today, with the no-Agent decision blocked as described above; related transport: #9235; replaces the stale #9139 approach.
Follows merged Go, Java, Python, and Ruby implementations, with PII guidance and JS as additional references.

Key decisions / assumptions

Private by default. Hash valid targeting keys at output, omit context, and allowlist error codes. Only boolean true for observeFullEvaluationData allows raw keys and bounded context; consent and timestamp are captured once per evaluation.
One consent value across merged RCM configs. Full data requires consent from every currently active file (AND), recomputed on each update—not combined with historical configurations. Any false, absent, null, or malformed value selects protected mode. Existing flag-merging behavior stays unchanged.
Background processing. The caller captures metadata/bounded context and enqueues without waiting for space. A dedicated worker aggregates and sends, batching partial queues for up to 50 ms. Explicit flush and shutdown wake it immediately; periodic sends remain every 10 seconds. Flush/shutdown waits are bounded. The five-second request timeout does not bound legacy pipe/socket connection setup; that existing shared-transport gap is deferred from this PR.
Stable service tags. Capture service/env/version when the writer starts, after replaying the latest settings, and keep them for its lifetime—as Go/Java/Ruby/JS do. Later tag changes require writer recreation; Agent-address updates remain independent.
Bounded entries, not a fixed memory ceiling. Queue: 4,096, matching Go/Java/Python/Ruby; full buckets: 10,000 per flag / 131,072 globally; degraded buckets: 32,768. Degraded events omit key/context and exclude consent from bucket identity; exhausted capacity drops telemetry, not evaluations.
Bounded consented context. Capture at most 256 fields with depth/size/visit limits, without scanning or sorting arbitrarily large inputs. Rare hash collisions in oversized maps can change the retained subset; only telemetry is affected.
Scoped transport. Use the Agent’s EVP v2 route, with no direct-intake fallback, discovery, or retries. Delivery is best-effort; Agent 7.84.0 preserved SDK identity headers in testing, while 7.83.1 stripped them.

Measured performance overhead

Historical comparison: b32fca42cc vs master f0114c8c71, before the worker-batching change; 4,096-entry queue. ARM64 Linux/glibc, .NET 10.0.12; 100,000-evaluation bursts, medians of three runs. CPUs are affinity slots, not quotas; process CPU/allocation include final flush, caller latency does not.

CPUs	Mode	Evaluation p50 / p95 (µs)	CPU (µs/evaluation)	Allocation (bytes/evaluation)
1	Baseline	48.47 / 60.21	50.47	6,088
1	Events disabled	55.93 / 68.32	56.44	6,584
1	Default protected	67.77 / 112.38	79.26	6,837
1	Full consent	70.08 / 139.30	91.60	11,472
2	Baseline	10.85 / 11.38	11.86	6,088
2	Events disabled	12.23 / 12.79	13.49	6,584
2	Default protected	23.42 / 27.38	34.74	6,836
2	Full consent	28.18 / 31.96	43.64	11,470

Default protected mode adds ~19 µs / ~13 µs median evaluation latency and ~57% / ~193% process CPU per evaluation on one/two CPUs. Disabling events avoids the writer but is not zero-overhead.
Consented contexts add allocation. All 88 trials passed the delivery/privacy audit, including paced, sparse, high-cardinality, and oversized-context workloads; all 3,136,867 expected observations, including warmup, were retained.
These synthetic measurements make the cost visible; they do not establish acceptable production overhead. .NET reviewer assessment remains open. A later paired worker-batching comparison improved protected burst latency/CPU, with a documented overload/drop tradeoff; it is not a fresh comparison against master. Hook-list reuse (FFL-3356) is now included through merged #9334; the historical measurements above predate that change.

Historical methodology, full workload matrix, and memory results. Earlier experiments and validation history used different artifacts/settings and are not an isolated optimization comparison.

Test coverage

Latest local post-merge validation at cef427882a: targeted feature-flag, telemetry, and metadata tests: 1,312 passed / two skipped on each of .NET 10 and .NET Core 3.1. Tracer builds passed for all four target frameworks with zero warnings/errors. Subsequent commits changed only CODEOWNERS; these runs do not replace final-head external/performance validation.
At 7fe8e0651f: 780 feature-flag tests passed / two platform skips; benchmark project built with zero warnings/errors. Full tracer suite: 13,507 passed / 71 skipped / zero failed. Writer-attribution fix and regression coverage.
At b32fca42cc: 746 feature-flag tests passed / one skipped; full tracer suite 13,473 passed / 70 skipped / zero failed. Benchmark project built without warnings/errors.
System-tests at b32fca42cc: all 51 Agent-only FFE tests passed, including all 12 EVP cases; baseline passed the other 39 and failed only those 12 unimplemented cases. All 12,000 degradation observations retained.
Dogfooding at b32fca42cc: identical results across 81 live, 24 controlled-consent, and six no-Agent requests. Verified emission, SDK identity, strict consent, and mixed-version/uninstrumented safety.
Signed companion harness fixes: system-tests #7951 and ffe-dogfooding #131, both awaiting review. Earlier evidence is historical.

Reviewer QA

Compare flag results with emission on/off and with the Agent disconnected. Verify strict consent, protected payloads, and unchanged exposure/span-enrichment behavior; disabling emission must produce no new flagevaluation events.

Blast radius

Updates the provider/tracer metadata bridge and adds a hook, worker, Agent requests, and diagnostic metrics. Both updated components are required for full functionality; mixed-version and uninstrumented behavior have dedicated tests.
Events default on when feature flagging is active; DD_FLAGGING_EVALUATION_COUNTS_ENABLED=false disables them. Hook registration is decided when the provider is constructed; enabling events after starting disabled requires provider recreation or an application restart. Adds CPU/allocation and best-effort shutdown work; consented snapshots retain more data.
No changes to targeting rules, returned values, exposure payloads, or configuration-fetching transport. Exposure Agent requests also receive timeout protection, subject to the legacy connection-setup limitation above. Existing Remote Config and Agentless configuration sources remain supported; event delivery is Agent-only.

External follow-ups

External/performance results above predate the latest review fixes and need a final-head refresh. Remaining coordination, in addition to the no-Agent blocker above:

Release owner: confirm provider/tracer release versions.
Companion harness PRs #7951 / #131: review and manual merge.

…dotnet # Conflicts: # Datadog.Trace.Build.g.sln # Datadog.Trace.sln

dd-trace-dotnet-ci-bot · 2026-10-01T21:47:56Z

Execution-Time Benchmarks Report ⏱️

Execution-time results for samples comparing This PR (9351) and master.

✅ No regressions detected

📄 View the full report (charts + all metrics) →

pr-commenter · 2026-10-01T21:56:14Z

Benchmarks

Benchmark execution time: 2026-10-19 15:50:16

Comparing candidate commit 27f6512 in PR branch vickie/ffe-evp-pii-dotnet with baseline commit 189ec8f in branch master.

📊 Benchmarking dashboard

Found 0 performance improvements and 0 performance regressions! Performance is the same for 72 metrics, 0 unstable metrics, 11 known flaky benchmarks, 115 flaky benchmarks without significant changes.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

🟩 = significantly better candidate vs. baseline
🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

Known flaky benchmarks

These benchmarks are marked as flaky and will not trigger a failure. Modify FLAKY_BENCHMARKS_REGEX to control which benchmarks are marked as flaky.

scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan net472

🟩 throughput [+9365.590op/s; +11402.171op/s] or [+5.123%; +6.237%]

scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_AddEvent_Sampled net472

🟩 throughput [+9855.655op/s; +12919.313op/s] or [+5.977%; +7.835%]

scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_SetAttributes_Sampled net472

🟩 throughput [+10064.176op/s; +12571.178op/s] or [+6.869%; +8.580%]

scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_SetStatus_Sampled net472

🟩 throughput [+10587.737op/s; +12172.398op/s] or [+5.745%; +6.604%]

scenario:Benchmarks.Trace.AspNetCoreBenchmark.SendRequest net6.0

unstable execution_time [+27.624ms; +88.625ms] or [+6.090%; +19.539%]

scenario:Benchmarks.Trace.AspNetCoreBenchmark.SendRequest netcoreapp3.1

unstable execution_time [-87.408ms; -2.387ms] or [-19.382%; -0.529%]

scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces net472

unstable execution_time [+58.420ms; +117.803ms] or [+11.355%; +22.898%]
🟥 throughput [-97.813op/s; -42.836op/s] or [-14.892%; -6.522%]

scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice netcoreapp3.1

🟩 execution_time [-435.871µs; -158.357µs] or [-15.165%; -5.510%]
unstable throughput [+29.977op/s; +84.960op/s] or [+8.600%; +24.374%]

scenario:Benchmarks.Trace.CharSliceBenchmark.OriginalCharSlice netcoreapp3.1

unstable throughput [-26.321op/s; +1.875op/s] or [-9.643%; +0.687%]

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark netcoreapp3.1

🟥 allocated_mem [+17.582KB; +17.603KB] or [+6.859%; +6.867%]

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark netcoreapp3.1

unstable execution_time [-2.627µs; +4.719µs] or [-4.756%; +8.543%]

Known flaky benchmarks without significant changes:

scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan net6.0
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan netcoreapp3.1
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_AddEvent_Sampled net6.0
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_AddEvent_Sampled netcoreapp3.1
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_GetContext_Sampled net472
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_GetContext_Sampled net6.0
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_GetContext_Sampled netcoreapp3.1
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_SetAttributes_Sampled net6.0
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_SetAttributes_Sampled netcoreapp3.1
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_SetStatus_Sampled net6.0
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_SetStatus_Sampled netcoreapp3.1
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_UpdateName_Sampled net472
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_UpdateName_Sampled net6.0
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_UpdateName_Sampled netcoreapp3.1
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan net472
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan net6.0
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan netcoreapp3.1
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_AddEvent_Sampled net472
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_AddEvent_Sampled net6.0
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_AddEvent_Sampled netcoreapp3.1
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_GetContext_Sampled net472
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_GetContext_Sampled net6.0
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_GetContext_Sampled netcoreapp3.1
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_RecordException_Sampled net472
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_RecordException_Sampled net6.0
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_RecordException_Sampled netcoreapp3.1
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_SetAttributes_Sampled net472
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_SetAttributes_Sampled net6.0
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_SetAttributes_Sampled netcoreapp3.1
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_SetStatus_Sampled net472
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_SetStatus_Sampled net6.0
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_SetStatus_Sampled netcoreapp3.1
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_UpdateName_Sampled net472
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_UpdateName_Sampled net6.0
scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.TelemetrySpanBenchmark.StartSpan_UpdateName_Sampled netcoreapp3.1
scenario:Benchmarks.Trace.ActivityBenchmark.StartStopWithChild net472
scenario:Benchmarks.Trace.ActivityBenchmark.StartStopWithChild net6.0
scenario:Benchmarks.Trace.ActivityBenchmark.StartStopWithChild netcoreapp3.1
scenario:Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces net472
scenario:Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces net6.0
scenario:Benchmarks.Trace.AgentWriterBenchmark.WriteAndFlushEnrichedTraces netcoreapp3.1
scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody net472
scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody net6.0
scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleMoreComplexBody netcoreapp3.1
scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody net472
scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody net6.0
scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.AllCycleSimpleBody netcoreapp3.1
scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorMoreComplexBody net472
scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorMoreComplexBody net6.0
scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorMoreComplexBody netcoreapp3.1
scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody net472
scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody net6.0
scenario:Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody netcoreapp3.1
scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeArgs net472
scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeArgs net6.0
scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeArgs netcoreapp3.1
scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs net472
scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs net6.0
scenario:Benchmarks.Trace.Asm.AppSecEncoderBenchmark.EncodeLegacyArgs netcoreapp3.1
scenario:Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmark net472
scenario:Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmark net6.0
scenario:Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmark netcoreapp3.1
scenario:Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmarkWithAttack net472
scenario:Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmarkWithAttack net6.0
scenario:Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWafRealisticBenchmarkWithAttack netcoreapp3.1
scenario:Benchmarks.Trace.AspNetCoreBenchmark.SendRequest net472
scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces net6.0
scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces netcoreapp3.1
scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice net472
scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice net6.0
scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool net472
scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool net6.0
scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSliceWithPool netcoreapp3.1
scenario:Benchmarks.Trace.CharSliceBenchmark.OriginalCharSlice net472
scenario:Benchmarks.Trace.CharSliceBenchmark.OriginalCharSlice net6.0
scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch net472
scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch net6.0
scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch netcoreapp3.1
scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearchAsync net472
scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearchAsync net6.0
scenario:Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearchAsync netcoreapp3.1
scenario:Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync net472
scenario:Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync net6.0
scenario:Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync netcoreapp3.1
scenario:Benchmarks.Trace.ILoggerBenchmark.EnrichedLog net472
scenario:Benchmarks.Trace.ILoggerBenchmark.EnrichedLog net6.0
scenario:Benchmarks.Trace.ILoggerBenchmark.EnrichedLog netcoreapp3.1
scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark net472
scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark net6.0
scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark net472
scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark net6.0
scenario:Benchmarks.Trace.Log4netBenchmark.EnrichedLog net472
scenario:Benchmarks.Trace.Log4netBenchmark.EnrichedLog net6.0
scenario:Benchmarks.Trace.Log4netBenchmark.EnrichedLog netcoreapp3.1
scenario:Benchmarks.Trace.RedisBenchmark.SendReceive net472
scenario:Benchmarks.Trace.RedisBenchmark.SendReceive net6.0
scenario:Benchmarks.Trace.RedisBenchmark.SendReceive netcoreapp3.1
scenario:Benchmarks.Trace.SerilogBenchmark.EnrichedLog net472
scenario:Benchmarks.Trace.SerilogBenchmark.EnrichedLog net6.0
scenario:Benchmarks.Trace.SerilogBenchmark.EnrichedLog netcoreapp3.1
scenario:Benchmarks.Trace.SingleSpanAspNetCoreBenchmark.SingleSpanAspNetCore net472
scenario:Benchmarks.Trace.SingleSpanAspNetCoreBenchmark.SingleSpanAspNetCore net6.0
scenario:Benchmarks.Trace.SingleSpanAspNetCoreBenchmark.SingleSpanAspNetCore netcoreapp3.1
scenario:Benchmarks.Trace.SpanBenchmark.StartFinishScope net472
scenario:Benchmarks.Trace.SpanBenchmark.StartFinishScope net6.0
scenario:Benchmarks.Trace.SpanBenchmark.StartFinishScope netcoreapp3.1
scenario:Benchmarks.Trace.SpanBenchmark.StartFinishSpan net472
scenario:Benchmarks.Trace.SpanBenchmark.StartFinishSpan net6.0
scenario:Benchmarks.Trace.SpanBenchmark.StartFinishSpan netcoreapp3.1
scenario:Benchmarks.Trace.SpanBenchmark.StartFinishTwoScopes net472
scenario:Benchmarks.Trace.SpanBenchmark.StartFinishTwoScopes net6.0
scenario:Benchmarks.Trace.SpanBenchmark.StartFinishTwoScopes netcoreapp3.1
scenario:Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin net472
scenario:Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin net6.0
scenario:Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin netcoreapp3.1

…adata

…endencies

…dotnet

vjfridge · 2026-10-05T21:29:30Z

Supporting evidence: performance and validation history

Preserved from the PR description on October 5, 2026 so the overview can stay concise. This comment records existing evidence, not a new test or benchmark run. Unit validation is at b32fca42cc; external tests and performance measurements are historical, at the commits identified below. Final-head external validation and refreshed performance measurements remain outstanding.

Measured performance overhead

The historical measurements below used the former 16,384-observation queue. The current default is 4,096 for cross-SDK consistency; these results have not been rerun with the smaller queue.

These are local engineering measurements, not final-head measurements or a claim of performance equivalence. Baseline: 2514c73b48; measured implementation: 4de2e6c67c (same production code in 3cef2ec0d2). They predate the two review fixes and latest-master integration.

Native ARM64, Linux/musl, .NET 10.0.12; real OpenFeature/native-instrumentation path and a responsive local Agent fixture. Each burst run measured 100,000 evaluations, 64 subjects and eight attributes after warm-up. Values are medians of three serial runs. CPUs mean affinity slots, not container quotas. Caller latency excludes final flush; CPU and allocation include all process threads and final flush.

CPUs	Mode	Evaluation p50 / p95 (µs)	CPU (µs/evaluation)	Allocation (bytes/evaluation)
1	Baseline	46.12 / 56.25	47.03	6,088
1	Events disabled	53.05 / 75.64	57.05	6,584
1	Default protected	65.94 / 105.22	75.11	6,837
1	Full consent	71.98 / 134.29	89.92	11,472
2	Baseline	10.72 / 11.24	11.79	6,088
2	Events disabled	11.82 / 12.40	12.96	6,584
2	Default protected	22.53 / 26.23	30.64	6,836
2	Full consent	27.17 / 30.05	41.71	11,470

Protected-mode median latency increased by about 20 µs on one CPU / 12 µs on two CPUs, with process CPU per evaluation up approximately 60% / 160%. Disabling events avoids the writer, but not all added provider/metadata work: it is not a zero-overhead path.
Protected burst peak RSS was 142.10 vs 133.38 MiB baseline on one CPU and 136.50 vs 135.30 MiB on two CPUs. These are whole-process observations, not queue-only memory or worst-case bounds; allocation above measures garbage produced, not retained memory.
Paced (~1,000/s before processing cost), sparse (~100/s), high-cardinality and oversized-context cases were also exercised. These are test workloads, not customer averages. High-cardinality enabled runs reached roughly 167–185 MiB peak RSS and 134–638 ms final flush times; full-consent oversized contexts add substantial allocation despite bounded capture.
Profiling was followed by three worker-signaling experiments. None established a broad win across one/two CPUs, so none is retained; the original AutoResetEvent design stays. Provider hook-list reuse is a separate general optimization tracked in FFL-3356.

Test coverage

Queue alignment at b32fca42cc: 746 feature-flag tests passed / one skipped; full tracer unit suite: 13,473 passed / 70 skipped / zero failed (.NET 10). Benchmark project compiled with zero warnings/errors. The production-default capacity regression fails at 16,384 and passes at 4,096; this is not a new performance measurement.
Merged tree, committed as 7160ad3b03: 788 targeted tracer tests + 74 provider tests passed; one platform-specific skip. All four tracer and four provider framework builds, plus master’s startup-hook project, passed without warnings/errors. These checks ran on the resolved tree before its signed merge commit.
Pre-merge 02b6a03698: full tracer suite 13,390 passed / 70 skipped / zero failed; regression coverage for worker request-context retention and metadata-only error codes.
External validation at 3cef2ec0d2: system-tests 43 passed / zero failed / eight existing XPASS / 53 deselected, including all 12 Agent EVP cases. Identical baseline harness had 31 passes and 12 expected failures for absent emission. Direct-routing cases were outside scope, not reported as passing.
Dogfooding at 3cef2ec0d2, staging dev, Agent 7.84.0: baseline/candidate results matched for 81 live and 24 controlled requests, plus six no-Agent checks. Verified emitted observations, SDK identity, strict consent, and unchanged evaluation behavior.
Performance correctness: all 3,237,402 expected enabled observations across 88 trials were audited, including protected payloads and overflow degradation. This proves delivery to the responsive test fixture, not guaranteed delivery under failures.

Detailed design considerations

Key decisions / assumptions

One consent value across merged RCM configs. observeFullEvaluationData is an environment-wide setting, not a per-flag setting. When .NET merges multiple Remote Config files, the last file in the existing merge order sets consent for the whole resulting configuration, including flags retained from earlier files and missing/invalid flag evaluations. A final false, absent, null, or wrong-typed value selects protected mode. “Last” means merge order, not CreatedAt. Consent is still captured once per evaluation; changing .NET’s existing flag-merging behavior is outside this PR’s scope.
Consent belongs to the evaluation. Capture the configuration’s observeFullEvaluationData value and evaluation timestamp when evaluating the flag. Later configuration changes must not change that observation’s privacy treatment. Only boolean true opts into full data.
Protect output independently. Protected events omit context and hash non-empty, valid targeting keys when building output. Full-consent events may include the raw key and a bounded context snapshot. Invalid targeting keys are omitted, not used to discard the observation. Errors use an allowlist of stable codes, never raw messages.
Keep expensive processing off the evaluation call. The caller captures metadata and, only with consent, bounded context, then attempts queue admission without waiting for room. A dedicated worker aggregates, serializes, compresses, and sends. It sleeps when idle, flushes every 10 seconds or on explicit flush, and has a bounded shutdown wait. Caller-side capture and synchronization still have a measurable cost.
Bound retained entries, not total process memory. Queue: 4,096 observations, matching merged Go/Java/Python/Ruby; full aggregation: 10,000 buckets per flag / 131,072 globally; degraded aggregation: 32,768 buckets. Degraded events combine observations without targeting key or context; consent is not part of their identity. Once capacity is exhausted, telemetry can be dropped without changing evaluation results. These are not byte budgets or an RSS guarantee.
Bound consented context work. Capture at most 256 fields with depth, text, container, and total-visit limits. We do not scan/sort an arbitrarily large input just to choose fields. Rare hash collisions in rebuilt over-limit maps can change the retained subset and split telemetry buckets; flag results and protected-mode events are unaffected.
Keep transport scoped. Send gzip JSON through the Agent’s existing EVP v2 route. No direct-intake fallback, route discovery, or retries. Missing/incompatible Agents and failed sends can lose telemetry, but must not break evaluations. Agent 7.84.0 was verified to preserve SDK origin/version headers; 7.83.1 stripped those headers despite delivering events.

…dotnet

datadog-prod-us1-4

On .NET Core 3.1, a saturated Agent Unix socket backlog can block event delivery beyond the five-second deadline, eventually filling the evaluation queue and dropping subsequent events.

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit cef4278}

vjfridge · 2026-10-09T18:53:41Z

Note to self: ffe-dogfooding

PASS — no evaluation-result regressions in the tested matrix. Rerun 2026-10-09 on the native ARM64 workspace, comparing this PR at ff0308ff39 with master a5face8c30. Agent 7.84.0, .NET 8.0.31, staging datad0g.com / dev.

81 live requests: all HTTP 200; values and reasons matched master exactly (30 static, 51 existing error/default responses). The PR delivered all 81 evaluations in 17 event rows; master emitted none, as expected.
24 controlled requests through the real Agent: six each with absent, false, true, and malformed ("true") consent. Results matched master and all observations arrived. Only boolean true included raw targeting keys/context; other modes used the expected SHA-256 keys and omitted context/PII canaries. Context-bound checks passed.
Six requests with the Agent stopped: evaluation results remained unchanged on both builds; no event delivery was expected or observed.
Origin/version headers were dd-trace-dotnet / 3.56.0. Receipt was verified at the local mock intake. The Agent also targeted staging, but staging backend ingestion was not independently queried.

Used harness main 125956baa9 plus the local-runtime fix from ffe-dogfooding #131 (4e4cf31f53), and the checked-in local .NET Compose override with exact locally built provider/Bundle packages. Installed provider/tracer hashes matched both source builds; no published SDK substitution. Controlled configurations did not modify shared staging flags.

All cases passed first-run. Run-owned containers/network/volumes were removed. This rerun does not refresh performance, mixed-version, or agentless-delivery evidence.

Installed artifact identity (SHA-256)

Assembly	Master	PR
Provider DLL	`493bdf2ee47b267f233f87ff0af0087212f7a3f058d3f76f13f9edbface46a3b`	`bde41857a37e1fef73384f7d26bf82caa8f4a0c7dd10aeaaa9e2946f8625a43c`
Tracer DLL	`2c5a94be42bf2da562e32788eb1bd0f84aba92069a5249a01dc07d0d9a9d6992`	`93e8f317407733bd126bc82deb0f2196a288977d960c29fcc102a00436051d42`

vjfridge · 2026-10-09T18:54:06Z

Note to self: system-tests

PASS — all 51 selected Agent-only FFE tests pass at ff0308ff39; no regressions found against master a5face8c30 in this selection. Rerun 2026-10-09 on native Linux ARM64, .NET 8, Agent 7.84.2 (same image on both sides).

Selection	Master	PR
Focused Agent EVP	12 expected failures: events not implemented	12 passed
Full Agent-only FFE	39 passed + the same 12 expected failures	51 passed, zero failed

Same frozen harness on both sides: main e0ec61e73 plus system-tests #7951 (9cd5af79f), preserving context value types and starting bursts in a fresh flush window.
Exact local provider/tracer builds, with installed DLL hashes verified. A temporary manifest override enabled unreleased EVP tests on both builds. Focused/full selections deselected 2/53 tests respectively; this is Agent-only, not every FFE scenario.
Covered consent/privacy, context bounds, errors/runtime defaults, aggregation and degradation, alongside existing evaluations, exposures and metrics. Both PR runs captured all 12,000 degradation-case evaluations: 10,000 full-detail + 2,000 reduced-detail, with targeting key/context omitted from reduced-detail events.
Both PR selections passed first-run; no retries or SDK changes. All master failures were missing-event assertions, not evaluation failures. Temporary harness settings and original shared image tags were restored; test containers were removed.

Latest dogfooding results and artifact hashes.

Commands on each pinned local build:

./build.sh dotnet -w poc --docker-platform linux/arm64/v8 --binary-path <staged-local-artifacts> -i weblog
./run.sh FEATURE_FLAGGING_AND_EXPERIMENTATION --library dotnet tests/ffe/test_flag_eval_evp.py
./run.sh FEATURE_FLAGGING_AND_EXPERIMENTATION --library dotnet tests/ffe

vjfridge · 2026-10-09T19:53:11Z

/code blockers

gh-worker-devflow-routing-ef8351 · 2026-10-09T19:53:14Z

View all feedbacks in Devflow UI.

2026-10-09 19:53:13 UTC ℹ️ Start processing command /code blockers

2026-10-09 19:53:14 UTC ℹ️ Devflow:

Checking merge blockers for #9351...

2026-10-09 19:53:21 UTC ℹ️ Devflow: /code blockers

Detected 3 merge blocker(s) to address:

🔴 Blocking

Required check code_freeze has not reported
Hint: Ensure the workflow that produces this check is triggered.
Required check smoke_x64_installer_tests has not reported
Hint: Ensure the workflow that produces this check is triggered.
Required check system_tests has not reported
Hint: Ensure the workflow that produces this check is triggered.

…dotnet

Integrate Agentless evaluation delivery with the existing exposure transport policy while retaining privacy, payload limits, and bounded flushing. Keep separate transport instances, preserve Remote Config admission, and fix startup exporter settings replay.

…dotnet # Conflicts: # tracer/src/Datadog.Tracer.Native/Generated/generated_calltargets.g.cpp

vjfridge · 2026-10-17T01:15:19Z

@arp review

…dotnet

vjfridge added 10 commits September 28, 2026 18:20

feat(ffe): capture evaluation-time privacy consent

3716498

feat(ffe): capture bounded EVP observations in the provider

1eb4816

feat(ffe): aggregate and serialize privacy-safe flag evaluations

0712aae

feat(ffe): add Agent-only flagevaluation sender

0dcca8f

feat(ffe): add bounded background EVP delivery

7823738

feat(ffe): wire OpenFeature EVP instrumentation and configuration

4de2e6c

test(ffe): validate native EVP pipeline and performance

3cef2ec

[Feature Flags] Guard worker wake and empty flush lifecycle

501a1d3

[Feature Flags] Isolate EVP worker context and preserve error codes

02b6a03

Merge remote-tracking branch 'origin/master' into vickie/ffe-evp-pii-…

7160ad3

…dotnet # Conflicts: # Datadog.Trace.Build.g.sln # Datadog.Trace.sln

vjfridge added 3 commits October 1, 2026 21:56

[Feature Flags] Enable nullable checking in EVP helper files

b1ccab3

[Feature Flags] Attach privacy consent directly to evaluation returns

071dabf

[Feature Flags] Initialize privacy consent directly in evaluation met…

95df9d6

…adata

vjfridge added type:new-feature area:ffe labels Oct 2, 2026

vjfridge added 8 commits October 2, 2026 01:02

[Feature Flags] Regenerate trimming descriptor for worker context dep…

7584f5a

…endencies

[Feature Flags] Fix legacy EVP timeout and CI compatibility

0fc8325

[Feature Flags] Fix integration fixture shutdown and outage isolation

ec912cc

Merge remote-tracking branch 'origin/master' into vickie/ffe-evp-pii-…

ac9b0a7

…dotnet

Merge remote-tracking branch 'origin/master' into vickie/ffe-evp-pii-…

4507008

…dotnet

[Feature Flags] Resolve merged configuration consent last-wins

ab91101

[Feature Flags] Clarify privacy bounds and reuse payload keys

cfa3d17

[Feature Flags] Align EVP queue capacity across SDKs

b32fca4

This was referenced Oct 5, 2026

feat(openfeature): emit server-side EVP flagevaluation #9139

Closed

[dotnet] Preserve FFE context types and isolate burst windows DataDog/system-tests#7951

Closed

This comment was marked as outdated.

Sign in to view

Merge remote-tracking branch 'origin/master' into vickie/ffe-evp-pii-…

d59f3f8

…dotnet

datadog-prod-us1-4 Bot reviewed Oct 9, 2026

View reviewed changes

Comment thread tracer/src/Datadog.Trace/Agent/Transports/HttpStreamRequest.cs Resolved

[Feature Flags] Assign evaluation benchmark ownership to FFE

ff0308f

vjfridge added 12 commits October 13, 2026 16:24

[Feature Flags] Gate evaluation events on shared Agent discovery

9d5781c

Merge remote-tracking branch 'origin/master' into vickie/ffe-evp-pii-…

726b7e5

…dotnet

[FeatureFlags] Simplify redundant evaluation event tests

e84a4d7

Merge latest master into vickie/ffe-evp-pii-dotnet

e0fc50c

[Feature Flags] Wait for EVP delivery readiness in integration tests

f0d10b4

Merge branch 'master' into vickie/ffe-evp-pii-dotnet

e218945

Merge remote-tracking branch 'origin/master' into vickie/ffe-evp-pii-…

6c7cb37

…dotnet

Merge remote-tracking branch 'origin/master' into vickie/ffe-evp-pii-…

6e18142

…dotnet

[Feature Flags] Wait for delivery after provider recreation

00e5f15

Merge remote-tracking branch 'origin/master' into vickie/ffe-evp-pii-…

80aa709

…dotnet # Conflicts: # tracer/src/Datadog.Tracer.Native/Generated/generated_calltargets.g.cpp

Fix feature flag integration fixtures for shared EVP routing

e452d59

aarsilv mentioned this pull request Oct 18, 2026

[Feature Flags] Add per-subject evaluation telemetry opt-out #9426

Open

vjfridge added 2 commits October 19, 2026 14:51

Match feature flag discovery callbacks to exporter generation

8468cec

Merge remote-tracking branch 'origin/master' into vickie/ffe-evp-pii-…

27f6512

…dotnet

vjfridge enabled auto-merge (squash) October 19, 2026 16:50

vjfridge merged commit 29487ea into master Oct 19, 2026
147 of 148 checks passed

vjfridge deleted the vickie/ffe-evp-pii-dotnet branch October 19, 2026 19:00

github-actions Bot added this to the vNext-v3 milestone Oct 19, 2026

This was referenced Oct 21, 2026

Add provisional per-subject evaluation telemetry opt-out tests DataDog/system-tests#8080

Merged

[Feature Flags] Preserve 256 attributes alongside targeting key #9511

Merged

feat(openfeature): add shared agentless EVP fallback DataDog/dd-trace-rb#6574

Open

Conversation

vjfridge commented Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pre-merge blocker: no-Agent behavior

Summary of changes

Key decisions / assumptions

Measured performance overhead

Test coverage

Reviewer QA

Blast radius

External follow-ups

Uh oh!

dd-trace-dotnet-ci-bot Bot commented Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Execution-Time Benchmarks Report ⏱️

Uh oh!

pr-commenter Bot commented Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Explanation

More details about the CI and significant changes

Known flaky benchmarks

scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan net472

scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_AddEvent_Sampled net472

scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_SetAttributes_Sampled net472

scenario:Benchmarks.OpenTelemetry.InstrumentedApi.Trace.ActivityBenchmark.StartSpan_SetStatus_Sampled net472

scenario:Benchmarks.Trace.AspNetCoreBenchmark.SendRequest net6.0

scenario:Benchmarks.Trace.AspNetCoreBenchmark.SendRequest netcoreapp3.1

scenario:Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces net472

scenario:Benchmarks.Trace.CharSliceBenchmark.OptimizedCharSlice netcoreapp3.1

scenario:Benchmarks.Trace.CharSliceBenchmark.OriginalCharSlice netcoreapp3.1

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark netcoreapp3.1

scenario:Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark netcoreapp3.1

Uh oh!

vjfridge commented Oct 5, 2026 • edited by atlassian Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Supporting evidence: performance and validation history

Measured performance overhead

Test coverage

Key decisions / assumptions

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

datadog-prod-us1-4 Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vjfridge commented Oct 9, 2026

Note to self: ffe-dogfooding

Uh oh!

vjfridge commented Oct 9, 2026

Note to self: system-tests

Uh oh!

vjfridge commented Oct 9, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔴 Blocking

Uh oh!

vjfridge commented Oct 17, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

vjfridge commented Oct 1, 2026 •

edited

Loading

dd-trace-dotnet-ci-bot Bot commented Oct 1, 2026 •

edited

Loading

pr-commenter Bot commented Oct 1, 2026 •

edited

Loading

vjfridge commented Oct 5, 2026 •

edited by atlassian Bot

Loading

gh-worker-devflow-routing-ef8351 Bot commented Oct 9, 2026 •

edited

Loading