[leoromanovsky]

Description

Applications need to suppress evaluation telemetry for individual subjects without changing flag results or experiment exposure logging.

Support is disabled by default. Enable it with DD_FEATURE_FLAGS_EVALUATION_TELEMETRY_PER_SUBJECT_OPT_OUT_ENABLED=true or DataDogProvider(enable_evaluation_telemetry_opt_out_parameter=True). An explicit constructor value overrides the environment for that instance. Only boolean True in the context attribute dd_evaluation_telemetry_opt_out suppresses new evaluation events, evaluation metrics, and automatic flag span annotations.

The provider captures the final merged context at resolver entry, including contexts returned by before hooks, and isolates decisions across nested and concurrent evaluations. Existing queued telemetry, application hooks, and exposure behavior are preserved. Object evaluations return the supplied default and TYPE_MISMATCH for scalar/null JSON, including async evaluation, so error results retain the telemetry decision. The attribute remains available for targeting and exposure payloads.

Testing

At commit 795b8ba1d003865ee83aff07806135df1f2b2216, the candidate tree passed all 1,310 OpenFeature tests plus four Reno regression tests on Python 3.12 with OpenFeature 0.10.0. All 139 opt-out cases also passed in the repository's older OpenFeature environment. Repository lint checks and Reno lint/report generation passed. The new full-client benchmark covers support disabled, enabled with emission, and enabled with suppression; discovery includes both changed runtime files.

Local system-test receipt for both opt-out scenarios

Tested dd-trace-py commit: 795b8ba1d003865ee83aff07806135df1f2b2216.
System-tests commit: b76b165ebaeff5cf4458936cadc8e7259796dde8.

Reused the Python 3.12 weblog built with ./build.sh python --weblog-variant python3.12, with binaries/python-load-from-local pointing to the clean signed tracer checkout through the harness's read-only source mount. The image ID was sha256:6c4d0f7c9cfc57e4aa81e80cf830ef7469ba30309b4b65478c2eaefa9579b8d6.

Executed:

./run.sh FEATURE_FLAGGING_EVALUATION_TELEMETRY_OPT_OUT
./run.sh FEATURE_FLAGGING_EVALUATION_TELEMETRY_OPT_OUT_DISABLED

Test class	Enabled scenario	Default-disabled scenario
Test_FFE_Evaluation_Telemetry_Opt_Out_EVP	XPASS	XPASS
Test_FFE_Evaluation_Telemetry_Opt_Out_Metrics	XPASS	XPASS
Test_FFE_Evaluation_Telemetry_Opt_Out_Exposures	XPASS	XPASS
Test_FFE_Evaluation_Telemetry_Opt_Out_Trace_Annotations	XPASS	XPASS

Each scenario completed with four XPASS results and no failed assertions. The unchanged manifest's provisional missing_feature entry explains XPASS. These runs exercise environment activation in a local weblog; constructor activation is covered by unit tests. They are not staging or production evidence and do not establish that PR CI runs these scenarios.

Risks

This suppresses only the three evaluation-telemetry signals. Experiment exposures still contain the targeting key and context attributes; local debug logging remains separately controlled. Before-hook returned contexts apply only when resolution is reached: provider NOT_READY/FATAL and later hook failures require the opt-out attribute in the initial context. The final decision travels in result metadata across executor boundaries; custom resolver overrides must preserve that metadata.

Additional Notes

After the first SDK release containing this feature, replace the provisional system-tests manifest entry with that released version. This release-activation follow-up remains outside this PR.

[cit-pr-commenter-54b7da]

Codeowners resolved as

Resolved from the full PR diff against main using the target branch CODEOWNERS file.
CODEOWNERS team requests not listed below are not required by the current file set.

.github/workflows/changelog.yml                                         @DataDog/python-guild @DataDog/apm-core-python
.gitlab/benchmarks/slos/feature-flagging-and-experimentation-sdk.yml    @DataDog/feature-flagging-and-experimentation-sdk
benchmarks/Dockerfile                                                   @DataDog/apm-core-python
benchmarks/openfeature_opt_out/config.yaml                              @DataDog/apm-core-python
benchmarks/openfeature_opt_out/requirements_scenario.txt                @DataDog/apm-core-python
benchmarks/openfeature_opt_out/scenario.py                              @DataDog/apm-core-python
benchmarks/suitespec.yml                                                @DataDog/apm-core-python
ddtrace/internal/openfeature/_evaluation_telemetry.py                   @DataDog/feature-flagging-and-experimentation-sdk
ddtrace/internal/openfeature/_provider.py                               @DataDog/feature-flagging-and-experimentation-sdk
ddtrace/internal/settings/_supported_configurations.py                  @DataDog/apm-python
ddtrace/internal/settings/openfeature.py                                @DataDog/feature-flagging-and-experimentation-sdk
docs/api.rst                                                            @DataDog/python-guild
docs/configuration.rst                                                  @DataDog/python-guild
releasenotes/notes/openfeature-evaluation-telemetry-opt-out-2b7d82efa6eb2ae8.yaml  @DataDog/apm-python
releasenotes/notes/openfeature-object-type-errors-6edcf89f2750b9a2.yaml  @DataDog/apm-python
scripts/docs/build.sh                                                   @DataDog/python-guild
scripts/reno_compat.py                                                  @DataDog/python-guild
supported-configurations.json                                           @DataDog/apm-python
tests/internal/test_reno_compat.py                                      @DataDog/apm-core-python
tests/openfeature/test_evaluation_telemetry_opt_out.py                  @DataDog/feature-flagging-and-experimentation-sdk

[cit-pr-commenter-54b7da]

Dependency direction analysis

⚠️ Existing dependency direction violations

There are 104 dependency direction violations that already exist on the base branch and have not been changed by this PR.

Show existing violations (showing 5 of 104 highest severity)

ddtrace.internal.tracemethods -×-> ddtrace.trace  (internal-core -> product:tracing, score=122)
ddtrace.llmobs._integrations.google_adk -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=120)
ddtrace.llmobs._integrations.crewai -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=120)
ddtrace.llmobs._integrations.openai -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=120)
ddtrace.llmobs._evaluators.runner -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=120)

To see all violations, download the layers-base.json and layers-pr.json artifacts from this CI job and run:

uv run --script scripts/import-analysis/layers.py compare layers-base.json layers-pr.json

[datadog-datadog-prod-us1-2]

 

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 3d95a4f | Docs | View more details | Give us feedback!}

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-30T22:27:15.328060Z	3d95a4f	New commits
🔒 Security Review	✅ Completed 2026-10-30T22:28:44.715540Z	3d95a4f	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[datadog-datadog-prod-us1-2]

More details

The telemetry gate uses the final resolved context and keeps suppression state per evaluation, with no reportable defect identified. Runtime behavior remains unverified because the required OpenFeature and test dependencies were unavailable.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 956e6e0}

[aarsilv]

Thanks for iterating!

[gnufede]

Approving to unblock, but please address @KowalskiThomas' comments

[aarsilv]

🤖 Comment from Claude working fully autonomously overnight:

[P2] Please add a local system-tests run of the opt-out scenarios to the PR description. No CI job runs those tests against this PR yet.

The matching tests exist on the system-tests default branch: tests/ffe/test_evaluation_telemetry_opt_out.py (DataDog/system-tests#8080). They cover evaluation events, metrics, exposures and trace annotations. Three things keep them out of this PR's CI:

• manifests/python.yml:2247 marks the file missing_feature ("enable after the SDK release").
• The two scenarios are not in any default scenario group (feature_flagging.py:37).
• .github/workflows/system-tests.yml here pins system-tests at 648ead29e24f9c64f28bc4893f16fe6963c74d4f, which does not contain the tests.

A unit-test pass is not the same evidence. The system tests run the built tracer in a weblog and assert the payloads it sends.

Suggested run and receipt

From a system-tests checkout, with this PR's head loaded into the Python weblog (system-tests docs/execute/binaries.md explains how to use a custom tracer):

./build.sh python
./run.sh FEATURE_FLAGGING_EVALUATION_TELEMETRY_OPT_OUT
./run.sh FEATURE_FLAGGING_EVALUATION_TELEMETRY_OPT_OUT_DISABLED

The manifest marks the file missing_feature, so the run reports these tests as expected failures (xfail / xpass), not plain passes. Please report the outcome per test class, or change the manifest entry locally for the run.

Receipt fields that make the run checkable:

• dd-trace-py SHA under test, and how the weblog loaded it
• system-tests SHA, scenarios and commands
• per-class results for Test_FFE_Evaluation_Telemetry_Opt_Out_EVP, _Metrics, _Exposures and _Trace_Annotations

The Python weblogs construct DataDogProvider() and set the environment variable in the scenario, so these tests cover the environment path. The constructor path stays covered by the unit tests in this PR.

[aarsilv]

🤖 Comment from Claude working fully autonomously overnight:

Reminder: after the release that contains this change, replace the provisional missing_feature entry for tests/ffe/test_evaluation_telemetry_opt_out.py in system-tests manifests/python.yml with the released version. The system-tests FFE README asks for this ("Replace each provisional manifest entry with the first released SDK version containing its opt-out implementation"). This is a follow-up in system-tests, not a change to this PR.

[leoromanovsky]

FFE reviewer selection

Reviewers: @vjfridge, @sameerank

Why: Privacy opt-out and event logging rules — The PR adds a per-subject opt-out for evaluation telemetry. Reviewers come from the matched pod.

_{🤖 Automated selection.}

Selection details

Initiative: project / ufc-compliance

Why this match: The PR adds a per-subject opt-out for evaluation telemetry. It is off by default and can be turned on with an environment setting or a provider constructor argument. When an evaluation context sets dd_evaluation_telemetry_opt_out=True, the provider stops emitting evaluation EVP events, evaluation metrics and flag span annotations. Flag results and exposure logging stay the same. This matches the project's per-context suppression of evaluation event egress for user opt-out.

• @vjfridge — primary (flag-evaluation-evp): Matched pod member. Existing review coverage preserved.
• @sameerank — floating (flag-evaluation-evp): Coverage and knowledge sharing. Covers: openfeature-architecture. Existing review coverage preserved.

Additional review: openfeature-architecture — The provider's evaluation contract changes. Object evaluations that resolve to scalar or null JSON now return the supplied default with Reason.ERROR and TYPE_MISMATCH instead of the raw value, and this has its own fix release note. The provider also adds a new constructor parameter, puts a new internal key in the resolution flag_metadata, and wraps the provider hooks in a per-call gating hook that removes that metadata key before the SDK returns the result. Coverage: @sameerank. Evidence: ddtrace/internal/openfeature/_provider.py, ddtrace/internal/openfeature/_evaluation_telemetry.py

The PR author was excluded. Existing reviewers and teams were preserved.

Debug or fix a problem

Workflow run · Bits Code session

• Wrong project or pod? Clarify this PR’s purpose, or correct the reviewer registry and publish the change. Sheet edits take effect after publication.
• Need another reviewer now? Request them in GitHub. This workflow only adds requests and keeps existing reviewers.
• Stuck beyond five minutes or failed? Inspect the run’s failed step and available Bits session. After fixing the cause, rerun the workflow with this PR URL and dry_run=false. Unchanged completed runs are skipped. If the receipt says needs_reconciliation or a write outcome is uncertain, ask the workflow owner to reconcile GitHub and the receipt before retrying; do not delete the lock blindly.
• Report a problem: Comment on the reviewer-routing skill PR with this PR URL, the workflow run link, and the expected initiative or reviewer correction.

Evidence and run details

✅ Complete — reviewer coverage verified. The workflow checked pending requests or submitted reviews on the evaluated head.

Generated asynchronously by the FFE reviewer-routing workflow.

Changed paths: ddtrace/internal/openfeature/_evaluation_telemetry.py, ddtrace/internal/openfeature/_provider.py, ddtrace/internal/settings/openfeature.py, ddtrace/internal/settings/_supported_configurations.py, supported-configurations.json, tests/openfeature/test_evaluation_telemetry_opt_out.py

Evaluated head: 3d95a4fbe5ca363166a2997fca3f649645e566f7

Registry revision: revision:efde5129-b144-41ab-b630-1adf6ad22901

Pinned skill source: 7e3283c602f6b18a7e3e623eecb52d3a9c091a2e

Instructions: workflow-injected snapshot (no plugin loaded)

Instruction snapshot SHA-256: 52ed8cb68675dde1a47addaefe4a9500891c24014abf7dade6dfe38cf836c1d2

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2b79b4acae

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[leoromanovsky]

🤖 Comment from Codex working with @leoromanovsky:

Added the local system-test receipt requested in this comment to the PR description. Both enabled and default-disabled scenarios completed with four XPASS results at tracer commit 2b79b4acaec272c59be3aeddeb4831bf76332ed9, using system-tests b76b165ebaeff5cf4458936cadc8e7259796dde8. The description records the source-loading check and each class's result. The provisional manifest entry remains unchanged.

The release-activation reminder remains a follow-up after the first SDK release containing this feature; it does not require a change in this PR.

[pr-commenter]

Benchmarks

Benchmark execution time: 2026-10-30 22:44:41

Comparing candidate commit 3d95a4f in PR branch leoromanovsky/evaluation-telemetry-opt-out with baseline commit 04fd18b in branch main.

📊 Benchmarking dashboard

Found 1 performance improvements and 0 performance regressions! Performance is the same for 610 metrics, 10 unstable metrics, 1 flaky benchmarks without significant changes.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

• 🟩 = significantly better candidate vs. baseline
• 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:openfeatureoptout-enabled-suppressed

• 🟩 execution_time [-28.459µs; -28.052µs] or [-40.547%; -39.967%]

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:coreapiscenario-context_with_data_listeners

• unstable execution_time [-720.280ns; +729.166ns] or [-6.709%; +6.792%]

scenario:coreapiscenario-core_dispatch_1_listener

• unstable execution_time [-36.607ns; +34.667ns] or [-5.303%; +5.022%]

scenario:coreapiscenario-core_dispatch_50_listeners

• unstable execution_time [-1880.919ns; +1920.773ns] or [-9.437%; +9.636%]

scenario:coreapiscenario-core_dispatch_exception_listeners

• unstable execution_time [-1967.469ns; +1672.944ns] or [-10.215%; +8.686%]

scenario:coreapiscenario-core_dispatch_listeners

• unstable execution_time [-373.231ns; +382.013ns] or [-8.548%; +8.749%]

scenario:coreapiscenario-core_dispatch_no_args_listeners

• unstable execution_time [-225.945ns; +223.543ns] or [-8.357%; +8.268%]

scenario:coreapiscenario-core_dispatch_with_results_1_listener

• unstable execution_time [-89.693ns; +91.420ns] or [-6.318%; +6.440%]

scenario:coreapiscenario-core_dispatch_with_results_50_listeners

• unstable execution_time [-4734.993ns; +4909.765ns] or [-9.467%; +9.816%]

scenario:coreapiscenario-core_dispatch_with_results_listeners

• unstable execution_time [-1077.069ns; +919.126ns] or [-10.095%; +8.615%]

scenario:packagesupdateimporteddependencies-import_many_stdlib_cached

• unstable execution_time [-51.362µs; +60.714µs] or [-8.745%; +10.337%]

Known flaky benchmarks

These benchmarks are marked as flaky and will not trigger a failure. Modify FLAKY_BENCHMARKS_REGEX to control which benchmarks are marked as flaky.

Known flaky benchmarks without significant changes:

• scenario:packagespackageforrootmodulemapping-cache_off

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 52627ff359

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[datadog-datadog-prod-us1-2]

The new benchmark requires a constructor option absent from the baseline tracer, preventing normal baseline/candidate comparisons. It needs capability-aware provider construction and telemetry expectations.

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 795b8ba}