test(ffe): enable experimentation opt-out mode coverage for Go and Java by leoromanovsky · Pull Request #8244 · DataDog/system-tests

leoromanovsky · 2026-10-29T23:07:37Z

Motivation

Subjects who opt out of experimentation must retain ordinary feature delivery, and future-only opt-out must retain experiments activated before their cutoff. The canonical suite currently reads baseline results without enabling either opt-out mode, so it cannot catch regressions in these policies.

Changes

Run the immutable canonical opt-out cases in four separate application configurations: default-disabled, explicit disabled, immediate, and future_experiments_only. Enabled modes use their capability-specific expectations and assert values, reasons, and error codes, including invalid contexts, cutoff equality, unknown activation, and ordinary allocations. The settled doLog behavior is unchanged.

Enable this suite from Go 2.13.0-dev and Java 1.69.0-SNAPSHOT. Go's merge already contains both modes; v2.12.0-rc.1 does not. Java activation depends on dd-trace-java#12950. Other SDK manifests identify their pending implementation or dependency requirements instead of claiming unsupported coverage. These are development-build thresholds, not stable-release claims.

Stacked on canonical fixtures #8139. Merge after that parent and the Java implementation. The existing evaluation-telemetry opt-out scenarios are separate and unchanged.

Validation

Normal manifest activation, without --runxfail: 8 configurations passed, 320 canonical evaluations, zero skips or expected failures.

SDK	Source revision	Result
Go `2.13.0-dev`	`fd09513977a448b87ca402252944a2c55d81eb01`	Four configurations passed; 160 evaluations.
Java `1.69.0-SNAPSHOT` candidate	`22bc2619cec98102c24b6c9eb1060cd8378ba1fe`	Four configurations passed; 160 evaluations.

Both enabled-mode Java negative controls failed when the application was forced to disable opt-out; the restored tests pass. Repository formatting, typing, import, manifest, shell, and Node checks pass. Java artifacts were built from the changes committed at the listed revision and carry the parent build tag +88354535bf.

This validates local public-provider evaluation through Remote Config. Exposure/evaluation delivery, standalone transport, other SDKs, and hosted CI are not covered by these runs.

Workflow

⚠️ Create your PR as draft ⚠️
Work on you PR until the CI passes
Mark it as ready for review
- Tests, manifest, weblog are modified -> you'll need a review from system-tests-reviewers: ask to one of youre co-worker familiar with the tested feature.
- Framework is modified, or non obvious usage of it -> get a review from system-tests-core (slack)

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Replace local fixture copies with the shared corpus, preserve extracted metadata and span inputs, and add a reviewed snapshot update workflow. Environment: Datadog workspace

Expose Node.js error codes and Ruby resolution details to canonical assertions. Convert Ruby numeric defaults to Float before calling the OpenFeature client. Environment: Datadog workspace

Keep the unified FFE mock backend loading the canonical fixture snapshot and preserve updater ownership. Environment: Datadog workspace

Co-authored-by: Codex GPT-5 <noreply@localhost>

Preserve the canonical FFE fixture path through the mock backend container refactor and retain the merged platform applicability filters. Environment: Datadog workspace

Download dd-openfeature.jar from the selected Java snapshot branch so FFE tests do not pair the development agent with an incompatible release provider. Cover default and custom branches, Java Lambda, and missing providers. Environment: Datadog workspace

Preserve canonical fixture discovery and platform filters. Keep the development OpenFeature provider out of Java Lambda builds, cover both loader paths, and document unasserted split-feature metadata.

The 0.5.4 release moved back to OpenTelemetry 0.32, which lacks the compression fallback and scope schema URL behavior present in 0.33. Preserve the existing minimum versions and declare only the released 0.5.4 unsupported; keep the 0.5.4 development build enabled. Environment: Datadog workspace

Exercise canonical capability expectations with disabled, immediate, and future-only process settings. Activate the validated development versions and document pending SDK dependencies. Environment: Datadog workspace

github-actions · 2026-10-29T23:08:04Z

CODEOWNERS have been resolved as:

manifests/cpp.yml                                                       @DataDog/system-tests-reviewers
manifests/dotnet.yml                                                    @DataDog/system-tests-reviewers
manifests/golang.yml                                                    @DataDog/system-tests-reviewers
manifests/java.yml                                                      @DataDog/system-tests-reviewers
manifests/nodejs.yml                                                    @DataDog/system-tests-reviewers
manifests/php.yml                                                       @DataDog/system-tests-reviewers
manifests/python.yml                                                    @DataDog/system-tests-reviewers
manifests/ruby.yml                                                      @DataDog/system-tests-reviewers
manifests/rust.yml                                                      @DataDog/system-tests-reviewers
tests/ffe/README.md                                                     @DataDog/feature-flagging-and-experimentation-sdk
tests/parametric/test_ffe/test_dynamic_evaluation.py                    @DataDog/feature-flagging-and-experimentation-sdk

datadog-datadog-prod-us1 · 2026-10-29T23:27:08Z

❌ Errors

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 2 Pipeline jobs failed

Testing the test | System Tests (java, dev) / parametric / parametric (2) — ❌ 2 tests failed · 🔧 Needs a code fix, caused by this PR

View more details · View in GitHub Actions

❌

tests.parametric.test_ffe.test_dynamic_evaluation.Test_Feature_Flag_Experimentation_Opt_Out.test_canonical_modes[future_experiments_only, parametric-java]

from system_tests_suite

AssertionError: (&#39;experimentationOptOutFutureExperimentsOnly&#39;, &#39;test-case-experimentation-opt-out-future.json&#39;, 1, {&#39;reason&#39;: &#39;DEFAULT...: 1793316327185, &#39;__dd_exposure_cache_hit&#39;: True, &#39;allocationKey&#39;: &#39;50/50 split&#39;, ...}, &#39;reason&#39;: &#39;SPLIT&#39;, &#39;value&#39;: 1})
assert 1 == 0
 &#43;  where 1 = &lt;built-in method get of dict object at 0x7fe199d850c0&gt;(&#39;value&#39;)
 &#43;    where &lt;built-in method get of dict object at 0x7fe199d850c0&gt; = {&#39;errorCode&#39;: None, &#39;flagMetadata&#39;: {&#39;__dd_do_log&#39;: True, &#39;__dd_eval_timestamp_ms&#39;: 1793316327185, &#39;__dd_exposure_cache_hit&#39;: True, &#39;allocationKey&#39;: &#39;50/50 split&#39;, ...}, &#39;reason&#39;: &#39;SPLIT&#39;, &#39;value&#39;: 1}.get

self = &lt;tests.parametric.test_ffe.test_dynamic_evaluation.Test_Feature_Flag_Experimentation_Opt_Out object at 0x7fe1a74809b0&gt;
library_env = {&#39;DD_EXPERIMENTAL_FLAGGING_PROVIDER_ENABLED&#39;: &#39;true&#39;, &#39;DD_FEATURE_FLAGS_CONFIGURATION_SOURCE&#39;: &#39;remote_config&#39;, &#39;DD_FE...E_FLAGS_EXPERIMENTATION_OPT_OUT_MODE&#39;: &#39;future_experiments_only&#39;, &#39;DD_REMOTE_CONFIG_POLL_INTERVAL_SECONDS&#39;: &#39;0.2&#39;, ...}
test_agent = &lt;utils.docker_fixtures._test_agent.TestAgentAPI object at 0x7fe1a44ba3f0&gt;
test_library = &lt;utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7fe199f42c00&gt;

...

❌

tests.parametric.test_ffe.test_dynamic_evaluation.Test_Feature_Flag_Experimentation_Opt_Out.test_canonical_modes[immediate, parametric-java]

from system_tests_suite

AssertionError: (&#39;experimentationOptOutImmediate&#39;, &#39;test-case-experimentation-opt-out.json&#39;, 0, {&#39;errorCode&#39;: &#39;INVALID_CONTEXT&#39;, &#39;reas...de&#39;: &#39;FLAG_NOT_FOUND&#39;, &#39;flagMetadata&#39;: {&#39;observe_full_evaluation_data&#39;: False}, &#39;reason&#39;: &#39;ERROR&#39;, &#39;value&#39;: &#39;default&#39;})
assert &#39;FLAG_NOT_FOUND&#39; == &#39;INVALID_CONTEXT&#39;
  - INVALID_CONTEXT
  &#43; FLAG_NOT_FOUND

self = &lt;tests.parametric.test_ffe.test_dynamic_evaluation.Test_Feature_Flag_Experimentation_Opt_Out object at 0x7fe1a7480920&gt;
library_env = {&#39;DD_EXPERIMENTAL_FLAGGING_PROVIDER_ENABLED&#39;: &#39;true&#39;, &#39;DD_FEATURE_FLAGS_CONFIGURATION_SOURCE&#39;: &#39;remote_config&#39;, &#39;DD_FEATURE_FLAGS_EXPERIMENTATION_OPT_OUT_MODE&#39;: &#39;immediate&#39;, &#39;DD_REMOTE_CONFIG_POLL_INTERVAL_SECONDS&#39;: &#39;0.2&#39;, ...}
test_agent = &lt;utils.docker_fixtures._test_agent.TestAgentAPI object at 0x7fe1a44ba3f0&gt;
test_library = &lt;utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7fe19a24bf80&gt;

...

Testing the test | all-jobs-are-green

View more details · View in GitHub Actions

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 052a944 | Docs | View more details | Give us feedback!}

leoromanovsky and others added 12 commits October 21, 2026 16:56

test(ffe): consume canonical fixture snapshot

e40fc66

Replace local fixture copies with the shared corpus, preserve extracted metadata and span inputs, and add a reviewed snapshot update workflow. Environment: Datadog workspace

test(ffe): preserve evaluation details in parametric adapters

9e3ad6a

Expose Node.js error codes and Ruby resolution details to canonical assertions. Convert Ruby numeric defaults to Float before calling the OpenFeature client. Environment: Datadog workspace

Merge main into leo/ffe-canonical-20261021

abe62eb

Keep the unified FFE mock backend loading the canonical fixture snapshot and preserve updater ownership. Environment: Datadog workspace

test: honor FFE evaluation case platform skip/include filters (#8159)

1e5e179

Co-authored-by: Codex GPT-5 <noreply@localhost>

Merge main into leo/ffe-canonical-20261021

a612776

Preserve the canonical FFE fixture path through the mock backend container refactor and retain the merged platform applicability filters. Environment: Datadog workspace

Merge main into leo/ffe-canonical-20261021

9ed22a4

Preserve canonical fixture discovery and platform filters. Keep the development OpenFeature provider out of Java Lambda builds, cover both loader paths, and document unasserted split-feature metadata.

Assert canonical split-feature metadata at hook and intake boundaries

88bdb42

Gate split-feature capture for unsupported C++ and Rust clients

c7587a8

Merge main into leo/ffe-canonical-20261021

c8ced27

test(ffe): enable timestamp opt-out mode coverage for Go and Java

052a944

Exercise canonical capability expectations with disabled, immediate, and future-only process settings. Activate the validated development versions and document pending SDK dependencies. Environment: Datadog workspace

Base automatically changed from leo/ffe-canonical-20261021 to main October 29, 2026 23:56

Navigation Menu

Repository navigation

test(ffe): enable experimentation opt-out mode coverage for Go and Java - #8244

test(ffe): enable experimentation opt-out mode coverage for Go and Java#8244
leoromanovsky wants to merge 12 commits into
mainfrom
leo/ffe-opt-out-modes-20261029

leoromanovsky commented Oct 29, 2026

Uh oh!

github-actions Bot commented Oct 29, 2026

Uh oh!

datadog-datadog-prod-us1 Bot commented Oct 29, 2026 •

edited by datadog-prod-us1-6 Bot

Loading

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

leoromanovsky commented Oct 29, 2026

Motivation

Changes

Validation

Workflow

Uh oh!

github-actions Bot commented Oct 29, 2026

Uh oh!

datadog-datadog-prod-us1 Bot commented Oct 29, 2026 • edited by datadog-prod-us1-6 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ Errors

ℹ️ Info

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

datadog-datadog-prod-us1 Bot commented Oct 29, 2026 •

edited by datadog-prod-us1-6 Bot

Loading