test(ffe): enforce dependent flag RC capability contract by hhan2024 · Pull Request #8323 · DataDog/system-tests

hhan2024 · 2026-11-03T18:35:41Z

Motivation

An SDK that advertises dependent-flag support must send RC capability bit 50 before receiving dependent configuration. Registering the bit in #8293 does not verify actual SDK requests or that the advertised feature works.

Changes

Add a shared parametric test requiring bit 50 on every observed FFE_FLAGS request, including initial polls and polls after configuration updates. Requests for unrelated RC products do not affect the assertion.
Deliver a prerequisite configuration only after observing the capability. Evaluate matching and nonmatching variant keys (deliberately distinct from flag values), then verify the changed result after an RC update. No configuration is injected through the application endpoint.
Add a legacy-build test requiring ordinary RC flag evaluation and no bit 50. No traffic cannot produce a passing negative assertion.
Add six harness regressions covering base64 and byte-array masks, intermittent omissions, false advertisement, and absent traffic.
Document supported/legacy build selection and forced execution for unreleased native combinations.

Activation and merge requirements

Python's positive test remains missing_feature until a release containing dd-trace-py #21346 is confirmed. Run implementation builds with -F to make failures blocking, then activate the confirmed release threshold.
Python 4.16.0 is the explicitly tested legacy baseline.
All SDKs retain missing-feature declarations until their supporting SDK/native releases are available. Their feature PRs can run the positive class with -F to make failures blocking, then replace the manifest declaration with the confirmed release threshold.
Ruby must test the actual native package as well as the Ruby SDK version.
This uses the shared test agent to model capability-gated delivery; production backend targeting is outside this test.

Validation

Current dd-trace-py #21346 head f0d8327aeb90ce7de894fa09f984310cf4d8b275 / Python 3.11, using its local source and compiled native extensions: 1 passed in 23.69s, with the positive class forced using -F (not an expected failure). This checks initial bit 50 advertisement, matching/nonmatching variant keys, the RC update, and subsequent polls.
Released Python SDK 4.16.0: 1 passed for legacy RC delivery and absent bit 50.
TEST_THE_TEST assertion regressions: 6 passed.
./format.sh: passed, including mypy, import rules, formatting, manifests/YAML, shell and Node checks.
Live Ruby/PHP and other SDK runs remain for their integration CI; no cross-language pass is claimed.

Check bit 50 on every FFE RC poll before delivery and after updates, exercise variant-key prerequisites through Agent RC, and cover a known legacy SDK without support. Environment: Datadog workspace Co-Authored-By: Codex GPT-6 <noreply@localhost>

github-actions · 2026-11-03T18:37:22Z

CODEOWNERS have been resolved as:

tests/parametric/test_ffe/test_dependent_flags_rc.py                    @DataDog/feature-flagging-and-experimentation-sdk
tests/test_the_test/test_ffe_rc_capabilities.py                         @DataDog/system-tests-reviewers
docs/understand/scenarios/parametric.md                                 @DataDog/system-tests-core
manifests/cpp.yml                                                       @DataDog/system-tests-reviewers
manifests/dotnet.yml                                                    @DataDog/system-tests-reviewers
manifests/golang.yml                                                    @DataDog/system-tests-reviewers
manifests/java.yml                                                      @DataDog/system-tests-reviewers
manifests/nodejs.yml                                                    @DataDog/system-tests-reviewers
manifests/php.yml                                                       @DataDog/system-tests-reviewers
manifests/python.yml                                                    @DataDog/system-tests-reviewers
manifests/ruby.yml                                                      @DataDog/system-tests-reviewers
manifests/rust.yml                                                      @DataDog/system-tests-reviewers
tests/parametric/test_ffe/utils.py                                      @DataDog/feature-flagging-and-experimentation-sdk
utils/docker_fixtures/_test_clients/_test_client_parametric.py          @DataDog/system-tests-core

datadog-datadog-prod-us1 · 2026-11-03T19:16:04Z

❌ Errors

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 5 Pipeline jobs failed

Testing the test | System Tests (dotnet, dev) / parametric / parametric (2) — ❌ 4 tests failed

View more details · View in GitHub Actions
❌ tests.parametric.test_ffe.test_configuration_sources.Test_Feature_Flag_Configuration_Source_Cold_Failure_And_Recovery.test_bad_to_unchanged_cold_preserves_no... from system_tests_suite
 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
utils/docker_fixtures/_test_clients/_test_client_parametric.py:838: in ffe_start
    resp = self._session.post(self._url(&#34;/ffe/start&#34;), json=payload, timeout=30)
venv/lib/python3.12/site-packages/requests/sessions.py:637: in post
    return self.request(&#34;POST&#34;, url, data=data, json=json, **kwargs)
venv/lib/python3.12/site-packages/requests/sessions.py:589: in request
    resp = self.send(prep, **send_kwargs)
venv/lib/python3.12/site-packages/requests/sessions.py:703: in send
    r = adapter.send(request, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
...
❌ tests.parametric.test_ffe.test_configuration_sources.Test_Feature_Flag_Configuration_Source_Cold_Failure_And_Recovery.test_malformed_cold[library_env0, param... from system_tests_suite
 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
utils/docker_fixtures/_test_clients/_test_client_parametric.py:838: in ffe_start
    resp = self._session.post(self._url(&#34;/ffe/start&#34;), json=payload, timeout=30)
venv/lib/python3.12/site-packages/requests/sessions.py:637: in post
    return self.request(&#34;POST&#34;, url, data=data, json=json, **kwargs)
venv/lib/python3.12/site-packages/requests/sessions.py:589: in request
    resp = self.send(prep, **send_kwargs)
venv/lib/python3.12/site-packages/requests/sessions.py:703: in send
    r = adapter.send(request, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
...
↳ and 2 more — View all
Testing the test | System Tests (nodejs, prod) / parametric / parametric (1) — ❌ 2 tests failed

View more details · View in GitHub Actions
❌ tests.parametric.test_ffe.test_configuration_sources.Test_Feature_Flag_Configuration_Source_Cold_Failure_And_Recovery.test_malformed_cold[library_env0, param... from system_tests_suite
 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
utils/docker_fixtures/_test_clients/_test_client_parametric.py:838: in ffe_start
    resp = self._session.post(self._url(&#34;/ffe/start&#34;), json=payload, timeout=30)
venv/lib/python3.12/site-packages/requests/sessions.py:637: in post
    return self.request(&#34;POST&#34;, url, data=data, json=json, **kwargs)
venv/lib/python3.12/site-packages/requests/sessions.py:589: in request
    resp = self.send(prep, **send_kwargs)
venv/lib/python3.12/site-packages/requests/sessions.py:703: in send
    r = adapter.send(request, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
...
❌ tests.parametric.test_ffe.test_configuration_sources.Test_Feature_Flag_Configuration_Source_Cold_Failure_And_Recovery.test_request_timeout_cold[library_env0,... from system_tests_suite
 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
utils/docker_fixtures/_test_clients/_test_client_parametric.py:838: in ffe_start
    resp = self._session.post(self._url(&#34;/ffe/start&#34;), json=payload, timeout=30)
venv/lib/python3.12/site-packages/requests/sessions.py:637: in post
    return self.request(&#34;POST&#34;, url, data=data, json=json, **kwargs)
venv/lib/python3.12/site-packages/requests/sessions.py:589: in request
    resp = self.send(prep, **send_kwargs)
venv/lib/python3.12/site-packages/requests/sessions.py:703: in send
    r = adapter.send(request, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
...
Testing the test | System Tests (nodejs, dev) / parametric / parametric (2) — 🔄 Retry may pass, looks flaky

View more details · View in GitHub Actions
View all 5 failed jobs.

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 08dda62 | Docs | View more details | Give us feedback!}

Keep the positive RC capability test missing_feature until a Python release containing dd-trace-py #21346 is confirmed, and align the activation docs. Environment: Datadog workspace Co-Authored-By: Codex GPT-6 <noreply@localhost>

chatgpt-codex-connector · 2026-11-04T18:35:23Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-04T18:37:21.775891Z	`43090f4`	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Use the shared dependent-flag capability in assertions and capability constants for regression fixtures instead of hardcoded bit positions. Environment: Datadog workspace Co-Authored-By: Codex GPT-6 <noreply@localhost>

Copilot

🟡 Changes recommended

The tests exclude Go GET polls, require provider readiness before configuration delivery, and can mistake a prior configuration’s ACK for the update’s, causing false failures and unreliable update verification.

3 open issues

What changed in this PR

Adds shared parametric coverage for dependent flags over Remote Configuration, extending bit 50 registration into capability-advertisement and evaluation checks.

Changes:

Tests supported and legacy builds, including variant-key matching and configuration updates.
Adds six harness regressions for capability decoding, missing or false advertisements, and absent traffic.
Adds manifest declarations and documents forced execution until supporting releases are confirmed.

File	Description
`tests/test_the_test/test_ffe_rc_capabilities.py`	Adds capability assertion regressions.
`tests/parametric/test_ffe/utils.py`	Adds per-request bit 50 validation.
`tests/parametric/test_ffe/test_dependent_flags_rc.py`	Adds supported and legacy RC contract tests.
`manifests/rust.yml`	Declares pending support and legacy exclusion.
`manifests/ruby.yml`	Declares pending support and legacy exclusion.
`manifests/python.yml`	Declares pending support and pins legacy coverage to 4.16.0.
`manifests/php.yml`	Declares pending support and legacy exclusion.
`manifests/nodejs.yml`	Declares pending support and legacy exclusion.
`manifests/java.yml`	Declares pending support and legacy exclusion.
`manifests/golang.yml`	Declares pending support and legacy exclusion.
`manifests/dotnet.yml`	Declares pending support and legacy exclusion.
`manifests/cpp.yml`	Declares pending support and legacy exclusion.
`docs/understand/scenarios/parametric.md`	Documents build selection, forced runs, and release activation.

_{Review effort: Balanced}

typotter

please address copilot's findings

Include GET polls, require changed cached content with its ACK, and coordinate provider initialization with capability-gated RC delivery. Add harness regressions and bound the startup HTTP request. Environment: Datadog workspace Co-Authored-By: Codex GPT-6 <noreply@localhost>

Move the RC transport helpers into the existing FFE utils.py and update imports and mock targets so the test discovery convention check passes. Environment: Datadog workspace Co-Authored-By: Codex GPT-6 <noreply@localhost>

hhan2024 marked this pull request as ready for review November 4, 2026 18:35

hhan2024 requested review from a team as code owners November 4, 2026 18:35

test(ffe): use registered RC capability constants

d49323c

Use the shared dependent-flag capability in assertions and capability constants for regression fixtures instead of hardcoded bit positions. Environment: Datadog workspace Co-Authored-By: Codex GPT-6 <noreply@localhost>

hhan2024 requested review from typotter and a balanced review from Copilot November 4, 2026 19:48

Copilot started reviewing on behalf of hhan2024 November 4, 2026 20:16

View session

Copilot AI reviewed Nov 4, 2026

View reviewed changes

Comment thread tests/parametric/test_ffe/test_dependent_flags_rc.py Resolved

Comment thread tests/parametric/test_ffe/test_dependent_flags_rc.py Resolved

Comment thread tests/parametric/test_ffe/test_dependent_flags_rc.py Resolved

typotter requested changes Nov 5, 2026

View reviewed changes

hhan2024 and others added 2 commits November 5, 2026 22:04

test(ffe): keep RC helpers in the supported utils module

08dda62

Move the RC transport helpers into the existing FFE utils.py and update imports and mock targets so the test discovery convention check passes. Environment: Datadog workspace Co-Authored-By: Codex GPT-6 <noreply@localhost>

hhan2024 mentioned this pull request Nov 5, 2026

test(ffe): cover delayed capability refresh through a real Agent #8377

Draft

Navigation Menu

Repository navigation

test(ffe): enforce dependent flag RC capability contract - #8323

test(ffe): enforce dependent flag RC capability contract#8323
hhan2024 wants to merge 5 commits into
mainfrom
helen/dependent-flags-system-tests

hhan2024 commented Nov 3, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Nov 3, 2026 •

edited

Loading

Uh oh!

datadog-datadog-prod-us1 Bot commented Nov 3, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot commented Nov 4, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

typotter left a comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

hhan2024 commented Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Activation and merge requirements

Validation

Uh oh!

github-actions Bot commented Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-datadog-prod-us1 Bot commented Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ Errors

ℹ️ Info

Uh oh!

chatgpt-codex-connector Bot commented Nov 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

typotter left a comment

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

hhan2024 commented Nov 3, 2026 •

edited

Loading

github-actions Bot commented Nov 3, 2026 •

edited

Loading

datadog-datadog-prod-us1 Bot commented Nov 3, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Nov 4, 2026 •

edited

Loading