Skip to content

[FFL-1694] test(ffe): add dependent flag evaluation coverage - #8049

Closed
typotter wants to merge 8 commits into
mainfrom
dependent-flags-variant-key-e2e
Closed

typotter wants to merge 8 commits into
mainfrom
dependent-flags-variant-key-e2e

Conversation

@typotter

@typotter typotter commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Motivation

Validate variant-key dependent flag evaluation results through the existing parametric FFE dynamic-evaluation harness.

Linear: FFL-1694 — [SDK][JS] Support Dependent Flags

Changes

  • import the dependent graphs and evaluation cases from the canonical ffe-system-test-data snapshot
  • discover the dependent fixture exactly like every other canonical evaluation-case file
  • run every imported case through Test_Feature_Flag_Dynamic_Evaluation.test_ffe_flag_evaluation
  • validate each result field present in the fixture (value, reason, and optionally errorCode)
  • cover direct and transitive success, short-circuiting, memoization, missing prerequisites, depth limits, and cycles
  • return OpenFeature resolution details from the Node parametric endpoint so the shared runner can validate those result fields

This PR deliberately adds no dependent-flags-specific loader, test method, or manifest exemption. SDKs without dependent-flags support will fail the imported cases until case-level skip support is added separately.

Validation

  • ./format.sh --check
  • TEST_LIBRARY=nodejs venv/bin/python -m pytest -S PARAMETRIC --collect-only -q tests/parametric/test_ffe/test_dynamic_evaluation.py (46 tests; dependent fixtures collected by the generic test method)

Stack

FFE reviewer selection

Reviewers: @leoromanovsky, @sameerank

Why: Global fallback — The PR adds parametric test coverage for evaluating dependent (prerequisite) flags. Global reviewers were selected because no project or SME pod matched.

🤖 Automated selection.

Selection details

Initiative: global / global

Why this match: The PR adds parametric test coverage for evaluating dependent (prerequisite) flags. It imports a dependent-flags evaluation-case fixture and an updated UFC config from the canonical ffe-system-test-data snapshot, and makes the shared runner assert value, reason and errorCode for each case. That work is not Bandits V0, which explicitly assigns prerequisites to a separate dependent-flags initiative. It is not Flag evaluation on EVP, because there are no EVP telemetry or consent changes. It is not Guided and observable onboarding, because the purpose is not onboarding diagnostics. The supplied catalog has no dependent-flags project and no fallback entry, so this goes to global ownership.

  • @leoromanovsky — final owner (final-review-owners): Global pool filling a pod coverage gap. Existing review coverage preserved.
  • @sameerank — final owner (final-review-owners): Global pool filling a pod coverage gap. Existing review coverage preserved.

The PR author was excluded. Existing reviewers and teams were preserved.

Debug or fix a problem

Workflow run · Bits Code session

  • Wrong project or pod? Clarify this PR’s purpose, or correct the reviewer registry and publish the change. Sheet edits take effect after publication.
  • Need another reviewer now? Request them in GitHub. This workflow only adds requests and keeps existing reviewers.
  • Stuck beyond five minutes or failed? Inspect the run’s failed step and available Bits session. After fixing the cause, rerun the workflow with this PR URL and dry_run=false. Unchanged completed runs are skipped. If the receipt says needs_reconciliation or a write outcome is uncertain, ask the workflow owner to reconcile GitHub and the receipt before retrying; do not delete the lock blindly.
  • Report a problem: Comment on the reviewer-routing skill PR with this PR URL, the workflow run link, and the expected initiative or reviewer correction.
Evidence and run details

✅ Complete — reviewer coverage verified. The workflow checked pending requests or submitted reviews on the evaluated head.

Generated asynchronously by the FFE reviewer-routing workflow.

Changed paths: tests/parametric/test_ffe/ffe-system-test-data/evaluation-cases/test-case-dependent-flags.json, tests/parametric/test_ffe/ffe-system-test-data/ufc-config.json, tests/parametric/test_ffe/test_dynamic_evaluation.py, tests/parametric/test_ffe/ffe-system-test-data/SOURCE.md

Evaluated head: 0cebbe7939e2c5250128ba31dd4eb4c1131598b0

Registry revision: revision:1946aa2a-530e-487a-84a7-83aff44a1089

Pinned skill source: d214cfc65ae154b97d39af5cf32caf48aaec9bf0

Instructions: workflow-injected snapshot (no plugin loaded)

Instruction snapshot SHA-256: f23b2a88c13c0da90e2dd58246a14f257af91e2195c555039bd555b52e6f3516

@linear-code

linear-code Bot commented Oct 14, 2026

Copy link
Copy Markdown

FFL-1694

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

CODEOWNERS have been resolved as:

tests/parametric/test_ffe/ffe-system-test-data/evaluation-cases/test-case-dependent-flags.json  @DataDog/feature-flagging-and-experimentation-sdk
tests/parametric/test_ffe/ffe-system-test-data/SOURCE.md                @DataDog/feature-flagging-and-experimentation-sdk
tests/parametric/test_ffe/ffe-system-test-data/ufc-config.json          @DataDog/feature-flagging-and-experimentation-sdk
tests/parametric/test_ffe/test_dynamic_evaluation.py                    @DataDog/feature-flagging-and-experimentation-sdk

@datadog-datadog-prod-us1

datadog-datadog-prod-us1 Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

Pipelines  Tests

✨ Unblock PR with BitsAI

❌ Errors

Your PR has failed checks. Please review the issues below and take necessary action before merging.

🚦 13 Pipeline jobs failed

Testing the test | System Tests (golang, dev) / parametric / parametric (2) — ❌ 1 test failed · 🔧 Needs a code fix, caused by this PR

View more details · View in GitHub Actions

❌ tests.parametric.test_ffe.test_dynamic_evaluation.Test_Feature_Flag_Dynamic_Evaluation.test_ffe_flag_evaluation[test-case-dependent-flags.json-library_env0, ... from system_tests_suite
AssertionError: Test case 0 in test-case-dependent-flags.json failed: flag='dependent-direct-root', targetingKey='dependent-direct', field='value', expected='direct-match', actual='direct-fallback'
assert 'direct-fallback' == 'direct-match'
  - direct-match
  + direct-fallback

self = <tests.parametric.test_ffe.test_dynamic_evaluation.Test_Feature_Flag_Dynamic_Evaluation object at 0x7f9dc6bb5700>
test_case_file = 'test-case-dependent-flags.json'
test_agent = <utils.docker_fixtures._test_agent.TestAgentAPI object at 0x7f9dc3afef00>
test_library = <utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7f9db97c1520>

...
Testing the test | System Tests (java, prod) / parametric / parametric (1) — ❌ 1 test failed · 🔧 Needs a code fix, caused by this PR

View more details · View in GitHub Actions

❌ tests.parametric.test_ffe.test_dynamic_evaluation.Test_Feature_Flag_Dynamic_Evaluation.test_ffe_flag_evaluation[test-case-dependent-flags.json-library_env0, ... from system_tests_suite
AssertionError: Test case 0 in test-case-dependent-flags.json failed: flag='dependent-direct-root', targetingKey='dependent-direct', field='value', expected='direct-match', actual='direct-fallback'
assert 'direct-fallback' == 'direct-match'
  - direct-match
  + direct-fallback

self = <tests.parametric.test_ffe.test_dynamic_evaluation.Test_Feature_Flag_Dynamic_Evaluation object at 0x7fc795dadb50>
test_case_file = 'test-case-dependent-flags.json'
test_agent = <utils.docker_fixtures._test_agent.TestAgentAPI object at 0x7fc792d19ac0>
test_library = <utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7fc788271c10>

...
Testing the test | System Tests (nodejs, dev) / parametric / parametric (2) — ❌ 1 test failed · 🔧 Needs a code fix, caused by this PR

View more details · View in GitHub Actions

❌ tests.parametric.test_ffe.test_dynamic_evaluation.Test_Feature_Flag_Dynamic_Evaluation.test_ffe_flag_evaluation[test-case-dependent-flags.json-library_env0, ... from system_tests_suite
AssertionError: Test case 0 in test-case-dependent-flags.json failed: flag='dependent-direct-root', targetingKey='dependent-direct', field='value', expected='direct-match', actual='caller-default'
assert 'caller-default' == 'direct-match'
  - direct-match
  + caller-default

self = <tests.parametric.test_ffe.test_dynamic_evaluation.Test_Feature_Flag_Dynamic_Evaluation object at 0x7f7dd38556d0>
test_case_file = 'test-case-dependent-flags.json'
test_agent = <utils.docker_fixtures._test_agent.TestAgentAPI object at 0x7f7dc62e2d80>
test_library = <utils.docker_fixtures._test_clients._test_client_parametric.ParametricTestClientApi object at 0x7f7dc600e150>

...

View all 13 failed jobs.

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 0cebbe7 | Docs | View more details | Give us feedback!

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Activation gating, scenario-aborting setup validation, and missing graph telemetry assertions remain unresolved.

3 open findings
What changed in this PR

Adds dependent-flag evaluation coverage to the FFE system-test harness, with Node activation intended for stacked PR #8050.

Changes:

  • Adds dependency graphs and seven evaluation cases.
  • Introduces end-to-end root-result checks and fixture loading.
  • Adds optional detailed responses to the Node /ffe endpoint.
File Description
utils/​build/​docker/​nodejs/​express/​app.js Supports detailed flag evaluation responses.
tests/​parametric/​test_ffe/​test-case-dependent-flags.json Defines seven dependency evaluation cases.
tests/​parametric/​test_ffe/​test_dynamic_evaluation.py Excludes dependent cases from generic discovery.
tests/​parametric/​test_ffe/​flags-v1.json Adds dependency graphs and a depth limit.
tests/​ffe/​utils/​canonical.py Loads checked-in fixtures.
tests/​ffe/​test_dependent_flags.py Exercises dependent-flag root results.
tests/​ffe/​README.md Documents branch-testing commands.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread tests/ffe/test_dependent_flags.py Resolved
Comment thread tests/ffe/test_dependent_flags.py Resolved
Comment thread tests/ffe/test_dependent_flags.py Resolved
@typotter
typotter force-pushed the dependent-flags-variant-key-e2e branch from 5a23492 to 5fcbf37 Compare October 14, 2026 17:05
@typotter typotter changed the title [FFL-1694] test(ffe): add dependent flag system coverage [FFL-1694] test(ffe): add dependent flag evaluation coverage Oct 14, 2026
@typotter
typotter force-pushed the dependent-flags-variant-key-e2e branch from 5fcbf37 to 7b19f61 Compare October 14, 2026 17:21
@typotter
typotter requested a balanced review from Copilot October 14, 2026 18:05

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Provider-readiness handling needs revision, and shared fixture compatibility across SDKs remains unverified.

1 open finding
3 resolved since last review

🧠 Review effort: Balanced

Comment thread tests/ffe/test_dependent_flags.py Resolved
@typotter
typotter force-pushed the dependent-flags-variant-key-e2e branch 2 times, most recently from 36c286e to e36f568 Compare October 15, 2026 05:14
Add dependent-flag UFC fixtures and validate successful and failed root evaluation results through the FFE system-test harness.

Environment: Datadog workspace

Co-Authored-By: Codex GPT-5 <noreply@localhost>
@typotter
typotter force-pushed the dependent-flags-variant-key-e2e branch from e36f568 to deec34b Compare October 15, 2026 17:34
@typotter
typotter requested a balanced review from Copilot October 15, 2026 17:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Resolve fixture discovery and evaluation-detail conflicts with current system-tests main.

Environment: Datadog workspace

Co-Authored-By: Codex GPT-5 <noreply@localhost>
@typotter
typotter marked this pull request as ready for review October 20, 2026 16:37
@typotter
typotter requested review from a team as code owners October 20, 2026 16:37
@typotter
typotter requested review from leoromanovsky and removed request for a team October 20, 2026 16:37
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-22T16:38:06.720822Z 0cebbe7 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The expanded shared fixture reaches already-active SDK tests, and cross-SDK configuration compatibility remains unverified.

No open issues 🎉

Review effort: Balanced

Stack #8049 on Leo's #8139 branch, source the UFC and root cases from ffe-system-test-data #45, and manifest-gate dependency cases by UFC structure.

Environment: Datadog workspace

Co-Authored-By: Codex GPT-5 <noreply@localhost>
@typotter
typotter requested a review from a team as a code owner October 21, 2026 20:44
@typotter
typotter changed the base branch from main to leo/ffe-canonical-20261021 October 21, 2026 20:44
typotter added a commit that referenced this pull request Oct 21, 2026
Stack #8051 on the refreshed #8049 branch, preserve the canonical snapshot unchanged, and keep richer telemetry expectations in a system-test-only case overlay.

Environment: Datadog workspace

Co-Authored-By: Codex GPT-5 <noreply@localhost>
@typotter
typotter requested a balanced review from Copilot October 21, 2026 20:57

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The shared assertion refactor rejects valid metadata-bearing fixtures and no longer rejects unexpected error codes on successful evaluations.

2 open issues

Review effort: Balanced

@typotter
typotter marked this pull request as draft October 21, 2026 21:28
Limit response assertions to API result fields, preserve successful error-code absence checks, and refresh dependent fixtures.

Environment: Datadog workspace

Co-Authored-By: Codex GPT-5 <noreply@localhost>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No blocking issues remain; existing result checks are preserved, and dependent coverage is explicitly gated pending SDK support.

No open issues 🎉

2 resolved since last review

Review effort: Balanced

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

Existing fixture coverage and result checks are preserved, dependent tests are explicitly deactivated, and no blocking issues remain.

No open issues 🎉

Review effort: Balanced

Remove dependency-specific case partitioning and manifest exemptions so every imported evaluation case uses the shared dynamic-evaluation path.

Environment: Datadog workspace

Co-Authored-By: Codex GPT-5 <noreply@localhost>
@typotter
typotter marked this pull request as ready for review October 22, 2026 16:35
@typotter
typotter requested a balanced review from Copilot October 22, 2026 16:36

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No blocking issues remain in the fixture graphs or shared assertions, and unsupported-SDK failures are explicitly intentional.

No open issues 🎉

Review effort: Balanced

@leoromanovsky leoromanovsky left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

approving if you want to move forward here while the shared files mode is in progress

@leoromanovsky

leoromanovsky commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

FFE reviewer selection

Reviewers: @hhan2024, @sameerank, @leoromanovsky

Why: Dependent Flags — The PR adds shared cross-SDK parametric system tests for evaluating prerequisite (dependent) flags. Reviewers come from the matched pod and the required specialist pods.

🤖 Automated selection.

Selection details

Initiative: project / dependent-flags

Why this match: The PR adds shared cross-SDK parametric system tests for evaluating prerequisite (dependent) flags. The new dependent-flags evaluation cases cover direct and transitive prerequisite matches by variant key, ONE_OF and NOT_ONE_OF prerequisite conditions, short-circuiting, memoization, missing prerequisites, depth limits and cycles. The canonical UFC config adds the dependent flag graphs and maxDependencyDepth. The generic dynamic-evaluation test runs these cases. The rest of the canonical-snapshot migration supports this coverage.

Why these owners: Eligible reviewers have already approved the current head; no additional reviewers are needed for the responsibilities those approvals cover.

  • @hhan2024 — primary (dependent-flags): Matched pod member. Existing review coverage preserved.
  • @sameerank — floating (dependent-flags): Coverage and knowledge sharing. Existing review coverage preserved.
  • @leoromanovsky — specialist (openfeature-architecture): Additional subject expertise. Covers: openfeature-architecture. Existing review coverage preserved.

Additional review: openfeature-architecture — The PR changes the shared evaluation-result contract the parametric harness expects from every SDK. The Ruby endpoint switches from fetch_*_value to fetch_*_details and now returns the OpenFeature reason, errorCode and flagMetadata. The Node endpoint now returns errorCode. The shared runner now checks reason and errorCode on every case, not just value, and treats an unexpected errorCode as a failure. Coverage: @leoromanovsky. Evidence: utils/build/docker/ruby/parametric/server.rb, utils/build/docker/nodejs/parametric/server.js, tests/parametric/test_ffe/test_dynamic_evaluation.py

The PR author was excluded. Existing reviewers and teams were preserved.

Debug or fix a problem

Workflow run · Bits Code session

  • Wrong project or pod? Clarify this PR’s purpose, or correct the reviewer registry and publish the change. Sheet edits take effect after publication.
  • Need another reviewer now? Request them in GitHub. This workflow only adds requests and keeps existing reviewers.
  • Stuck beyond five minutes or failed? Inspect the run’s failed step and available Bits session. After fixing the cause, rerun the workflow with this PR URL and dry_run=false. Unchanged completed runs are skipped. If the receipt says needs_reconciliation or a write outcome is uncertain, ask the workflow owner to reconcile GitHub and the receipt before retrying; do not delete the lock blindly.
  • Report a problem: Comment on the reviewer-routing skill PR with this PR URL, the workflow run link, and the expected initiative or reviewer correction.
Evidence and run details

✅ Complete — reviewer coverage verified. The workflow checked pending requests or submitted reviews on the evaluated head.

Generated asynchronously by the FFE reviewer-routing workflow.

Changed paths: tests/parametric/test_ffe/ffe-system-test-data/evaluation-cases/test-case-dependent-flags.json, tests/parametric/test_ffe/ffe-system-test-data/ufc-config.json, tests/parametric/test_ffe/test_dynamic_evaluation.py

Evaluated head: 0cebbe7939e2c5250128ba31dd4eb4c1131598b0

Registry revision: revision:c041715e-7c35-40a5-bc60-5180aab7a600

Pinned skill source: 7e3283c602f6b18a7e3e623eecb52d3a9c091a2e

Instructions: workflow-injected snapshot (no plugin loaded)

Instruction snapshot SHA-256: 52ed8cb68675dde1a47addaefe4a9500891c24014abf7dade6dfe38cf836c1d2

Base automatically changed from leo/ffe-canonical-20261021 to main October 29, 2026 23:56
@typotter

Copy link
Copy Markdown
Contributor Author

Superseded by #8139 (canonical snapshot import), #8190 (separately activated dependent-flags harness), and #8260 (canonical evaluation-suites import and shared runner). The old branch duplicates fixture/import history and its pre-unification CI failures are no longer actionable. Per-SDK activation and telemetry work will be restacked on #8260.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants