[typotter]

Motivation

Dependent Flags work needs a core set of tests to re-use across SDKs

Changes

Added dependent flags as a self-contained evaluation-suites/dependent-flags/ suite with its own UFC and cases. The legacy root UFC and evaluation-cases/ corpus remain unchanged.

Fixture behavior added by this PR

flowchart TB
    Load["Load ufc-config.json"] --> Case["Read dependent evaluation case"]
    Case --> Traverse{"Reached dependency graph"}

    Traverse --> Success["Direct or transitive success"]
    Traverse --> Short["Short-circuited branch"]
    Traverse --> Memo["Repeated prerequisite"]
    Traverse --> Failure["Missing, cycle, or depth failure"]

    Success --> RootSuccess["Expected root value and reason"]
    Short --> RootFallback["Expected root fallback value and reason"]
    Memo --> RootMemo["Expected successful root value and reason"]
    Failure --> Defaults["Expected caller default and ERROR reason"]

    classDef changed fill:#1f6feb,color:#fff,stroke:#0d419d,stroke-width:3px
    class Success,Short,Memo,Failure,RootSuccess,RootFallback,RootMemo,Defaults changed

Loading

Downstream conformance classification decision

This PR renames the existing-drift outcome to inconclusive when both the merge-base and proposed fixture suites fail.

A whole-suite exit code only establishes that each revision has at least one failure; it cannot establish that every failure observed with the proposed corpus already existed at the merge base. Calling that state existing-drift could therefore misclassify a new dependent-flags incompatibility as pre-existing. inconclusive records the evidence without making that attribution.

The downstream jobs remain advisory. Only a passing base followed by a failing head is classified as new-regression and exits non-zero; base-fail/head-fail results remain visible through the job summary and uploaded logs for case-level review.

Compatibility and risk

• Existing core-only consumers continue loading the same root ufc-config.json and evaluation-cases/ corpus unchanged.
• Consumers explicitly activate evaluation-suites/dependent-flags/ only after implementing dependent-flag UFC syntax.
• Follow-up PR feat: add evaluation case applicability controls #56 adds the common suite discovery, validation, and include/exclude contract.

[linear-code]

FFL-1695

[Copilot]

🟡 Changes recommended

Missing negative variant-membership cases allow incorrect dependency matching implementations to pass.

1 open issue

• Add negative tests for ONE_OF and NOT_ONE_OF membership · New

What changed in this PR

Adds shared dependent-flag conformance fixtures for SDK consumers.

Changes:

• Adds dependency graphs and seven evaluation cases covering success, short-circuiting, repeated prerequisites, and failures.
• Sets the dependency-depth limit to two.
• Classifies downstream runs where both revisions fail as inconclusive and updates documentation.

File	Description
ufc-config.json	Adds dependent flags and the depth limit.
README.md	Documents fixtures and compatibility classifications.
evaluation-cases/​test-case-dependent-flags.json	Adds seven dependency evaluation cases.
ci/​run-downstream-conformance.sh	Reports dual failures as inconclusive.

_{Review effort: Balanced}

[Copilot]

🟢 Approval recommended

Fixture graphs and expected outcomes are consistent, and compatibility reporting matches the documentation.

1 open issue

• Add negative tests for ONE_OF and NOT_ONE_OF membership

_{Review effort: Balanced}

[Copilot]

🟢 Approval recommended

The expectations align with the dependency graphs, the classification change matches its documentation, and no blocking issues remain.

No open issues 🎉

1 resolved since last review

• Add negative tests for ONE_OF and NOT_ONE_OF membership

_{Review effort: Balanced}

[Copilot]

🟢 Approval recommended

The fixture expectations align with the dependency graphs, and the compatibility-reporting change matches its documentation without introducing blocking issues.

No open issues 🎉

_{Review effort: Balanced}

[leoromanovsky]

fair

[aarsilv]

🤖 Comment from Claude working with Aaron Silverman:

[P2] Follow-up for this suite, as it stands on main (b57eac6): some cases have no telemetry assertions, and some behaviors have no case.

• Six of the 14 cases have no expectations, so the harness performs no telemetry assertions for them: dependent-direct, dependent-transitive, dependent-memoized, dependent-atomic-error, dependent-over-depth and dependent-cycle. For example, an SDK that sends the leaf exposure before the missing prerequisite aborts dependent-atomic-error still passes.
• No case covers a configuration that contains a disabled prerequisite. The design forbids adding one; a case would pin what SDKs do if one arrives anyway.
• No case covers a prerequisite with doLog: false, or a root with doLog: false whose prerequisite logs. Exposure logging applies per flag.
• No case covers a rule that matches but whose allocation selects no split, so the root falls through.
• No case covers non-STRING prerequisites (BOOLEAN, JSON). These check that the comparison uses the variant key, not the value.
• Only dependent-memoized exercises a successful NOT_ONE_OF comparison, and it has no telemetry expectations.
• No case covers the default depth (no evaluatorParams) or an invalid maxDependencyDepth. These need their own suite directories, because evaluatorParams applies to the whole configuration.

Suggested change: add expectations to the six cases, and add cases for the missing behaviors in a follow-up PR. The cycle and depth expectations need the evaluators to agree first on which intermediate prerequisites get an evaluation event.

[aarsilv]

🤖 Comment from Claude working with Aaron Silverman:

[P1] At current main (b57eac6), this case expects no prerequisite exposure, which conflicts with the worked example in the dependent-flags design. We would like to follow the example.

The design's example uses a rule that requires both a user attribute (country) and a prerequisite flag's value. For a user who matches the attribute but whose prerequisite evaluates to the other variant, the root serves its fallback, and the example lists both the root and the prerequisite for exposure handling (subject to the usual logging rules and deduplication): the prerequisite decided the root's result. The design's numbered rule about rejected rules can be read more broadly, so the design is ambiguous here; the example and its stated reasoning are the clearer guide.

#58 added telemetry expectations to this case and to dependent-not-one-of-hit. Both now expect only the root exposure (noUnmatchedEvents: true), although the prerequisite's own comparison rejects the rule.

Suggested change:

• When a flagEvaluation condition is the condition that rejects the rule, that prerequisite (and the prerequisites on its own selected path) gets an exposure, subject to its doLog, when the root succeeds.
• Keep "no exposure" when a different condition rejects the rule (dependent-inapplicable-rule, dependent-transitive-inapplicable-rule), and when the root fails.
• Add {"flag": "dependent-leaf-on"} to the exposures of dependent-one-of-miss and dependent-not-one-of-hit on main.

All evaluators and DataDog/system-tests#8051 follow the current expectation, so this also needs evaluator updates (libdatadog, openfeature-js-client, Go, Java, .NET) and the matching isPresentInAssignment meaning in precomputed responses.

[aarsilv]

🤖 Comment from Claude working with Aaron Silverman:

[P2] Please pin the root error code in this case and in the depth and cycle cases. When a prerequisite fails, the root should report the general evaluation error, and the prerequisite's own evaluation event should keep the specific code.

Today the implementations in this stack disagree, and the shared evaluation suite does not decide:

• feat(openfeature): support dependent flags on Python delivery paths dd-trace-py#21346 and feat(ffe): support dependent flag evaluations dd-trace-php#4421 map the libdatadog dependency failure to GENERAL on the root (and so does the feat(ffe): expose prerequisite evaluations through C FFI libdatadog#2791 C FFI that feat(open_feature): support dependent flag evaluations dd-trace-rb#6613 will use once its native library ships).
• [FFL-1694] feat(core): evaluate variant-key flag dependencies openfeature-js-client#414, feat: Dependent Flags dd-trace-go#5799, feat: Dependent Flags dd-trace-java#13166, feat: Dependent Flags dd-trace-dotnet#9589 and feat(openfeature): support dependent flag telemetry dd-trace-py#21494 copy the prerequisite's code to the root: FLAG_NOT_FOUND for a missing prerequisite, PARSE_ERROR for a malformed one.

We propose GENERAL on the root. FLAG_NOT_FOUND on a root that exists is misleading: OpenFeature reports a flag-not-found error, and metrics tag error.type=flag_not_found for a real flag. GENERAL on the root also matches the no-assignment case that #60 added.

Suggested change on main: add "errorCode": "GENERAL" to the result of dependent-atomic-error-root, dependent-depth-three-root and dependent-cycle-a. Add expectations.evaluationEvents to the atomic-error case, so the missing prerequisite's own event carries FLAG_NOT_FOUND.