[typotter]

Motivation

#8260 has landed the canonical rules-based corpus and feature-suite import on main. This PR makes the existing dynamic-evaluation harness consume optional telemetry expectations through the same suite-directory-driven execution path used for result assertions. Core cases, dependent flags, and later feature suites therefore share one runner while retaining separate manifest activation nodes.

Changes

• discover each UFC-plus-cases suite through the common loader, but defer feature UFC parsing until an applicable activated test executes
• keep a visible failing placeholder when an activated feature suite is missing instead of silently collecting an empty parameter set
• run values, reasons, error codes, and optional telemetry expectations through one shared case executor
• enable the custom FFE intake only for case files that carry telemetry expectations, preserving pooled test-agent execution for the current core corpus
• extend the existing ffe_events_from_requests helper to distinguish an explicit null targeting key from no filter and to filter already-decoded events by raw or hashed targeting key
• consume canonical event matchers, counts, exact-event semantics, and platform _skip/_include annotations
• use an unchanged, corpus-owned static doLog=true flag as the post-batch marker and wait for both telemetry writers before checking event absence or exact counts
• reject ambiguous telemetry expectations whose targeting key is reused within the same case file; the wire events cannot reliably attribute aggregated or deduplicated events back to individual evaluations
• parse each captured request snapshot once per poll, preserve the 30-second bound with lower polling/log amplification, and include marker state in timeout diagnostics
• keep test_ffe_flag_evaluation and test_ffe_dependent_flags as separate manifest activation nodes; each node differs only in the suite directory supplied to the common harness
• keep canonical UFC and cases unchanged and owned by ffe-system-test-data

Validation:

• ./format.sh — passed, including repository-wide mypy, import policy, Ruff, YAML, parser, shell, and fixture checks
• focused module controls — passed for stable parameter IDs, pooled-agent inputs, corpus-owned marker selection, null-target filtering, repeated-target expectation handling, and matcher negative controls
• git diff --check — passed
• local Crucible review and verifier iterations — completed; final verifier found no remaining actionable issue
• existing tests/test_the_test/test_ffe_split_features.py invocation could not start because the local system_tests/weblog image is absent; no test-harness tests were added
• end-to-end SDK execution remains in the stacked per-SDK activation PRs; this push has triggered fresh CI

Stack:

• base: main; test(ffe): import canonical feature suites #8260 and test(ffe): register dependent flag RC capability #8293 are merged
• per-SDK activation PRs: [nodejs@typo/dependent-flags-evaluator-system-tests] [FFL-1694] test(nodejs): activate dependent flag coverage #8050 and [dotnet@typo/dependent-flags-returned-batch] [FFL-1694] test(dotnet): activate dependent flag coverage #8230–[ruby@typo/dependent-flags-returned-batch] [FFL-1694] test(ruby): activate dependent flag coverage #8235

Workflow

1. ⚠️ Create your PR as draft ⚠️
2. Work on you PR until the CI passes
3. Mark it as ready for review
   • Tests, manifest, weblog are modified -> you will need a review from system-tests-reviewers: ask one of your coworkers familiar with the tested feature.
   • Framework is modified, or non obvious usage of it -> get a review from system-tests-core (slack)

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

[linear-code]

FFL-1694

[github-actions]

CODEOWNERS have been resolved as:

tests/parametric/test_ffe/test_dynamic_evaluation.py                    @DataDog/feature-flagging-and-experimentation-sdk
tests/parametric/test_ffe/utils.py                                      @DataDog/feature-flagging-and-experimentation-sdk

[datadog-official]

 

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: d3cec03 | Docs | View more details | Give us feedback!}

[Copilot]

🟡 Changes recommended

Cycle assertions currently accept duplicate evaluations aggregated into a single event.

1 open finding

• Require single evaluation count for cycle flags · New

What changed in this PR

Adds telemetry validation to the dependent-flag evaluation coverage from #8049, with SDK activation remaining in #8050.

Changes:

• Adds evaluation-event extraction and telemetry matching.
• Checks graph telemetry, memoization, short-circuiting, and exposure rollback.
• Adds matcher self-tests and documents assertion timing.

File	Description
tests/​parametric/​test_ffe/​test-case-dependent-flags.json	Defines expected graph telemetry.
tests/​ffe/​utils/​evaluations.py	Extracts evaluation events by flag and subject.
tests/​ffe/​utils/​dependent_flags.py	Adds telemetry event matching.
tests/​ffe/​test_tests.py	Tests success and error matching.
tests/​ffe/​test_dependent_flags.py	Validates captured graph telemetry.
tests/​ffe/​README.md	Documents post-teardown telemetry assertions.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-11-03T11:50:55.916814Z	c83c928	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[Copilot]

🟡 Changes recommended

Telemetry validation can finish before delayed forbidden exposures arrive, allowing rollback regressions to pass.

1 open issue

• Wait for telemetry writers to drain before validating event absence · New

1 resolved since last review

• Require single evaluation count for cycle flags

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Reliable completion of asynchronous telemetry capture needs human verification before approval.

1 open issue

• Wait for telemetry writers to drain before validating event absence

_{Review effort: Balanced}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c83c9287c6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[aarsilv]

🤖 Comment from Claude working with Aaron Silverman:

[P2] Cases without expectations skip the telemetry check here, and the shared suite has gaps that this harness inherits.

• Six of the 14 cases have no expectations, so the harness performs no telemetry assertions for them: dependent-direct, dependent-transitive, dependent-memoized, dependent-atomic-error, dependent-over-depth and dependent-cycle. For example, an SDK that sends the leaf exposure before the missing prerequisite aborts dependent-atomic-error still passes.
• No case covers a configuration that contains a disabled prerequisite. The design forbids adding one; a case would pin what SDKs do if one arrives anyway.
• No case covers a prerequisite with doLog: false, or a root with doLog: false whose prerequisite logs. Exposure logging applies per flag.
• No case covers a rule that matches but whose allocation selects no split, so the root falls through.
• No case covers non-STRING prerequisites (BOOLEAN, JSON). These check that the comparison uses the variant key, not the value.
• Only dependent-memoized exercises a successful NOT_ONE_OF comparison, and it has no telemetry expectations.
• No case covers the default depth (no evaluatorParams) or an invalid maxDependencyDepth. These need their own suite directories, because evaluatorParams applies to the whole configuration.

The suite lives in ffe-system-test-data. Expectations and cases added to the existing suite directory reach this harness once its snapshot is refreshed. New suite directories (for example, for depth limits) also need wiring here, because this module hard-codes the dependent-flags suite path. Until then, these telemetry assertions and configuration scenarios are missing from the shared suite.

[aarsilv]

🤖 Comment from Claude working with Aaron Silverman:

[P2] Three ways the dependent-flags telemetry check can pass when it should fail.

• Events without a matching identifier are dropped. ffe_events_for_target keeps only events whose identifier (subject.id for exposures, targeting_key for evaluation events) matches the raw key or sha256_<hex>. For a case with a targeting key, a forbidden prerequisite exposure with no subject, or with a key in another format, is never compared, so noUnmatchedEvents still passes.
• Typos in expectations weaken the check without an error. A misspelled noUnmatchedEvent turns off exact mode. A misspelled exposure key makes the expected exposure list empty while exact mode stays on. _count: true is accepted as 1.
• Remote Config bit 50 is never asserted. Capabilities.FFE_FLAG_DEPENDENT_FLAG_VARIANT_KEY_V1 is registered, but no test reads it. The test agent serves the dependent configuration to every SDK, so an otherwise correct SDK passes this test without advertising bit 50. Per the dependent-flags design, capability-gated delivery would then omit dependent flags for it.

I confirmed the first two by running the helper functions directly on synthetic events.

Suggested changes (sketches; I did not run them in the harness):

Suggested code

# Extend the existing import from utils.dd_constants with Capabilities.
# In test_ffe_dependent_flags, before _assert_evaluation_cases:
test_agent.assert_rc_capabilities({Capabilities.FFE_FLAG_DEPENDENT_FLAG_VARIANT_KEY_V1})

Suggested code

_EXPECTATION_KEYS = frozenset({"exposures", "evaluationEvents", "noUnmatchedEvents"})
_MATCHER_KEYS = frozenset({"_count", "_skip", "_include", "allocation", "errorCode", "flag", "runtimeDefaultUsed", "variant"})

def _validate_expectations(case):
    exp = case["expectations"]
    assert isinstance(exp, dict) and exp.keys() <= _EXPECTATION_KEYS, f"unknown expectation keys: {sorted(exp)}"
    assert type(exp.get("noUnmatchedEvents", False)) is bool
    for kind in ("exposures", "evaluationEvents"):
        for matcher in exp.get(kind, []):
            assert isinstance(matcher, dict), f"{kind} matcher must be an object"
            assert matcher.keys() <= _MATCHER_KEYS, f"unknown {kind} matcher fields {sorted(matcher.keys() - _MATCHER_KEYS)}"

# Call it for each case that has "expectations", before Remote Config setup.

Also change isinstance(count, int) to type(count) is int, and fail on events that no case targeting key (or the drain and readiness markers) can claim.

[aarsilv]

🤖 Comment from Claude working with Aaron Silverman:

Heads up: three per-SDK activation PRs that build on this one enable the dependent-flags test for SDK versions that are already released and likely do not implement the feature.

Activation PR	Manifest gate for test_ffe_dependent_flags	Effect
#8050 (Node.js)	*ref_5_77_0	enabled for released Node.js versions from 5.77.0 on
#8230 (.NET)	v3.57.0-dev	enabled for released .NET 3.57.0
#8231 (PHP)	v1.26.0-dev	enabled for released PHP 1.26.x

The Node.js tracer, .NET and PHP implementation PRs for this feature (DataDog/dd-trace-js#10749, DataDog/dd-trace-dotnet#9589, DataDog/dd-trace-php#4421) have not merged, and Node.js 6.23.0, .NET 3.57.0 and PHP 1.26.2 are published releases. So these gates would likely run the test against released SDKs that do not implement dependent flags. Gate each on the first version that will contain the feature, or keep missing_feature on main and use the branch override for branch CI. Also, #8050, #8231 and #8235 target SDK branches that have no PR (for Node.js, a branch other than DataDog/dd-trace-js#10749).