Skip to content

Test dependent flags with shared FFE corpus - #13237

Draft
typotter wants to merge 13 commits into
typo/dependent-flags-returned-batchfrom
typo/dependent-flags-canonical-corpus
Draft

typotter wants to merge 13 commits into
typo/dependent-flags-returned-batchfrom
typo/dependent-flags-canonical-corpus

Conversation

@typotter

@typotter typotter commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

What Does This Do

Runs the shared FFE core and dependent-flags evaluation suites in the Java tracer and updates the ffe-system-test-data pointer from 8c4d2410 to current upstream 9afc3f75.

The canonical runner now discovers evaluation-suites/dependent-flags, applies the corpus _skip/_include rules for platform java, and executes every applicable case with isolated evaluator, exposure-cache, evaluation-writer, and event state. It validates results plus authored exposure and evaluation-event expectations, including counts, error codes, and noUnmatchedEvents.

Motivation

The existing runner only loaded the repository-root evaluation-cases directory. That allowed the Java dependent-flags implementation to pass its local tests without exercising the shared dependent-flags corpus or its transactional exposure and failure-telemetry contracts.

Additional Notes

  • This PR is stacked on feat: Dependent Flags #13166 because the shared dependent-flags suite requires that evaluator implementation. The PR can be retargeted to master after feat: Dependent Flags #13166 merges.
  • Existing core-only experimentation opt-out and metadata assertions remain scoped to the core suite.
  • The focused canonical run executes 458 tests, including all 14 dependent-flags cases.
  • Corpus-only pointer updates select the dd-openfeature canonical fixture tests in CI.
  • ./gradlew :dd-openfeature:test --tests 'com.datadog.openfeature.CanonicalFixtureTest' :dd-openfeature:spotlessCheck passes.
  • git diff --check passes.

Contributor Checklist

  • Format the title according to the contribution guidelines.
  • Assign the type: and comp: labels.
  • No source files were added, moved, or deleted.
  • No public configuration or documentation changes are required.

Jira ticket: N/A

typotter and others added 8 commits October 29, 2026 06:41
Add strict dependent-flag evaluation, ordered telemetry and transactional exposures, and the agentless capability header.

Environment: Datadog workspace

Co-Authored-By: Codex GPT-5 <noreply@localhost>
Publish detached prerequisite evaluations only after graph completion while keeping dependency exposures transactional.

Environment: Datadog workspace

Co-Authored-By: Codex GPT-5 <noreply@localhost>
Scope returned dependency exposures to their root hook lifecycle and isolate their delivery from ordinary root telemetry.

Environment: Datadog workspace

Co-Authored-By: Codex GPT-5 <noreply@localhost>
Integrate current evaluator opt-out behavior and align dependency telemetry with the existing hook lifecycle.

Environment: Datadog workspace

Co-Authored-By: Codex GPT-5 <noreply@localhost>
Reuse dependency snapshots, preserve invalid-depth telemetry, and scale exposure deduplication.

Environment: Datadog workspace

Co-Authored-By: Codex GPT-5 <noreply@localhost>
Bind returned telemetry to its evaluation lifecycle and avoid ordinary-path and repeated snapshot allocations.

Environment: Datadog workspace

Co-Authored-By: Codex GPT-5 <noreply@localhost>
Pin configuration snapshots across client hooks and structurally memoize shared exposure traces.

Environment: Datadog workspace

Co-Authored-By: Codex GPT-5 <noreply@localhost>
Claim pinned snapshots once and preserve wrapped telemetry setup.

Environment: Datadog workspace

Co-Authored-By: Codex GPT-5 <noreply@localhost>
@typotter typotter added tag: ai generated Largely based on code generated by an AI or LLM tag: no release notes Changes to exclude from release notes type: feature Enhancements and improvements comp: testing Testing comp: openfeature OpenFeature labels Nov 5, 2026
@datadog-prod-us1-6

datadog-prod-us1-6 Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 59.68% (+7.41%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 0604842 | Docs | Give us feedback!

@dd-octo-sts

dd-octo-sts Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 13.99 s 13.96 s [-0.6%; +1.0%] (no difference)
startup:insecure-bank:tracing:Agent 12.88 s 12.94 s [-1.1%; +0.3%] (no difference)
startup:petclinic:appsec:Agent 17.23 s 17.10 s [-0.1%; +1.5%] (no difference)
startup:petclinic:iast:Agent 17.05 s 17.02 s [-0.5%; +0.9%] (no difference)
startup:petclinic:profiling:Agent 16.68 s 17.06 s [-3.3%; -1.2%] (significantly better)
startup:petclinic:sca:Agent 17.22 s 17.00 s [+0.5%; +2.1%] (maybe worse)
startup:petclinic:tracing:Agent 16.25 s 16.16 s [-0.4%; +1.5%] (no difference)

Commit: 06048429 · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

Keep prerequisite telemetry and customer exposure callbacks aligned while making dependency parsing forward-compatible.

Environment: Datadog workspace

Co-Authored-By: Codex GPT-5 <noreply@localhost>
@typotter typotter mentioned this pull request Nov 5, 2026
typotter and others added 4 commits November 5, 2026 22:56
Bind dependency publication to the exact evaluation and provider scope, and preserve lifecycle identity through the deprecated API delegate.

Environment: Datadog workspace

Co-Authored-By: Codex GPT-5 <noreply@localhost>
Exercise production provider hooks and greedily match emitted telemetry.

Environment: Datadog workspace

Co-Authored-By: Codex GPT-5 <noreply@localhost>
Select the canonical fixture tests for corpus-only updates and cover selector and count semantics.

Environment: Datadog workspace

Co-Authored-By: Codex GPT-5 <noreply@localhost>
@typotter
typotter force-pushed the typo/dependent-flags-canonical-corpus branch from 10944a8 to 0604842 Compare November 5, 2026 22:58
@typotter
typotter force-pushed the typo/dependent-flags-returned-batch branch from d948973 to edf686b Compare November 5, 2026 23:14

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: openfeature OpenFeature comp: testing Testing tag: ai generated Largely based on code generated by an AI or LLM tag: no release notes Changes to exclude from release notes type: feature Enhancements and improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant