Skip to content

chore(openfeature): prepare evaluator-aware agentless capabilities - #21335

Open
hhan2024 wants to merge 4 commits into
mainfrom
helen/dependent-flags-python-agentless
Open

hhan2024 wants to merge 4 commits into
mainfrom
helen/dependent-flags-python-agentless

Conversation

@hhan2024

@hhan2024 hhan2024 commented Oct 28, 2026 •

Copy link
Copy Markdown

Description

Prepare agentless feature-flag requests to send cap_dependent-flags-variant-key=v1 in the query string only when the native evaluator binding explicitly reports dependent-flag-variant-key-v1. The optional HTTP capability header is no longer sent. Existing capability parameters in custom endpoints are normalized to one supported value or removed for unsupported evaluators; unrelated query values and their encoding are preserved.

Add a shared capability reporter for configuration transports; older native extensions without the reporter advertise no optional features. The reporter deliberately returns no capabilities today: the linked evaluator does not support prerequisite evaluation. Activation requires native evaluation, error propagation, and exposure-reporting support plus corresponding integration tests.

Testing

  • scripts/run-tests --venv 616dbb5f2c38 -- -q -k test_agentless_poller: 85 tests passed on Python 3.11.
  • Coverage includes absent/empty/unknown capabilities, positive native support, initial and conditional requests, existing and duplicate query parameters, custom URL encoding, and the currently unsupported native evaluator.
  • Formatting, full scripts/lint checks, scoped typing, and type-comment checks pass.

Risks

A future positive capability declaration must cover both evaluation and exposure semantics; request transport support alone is insufficient. Capability selection is fixed when the source is constructed so initial and conditional polls use the same URL variant.

Additional Notes

No release note: preparatory internal plumbing does not enable a customer-facing feature.

Benchmark discovery finds the exposure aggregation scenario, which does not exercise the capability reporter or HTTP poller; performance coverage for that path remains a gap.

Advertise optional UFC extensions only when the native evaluator binding reports support. Keep dependent flags unadvertised until evaluation and exposure support are implemented.

Environment: Datadog workspace

Co-Authored-By: Codex GPT-6 <noreply@localhost>
@cit-pr-commenter-54b7da

Copy link
Copy Markdown

Codeowners resolved as

Resolved from the full PR diff against main using the target branch CODEOWNERS file.
CODEOWNERS team requests not listed below are not required by the current file set.

ddtrace/internal/native/_native.pyi                                     @DataDog/apm-core-python
ddtrace/internal/openfeature/_agentless_source.py                       @DataDog/feature-flagging-and-experimentation-sdk
ddtrace/internal/openfeature/_native.py                                 @DataDog/feature-flagging-and-experimentation-sdk
src/native/ffe.rs                                                       @DataDog/feature-flagging-and-experimentation-sdk
tests/openfeature/test_agentless_poller.py                              @DataDog/feature-flagging-and-experimentation-sdk

@cit-pr-commenter-54b7da

cit-pr-commenter-54b7da Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Dependency direction analysis

⚠️ Existing dependency direction violations

There are 104 dependency direction violations that already exist on the base branch and have not been changed by this PR.

Show existing violations (showing 5 of 104 highest severity)
ddtrace.internal.tracemethods -×-> ddtrace.trace  (internal-core -> product:tracing, score=122)
ddtrace.llmobs._evaluators.runner -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=120)
ddtrace.llmobs._integrations.bedrock -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=120)
ddtrace.llmobs._integrations.bedrock_agents -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=120)
ddtrace.aiguard._api_client -×-> ddtrace.trace  (product:aiguard -> product:tracing, score=120)

To see all violations, download the layers-base.json and layers-pr.json artifacts from this CI job and run:

uv run --script scripts/import-analysis/layers.py compare layers-base.json layers-pr.json

@hhan2024 hhan2024 added the changelog/no-changelog A changelog entry is not required for this PR. label Oct 28, 2026 — with ChatGPT Codex Connector
@datadog-datadog-prod-us1

datadog-datadog-prod-us1 Bot commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Pipelines  Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 0768653 | Docs | View more details | Give us feedback!

@pr-commenter

pr-commenter Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-11-05 18:07:15

Comparing candidate commit 0768653 in PR branch helen/dependent-flags-python-agentless with baseline commit 3001869 in branch main.

📊 Benchmarking dashboard

Found 2 performance improvements and 0 performance regressions! Performance is the same for 591 metrics, 10 unstable metrics, 1 flaky benchmarks without significant changes.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:httppropagationextract-empty_headers

  • 🟩 execution_time [-68.767ns; -57.268ns] or [-8.928%; -7.435%]

scenario:httppropagationextract-wsgi_empty_headers

  • 🟩 execution_time [-71.028ns; -61.109ns] or [-9.226%; -7.937%]

Unstable benchmarks

These benchmarks have a confidence interval too wide to call a change; treat them as noise rather than signal.

scenario:coreapiscenario-context_with_data_listeners

  • unstable execution_time [-641.028ns; +808.571ns] or [-6.025%; +7.600%]

scenario:coreapiscenario-core_dispatch_1_listener

  • unstable execution_time [-26.680ns; +45.481ns] or [-3.903%; +6.653%]

scenario:coreapiscenario-core_dispatch_50_listeners

  • unstable execution_time [-1740.778ns; +2053.607ns] or [-8.802%; +10.384%]

scenario:coreapiscenario-core_dispatch_exception_listeners

  • unstable execution_time [-1854.553ns; +1776.574ns] or [-9.706%; +9.298%]

scenario:coreapiscenario-core_dispatch_listeners

  • unstable execution_time [-252.163ns; +501.939ns] or [-5.934%; +11.812%]

scenario:coreapiscenario-core_dispatch_no_args_listeners

  • unstable execution_time [-204.067ns; +241.331ns] or [-7.643%; +9.039%]

scenario:coreapiscenario-core_dispatch_with_results_1_listener

  • unstable execution_time [-77.798ns; +100.281ns] or [-5.524%; +7.121%]

scenario:coreapiscenario-core_dispatch_with_results_50_listeners

  • unstable execution_time [-4369.096ns; +5291.761ns] or [-8.722%; +10.564%]

scenario:coreapiscenario-core_dispatch_with_results_listeners

  • unstable execution_time [-971.458ns; +1035.648ns] or [-9.156%; +9.761%]

scenario:packagesupdateimporteddependencies-import_many_stdlib_cached

  • unstable execution_time [-47.988µs; +63.482µs] or [-8.240%; +10.901%]

Known flaky benchmarks

These benchmarks are marked as flaky and will not trigger a failure. Modify FLAKY_BENCHMARKS_REGEX to control which benchmarks are marked as flaky.

Known flaky benchmarks without significant changes:

  • scenario:packagespackageforrootmodulemapping-cache_off

@hhan2024
hhan2024 marked this pull request as ready for review November 2, 2026 14:39
@hhan2024
hhan2024 requested review from a team as code owners November 2, 2026 14:39
@hhan2024
hhan2024 requested review from emmettbutler and removed request for a team November 2, 2026 14:39
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 2, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-05T17:33:29.000418Z 0768653 New commits
🔒 Security Review ✅ Completed 2026-11-05T17:34:33.924393Z 0768653 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@leoromanovsky

leoromanovsky commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

FFE reviewer selection

Reviewers: @typotter, @sameerank

Why: Dependent Flags — The PR prepares the Python SDK for dependent (prerequisite) flags. Reviewers come from the matched pod.

🤖 Automated selection.

Selection details

Initiative: project / dependent-flags

Why this match: The PR prepares the Python SDK for dependent (prerequisite) flags. It adds a native evaluator capability reporter (ffe.get_evaluator_capabilities in Rust, which returns nothing for now because prerequisite evaluation isn't supported yet) and a Python wrapper with a DEPENDENT_FLAG_VARIANT_KEY_V1 token. The agentless poller sends the X-DD-FEATURE-FLAGS-CAPABILITIES: dependent-flag-variant-key-v1 header only when the evaluator reports that capability. The agentless request path is just where the negotiation happens: the change does not alter configuration delivery itself, so it does not fit deployment-modes. No openfeature-architecture review is needed. The new capability token is a dependent-flags contract, not a provider-lifecycle or evaluation-metadata change, and today it returns an empty list, so runtime behavior is unchanged.

  • @typotter — primary (dependent-flags): Matched pod member. Existing review coverage preserved.
  • @sameerank — floating (dependent-flags): Coverage and knowledge sharing. Existing review coverage preserved.

The PR author was excluded. Existing reviewers and teams were preserved.

Debug or fix a problem

Workflow run · Bits Code session

  • Wrong project or pod? Clarify this PR’s purpose, or correct the reviewer registry and publish the change. Sheet edits take effect after publication.
  • Need another reviewer now? Request them in GitHub. This workflow only adds requests and keeps existing reviewers.
  • Stuck beyond five minutes or failed? Inspect the run’s failed step and available Bits session. After fixing the cause, rerun the workflow with this PR URL and dry_run=false. Unchanged completed runs are skipped. If the receipt says needs_reconciliation or a write outcome is uncertain, ask the workflow owner to reconcile GitHub and the receipt before retrying; do not delete the lock blindly.
  • Report a problem: Comment on the reviewer-routing skill PR with this PR URL, the workflow run link, and the expected initiative or reviewer correction.
Evidence and run details

✅ Complete — reviewer coverage verified. The workflow checked pending requests or submitted reviews on the evaluated head.

Generated asynchronously by the FFE reviewer-routing workflow.

Changed paths: src/native/ffe.rs, ddtrace/internal/openfeature/_native.py, ddtrace/internal/openfeature/_agentless_source.py, ddtrace/internal/native/_native.pyi, tests/openfeature/test_agentless_poller.py

Evaluated head: 290b9fba2b2d8ef7ac60e51616884bccd1186fac

Registry revision: revision:efde5129-b144-41ab-b630-1adf6ad22901

Pinned skill source: 7e3283c602f6b18a7e3e623eecb52d3a9c091a2e

Instructions: workflow-injected snapshot (no plugin loaded)

Instruction snapshot SHA-256: 52ed8cb68675dde1a47addaefe4a9500891c24014abf7dade6dfe38cf836c1d2

@datadog-datadog-prod-us1 datadog-datadog-prod-us1 Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

Capability advertisement requires explicit native support. The current evaluator reports no capabilities, and older extensions fall back to advertising none, preserving existing request behavior.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit 290b9fb

@sameerank sameerank added the crucible: reviewing A Crucible review has started; this PR has reviewer attention. label Nov 2, 2026
Comment thread ddtrace/internal/native/_native.pyi Outdated
Comment on lines 1011 to 1016
@staticmethod
def get_evaluator_capabilities() -> list[str]: ...

"""
Native Feature Flags and Experimentation module.
"""

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Comment from Claude working with Sameeran Kunche:

[P3] Put the class ffe docstring back above the new get_evaluator_capabilities stub.

A string is a class docstring only when it is the first statement in the class body. With the stub above it, the string on lines 1014-1016 is a bare expression. So the stub has no class docstring for ffe, which can affect the description that tools such as IDEs show. Runtime behavior does not change.

The sibling stubs logger and config put the docstring first. #21346 also moves it back. Fixing it here keeps that hunk out of the feature PR.

Suggested patch
diff --git a/ddtrace/internal/native/_native.pyi b/ddtrace/internal/native/_native.pyi
index c5cb0bee5c..f91e80610b 100644
--- a/ddtrace/internal/native/_native.pyi
+++ b/ddtrace/internal/native/_native.pyi
@@ -1008,13 +1008,11 @@ class SerializationError(Exception):
     ...
 
 class ffe:
+    """Native Feature Flags and Experimentation module."""
+
     @staticmethod
     def get_evaluator_capabilities() -> list[str]: ...
 
-    """
-    Native Feature Flags and Experimentation module.
-    """
-
     class FlagType(Enum):
         String = ...
         Integer = ...

This uses the same one-line text as #21346. A local merge of this change with the #21346 head (9d2a8a5) has no conflict.

How I checked

I parsed the stub with Python's ast module. ast.get_docstring() for class ffe returns 'Native Feature Flags and Experimentation module.' at the base (3001869) and None at this head (290b9fb). The stub has 58 top-level classes, and 40 of them carry a description string. At the base, all 40 put it first. At this head, ffe is the only exception. With the patch, ffe has its docstring again, and scripts/lint fmt passes.

[
((), None),
(("unknown-feature-v1",), None),
(("dependent-flag-variant-key-v1",), "dependent-flag-variant-key-v1"),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Comment from Claude working with Sameeran Kunche:

[P3] Add a test row where the reporter returns the dependent-flag token together with a token Python does not know.

_headers() sends only DEPENDENT_FLAG_VARIANT_KEY_V1, even when the reporter returns other tokens. The ("unknown-feature-v1",) row checks that an unknown token alone is not sent. No row checks the same filter when the dependent-flag token is also present.

So a later change that forwards every reported token would pass the current tests. For example, headers["X-DD-FEATURE-FLAGS-CAPABILITIES"] = ",".join(get_evaluator_capabilities()), with the guard on _agentless_source.py:320 kept. Only the dependent-only row reaches that assignment, and its join is the same single token.

The new row uses a list, which is also what the real binding returns. The current rows use tuples.

Suggested patch
diff --git a/tests/openfeature/test_agentless_poller.py b/tests/openfeature/test_agentless_poller.py
index a8c3523710..23f8a346df 100644
--- a/tests/openfeature/test_agentless_poller.py
+++ b/tests/openfeature/test_agentless_poller.py
@@ -234,6 +234,7 @@ def test_api_key_header_present_and_absent(harness):
         ((), None),
         (("unknown-feature-v1",), None),
         (("dependent-flag-variant-key-v1",), "dependent-flag-variant-key-v1"),
+        (["unknown-feature-v1", "dependent-flag-variant-key-v1"], "dependent-flag-variant-key-v1"),
     ],
 )
 def test_evaluator_capability_header_on_initial_and_conditional_requests(harness, monkeypatch, capabilities, expected):
What I checked, and what I could not run

I applied the row to this head in a separate worktree, and scripts/lint fmt passes. I could not run the tests: scripts/run-tests needs Docker, and none was available. That the forwarding change passes the current rows and fails the new row comes from tracing the current test inputs by hand, not from a run.

@sameerank sameerank left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both comments are non-blocking/nits!

This review also pointed to a couple suggestions in your other draft PR

Restore the native ffe stub docstring and cover filtering mixed evaluator capabilities on initial and conditional requests.

Environment: Datadog workspace

Co-Authored-By: Codex GPT-6 <noreply@localhost>
…eter

Use the evaluator capability report to select the dependent-flag URL variant, preserve unrelated query encoding, and cover conditional requests and duplicate parameters.

Environment: Datadog workspace

Co-Authored-By: Codex GPT-6 <noreply@localhost>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/no-changelog A changelog entry is not required for this PR. crucible: reviewing A Crucible review has started; this PR has reviewer attention.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants