Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/update-ffe-fixtures.yml
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@ jobs:

## Changes

Refresh the checked-in snapshot to [${{ steps.fixtures.outputs.source_commit }}](https://github.com/DataDog/ffe-system-test-data/commit/${{ steps.fixtures.outputs.source_commit }}), containing ${{ steps.fixtures.outputs.fixture_count }} evaluation cases.
Refresh the checked-in snapshot to [${{ steps.fixtures.outputs.source_commit }}](https://github.com/DataDog/ffe-system-test-data/commit/${{ steps.fixtures.outputs.source_commit }}), containing ${{ steps.fixtures.outputs.fixture_count }} evaluation cases and ${{ steps.fixtures.outputs.selection_fixture_count }} selection SDK cases.

Ordinary test runs use the snapshot without network access. Resolve newly exposed evaluator failures before merging; correct shared expectations upstream only when they are wrong. Scheduled updates leave an existing update PR untouched to preserve review and fixes.

Expand Down
6 changes: 4 additions & 2 deletions docs/understand/scenarios/parametric.md
Original file line number Diff line number Diff line change
Expand Up @@ -220,14 +220,16 @@ Not all endpoint implementations per language are up to spec with regards to the

#### Standalone Rules Configuration

`test_library.ffe_fetch_rules_configuration(options)` calls `POST /ffe/fetch-rules-configuration` (currently implemented in Node.js only). It fetches and serializes configuration using the public standalone FFE helpers in a fresh process, without initializing a tracer or OpenFeature provider. The response is either `{ "wire": "..." }` or `{ "error": { "name": "ConfigurationFetchError", "code": "...", "message": "...", "status": 403 } }`. `status` is present only for HTTP failures. Unexpected failures return HTTP 500.
`test_library.ffe_fetch_rules_configuration(options)` calls `POST /ffe/fetch-rules-configuration` (currently implemented in Node.js only). It fetches and serializes configuration using the public standalone FFE helpers in a fresh process, without initializing a tracer or OpenFeature provider. The response is either `{ "wire": "...", "diagnostics": [] }` or `{ "error": { "name": "ConfigurationFetchError", "code": "...", "message": "...", "status": 403 }, "diagnostics": [] }`. `status` is present only for HTTP failures. The adapter captures the helper's logger messages in `diagnostics` so tests can assert dropped or server-ignored selection diagnostics. Unexpected failures return HTTP 500.

Options use the Node helper's names (`distributionChannel`, `apiKey`, `clientToken`, `env`, `site`, `timeoutMs`). The test-only `abort` option supplies an already-aborted signal; `abortAfterMs` cancels an in-flight operation after the specified delay. `SYSTEM_TESTS_FFE_RULES_BASE_URL` routes the helper's custom Fetch transport to the mock backend, preserving paths, queries, headers, and redirect/cancellation options. No browser handoff is performed.
Options use the Node helper's names (`distributionChannel`, `apiKey`, `clientToken`, `env`, `site`, `flagSelection`, `timeoutMs`). The test-only `abort` option supplies an already-aborted signal; `abortAfterMs` cancels an in-flight operation after the specified delay. `SYSTEM_TESTS_FFE_RULES_BASE_URL` routes the helper's custom Fetch transport to the mock backend, preserving paths, queries, headers, and redirect/cancellation options. No browser handoff is performed.

The shared `mocked_backend` fixture serves these protobuf responses. Import shared FFE constants and fixture helpers from `utils.mocked_backend.ffe`. On the server distribution endpoint, explicitly accepting `application/protobuf` selects protobuf, including in comma-separated `Accept` lists with a positive quality value. Wildcard-only requests and existing agentless provider tests continue to receive JSON.

Use `mocked_backend.client.ffe_set_response` or `mocked_backend.client.ffe_set_responses` with `response_format="protobuf"` to configure these responses; the default format is JSON. Each format has an independent response timeline, and controls reject response IDs unsupported by the selected format. `mocked_backend.client.ffe_reset` resets both timelines and the shared request history. Use `mocked_backend.ffe_get_requests` to inspect paths, queries, and headers without including fixture control traffic.

The protobuf response IDs `selection_ignored_empty`, `selection_ignored_tag_load_failed`, and `selection_ignored_unknown` return the full rules fixture with a `DD-Flag-Selection-Ignored` header set to an empty string, `tag_load_failed`, or `future_reason`, respectively. Use these shared responses to test selection diagnostics without replacing the rules endpoint handler. The `valid` response omits the header.

#### Effective OpenTelemetry configuration

Use `test_library.otel_effective_config()` to inspect resolved OTLP exporter timeouts.
Expand Down
1 change: 1 addition & 0 deletions manifests/nodejs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2706,6 +2706,7 @@ manifest:
tests/parametric/test_ffe/test_dynamic_evaluation.py::Test_Feature_Flag_Split_Features: incomplete_test_app (Split-feature hook capture is currently implemented only in the Java parametric client)
tests/parametric/test_ffe/test_exposure_hooks.py: incomplete_test_app (Exposure hook observations are not implemented)
tests/parametric/test_ffe/test_rules_configuration.py: 7.0.0-pre # Exact development version until the first containing release is confirmed.
tests/parametric/test_ffe/test_rules_configuration.py::Test_FFE_Rules_Configuration_Selection: missing_feature (FFL-3364 requires the Node flag-selection dependency update in dd-trace-js)
tests/parametric/test_ffe/test_serial_id_metadata.py: v6.18.0
tests/parametric/test_ffe/test_serial_id_metadata.py::Test_FFE_Serial_Id_Metadata_Go: irrelevant (Legacy Go metadata key)
tests/parametric/test_ffe/test_span_enrichment.py: # TODO: a lower version might be supported
Expand Down
10 changes: 8 additions & 2 deletions tests/ffe/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,8 @@ first, then refresh the snapshot:
python3 utils/scripts/update_ffe_fixtures.py --ref <branch-tag-or-commit>
```

The updater imports `ufc-config.json` and `evaluation-cases/*.json` only. Ordinary
test runs use those checked-in files without submodules or network access. A
The updater imports `ufc-config.json`, `evaluation-cases/*.json`, and
`flag-selection/selection-vectors.json`. Ordinary test runs use those checked-in files without submodules or network access. A
weekly/manual workflow opens a signed draft PR when their contents change and
leaves any existing fixture update PR untouched. Review and resolve newly exposed
SDK failures before merging an update.
Expand Down Expand Up @@ -65,6 +65,12 @@ Standalone rules-configuration helpers are covered in `tests/parametric/test_ffe

These tests report to [Offline initialization and portable configuration helpers (#562)](https://feature-parity.us1.prod.dog/?feature=562). Current coverage is limited to Node's rules-based fetching and serialization; precomputed helpers and provider offline initialization are not covered here.

`Test_FFE_Rules_Configuration_Selection` covers tag/key-pattern query parameters on both distribution channels, invalid/URL-overflow selections dropped with diagnostics, quiet empty-selection omission, and retention of the full response when the backend ignores selection. It also checks diagnostics for nonempty `DD-Flag-Selection-Ignored` response headers, including unknown reasons. The mock does not implement backend filtering. These cases require [openfeature-js-client #445](https://github.com/DataDog/openfeature-js-client/pull/445) in the `dd-trace` dependency and remain gated in the Node manifest until a containing build is available. Other SDKs retain the existing file-level gate.

The class also runs the canonical selection vectors' `sdk` expectations through real HTTP requests, grouped by distribution channel to share container startup. These cover Unicode `White_Space` trimming, UTF-8 byte ordering and length, the 31/32-value and 200/201-byte boundaries, and drop-reason precedence. Raw-query percent-decoding and server-side `selectedKeys` expectations are intentionally excluded. The fixture updater validates the SDK vector shape before replacing the snapshot, and the existing update workflow tracks changes to both evaluator and selection fixtures.

To validate an unreleased build containing the selection change, use the [custom Node tracer instructions](../../docs/execute/binaries.md#nodejs-library), select `tests/parametric/test_ffe/test_rules_configuration.py::Test_FFE_Rules_Configuration_Selection`, and pass the same node ID to `-F`. Once released, update the class gate and the existing file-level version restriction together.

With Docker and the Python 3.12 runner installed:

```bash
Expand Down
2 changes: 1 addition & 1 deletion tests/parametric/test_ffe/ffe-system-test-data/SOURCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,4 @@ Do not edit these fixtures directly in system-tests. Add or update shared FFE be
ffe-system-test-data first, then refresh this snapshot.

The weekly update workflow runs `python3 utils/scripts/update_ffe_fixtures.py` and opens a signed
draft test PR only when the server evaluator fixture contents change.
draft test PR only when the evaluator or selection fixture contents change.
Loading
Loading