[FFL-3567] Add the flag-selection evaluation suite

CONTRIBUTING.md

@@ -62,6 +62,8 @@ When a new feature's UFC syntax itself is not supported by every SDK, create a
 self-contained `evaluation-suites/<suite>/` directory containing its own
 `ufc-config.json` and `cases/`. Do not add that syntax to the legacy root config:
 a suite the consumer has not enabled must not have its configuration parsed.
+A suite may also hold `flag-tags.json` when its server double needs flag tags,
+as `flag-selection` does; see [flag selection suite](README.md#flag-selection-suite).
 
 For selected split/allocation metadata, add optional `result.serialId` (integer or
 `null`) and `result.doLog` (boolean) expectations. A serial ID of `0` is valid;

README.md

@@ -87,9 +87,13 @@ ffe-system-test-data/
 ├── evaluation-cases/
 │   └── test-*.json          # Evaluation test case files
 ├── evaluation-suites/       # Optional self-contained, explicitly activated feature suites
-│   └── dependent-flags/
-│       ├── ufc-config.json  # UFC containing only dependent-flag fixtures
-│       └── cases/           # Evaluation cases for this feature
+│   ├── dependent-flags/
+│   │   ├── ufc-config.json  # UFC containing only dependent-flag fixtures
+│   │   └── cases/           # Evaluation cases for this feature
+│   └── flag-selection/
+│       ├── ufc-config.json  # Flags a server double narrows to the SDK's flag selection
+│       ├── flag-tags.json   # Tags of those flags, which UFC does not carry
+│       └── cases/           # One flag selection per case file
 ├── precomputed-assignments/
 │   ├── README.md            # Precomputed assignment fixture format
 │   └── cases/               # Precomputed assignment fixtures, discovered recursively
@@ -230,6 +234,9 @@ Each evaluation case uses a universal schema with the following fields:
 | `result.doLog` | boolean | Optional expected selected allocation's logging decision |
 | `result.features` | object | Optional exact feature maps for `HOOK`, `EXPOSURE`, and `EVALUATION`; empty maps assert no features, omission leaves routing unspecified |
 | `capabilities` | object | Optional expected results when a named capability is enabled; uses the same result format |
+| `providerOptions.flagSelection` | object | Optional raw flag selection, `{"tags": [...], "keyPatterns": [...]}`, passed untouched to the SDK's flag selection provider option. See [flag selection suite](#flag-selection-suite) |
+| `expectations.configurationRequest` | object | Optional selection the SDK sends on its configuration request: canonical `selectedFlagTags` and `selectedFlagKeyPatterns` lists, or `{"flagSelection": null}` when it sends none |
+| `configurationResponse.flagSelectionIgnored` | string | Optional reason the server double gives for ignoring the valid selection it receives |
 | `_skip` | array | Optional platforms that exclude this case |
 | `_include` | array | Optional exclusive list of platforms that include this case |
 
@@ -458,7 +465,9 @@ SDK sends, or the reason the SDK drops it. Validate them locally with
 `python3 ci/validate-flag-selection.py`.
 
 See [flag-selection/README.md](flag-selection/README.md) for the semantics and
-how to run the cases against each request form.
+how to run the cases against each request form. The
+[flag selection suite](#flag-selection-suite) runs selections through an SDK
+end to end.
 
 ## Automated Validation
 
@@ -468,6 +477,8 @@ object keys, malformed evaluation-case envelopes, mismatched flag or variation
 map keys, and references to unknown flags without a `FLAG_NOT_FOUND`
 expectation. Capability expectations may instead use `INVALID_CONTEXT` when
 context validation precedes flag lookup. The baseline still requires `FLAG_NOT_FOUND`.
+In suites, it also checks that `flag-tags.json` names only the suite's flags, the
+shape of the flag selection case fields, and that they are the same across a file.
 
 The validator deliberately does not fully schema-check allocation internals.
 Some fixtures contain malformed flag fields on purpose to verify that consumers
@@ -578,6 +589,41 @@ and `cases/` directory.
 | Suite | Description |
 |-------|-------------|
 | `dependent-flags` | Direct and transitive dependencies, short-circuiting, repeated prerequisites, failure propagation, depth limiting, and cycles |
+| `flag-selection` | SDK flag selection end to end: tags, key patterns, both, canonical form, a tag with interior whitespace, an empty selection, selections the SDK drops, and a selection the server ignores. Needs a server double; see below |
+
+### Flag selection suite
+
+`evaluation-suites/flag-selection/` checks that an SDK sends its flag selection
+on the agentless configuration request and serves only the flags the server
+returns. Each case file is one provider start, so every case in a file carries
+the same `providerOptions`, `expectations.configurationRequest`, and
+`configurationResponse`; the validator enforces this.
+
+The runner cannot load the suite's `ufc-config.json` directly. It serves it
+through a server double that applies the selection in the request with the
+[flag selection semantics](flag-selection/README.md) and the tags in
+`flag-tags.json` (a flag that is not listed has no tags), and returns only the
+selected flags. For each case file:
+
+1. Configure the server double. When `configurationResponse.flagSelectionIgnored`
+   is set, it answers a valid selection with every flag and that reason in the
+   `DD-Flag-Selection-Ignored` header.
+2. Start the provider in agentless mode against the server double, passing
+   `providerOptions.flagSelection` untouched to the SDK's flag selection option.
+   Omit the option when the field is absent.
+3. Check the first configuration request: it carries each
+   `expectations.configurationRequest` value as a repeated `selected_flag_tags`
+   or `selected_flag_key_patterns` parameter, in order, or no selection
+   parameter for `{"flagSelection": null}`.
+4. Evaluate the applicable cases as usual. A flag the server omitted evaluates
+   to the caller default with `ERROR` / `FLAG_NOT_FOUND`.
+
+An SDK sends no selection when the selection is empty after canonicalization,
+breaks a limit in [flag-selection/README.md](flag-selection/README.md#semantics),
+or would make the percent-encoded request URL longer than 8192 bytes
+([Fastly's URL limit](https://docs.fastly.com/products/network-services-resource-limits)).
+The long-URL case uses non-ASCII tags, so the URL is over that limit only once
+percent-encoded.
 
 ## Origin
 

ci/test_validate_fixtures.py

@@ -168,6 +168,128 @@ def test_self_contained_suite_rejects_missing_or_extra_entries(self) -> None:
                 validator._validate_suite(config_path, cases, self.platforms, self_contained=True)
 
 
+class FlagSelectionSuiteValidationTest(unittest.TestCase):
+    platforms = frozenset({"php", "python"})
+    path = validator.EVALUATION_SUITES_DIRECTORY / "example" / "cases" / "test-case.json"
+
+    @staticmethod
+    def case(**fields: object) -> dict[str, object]:
+        return {
+            **EvaluationSuiteValidationTest.case("suite-flag"),
+            "providerOptions": {"flagSelection": {"tags": [" team:checkout "]}},
+            "expectations": {
+                "configurationRequest": {"selectedFlagTags": ["team:checkout"], "selectedFlagKeyPatterns": []}
+            },
+            **fields,
+        }
+
+    def validate_case(self, case: dict[str, object]) -> None:
+        validator._validate_evaluation_case(self.path, 0, case, {"suite-flag": {}}, self.platforms)
+
+    def write_suite(self, directory: str, flag_tags: object, cases: list[dict[str, object]]) -> tuple[Path, Path]:
+        suite = Path(directory) / "example"
+        (suite / "cases").mkdir(parents=True)
+        config_path = suite / "ufc-config.json"
+        config_path.write_text(json.dumps(EvaluationSuiteValidationTest.config("suite-flag")), encoding="utf-8")
+        (suite / "flag-tags.json").write_text(json.dumps(flag_tags), encoding="utf-8")
+        (suite / "cases" / "test-case.json").write_text(json.dumps(cases), encoding="utf-8")
+        return config_path, suite / "cases"
+
+    def test_suite_accepts_flag_tags_for_its_own_flags(self) -> None:
+        with tempfile.TemporaryDirectory() as temporary_directory:
+            config_path, cases = self.write_suite(
+                temporary_directory, {"flags": [{"key": "suite-flag", "tags": ["team:checkout"]}]}, [self.case()]
+            )
+            self.assertEqual(
+                (1, 1), validator._validate_suite(config_path, cases, self.platforms, self_contained=True)
+            )
+
+            (config_path.parent / "unexpected.json").write_text("{}", encoding="utf-8")
+            with self.assertRaisesRegex(validator.FixtureValidationError, "unexpected suite entry"):
+                validator._validate_suite(config_path, cases, self.platforms, self_contained=True)
+
+    def test_flag_tags_reject_invalid_shapes_and_unknown_flags(self) -> None:
+        invalid = (
+            ({"flags": [{"key": "root-only-flag", "tags": []}]}, "must name a flag"),
+            ({"flags": [{"key": "suite-flag", "tags": []}, {"key": "suite-flag", "tags": []}]}, "is repeated"),
+            ({"flags": [{"key": "suite-flag", "tags": ["", "team:checkout"]}]}, "non-empty strings"),
+            ({"flags": [{"key": "suite-flag"}]}, "only key and tags"),
+            ({"flags": []}, "non-empty array"),
+            ({"flags": [], "extra": True}, "only flags"),
+        )
+        for flag_tags, message in invalid:
+            with self.subTest(flag_tags=flag_tags), tempfile.TemporaryDirectory() as temporary_directory:
+                config_path, cases = self.write_suite(temporary_directory, flag_tags, [self.case()])
+                with self.assertRaisesRegex(validator.FixtureValidationError, message):
+                    validator._validate_suite(config_path, cases, self.platforms, self_contained=True)
+
+    def test_provider_start_fields_accept_valid_shapes(self) -> None:
+        self.validate_case(self.case())
+        self.validate_case(self.case(configurationResponse={"flagSelectionIgnored": "tag_load_failed"}))
+        self.validate_case(
+            self.case(
+                providerOptions={"flagSelection": {"tags": [], "keyPatterns": []}},
+                expectations={"configurationRequest": {"flagSelection": None}},
+            )
+        )
+
+    def test_provider_start_fields_reject_invalid_shapes(self) -> None:
+        no_selection = {"configurationRequest": {"flagSelection": None}}
+        invalid = (
+            ({"providerOptions": []}, "providerOptions must be an object"),
+            ({"providerOptions": {"flagSelections": {}}}, "unknown fields: flagSelections"),
+            ({"providerOptions": {"flagSelection": {"tags": [1]}}}, "string arrays"),
+            ({"providerOptions": {"flagSelection": {"labels": []}}}, "string arrays"),
+            ({"expectations": {}}, "requires expectations.configurationRequest"),
+            ({"expectations": {"configurationRequest": {"flagSelection": False}}}, "flagSelection\": null"),
+            (
+                {
+                    "expectations": {
+                        "configurationRequest": {"selectedFlagTags": ["b", "a"], "selectedFlagKeyPatterns": []}
+                    }
+                },
+                "sorted by UTF-8 bytes",
+            ),
+            (
+                {
+                    "expectations": {
+                        "configurationRequest": {"selectedFlagTags": ["a", "a"], "selectedFlagKeyPatterns": []}
+                    }
+                },
+                "without duplicates",
+            ),
+            (
+                {"expectations": {"configurationRequest": {"selectedFlagTags": [], "selectedFlagKeyPatterns": []}}},
+                "sends no selection",
+            ),
+            ({"configurationResponse": {"flagSelectionIgnored": "unknown_reason"}}, "must be one of"),
+            ({"configurationResponse": {"reason": "tag_load_failed"}}, "only flagSelectionIgnored"),
+            (
+                {"expectations": no_selection, "configurationResponse": {"flagSelectionIgnored": "tag_load_failed"}},
+                "sends a selection",
+            ),
+        )
+        for fields, message in invalid:
+            with self.subTest(fields=fields):
+                with self.assertRaisesRegex(validator.FixtureValidationError, message):
+                    self.validate_case(self.case(**fields))
+
+    def test_cases_in_a_file_share_one_provider_start(self) -> None:
+        other_start = self.case(
+            providerOptions={"flagSelection": {"tags": ["team:search"]}},
+            expectations={
+                "configurationRequest": {"selectedFlagTags": ["team:search"], "selectedFlagKeyPatterns": []}
+            },
+        )
+        for cases in ([self.case(), other_start], [self.case(), EvaluationSuiteValidationTest.case("suite-flag")]):
+            with self.subTest(cases=cases), tempfile.TemporaryDirectory() as temporary_directory:
+                config_path, cases_directory = self.write_suite(
+                    temporary_directory, {"flags": [{"key": "suite-flag", "tags": []}]}, cases
+                )
+                with self.assertRaisesRegex(validator.FixtureValidationError, "case 1 .* must equal case 0's"):
+                    validator._validate_suite(config_path, cases_directory, self.platforms, self_contained=True)
+
+
 class EvaluationPlatformValidationTest(unittest.TestCase):
     platforms = frozenset({"php", "python"})
     path = validator.EVALUATION_CASES_DIRECTORY / "test-case-filter.json"