feat: add evaluation case applicability controls

.github/workflows/downstream-compatibility.yml

@@ -6,6 +6,7 @@ on:
       - .github/workflows/downstream-compatibility.yml
       - ci/run-downstream-conformance.sh
       - evaluation-cases/**
+      - evaluation-suites/**
       - ufc-config.json
 
 permissions:

.github/workflows/validate-fixtures.yml

@@ -7,6 +7,7 @@ on:
       - ci/validate-fixtures.py
       - ci/test_validate_fixtures.py
       - evaluation-cases/**
+      - evaluation-suites/**
       - ufc-config.json
 
 permissions:

CONTRIBUTING.md

@@ -52,6 +52,17 @@ projects ship support.
 3. If your test case requires a new flag, add the flag definition to `ufc-config.json`
 4. Keep the fixture SDK-neutral. Do not include SDK-specific fields such as `variant` or `flagMetadata`; downstream SDKs should derive those from `ufc-config.json` when they need them.
 
+Use `_skip` or `_include` only for a known cross-platform applicability gap.
+Never specify both. Entries must use a canonical identifier from
+`ci/validate-fixtures.py` and should include a concise reason. These controls
+apply only to rules-based evaluation cases. Do not edit precomputed-assignment
+fixtures, schemas, documentation, or validators when changing this contract.
+
+When a new feature's UFC syntax itself is not supported by every SDK, create a
+self-contained `evaluation-suites/<suite>/` directory containing its own
+`ufc-config.json` and `cases/`. Do not add that syntax to the legacy root config:
+a suite the consumer has not enabled must not have its configuration parsed.
+
 For selected split/allocation metadata, add optional `result.serialId` (integer or
 `null`) and `result.doLog` (boolean) expectations. A serial ID of `0` is valid;
 `null` asserts absence, while omitting the field leaves it unspecified. Keep
@@ -106,11 +117,14 @@ revision in downstream repos. Each rules-based consumer should:
 2. Automate fixture updates. For submodules, configure Dependabot with
    `package-ecosystem: "gitsubmodule"`; for snapshots, automate regeneration from
    the source revision.
-3. Load `ufc-config.json` and loop over every `evaluation-cases/*.json` file in
-   unit tests.
-4. Run the cases for every implemented capability as well as the baseline;
+3. Treat `ufc-config.json` and `evaluation-cases/` as the legacy `core` suite.
+4. Implement `_skip`/`_include` filtering before parametrization and side effects,
+   using the canonical platform identifiers from `ci/validate-fixtures.py`.
+5. Run applicable cases for every implemented capability as well as the baseline;
    follow the [runner requirements](README.md#implement-the-rules-based-runner).
-5. Preserve shared expectations unchanged. Avoid hand-maintained copies and
+6. Explicitly opt into supported `evaluation-suites/<suite>/` directories; do
+   not auto-enable new suites.
+7. Preserve shared expectations unchanged. Avoid hand-maintained copies and
    programmatic-only replacements for shared evaluator cases.
 
 Precomputed-assignment consumers use the same pinning and provenance rules with

README.md

@@ -80,7 +80,7 @@ ffe-system-test-data/
 ├── ufc-config.json          # Shared rules-based configuration, including malformed test inputs
 ├── evaluation-cases/
 │   └── test-*.json          # Evaluation test case files
-├── evaluation-suites/       # Self-contained, explicitly activated feature suites
+├── evaluation-suites/       # Optional self-contained, explicitly activated feature suites
 │   └── dependent-flags/
 │       ├── ufc-config.json  # UFC containing only dependent-flag fixtures
 │       └── cases/           # Evaluation cases for this feature
@@ -121,22 +121,47 @@ results in a downstream copy.
 
 ### Implement the rules-based runner
 
-1. Load `ufc-config.json` through the consumer's real configuration parser and
-   initialize its evaluator. Preserve the malformed inputs used by isolation tests.
-2. Discover every `evaluation-cases/*.json` file and parse its array. Keep case
+1. Choose one canonical platform identifier from `ci/validate-fixtures.py`.
+2. Select suites without loading configuration. `core` is selected by default;
+   feature suites are explicitly enabled by the consuming repository.
+3. Discover the selected suite's case files and parse each array. Keep case
+   order within each file, including repeated evaluations. Fail if a selected
+   fixture directory is missing or empty.
+4. Apply `_skip`/`_include` to each case for the runner's canonical platform
+   before test parametrization, setup, evaluation, telemetry expectations,
+   counts, or any other side effect. An excluded case must not execute and must
+   not influence drain markers or expected event counts.
+5. If no applicable cases remain in a suite, report it as fully excluded and do
+   not parse, send, or install that suite's UFC configuration.
+6. For a suite with applicable cases, load its adjacent `ufc-config.json`
+   through the consumer's real configuration parser and initialize its evaluator.
+   Preserve malformed inputs used by isolation tests.
+7. Run every remaining baseline case with default provider options, using
+   `flag`, `variationType`, `defaultValue`, `targetingKey`, and `attributes` as
+   supplied. Assert the expected value and reason, and any specified error code.
+   Consumers exposing selected metadata must also check supplied `serialId` and
+   `doLog` expectations against the actual selection.
+8. For each capability the consumer implements, map its name to the native
+   option and rerun every applicable case carrying that capability. Assert its
+   expected result instead of the baseline result. Run capabilities separately.
+9. Report the fixture SHA and executed and excluded case counts, including
+   counts per supported capability. Fail if a supported capability discovers no
+   applicable cases. Excluding every capability case must not look like
+   successful capability support.
+
+The legacy `core` suite uses root `ufc-config.json` and `evaluation-cases/`.
+Feature suites use `evaluation-suites/<suite>/ufc-config.json` and the adjacent
+`cases/` directory. Never parse the configuration of an unselected suite.
+
+For backward-compatible core-only consumers, the equivalent minimum flow is:
+
+1. Discover every `evaluation-cases/*.json` file and parse its array. Keep case
    order within each file, including repeated evaluations. Fail if the fixture
    directory is missing or empty.
-3. Run every baseline case with default provider options, using `flag`,
-   `variationType`, `defaultValue`, `targetingKey`, and `attributes` as supplied.
-   Assert the expected value and reason, and any specified error code. Consumers
-   exposing selected metadata must also check the supplied `serialId` and `doLog`
-   expectations against the actual selection.
-4. For each capability the consumer implements, map its name to the native
-   option and rerun every case carrying that capability. Assert its expected
-   result instead of the baseline result. Run capabilities separately.
-5. Report the fixture SHA and executed case counts, including counts per
-   supported capability. Fail if a supported capability discovers no cases.
-   Skipping every capability case must not look like successful capability support.
+2. Apply the case filters. If none remain, report `core` as fully excluded and
+   stop without loading configuration.
+3. Load root `ufc-config.json`, then run the applicable cases using the remaining
+   steps above.
 
 The adapter may translate API names and native result containers. It must
 preserve the meaning and precision of the inputs and expected results, including
@@ -199,6 +224,8 @@ Each evaluation case uses a universal schema with the following fields:
 | `result.doLog` | boolean | Optional expected selected allocation's logging decision |
 | `result.features` | object | Optional exact feature maps for `HOOK`, `EXPOSURE`, and `EVALUATION`; empty maps assert no features, omission leaves routing unspecified |
 | `capabilities` | object | Optional expected results when a named capability is enabled; uses the same result format |
+| `_skip` | array | Optional platforms that exclude this case |
+| `_include` | array | Optional exclusive list of platforms that include this case |
 
 Example:
 
@@ -215,6 +242,19 @@ Example:
 ]
 ```
 
+`_skip` and `_include` are mutually exclusive non-empty arrays of objects shaped
+as `{"platform": "python", "reason": "optional explanation"}`. Each object
+has exactly one required canonical `platform` and an optional non-empty
+`reason`; a platform may appear only once. With neither field, the case applies
+to every platform. `_skip` excludes listed platforms. `_include` includes only
+listed platforms. The canonical platform list and fixture validation are in
+[`ci/validate-fixtures.py`](ci/validate-fixtures.py).
+
+The root configuration and cases form the legacy `core` suite. New feature
+suites may be self-contained under `evaluation-suites/<suite>/`, with their own
+`ufc-config.json` and `cases/` directory. Consumers explicitly opt into suites
+and must not load the configuration for a suite they have not enabled.
+
 ### Optional capabilities
 
 Every case still runs with default provider options and checks `result`.

ci/test_validate_fixtures.py

@@ -1,7 +1,9 @@
-"""Regression checks for optional selected split/allocation expectations."""
+"""Regression checks for rules-based evaluation fixture validation."""
 
 import importlib.util
+import json
 from pathlib import Path
+import tempfile
 import unittest
 
 
@@ -90,5 +92,114 @@ def test_capability_feature_expectations_are_also_validated(self) -> None:
             )
 
 
+class EvaluationSuiteValidationTest(unittest.TestCase):
+    platforms = frozenset({"php", "python"})
+
+    @staticmethod
+    def config(flag: str) -> dict[str, object]:
+        return {
+            "createdAt": "2026-01-01T00:00:00Z",
+            "format": "SERVER",
+            "environment": {"name": "test"},
+            "flags": {
+                flag: {
+                    "key": flag,
+                    "variations": {"on": {"key": "on", "value": True}},
+                    "allocations": [],
+                }
+            },
+        }
+
+    @staticmethod
+    def case(flag: str) -> dict[str, object]:
+        return {
+            "flag": flag,
+            "variationType": "BOOLEAN",
+            "defaultValue": False,
+            "targetingKey": "subject",
+            "attributes": {},
+            "result": {"value": True, "reason": "STATIC"},
+        }
+
+    def test_legacy_and_directory_core_cannot_both_exist(self) -> None:
+        suite_path = validator.EVALUATION_SUITES_DIRECTORY / "core"
+        with self.assertRaisesRegex(validator.FixtureValidationError, "cannot exist in both"):
+            validator._validate_suite_id(suite_path, legacy_core_exists=True)
+
+    def test_self_contained_suite_uses_its_adjacent_config(self) -> None:
+        with tempfile.TemporaryDirectory() as temporary_directory:
+            suite = Path(temporary_directory) / "example"
+            cases = suite / "cases"
+            cases.mkdir(parents=True)
+            config_path = suite / "ufc-config.json"
+            config_path.write_text(json.dumps(self.config("suite-flag")), encoding="utf-8")
+            (cases / "test-suite.json").write_text(
+                json.dumps([self.case("suite-flag")]), encoding="utf-8",
+            )
+
+            self.assertEqual(
+                (1, 1),
+                validator._validate_suite(config_path, cases, self.platforms, self_contained=True),
+            )
+
+            (cases / "test-suite.json").write_text(
+                json.dumps([self.case("root-only-flag")]), encoding="utf-8",
+            )
+            with self.assertRaisesRegex(validator.FixtureValidationError, "unknown flag 'root-only-flag'"):
+                validator._validate_suite(config_path, cases, self.platforms, self_contained=True)
+
+    def test_self_contained_suite_rejects_missing_or_extra_entries(self) -> None:
+        with tempfile.TemporaryDirectory() as temporary_directory:
+            suite = Path(temporary_directory) / "example"
+            suite.mkdir()
+            config_path = suite / "ufc-config.json"
+            config_path.write_text(json.dumps(self.config("suite-flag")), encoding="utf-8")
+            cases = suite / "cases"
+
+            with self.assertRaisesRegex(validator.FixtureValidationError, "must be a directory"):
+                validator._validate_suite(config_path, cases, self.platforms, self_contained=True)
+
+            cases.mkdir()
+            (cases / "test-suite.json").write_text(
+                json.dumps([self.case("suite-flag")]), encoding="utf-8",
+            )
+            (suite / "unexpected.json").write_text("{}", encoding="utf-8")
+            with self.assertRaisesRegex(validator.FixtureValidationError, "unexpected suite entry"):
+                validator._validate_suite(config_path, cases, self.platforms, self_contained=True)
+
+
+class EvaluationPlatformValidationTest(unittest.TestCase):
+    platforms = frozenset({"php", "python"})
+    path = validator.EVALUATION_CASES_DIRECTORY / "test-case-filter.json"
+
+    def validate(self, value: dict[str, object]) -> None:
+        validator._validate_platform_filter(self.path, "case 0", value, self.platforms)
+
+    def test_canonical_platforms_cover_annotated_sdks(self) -> None:
+        self.assertIn("php", validator.CANONICAL_PLATFORMS)
+        self.assertIn("python", validator.CANONICAL_PLATFORMS)
+
+    def test_skip_and_include_accept_canonical_platforms(self) -> None:
+        self.validate({"_skip": [{"platform": "php", "reason": "Known gap."}]})
+        self.validate({"_include": [{"platform": "python"}]})
+
+    def test_skip_and_include_are_mutually_exclusive(self) -> None:
+        with self.assertRaisesRegex(validator.FixtureValidationError, "mutually exclusive"):
+            self.validate({"_skip": [{"platform": "php"}], "_include": [{"platform": "python"}]})
+
+    def test_filters_reject_invalid_shapes_and_platforms(self) -> None:
+        invalid = (
+            ({"_skip": []}, "non-empty array"),
+            ({"_skip": [{"platform": "unknown"}]}, "canonical platform"),
+            ({"_skip": [{"platform": "php"}, {"platform": "php"}]}, "repeats platform"),
+            ({"_skip": [{"platform": "php", "reason": ""}]}, "non-empty string"),
+            ({"_skip": [{"platform": "php", "ticket": "ABC-1"}]}, "unexpected fields"),
+        )
+        for value, message in invalid:
+            with self.subTest(value=value):
+                with self.assertRaisesRegex(validator.FixtureValidationError, message):
+                    self.validate(value)
+
+
 if __name__ == "__main__":
     unittest.main()