Extract shared response signing conformance fixtures

.github/workflows/validate-response-signing.yml

@@ -0,0 +1,34 @@
+name: Response signing fixture validation
+
+on:
+  pull_request:
+    paths:
+      - .github/workflows/validate-response-signing.yml
+      - ci/validate-response-signing.py
+      - ci/test_validate_response_signing.py
+      - schemas/response-signing-v1.schema.json
+      - response-signing/**
+
+permissions:
+  contents: read
+
+concurrency:
+  group: response-signing-validation-${{ github.event.pull_request.number }}
+  cancel-in-progress: true
+
+jobs:
+  validate:
+    name: Validate response signing fixtures
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    steps:
+      - name: Check out fixtures
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+        with:
+          persist-credentials: false
+      - name: Install dependencies
+        run: pip install jsonschema
+      - name: Validate response signing fixtures
+        run: python3 ci/validate-response-signing.py
+      - name: Test response signing fixture validation
+        run: python3 -m unittest discover -s ci -p 'test_validate_response_signing.py'

README.md

@@ -53,6 +53,7 @@ behavior and does not need an update whenever a consumer ships support.
 | --- | --- | --- |
 | An SDK or backend that evaluates UFC rules | [`ufc-config.json`](ufc-config.json), [`evaluation-cases/`](evaluation-cases/), and [optional capabilities](#optional-capabilities) | Load the configuration through the real ingestion path and evaluate each case with the native evaluator. |
 | An SDK that consumes precomputed assignments | [Precomputed fixture guide](precomputed-assignments/README.md) and [schema](schemas/precomputed-assignment.schema.json) | Supply the mocked response, run the ordered evaluations, and check results and emitted events. |
+| An assignment signer or response verifier | [Response signing fixtures](response-signing/README.md) and [schema](schemas/response-signing-v1.schema.json) | Check exact request/response transcripts, certificate chains and acceptance results with the native implementation. |
 
 These families have different formats and runners. Passing one does not
 establish coverage of the other. When changing the contract, also read

ci/test_validate_response_signing.py

@@ -0,0 +1,78 @@
+"""Regression checks for unambiguous response-signing fixture inputs."""
+
+import contextlib
+import hashlib
+import importlib.util
+import io
+import json
+from pathlib import Path
+import shutil
+import tempfile
+import unittest
+
+
+ROOT = Path(__file__).resolve().parents[1]
+SPEC = importlib.util.spec_from_file_location(
+    "validate_response_signing", ROOT / "ci/validate-response-signing.py"
+)
+assert SPEC is not None and SPEC.loader is not None
+validator = importlib.util.module_from_spec(SPEC)
+SPEC.loader.exec_module(validator)
+
+
+class ResponseSigningValidationTest(unittest.TestCase):
+    def test_adversarial_corpus_including_duplicate_http_headers_is_valid(self):
+        with contextlib.redirect_stdout(io.StringIO()):
+            validator.validate(ROOT)
+
+    def test_duplicate_object_keys_are_rejected_in_every_input(self):
+        mutations = [
+            ("schemas/response-signing-v1.schema.json", '"title":', '"title": "ambiguous", "title":'),
+            (
+                "response-signing/v1/manifest.json",
+                '"schema_version": 1',
+                '"schema_version": 0, "schema_version": 1',
+            ),
+            (
+                "response-signing/v1/corpus.json",
+                '"expected_accept": true',
+                '"expected_accept": false, "expected_accept": true',
+            ),
+            (
+                "response-signing/v1/nested-der.json",
+                '"expected_accept": true',
+                '"expected_accept": false, "expected_accept": true',
+            ),
+            (
+                "response-signing/v1/isolated-certificate-rules.json",
+                '"expected_accept": true',
+                '"expected_accept": false, "expected_accept": true',
+            ),
+            (
+                "response-signing/v1/corpus.json",
+                '"expected_accept": true',
+                '"expected_accept": false, "expected\\u005faccept": true',
+            ),
+        ]
+        for relative, old, new in mutations:
+            with self.subTest(document=relative, replacement=new), tempfile.TemporaryDirectory() as directory:
+                root = Path(directory)
+                for name in ("schemas", "response-signing"):
+                    shutil.copytree(ROOT / name, root / name)
+                path = root / relative
+                raw = path.read_text()
+                self.assertIn(old, raw)
+                path.write_text(raw.replace(old, new, 1))
+                if path.name in ("corpus.json", "nested-der.json", "isolated-certificate-rules.json"):
+                    # Keep integrity checks valid so rejection must come from JSON parsing.
+                    manifest_path = root / "response-signing/v1/manifest.json"
+                    manifest = json.loads(manifest_path.read_text())
+                    manifest["files"][path.name]["sha256"] = hashlib.sha256(path.read_bytes()).hexdigest()
+                    manifest_path.write_text(json.dumps(manifest))
+                with contextlib.redirect_stdout(io.StringIO()):
+                    with self.assertRaisesRegex(ValueError, "duplicate object key"):
+                        validator.validate(root)
+
+
+if __name__ == "__main__":
+    unittest.main()

ci/validate-response-signing.py

@@ -0,0 +1,67 @@
+#!/usr/bin/env python3
+"""Validate the shape and pinned bytes of the response-signing conformance data."""
+
+import base64
+import hashlib
+import json
+from pathlib import Path
+
+import jsonschema
+
+
+def reject_duplicate_keys(pairs):
+    result = {}
+    for key, value in pairs:
+        if key in result:
+            raise ValueError(f"duplicate object key {key!r}")
+        result[key] = value
+    return result
+
+
+def validate(root):
+    directory = root / "response-signing/v1"
+    schema = json.loads(
+        (root / "schemas/response-signing-v1.schema.json").read_text(),
+        object_pairs_hook=reject_duplicate_keys,
+    )
+    jsonschema.Draft202012Validator.check_schema(schema)
+    manifest = json.loads(
+        (directory / "manifest.json").read_text(), object_pairs_hook=reject_duplicate_keys
+    )
+    assert manifest["schema_version"] == 1
+    assert set(manifest["files"]) == {"corpus.json", "nested-der.json", "isolated-certificate-rules.json"}
+    for name, expected in manifest["files"].items():
+        raw = (directory / name).read_bytes()
+        assert hashlib.sha256(raw).hexdigest() == expected["sha256"], name
+        corpus = json.loads(raw, object_pairs_hook=reject_duplicate_keys)
+        jsonschema.validate(corpus, schema)
+        cases = corpus["cases"]
+        ids = [case["id"] for case in cases]
+        assert len(ids) == len(set(ids)), f"{name}: duplicate case id"
+        assert len(cases) == expected["response_cases"], name
+        assert sum(case["expected_accept"] for case in cases) == expected["accepted"], name
+        for case in cases:
+            for message in (case["request"], case["response"]):
+                base64.b64decode(message["body_base64"], validate=True)
+            if case["expected_accept"]:
+                assert case["rejection_reason"] is None, case["id"]
+                assert case["transcript_base64"], case["id"]
+                for message in (case["request"], case["response"]):
+                    assert [v for k, v in message["headers"] if k.lower() ==
+                            "x-datadog-feature-flags-signature-version"] == ["1"], case["id"]
+                assert base64.b64decode(case["transcript_base64"], validate=True).startswith(
+                    b"datadog.feature-flags.precomputed-assignments.v1\0"
+                ), case["id"]
+            if case.get("transcript_base64") is not None:
+                base64.b64decode(case["transcript_base64"], validate=True)
+        signing_cases = corpus.get("signing_cases", [])
+        assert len(signing_cases) == expected["signing_cases"], name
+        signing_ids = [case["id"] for case in signing_cases]
+        assert len(signing_ids) == len(set(signing_ids)), name
+        for case in signing_cases:
+            assert case["case_id"] in ids, case["id"]
+        print(f"{name}: {len(cases)} response cases, {len(signing_cases)} signer cases validated")
+
+
+if __name__ == "__main__":
+    validate(Path(__file__).resolve().parents[1])

response-signing/README.md

@@ -0,0 +1,85 @@
+# Response signing conformance fixtures
+
+These fixtures define shared inputs and expected results for the v1 assignment
+response signer and client verifiers. Consumers pin an exact repository revision
+and run their native implementation against these bytes. The files are test data;
+clients do not fetch them when evaluating flags or verifying a response.
+
+## Suites
+
+| File | Coverage | Consumers |
+| --- | --- | --- |
+| `v1/corpus.json` | 46 response cases: 6 accepted and 40 rejected; 2 private-key matching cases | Verifiers consume `cases`; signers also consume `signing_cases` |
+| `v1/isolated-certificate-rules.json` | Accepted control and isolated certificate-size, leaf-CA and intermediate-CA rejections | Signers and verifiers |
+| `v1/nested-der.json` | Canonical and noncanonical nested INTEGER encodings in certificate extensions | Signers and verifiers |
+
+Version 1 is the first public response-signing protocol. Earlier POC v1/v2 and
+the implementation draft called v3 were never public release contracts. Public
+v1 requires its complete request/response transcript and leaf/intermediate chain;
+it does not accept the older POC format also named v1. X.509 v3 certificate
+encoding is independent of this protocol version and remains unchanged.
+
+These vectors regenerate the synthetic chains and signatures for public v1,
+preserving the earlier acceptance rules and adding rejection cases for prototype
+version headers and the abandoned v3 transcript domain. The [previous extraction](https://github.com/DataDog/ffe-system-test-data/tree/32f2249d66b83b507d68125fd8cda96648ab2bfd/response-signing/v3)
+remains available in history. `v1/manifest.json` records the new hashes and
+applicability counts; earlier validation receipts do not validate these new bytes.
+
+## Running a consumer
+
+1. Decode `body_base64` into exact bytes. Preserve header order, duplicates and
+   URL escaping: several rejection cases exercise those distinctions.
+   `response.body_base64` contains application bytes after HTTP content decoding.
+   When `wire_body_base64` is present, transport tests serve those compressed bytes
+   with the supplied encoding headers; direct verifier tests use `body_base64`.
+2. Supply the fixture's `anchor_der_base64` as the local trust anchor and `now`
+   as Unix epoch seconds, converting to the native clock unit when needed.
+   Use this fixed time even after the synthetic certificates expire.
+3. Verify each request/response pair and assert `expected_accept`. For accepted
+   cases, compare the constructed transcript with `transcript_base64` as well.
+   `rejection_reason` describes the violated rule; it is not a required native
+   exception string.
+4. Signers additionally load each `signing_cases` credential and its referenced
+   response case, then assert `expected_signing`. Clients exclude these cases
+   because they do not hold signing keys. The nested-DER supplement's credentials
+   let signers exercise the same accepted/rejected certificate boundary.
+
+The positive cases include leaf rotation and a later intermediate under the same
+root, without requiring the client to have seen earlier certificates. Negative
+cases cover unrelated roots, invalid chain profiles, expiry, changed request or
+response bytes, malformed proof headers and signature tampering. Each response
+carries its leaf and intermediate; verification requires no certificate fetch.
+These fixtures cover buffered response proofs, not a signed NDJSON protocol.
+
+Native adapters, secret-store access assertions, fail-open serving behavior and
+implementation-specific diagnostics belong in consumer repositories. Keep these
+fixtures independent of those adapters; never derive expected results from the
+implementation being tested.
+
+## Synthetic credentials
+
+All certificates, identities, tokens and private keys in this suite are synthetic
+test material. The included leaf private keys are intentionally public so signer
+implementations can be tested. They must never be trusted or provisioned outside
+tests. There are no production or staging credentials or customer responses here.
+
+## Validation and updates
+
+Run `python3 ci/validate-response-signing.py` with `jsonschema` installed. It checks
+the schema, exact fixture hashes, case identities, references and applicability
+counts. It deliberately allows malformed proof/certificate fields used by
+negative cases. Structural validation does not prove cryptographic correctness;
+run native signer and verifier tests when changing this contract.
+
+Review expectation changes together with the fixtures and update the manifest
+deliberately. Do not reformat signed body bytes, regenerate keys, or normalize
+headers during import. Follow the repository's [consumer pinning
+rules](../CONTRIBUTING.md#updating-downstream-repositories).
+
+The isolated certificate cases retain valid certificate and response signatures.
+The oversized leaf uses a noncritical DER OCTET STRING extension for padding;
+the leaf-CA case keeps leaf KeyUsage and no path-length constraint; the
+intermediate-CA case keeps CA KeyUsage and path length zero while omitting
+the default false CA flag. Each negative changes only the named profile rule.
+Consumers must explicitly load this supplement; a manifest entry alone does
+not exercise it. Its leaf credentials are public synthetic test material.