Skip to content

Commit 20ae514

Browse files
committed
feat(tck): emit a machine-readable conformance report
Collapsed from 22 commits when this branch was replayed onto the base's controlApi -> backend_type rename and its adopter tag gate, following the js-sdk-contrib replay of the same branch. Several of the commits fought the rename itself, so one squashed replay meant one conflict pass, and this repository squash-merges anyway. The report follows the rename: it writes backend.backendType through backend_type_of(), so a control that states none is reported as "custom", matching the schema change in spec#425. The report-binding fixture now runs ahead of the base's combined scenario gate, so a scenario skipped by any of the gates still reaches its suite. The original subjects, oldest first: - feat(tck): emit a machine-readable conformance report - feat(tck): identify a Scenario Outline row by its parameters - fix(tck): give a failed capability a reason, and stop claiming untested ones - fix(tck): read the tags of the Examples block a row came from - refactor(tck): carry results in Cucumber Messages - feat(tck): record which Cucumber Messages release produced the stream - fix(tck): say when a control does not report how it drives the backend - refactor(tck): rename @strict-numeric-typing to @numeric-coercion - fix(tck): a reserved capability must not be declarable - feat(tck): report an extension under its own uri prefix - feat(tck): refuse a report that did not run the canonical set - fix(tck): read the canonical count from the assets, not from a test - fix(tck): follow the canonical gherkin/ prefix through the report - fix(tck): stop reporting declaration.notApplicable - docs(tck): refresh the report numbers for 56 scenarios - fix(tck): follow the base's rename and the removal of features_path() - feat(tck)!: controlApi is always written, and the guessing goes away - refactor(tck)!: the environment variables drop their PROVIDER_ prefix - docs(tck): separate "answers the question" from "satisfies the capability" - docs(tck): restate the report documentation on the rewritten README - fix(tck): the mismatch matrix is eight rows, not eleven - docs(tck): follow the base branch's run numbers Signed-off-by: Simon Schrottner <simon.schrottner@flagsmith.com>
1 parent dd0d4b0 commit 20ae514

21 files changed

Lines changed: 4514 additions & 19 deletions

‎tools/openfeature-tck/.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,3 +5,6 @@
55
src/openfeature/contrib/tools/tck/gherkin/
66
src/openfeature/contrib/tools/tck/flag_data/
77
src/openfeature/contrib/tools/tck/control-api.yaml
8+
# Generated alongside them, from the submodule pin, so a conformance report can
9+
# name the spec revision it ran against.
10+
src/openfeature/contrib/tools/tck/spec_revision.json

‎tools/openfeature-tck/README.md‎

Lines changed: 80 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -371,6 +371,83 @@ self-tests declare less than they otherwise would: it cannot update its flag set
371371
hands each variant back untouched, so it does not attempt numeric coercion at all — a permitted
372372
choice rather than a defect, and the reason `@numeric-coercion` is simply not declared there.
373373

374+
## Conformance reports
375+
376+
Set `TCK_REPORT_DIR` and each suite writes **two** files: an envelope at `<dir>/<name>.json`,
377+
conforming to the [report schema][report-schema], and the results it points at, at
378+
`<dir>/<name>.ndjson`, which is a [Cucumber Messages][messages] stream.
379+
380+
```console
381+
$ TCK_REPORT_DIR=./reports pytest
382+
tck [in-memory]: report written to reports/in-memory.json with results in in-memory.ndjson (1 failed, 43 passed, 21 skipped)
383+
384+
$ jq -c .results reports/in-memory.json
385+
{"format":"cucumber-messages","location":"in-memory.ndjson","digest":"sha256:c7e12a…"}
386+
```
387+
388+
It is an environment variable rather than a `TckConfig` field so that emitting a report is a property
389+
of the *run* and not of the code: CI sets it, a developer running the suite locally does not, and no
390+
adopter changes a line to publish one. Unset means no report, which is not an error, and several
391+
suites in one session each write their own pair, so flagd's two resolvers do not collide.
392+
393+
**A partial run is not a conformance run.** `-k`, `-m`, `--deselect`, or a test module that stopped
394+
calling `scenarios()` on the canonical path each run fewer scenarios, and none of them is an error to
395+
pytest — Go measured the consequence: a green run and a well-formed report covering one scenario out
396+
of the whole canonical set, with nothing in the document saying so. So every run is checked against
397+
the scenarios this distribution ships, and one that did not execute all of them writes no report and
398+
names what it missed. A capability-gated skip **has** run — the question was put and declined — so
399+
declining never trips this, and your own extension scenarios are not counted towards the canonical
400+
set and cannot close a gap in it. `TCK_PARTIAL=1` lets you work on a single scenario without the
401+
guard failing the run; it still writes no report, and Java spells the same escape hatch the same way.
402+
403+
**Why the results are not our format.** Per-scenario outcomes, tags, Scenario Outline row identity
404+
and the executed feature source are already specified by Cucumber Messages, which is maintained,
405+
cross-language, schema'd and emitted natively by cucumber-jvm; defining them again would create a
406+
second format to version and two places for the same fact to disagree. So the envelope says what was
407+
tested and what the provider claims, and the payload says what happened. The payload is referenced
408+
rather than inlined because it carries the feature sources and is far larger than the envelope, and
409+
`results.digest` is a SHA-256 over the exact bytes written, so a consumer can tell that what it
410+
fetched is what the envelope described. Two things Messages cannot carry stay in the envelope:
411+
`declaration`, which is an *input* to reading the results rather than a summary of them — only it
412+
says whether a skipped scenario was declined — and the tested subject, for which no standard results
413+
format has a slot.
414+
415+
**Reading the payload.** Appendix F requires a scenario skipped for an undeclared capability to be
416+
reported as skipped with the reason and never as passed, and a consumer cannot check that against a
417+
summary line, so the stream carries every scenario the run collected, gate-skipped ones included, as
418+
Cucumber's own `SKIPPED`. Each is a `TestCase` referring to a `Pickle`, and a test case is as bad as
419+
its worst step. Every test case carries two hook steps as well as its Gherkin steps, because pytest
420+
runs a scenario in three phases and only the middle one executes steps: the before-hook is where a
421+
capability skip's reason lands and the after-hook where a teardown failure does. The capability
422+
responsible for a skip follows from the pickle's tags and the envelope's `declaration`, which is why
423+
it is not transported once per scenario. And a pickle's `astNodeIds` are `[scenario id, table row
424+
id]`, resolving in the `GherkinDocument` to exactly the cells the feature file wrote — which is what
425+
tells the eight rows of the type-mismatch matrix apart, one of which differs in outcome from its seven
426+
siblings, exactly rather than by a naming convention every implementation would have to reproduce.
427+
428+
So the payload is not a transcription of pytest's summary. The one scenario the Python SDK cannot
429+
satisfy is marked `xfail`, so pytest counts it as expected and exits zero; the provider still did not
430+
satisfy it, and the stream says `FAILED`. The acknowledgement goes in the envelope's
431+
`knownDeviations` instead — an expected failure is a recorded deviation, not an excused one.
432+
433+
**What identifies a report.** `tck.specRevision` comes from `spec_revision.json`, which the asset
434+
sync generates from the submodule at build time, because the submodule is not in the wheel and an
435+
installed copy has nothing left to ask; a build that cannot reach git records `unknown` rather than
436+
inventing a commit. There is no asset tree hash: the payload's `Source` messages carry the executed
437+
feature files verbatim, which answers "did these two runs ask the same questions" directly rather
438+
than by proxy. `provider.name` is what the provider reports through its own metadata, not
439+
`TckConfig.name`, which is chosen to read well in a failure message — `flagd-rpc` — and is therefore
440+
reported as the *configuration*. `backend.backendType` comes from the control's `backend_type`, `"custom"` when it states none, and
441+
the whole `backend` block is always written, both being in the schema's `required` arrays: nothing
442+
is inferred from the control's class, which is the point.
443+
444+
One gap is this implementation's rather than the suite's: **pytest-bdd emits no Cucumber Messages.**
445+
It ships the legacy Cucumber JSON format and nothing for the ndjson protocol, so `messages.py` builds
446+
the stream from the official types and re-parses the feature files for the AST node ids a pickle
447+
refers to. If pytest-bdd ever emits Messages itself, that module should shrink to a shim. Whether a
448+
report belongs inside a provider's released artifact is open on
449+
[open-feature/spec#424](https://github.com/open-feature/spec/issues/424).
450+
374451
## Contributing
375452

376453
The Gherkin, the canonical flag set and the control-API document are **not owned by this
@@ -379,7 +456,7 @@ sdist at build time. So adopting needs no submodule and contributing does:
379456

380457
```bash
381458
git submodule update --init tools/openfeature-tck/spec
382-
poe test # syncs the assets first; 227 passed, 41 skipped, 2 xfailed, no Docker
459+
poe test # syncs the assets first; 295 passed, 41 skipped, 2 xfailed, no Docker
383460
```
384461

385462
The copies under `src/` are gitignored, generated and carry a `DO-NOT-EDIT.txt`: a change goes to
@@ -401,6 +478,8 @@ longer defeats it, because the pin is read by naming the superproject that git c
401478
remains exempt is an unpacked sdist, which has no submodule, no pin and nothing that could have
402479
drifted from one.
403480

481+
[report-schema]: https://github.com/open-feature/spec/blob/main/specification/assets/provider-tck/report/conformance-report.schema.json
482+
[messages]: https://github.com/cucumber/messages
404483
[appendix-a]: https://github.com/open-feature/spec/blob/main/specification/appendix-a-included-utilities.md
405484
[appendix-f]: https://github.com/open-feature/spec/blob/main/specification/appendix-f-provider-conformance.md
406485
[java-tck]: https://github.com/open-feature/java-sdk-contrib/tree/main/tools/tck

‎tools/openfeature-tck/hatch_build.py‎

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,15 +18,28 @@
1818
# the single definition of what gets copied where -- would not be importable.
1919
sys.path.insert(0, str(Path(__file__).parent))
2020

21-
from hatch_build_sync import FILES, PACKAGE_REL, SPEC_ASSETS, TREES, sync
21+
from hatch_build_sync import (
22+
FILES,
23+
PACKAGE_REL,
24+
REVISION_FILE,
25+
SPEC_ASSETS,
26+
TREES,
27+
sync,
28+
)
2229

2330

2431
class SpecAssetsCopyHook(BuildHookInterface):
2532
PLUGIN_NAME = "spec-assets-copy"
2633

2734
def initialize(self, version: str, build_data: dict) -> None:
2835
root = Path(self.root)
29-
copies = [root / PACKAGE_REL / dest for _, dest in TREES + FILES]
36+
# The generated revision file travels with the assets it describes. It
37+
# has to be built here rather than read at run time, because the
38+
# submodule that knows the answer is not in the wheel and a conformance
39+
# report has to name the revision it ran against.
40+
copies = [root / PACKAGE_REL / dest for _, dest in TREES + FILES] + [
41+
root / PACKAGE_REL / REVISION_FILE
42+
]
3043

3144
# Building from a checkout: refresh from the submodule, so what ships is
3245
# always the revision the pin names. Building from an sdist: there is no

‎tools/openfeature-tck/hatch_build_sync.py‎

Lines changed: 67 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@
1515

1616
from __future__ import annotations
1717

18+
import json
1819
import os
1920
import shutil
2021
import subprocess
@@ -25,7 +26,8 @@
2526
ROOT = Path(__file__).parent
2627
SPEC_DIRNAME = "spec"
2728
SPEC_ROOT = (ROOT / SPEC_DIRNAME).resolve()
28-
SPEC_ASSETS = (SPEC_ROOT / "specification/assets/provider-tck").resolve()
29+
ASSETS_PATH_IN_SPEC = "specification/assets/provider-tck"
30+
SPEC_ASSETS = (SPEC_ROOT / ASSETS_PATH_IN_SPEC).resolve()
2931
PACKAGE_REL = Path("src/openfeature/contrib/tools/tck")
3032
DEST_BASE = ROOT / PACKAGE_REL
3133

@@ -243,6 +245,28 @@ def checkout_pinned_spec(git: GitRunner = _run_git) -> str | None:
243245
return pinned
244246

245247

248+
REVISION_FILE = "spec_revision.json"
249+
"""Which revision of the specification the copied assets came from.
250+
251+
Recorded at build time because the answer is only available at build time: the
252+
submodule that holds it is not in the wheel, and a conformance report that cannot
253+
name the revision it ran against cannot be compared with another. It is generated
254+
by the same command that copies the assets, which is what keeps the two from
255+
disagreeing.
256+
257+
Not committed, for the same reason the assets are not: the submodule pin is the
258+
single record of which revision this package targets.
259+
"""
260+
261+
UNKNOWN_REVISION = "unknown"
262+
"""Seven characters, the minimum the report schema accepts.
263+
264+
A build that cannot reach git says it does not know rather than inventing a
265+
commit, and still produces a document that validates. Which happens for real:
266+
building from a source tarball has no ``.git`` to ask.
267+
"""
268+
269+
246270
def sync() -> None:
247271
checkout_pinned_spec()
248272

@@ -267,6 +291,48 @@ def sync() -> None:
267291
dest.unlink()
268292
shutil.copy2(SPEC_ASSETS / src_name, dest)
269293

294+
write_revision()
295+
296+
297+
def write_revision() -> None:
298+
"""Record the spec commit these copies came from.
299+
300+
The asset tree hash that used to accompany it is gone. It was carried so a
301+
consumer could tell whether two runs executed the same questions; the
302+
conformance report's results are now a Cucumber Messages stream, which
303+
carries the executed feature source itself and answers that directly rather
304+
than by proxy.
305+
"""
306+
commit = _git("rev-parse", "HEAD") or UNKNOWN_REVISION
307+
(DEST_BASE / REVISION_FILE).write_text(
308+
json.dumps({"specRevision": commit}, indent=2) + "\n",
309+
encoding="utf-8",
310+
)
311+
312+
313+
def _git(*args: str) -> str:
314+
"""Run git inside the submodule, returning its output or an empty string.
315+
316+
A build must not hard-fail because git is absent or the checkout is not a
317+
repository -- both are ordinary when building from an unpacked sdist. The
318+
failure is reported as a warning and the identity degrades to ``unknown``,
319+
which is legible in the resulting report rather than silently wrong.
320+
"""
321+
command = ["git", "-C", str(SPEC_ROOT), *args]
322+
try:
323+
completed = subprocess.run( # noqa: S603
324+
command, capture_output=True, check=True, text=True
325+
)
326+
except (OSError, subprocess.CalledProcessError) as error:
327+
warnings.warn(
328+
f"could not determine the spec revision ({' '.join(command)}: {error}); "
329+
f"conformance reports from this build will not name the revision they "
330+
f"ran against",
331+
stacklevel=2,
332+
)
333+
return ""
334+
return completed.stdout.strip()
335+
270336

271337
if __name__ == "__main__":
272338
sync()

‎tools/openfeature-tck/pyproject.toml‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,21 @@ dependencies = [
2222
# Same runner the flagd provider and the flagd testkit already use, so an
2323
# adopting module gains no new test framework.
2424
"pytest-bdd>=8.1.0,<9.0.0",
25+
# The conformance report's results are a Cucumber Messages stream rather
26+
# than a format this package defines. These two are the reference
27+
# implementations of the halves of that protocol: cucumber-messages is the
28+
# official Python types, published from the same repository as the protocol
29+
# itself, and gherkin-official is the parser that produces the
30+
# gherkinDocument and pickle messages. pytest-bdd already depends on
31+
# gherkin-official, so only the first is genuinely new -- and it has no
32+
# dependencies of its own.
33+
#
34+
# pytest-bdd ships no Messages emitter (its cucumber_json.py is the legacy
35+
# JSON format), so the stream is assembled here; assembling it from typed
36+
# messages rather than hand-written dicts is what keeps it from drifting
37+
# away from the protocol.
38+
"cucumber-messages>=34.0.0,<35.0.0",
39+
"gherkin-official>=29.0.0",
2540
]
2641
requires-python = ">=3.10"
2742

@@ -69,6 +84,10 @@ artifacts = [
6984
"src/openfeature/contrib/tools/tck/gherkin/",
7085
"src/openfeature/contrib/tools/tck/flag_data/",
7186
"src/openfeature/contrib/tools/tck/control-api.yaml",
87+
# Which spec revision those assets came from, generated beside them. The
88+
# submodule is not in the wheel, so a conformance report emitted by an
89+
# installed copy has no other way to name the revision it ran against.
90+
"src/openfeature/contrib/tools/tck/spec_revision.json",
7291
]
7392

7493
[tool.hatch.build.hooks.custom]
@@ -92,6 +111,21 @@ fixed_format_cache = true
92111
pretty = true
93112
strict = true
94113
disallow_any_generics = false
114+
# cucumber-messages and gherkin-official ship no py.typed. Both are annotated
115+
# internally, so following them gives real types for the messages this package
116+
# builds rather than the Any a plain `ignore_missing_imports` would hand back --
117+
# which is the point of using the typed library at all.
118+
follow_untyped_imports = true
119+
120+
[[tool.mypy.overrides]]
121+
# gherkin-official has no annotations at all, so following it turns every call
122+
# into a `no-untyped-call` error rather than into a type. pytest-bdd silences
123+
# the same import the same way. cucumber-messages is the opposite case -- fully
124+
# annotated, only missing py.typed -- and is followed, which is where the value
125+
# of using it rather than hand-written dicts actually lands.
126+
module = ["gherkin.*"]
127+
follow_untyped_imports = false
128+
ignore_missing_imports = true
95129

96130
# testcontainers 4.14 ships no py.typed for `testcontainers.compose`, so the one
97131
# lazy import in compose.py cannot be checked against it. Scoped to that module

‎tools/openfeature-tck/src/openfeature/contrib/tools/tck/__init__.py‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,7 @@ def tck_config():
7676

7777
import importlib.resources
7878

79+
from .canonical import PARTIAL_ENV
7980
from .capability import (
8081
DECLARABLE_CAPABILITIES,
8182
INEXPRESSIBLE_CAPABILITIES,
@@ -112,12 +113,14 @@ def tck_config():
112113
HttpControl,
113114
)
114115
from .inprocess import InProcessControl
116+
from .messages import MESSAGES_FORMAT
115117
from .provider import (
116118
CHANGING_FLAG_KEY,
117119
ControllableInMemoryProvider,
118120
canonical_flag_set,
119121
canonical_flags_json,
120122
)
123+
from .report import REPORT_DIR_ENV, SCHEMA_VERSION
121124
from .state import TckState
122125

123126
__all__ = [
@@ -130,7 +133,11 @@ def tck_config():
130133
"DEFAULT_STARTUP_TIMEOUT",
131134
"EXTENSIONS_DIRECTORY",
132135
"INEXPRESSIBLE_CAPABILITIES",
136+
"MESSAGES_FORMAT",
137+
"PARTIAL_ENV",
138+
"REPORT_DIR_ENV",
133139
"RESERVED_CAPABILITIES",
140+
"SCHEMA_VERSION",
134141
"BackendControl",
135142
"BackendEndpoint",
136143
"BackendType",

0 commit comments

Comments
 (0)