Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
40ae237
feat: add a schema for machine-readable provider conformance reports
aepfli Aug 24, 2026
81de892
feat: identify executed assets by git tree hash rather than a content…
aepfli Aug 24, 2026
a5282b3
feat: enforce reasons, and let a single scenario carry a known deviation
aepfli Aug 24, 2026
3e76494
feat: identify a Scenario Outline row by its parameters
aepfli Aug 24, 2026
86f74f8
docs: state that example key order is not significant
aepfli Aug 24, 2026
6724bd3
feat: carry results in Cucumber Messages instead of defining our own
aepfli Sep 10, 2026
693c7e4
feat: record which version of the results format was produced
aepfli Sep 10, 2026
0271ff9
docs: correct what specRevision alone has to carry, and how to read a…
aepfli Sep 10, 2026
ae68b55
feat: a capability the suite does not test must not be declared
aepfli Sep 10, 2026
20f09da
docs: rename the capability in the notApplicable illustration
aepfli Sep 11, 2026
68be0f9
feat: drop declaration.notApplicable, one skip carries its own reason
aepfli Sep 11, 2026
57d84ad
docs: say what the conformance report is, and what provenance is for
aepfli Sep 11, 2026
527794f
feat: require a deviation's summary, and stop implying its issue is m…
aepfli Sep 12, 2026
99e9adb
feat: require backend.controlApi, which decides what the rest of the …
aepfli Sep 12, 2026
a63d193
fix: the declaration no longer explains every skip, and said it did
aepfli Sep 13, 2026
9b7b7cd
feat!: name the report's control path backendType, and admit custom
aepfli Nov 2, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 55 additions & 0 deletions specification/appendix-f-provider-conformance.md
Original file line number Diff line number Diff line change
Expand Up @@ -643,6 +643,61 @@ provider on a 32-bit accessor leaves undeclared. Nothing above 2^53 − 1 is ask
cannot represent it, and what a provider owes a value that does not fit the requested accessor is
the open question in [open-feature/spec#430](https://github.com/open-feature/spec/issues/430).

## The conformance report

A run may emit a **conformance report**: a machine-readable document saying what was tested, what the
provider claimed, and where the results are. Its shape is fixed by
[`assets/provider-tck/report/conformance-report.schema.json`](./assets/provider-tck/report/conformance-report.schema.json),
and it is deliberately a transcription of that schema in every language rather than whatever each
language finds convenient, because the point of the format is that four implementations emit the
same thing.

Emitting one is optional. A developer running the suite locally wants a pass or a failure, not a
document; CI publishing a claim wants the document. Nothing about a provider's conformance depends
on whether a report was written.

The report is an **envelope**. It identifies the run and points at the results; it does not contain
them. The results are Cucumber Messages written alongside it, because per-scenario outcomes, tags,
Scenario Outline row identity and the executed feature source are all already specified there.
Restating them in a second format would create two places for the same fact to disagree.

Two fields carry the interpretation the results cannot:

- **`declaration.declared`** is every capability the configuration claims, as tags. It is an input to
reading the results rather than a summary of them: a skipped scenario says only that the question
was not put to this provider, and it is the declaration that says whether that is because the
provider declines the capability. Given the declaration and a scenario's tags, the reason for a
skip follows without being transported per scenario.
- **`knownDeviations`** are the gaps the provider author acknowledges. It sits beside the declaration
rather than inside it because it is not a claim about capabilities: an entry may concern a declared
capability, or a mandatory scenario belonging to no capability at all. What it qualifies is which
of the declaration's absences were decisions and which were defects — a distinction the results
cannot carry, because a skip looks the same either way. Omitted when empty, which is silence rather
than a claim of having no known gaps.

### Provenance

`provenance` is optional and records where a report came from: `runUrl`, `commit`, `timestamp` and an
optional signed `attestation`.

It exists because **a report is a claim, not an audit**. Everything else in the document is what the
provider says about itself, and a consumer comparing two providers' reports — or deciding whether to
publish a badge from one — has no way to tell a report produced by a CI run against a tagged commit
from one produced by hand on a laptop. `provenance` is what lets that consumer decide how much to
trust the rest.

It is optional rather than required because the facts in it are properties of the environment that
ran the suite, not of the provider or the TCK. A TCK implementation cannot invent them: outside CI
there is no run URL, and a working tree with uncommitted changes has no commit that describes what
was actually executed. A report that filled the field in anyway would be worse than one that omits
it, because the whole value of the field is that its contents can be checked.

**No TCK implementation is required to emit it, and at the time of writing none does.** It is
specified here so that the first one to need it does not invent a second shape for the same facts,
and so that a reader who finds the field absent knows that means "not stated" rather than "this
implementation calls it something else". Where a suite runs in CI and the information is available,
populating it is encouraged; a consumer must treat its absence as unremarkable.

## Implementing the suite in a language

A TCK implementation is the language-specific harness around these three artifacts. Each obligation
Expand Down
5 changes: 4 additions & 1 deletion specification/assets/provider-tck/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ These validate a **provider** against a real backend. For assets that validate a
| [`gherkin/reason.feature`](./gherkin/reason.feature) | the standard resolution reasons, gated behind `@standard-reasons` |
| [`flags/canonical-flags.json`](./flags/canonical-flags.json) | the flag set every scenario assumes |
| [`openapi/control-api.yaml`](./openapi/control-api.yaml) | the HTTP surface a backend under test must expose |
| [`report/conformance-report.schema.json`](./report/conformance-report.schema.json) | the shape of a machine-readable conformance report |

## Capabilities

Expand All @@ -26,10 +27,12 @@ asserts are in
[Appendix F](../../appendix-f-provider-conformance.md#capabilities-how-a-provider-says-what-it-cannot-do),
which owns them; repeating the table here would give it two homes that can disagree.

## These three travel together
## The three inputs travel together

A feature file that evaluates `boolean-flag` is meaningless without the flag definition, and a disconnect scenario is meaningless without the control endpoint that produces the disconnect. Changing one without the others breaks the suite in every language at once.

The report schema is not one of the three: it describes what a run *emits*, not what a run needs, and a suite that never writes a report is no less conformant for it.

## Five properties that are load-bearing

- **`missing-flag` must not exist** in the flag set. Its absence is what the `FLAG_NOT_FOUND` scenario tests. Seeding it turns that scenario green for the wrong reason.
Expand Down
Loading