feat: the flagd scenarios as Provider TCK extensions

.gherkin-lintrc

@@ -49,8 +49,8 @@
     "on",
     {
       "steps-length": {
-        "Background": 3,
-        "Scenario": 16
+        "Background": 4,
+        "Scenario": 17
       }
     }
   ]

tck-extensions/.gitattributes

@@ -0,0 +1,5 @@
+# Every language reads these files byte for byte, and the Go module zip is checksummed, so
+# normalise to LF whichever platform they are committed from.
+*.feature text eol=lf
+*.go      text eol=lf
+go.mod    text eol=lf

tck-extensions/README.md

@@ -0,0 +1,21 @@
+# TCK extensions
+
+The scenarios in `gherkin/`, rewritten as extensions for the
+[OpenFeature Provider TCK](https://github.com/open-feature/spec/issues/417). A flagd provider that
+adopts the TCK runs these alongside the TCK's own scenarios, instead of running `gherkin/` through a
+harness of its own.
+
+They ask only what the TCK's canonical scenarios do not. Each file's header names its source feature,
+the scenarios it leaves out, and why; most of those the TCK already asks.
+
+They are written in the TCK's step vocabulary (`a stable provider`, `a context with the following
+keys:`), plus flagd-specific steps an adopter supplies as extension glue: provider options
+(`an option "<name>" of type "<type>" with value "<value>"`) and connection control. Feature tags
+name the resolvers a scenario applies to (`@rpc`, `@in-process`, `@file`), as in `gherkin/`.
+
+Java, JavaScript, Python and .NET read this directory through the `test-harness` submodule. Go
+depends on it as a module, `github.com/open-feature/flagd-testbed/tck-extensions`, whose `FS`
+embeds the feature files, because a Go module zip carries a submodule without its files.
+
+`gherkin/` stays as it is for SDKs that still run it directly. A scenario added there should be
+added here too, or listed as not ported in the matching file's header.

tck-extensions/connection.feature

@@ -0,0 +1,118 @@
+@ext-connection
+Feature: TCK extension: flagd provider disconnect and reconnect
+
+  # Ported from gherkin/connection.feature.
+  #
+  # Not ported: the "stable" row of the Connection outline, which every canonical evaluation
+  # scenario asks, and the @unixsocket rows, which need a socket shared between host and container
+  # that the TCK stack does not mount. The testbed's "Provider unavailable" is kept although the
+  # canonical lifecycle.feature asks the same, because it holds flagd to a 3000ms bound where the
+  # canonical scenario allows 10000ms.
+  #
+  # The outlines over provider flavours are spelled out, one scenario per flavour, because a flavour
+  # here is a set of steps rather than a value. Reconnect timing is the testbed's "stable flagd
+  # provider", as in events.feature.
+  #
+  # @targetURI and @forbidden connect through the envoy sidecar in the testbed's docker-compose.yaml.
+  # certPath is relative to the testbed root; an adopter resolves it against its own checkout.
+
+  @rpc @in-process @file @customCert
+  Scenario: Connection with a custom certificate
+    Given an option "tls" of type "Boolean" with value "true"
+    And an option "certPath" of type "String" with value "ssl/custom-root-cert.crt"
+    And a stable provider with "ssl" configuration
+    And a Boolean-flag with key "boolean-flag" and a default value "false"
+    When the flag was evaluated with details
+    Then the resolved details value should be "true"
+
+  @rpc @in-process @file @reconnect
+  Scenario: Provider reconnection
+    Given an option "retryGracePeriod" of type "Integer" with value "5"
+    And an option "retryBackoffMaxMs" of type "Integer" with value "500"
+    And a stable provider
+    And a ready event handler
+    And a error event handler
+    When a ready event was fired
+    When the connection is lost
+    Then the error event handler should have been executed
+    When the connection is restored
+    Then the ready event handler should have been executed
+
+  @rpc @in-process @file @reconnect @customCert
+  Scenario: Provider reconnection with a custom certificate
+    Given an option "tls" of type "Boolean" with value "true"
+    And an option "certPath" of type "String" with value "ssl/custom-root-cert.crt"
+    And an option "retryGracePeriod" of type "Integer" with value "5"
+    And an option "retryBackoffMaxMs" of type "Integer" with value "500"
+    And a stable provider with "ssl" configuration
+    And a ready event handler
+    And a error event handler
+    When a ready event was fired
+    When the connection is lost
+    Then the error event handler should have been executed
+    When the connection is restored
+    Then the ready event handler should have been executed
+
+  @rpc @in-process @file @unavailable
+  Scenario: Provider unavailable
+    Given an option "deadlineMs" of type "Integer" with value "1000"
+    And a unavailable provider
+    And a error event handler
+    Then the error event handler should have been executed within 3000ms
+
+  # A forbidden response from flagd results in a fatal client state. "a unavailable provider" is the
+  # TCK's flavour that does not wait for readiness; the step before it points that provider at envoy.
+  @rpc @in-process @forbidden
+  Scenario: Provider forbidden
+    Given an option "fatalStatusCodes" of type "StringList" with value "PERMISSION_DENIED"
+    And the provider connects to envoy's forbidden listener
+    And a unavailable provider
+    And a error event handler
+    Then the error event handler should have been executed within 5000ms
+    And the client should be in fatal state
+
+  @targetURI @rpc
+  Scenario: Connection via TargetUri rpc
+    Given a target URI "envoy://localhost:<port>/rpc.service" routed through envoy
+    And a stable provider
+    And a Boolean-flag with key "boolean-flag" and a default value "false"
+    When the flag was evaluated with details
+    Then the resolved details value should be "true"
+
+  @targetURI @in-process
+  Scenario: Connection via TargetUri in-process
+    Given a target URI "envoy://localhost:<port>/sync.service" routed through envoy
+    And a stable provider
+    And a Boolean-flag with key "boolean-flag" and a default value "false"
+    When the flag was evaluated with details
+    Then the resolved details value should be "true"
+
+  @targetURI @in-process @reconnect
+  Scenario: Re-Connection via TargetUri in-process
+    Given a target URI "envoy://localhost:<port>/sync.service" routed through envoy
+    And an option "retryGracePeriod" of type "Integer" with value "5"
+    And an option "retryBackoffMaxMs" of type "Integer" with value "500"
+    And a stable provider
+    And a ready event handler
+    And a error event handler
+    When a ready event was fired
+    When the connection is lost
+    Then the error event handler should have been executed
+    When the connection is restored
+    Then the ready event handler should have been executed
+
+  @targetURI @in-process @reconnect @grace
+  Scenario: Re-Connection via TargetUri in-process with grace period
+    Given a target URI "envoy://localhost:<port>/sync.service" routed through envoy
+    And an option "retryGracePeriod" of type "Integer" with value "5"
+    And an option "retryBackoffMaxMs" of type "Integer" with value "500"
+    And a stable provider
+    And a ready event handler
+    And a stale event handler
+    And a error event handler
+    When a ready event was fired
+    When the connection is lost
+    Then the stale event handler should have been executed
+    Then the error event handler should have been executed
+    When the connection is restored
+    Then the ready event handler should have been executed

tck-extensions/context-enrichment.feature

@@ -0,0 +1,72 @@
+@ext-metadata @contextEnrichment
+Feature: TCK extension: Context Enrichment
+
+  # Ported from gherkin/contextEnrichment.feature. The testbed's Background "a stable
+  # flagd provider" moves into each scenario as the TCK's "a stable provider", because the @grace
+  # scenarios record options that have to come before it.
+  #
+  # The @grace scenarios set the testbed's retry timing (grace period 5s, backoff 500ms capped at
+  # 500ms) whatever the adoption's defaults, so the provider is back within the event timeout once
+  # the connection is restored. The RPC one also sets a 1000ms deadline, so an evaluation issued
+  # while flagd is down fails rather than outlasting the outage.
+  #
+  # The testbed's @caching is renamed @flagd-caching: the TCK reserves @caching and refuses a
+  # scenario that carries it.
+
+  @in-process @rpc
+  Scenario: Use enriched context
+    Given a stable provider
+    And a String-flag with key "flagd-context-aware" and a default value "not"
+    When the flag was evaluated with details
+    Then the resolved details value should be "INTERNAL"
+
+  @grace @in-process
+  Scenario: Use enriched context on connection error for IN-PROCESS
+    Given an option "retryGracePeriod" of type "Integer" with value "5"
+    And an option "retryBackoffMs" of type "Integer" with value "500"
+    And an option "retryBackoffMaxMs" of type "Integer" with value "500"
+    And a stable provider
+    And a String-flag with key "flagd-context-aware" and a default value "not"
+    And a stale event handler
+    And a ready event handler
+    When the flag was evaluated with details
+    Then the resolved details value should be "INTERNAL"
+    When the connection is lost
+    And a stale event was fired
+    When the flag was evaluated with details
+    Then the resolved details value should be "INTERNAL"
+    When the connection is restored
+    And a ready event was fired
+
+  @grace @rpc
+  Scenario: Use enriched context on connection error for RPC
+    Given an option "deadlineMs" of type "Integer" with value "1000"
+    And an option "retryGracePeriod" of type "Integer" with value "5"
+    And an option "retryBackoffMs" of type "Integer" with value "500"
+    And an option "retryBackoffMaxMs" of type "Integer" with value "500"
+    And a stable provider
+    And a String-flag with key "flagd-context-aware" and a default value "not"
+    And a stale event handler
+    And a ready event handler
+    When the flag was evaluated with details
+    Then the resolved details value should be "INTERNAL"
+    When the connection is lost
+    And a stale event was fired
+    When the flag was evaluated with details
+    Then the resolved details value should be "not"
+    When the connection is restored
+    And a ready event was fired
+
+  @rpc @flagd-caching
+  Scenario: Use enriched context on RPC connection will not cache the value
+    Given a stable provider
+    And a String-flag with key "flagd-context-aware" and a default value "not"
+    And a change event handler
+    And a ready event handler
+    When the flag was modified
+    And a change event was fired
+    And the flag was evaluated with details
+    Then the reason should be "TARGETING_MATCH"
+    # ensure that we do not cache a "TARGETING_MATCH", we should only cache evaluation with a "STATIC" reason
+    When the flag was evaluated with details
+    Then the reason should be "TARGETING_MATCH"

tck-extensions/disabled.feature

@@ -0,0 +1,39 @@
+@ext-evaluation @disabled @in-process @rpc
+Feature: TCK extension: Disabled flag evaluation
+
+  # Ported from gherkin/disabled.feature. A flag with state=DISABLED resolves
+  # successfully with reason=DISABLED; the evaluator substitutes the caller-provided default value
+  # and omits the variant. Relates to: https://github.com/open-feature/flagd/issues/1965
+  #
+  # The TCK's @disabled-flags scenarios ask the value for the four scalar flags and the reason for
+  # disabled-boolean-flag only, so the boolean row is left out here: the remaining rows add the
+  # DISABLED reason for each type and the structured flag.
+
+  Scenario Outline: Evaluating a disabled flag returns reason DISABLED and code default
+    Given an option "cache" of type "CacheType" with value "disabled"
+    And a stable provider
+    And a <type>-flag with key "<key>" and a default value "<default>"
+    When the flag was evaluated with details
+    Then the resolved details value should be "<default>"
+    And the reason should be "DISABLED"
+
+    Examples:
+      | key                   | type    | default |
+      | disabled-string-flag  | String  | bye     |
+      | disabled-integer-flag | Integer | 1       |
+      | disabled-float-flag   | Float   | 0.1     |
+      | disabled-object-flag  | Object  | {}      |
+
+  Scenario Outline: Flag disabled in one flag set, enabled in another
+    Given an option "cache" of type "CacheType" with value "disabled"
+    And an option "selector" of type "String" with value "<selector>"
+    And a stable provider
+    And a Boolean-flag with key "cross-flagset-flag" and a default value "false"
+    When the flag was evaluated with details
+    Then the resolved details value should be "<resolved_value>"
+    And the reason should be "<reason>"
+
+    Examples:
+      | selector                     | resolved_value | reason   |
+      | flags/allFlags.json          | false          | DISABLED |
+      | rawflags/selector-flags.json | true           | STATIC   |

tck-extensions/embed.go

@@ -0,0 +1,15 @@
+// Package tckextensions carries the flagd Provider TCK extension scenarios as a
+// Go module, so a Go flagd adoption can pin a revision of them in go.mod.
+//
+// Other languages read this directory through a git submodule. A Go module zip
+// carries a submodule only as a gitlink, without its files, so Go gets the same
+// bytes through this module instead. It holds no code beyond this file and has
+// no dependencies. See https://github.com/open-feature/spec/issues/417.
+package tckextensions
+
+import "embed"
+
+// FS holds the extension feature files, keyed by file name.
+//
+//go:embed *.feature
+var FS embed.FS

tck-extensions/evaluation.feature

@@ -0,0 +1,76 @@
+@ext-evaluation @rpc @in-process @file
+Feature: TCK extension: flagd evaluations
+
+  # Ported from gherkin/evaluation.feature. Asks what the TCK's evaluation.feature
+  # and reason.feature do not: the reason flagd reports for a zero value, a float zero value,
+  # zero values served by a targeting rule, and flags with no default variant.
+  #
+  # Not ported: "Resolve values" (the TCK's evaluation.feature asks it), the @deprecated
+  # @no-default outline, which @no-default-variant replaces, and the two @precision scenarios,
+  # "Resolves numeric values without loss of precision" and "Resolves an integer beyond 32 bits
+  # without loss of precision", which the TCK's own precision scenarios ask.
+
+  Background:
+    Given an option "cache" of type "CacheType" with value "disabled"
+    And a stable provider
+
+  Scenario Outline: Resolves zero value
+    Given a <type>-flag with key "<key>" and a default value "<default>"
+    When the flag was evaluated with details
+    Then the resolved details value should be "<resolved_value>"
+    And the reason should be "STATIC"
+
+    Examples:
+      | key               | type    | default | resolved_value |
+      | boolean-zero-flag | Boolean | true    | false          |
+      | string-zero-flag  | String  | hi      |                |
+      | integer-zero-flag | Integer | 1       | 0              |
+      | float-zero-flag   | Float   | 0.1     | 0.0            |
+
+  @targeting
+  Scenario Outline: Resolves zero value with targeting
+    Given a <type>-flag with key "<key>" and a default value "<default>"
+    And a context containing a key "email", with type "String" and with value "ballmer@macrosoft.com"
+    When the flag was evaluated with details
+    Then the resolved details value should be "<resolved_value>"
+    And the reason should be "TARGETING_MATCH"
+
+    Examples:
+      | key                        | type    | default | resolved_value |
+      | boolean-targeted-zero-flag | Boolean | true    | false          |
+      | string-targeted-zero-flag  | String  | hi      |                |
+      | integer-targeted-zero-flag | Integer | 1       | 0              |
+      | float-targeted-zero-flag   | Float   | 0.1     | 0.0            |
+
+  @targeting
+  Scenario Outline: Resolves zero value with targeting using default
+    Given a <type>-flag with key "<key>" and a default value "<default>"
+    And a context containing a key "email", with type "String" and with value "ballmer@none.com"
+    When the flag was evaluated with details
+    Then the resolved details value should be "<resolved_value>"
+    And the reason should be "DEFAULT"
+
+    Examples:
+      | key                        | type    | default | resolved_value |
+      | boolean-targeted-zero-flag | Boolean | true    | false          |
+      | string-targeted-zero-flag  | String  | hi      |                |
+      | integer-targeted-zero-flag | Integer | 1       | 0              |
+      | float-targeted-zero-flag   | Float   | 0.1     | 0.0            |
+
+  @no-default-variant
+  Scenario Outline: Resolves flag with no default variant correctly
+    Given a <type>-flag with key "<key>" and a default value "<code_default>"
+    And a context containing a key "email", with type "String" and with value "<email>"
+    When the flag was evaluated with details
+    Then the resolved details value should be "<resolved_value>"
+    And the reason should be "<reason>"
+
+    Examples:
+      | key                                         | type    | email              | code_default  | resolved_value | reason          |
+      | null-default-flag                           | Boolean |                    | true          | true           | DEFAULT         |
+      | null-default-flag                           | Boolean |                    | false         | false          | DEFAULT         |
+      | undefined-default-flag                      | Integer |                    | 100           | 100            | DEFAULT         |
+      | no-default-flag-null-targeting-variant      | String  | wozniak@orange.com | Inventor      | Inventor       | DEFAULT         |
+      | no-default-flag-null-targeting-variant      | String  | wozniak@orange.com | Founder       | Founder        | DEFAULT         |
+      | no-default-flag-null-targeting-variant      | String  | jobs@orange.com    | CEO           | CEO            | TARGETING_MATCH |
+      | no-default-flag-undefined-targeting-variant | String  | wozniak@orange.com | Retired       | Retired        | DEFAULT         |