diff --git a/.gherkin-lintrc b/.gherkin-lintrc index 36b2d0c..5679dc5 100644 --- a/.gherkin-lintrc +++ b/.gherkin-lintrc @@ -49,8 +49,8 @@ "on", { "steps-length": { - "Background": 3, - "Scenario": 16 + "Background": 4, + "Scenario": 17 } } ] diff --git a/tck-extensions/.gitattributes b/tck-extensions/.gitattributes new file mode 100644 index 0000000..8bb653e --- /dev/null +++ b/tck-extensions/.gitattributes @@ -0,0 +1,5 @@ +# Every language reads these files byte for byte, and the Go module zip is checksummed, so +# normalise to LF whichever platform they are committed from. +*.feature text eol=lf +*.go text eol=lf +go.mod text eol=lf diff --git a/tck-extensions/README.md b/tck-extensions/README.md new file mode 100644 index 0000000..7c1cd0f --- /dev/null +++ b/tck-extensions/README.md @@ -0,0 +1,21 @@ +# TCK extensions + +The scenarios in `gherkin/`, rewritten as extensions for the +[OpenFeature Provider TCK](https://github.com/open-feature/spec/issues/417). A flagd provider that +adopts the TCK runs these alongside the TCK's own scenarios, instead of running `gherkin/` through a +harness of its own. + +They ask only what the TCK's canonical scenarios do not. Each file's header names its source feature, +the scenarios it leaves out, and why; most of those the TCK already asks. + +They are written in the TCK's step vocabulary (`a stable provider`, `a context with the following +keys:`), plus flagd-specific steps an adopter supplies as extension glue: provider options +(`an option "" of type "" with value ""`) and connection control. Feature tags +name the resolvers a scenario applies to (`@rpc`, `@in-process`, `@file`), as in `gherkin/`. + +Java, JavaScript, Python and .NET read this directory through the `test-harness` submodule. Go +depends on it as a module, `github.com/open-feature/flagd-testbed/tck-extensions`, whose `FS` +embeds the feature files, because a Go module zip carries a submodule without its files. + +`gherkin/` stays as it is for SDKs that still run it directly. A scenario added there should be +added here too, or listed as not ported in the matching file's header. diff --git a/tck-extensions/connection.feature b/tck-extensions/connection.feature new file mode 100644 index 0000000..60ea89d --- /dev/null +++ b/tck-extensions/connection.feature @@ -0,0 +1,118 @@ +@ext-connection +Feature: TCK extension: flagd provider disconnect and reconnect + + # Ported from gherkin/connection.feature. + # + # Not ported: the "stable" row of the Connection outline, which every canonical evaluation + # scenario asks, and the @unixsocket rows, which need a socket shared between host and container + # that the TCK stack does not mount. The testbed's "Provider unavailable" is kept although the + # canonical lifecycle.feature asks the same, because it holds flagd to a 3000ms bound where the + # canonical scenario allows 10000ms. + # + # The outlines over provider flavours are spelled out, one scenario per flavour, because a flavour + # here is a set of steps rather than a value. Reconnect timing is the testbed's "stable flagd + # provider", as in events.feature. + # + # @targetURI and @forbidden connect through the envoy sidecar in the testbed's docker-compose.yaml. + # certPath is relative to the testbed root; an adopter resolves it against its own checkout. + + @rpc @in-process @file @customCert + Scenario: Connection with a custom certificate + Given an option "tls" of type "Boolean" with value "true" + And an option "certPath" of type "String" with value "ssl/custom-root-cert.crt" + And a stable provider with "ssl" configuration + And a Boolean-flag with key "boolean-flag" and a default value "false" + When the flag was evaluated with details + Then the resolved details value should be "true" + + @rpc @in-process @file @reconnect + Scenario: Provider reconnection + Given an option "retryGracePeriod" of type "Integer" with value "5" + And an option "retryBackoffMaxMs" of type "Integer" with value "500" + And a stable provider + And a ready event handler + And a error event handler + When a ready event was fired + When the connection is lost + Then the error event handler should have been executed + When the connection is restored + Then the ready event handler should have been executed + + @rpc @in-process @file @reconnect @customCert + Scenario: Provider reconnection with a custom certificate + Given an option "tls" of type "Boolean" with value "true" + And an option "certPath" of type "String" with value "ssl/custom-root-cert.crt" + And an option "retryGracePeriod" of type "Integer" with value "5" + And an option "retryBackoffMaxMs" of type "Integer" with value "500" + And a stable provider with "ssl" configuration + And a ready event handler + And a error event handler + When a ready event was fired + When the connection is lost + Then the error event handler should have been executed + When the connection is restored + Then the ready event handler should have been executed + + @rpc @in-process @file @unavailable + Scenario: Provider unavailable + Given an option "deadlineMs" of type "Integer" with value "1000" + And a unavailable provider + And a error event handler + Then the error event handler should have been executed within 3000ms + + # A forbidden response from flagd results in a fatal client state. "a unavailable provider" is the + # TCK's flavour that does not wait for readiness; the step before it points that provider at envoy. + @rpc @in-process @forbidden + Scenario: Provider forbidden + Given an option "fatalStatusCodes" of type "StringList" with value "PERMISSION_DENIED" + And the provider connects to envoy's forbidden listener + And a unavailable provider + And a error event handler + Then the error event handler should have been executed within 5000ms + And the client should be in fatal state + + @targetURI @rpc + Scenario: Connection via TargetUri rpc + Given a target URI "envoy://localhost:/rpc.service" routed through envoy + And a stable provider + And a Boolean-flag with key "boolean-flag" and a default value "false" + When the flag was evaluated with details + Then the resolved details value should be "true" + + @targetURI @in-process + Scenario: Connection via TargetUri in-process + Given a target URI "envoy://localhost:/sync.service" routed through envoy + And a stable provider + And a Boolean-flag with key "boolean-flag" and a default value "false" + When the flag was evaluated with details + Then the resolved details value should be "true" + + @targetURI @in-process @reconnect + Scenario: Re-Connection via TargetUri in-process + Given a target URI "envoy://localhost:/sync.service" routed through envoy + And an option "retryGracePeriod" of type "Integer" with value "5" + And an option "retryBackoffMaxMs" of type "Integer" with value "500" + And a stable provider + And a ready event handler + And a error event handler + When a ready event was fired + When the connection is lost + Then the error event handler should have been executed + When the connection is restored + Then the ready event handler should have been executed + + @targetURI @in-process @reconnect @grace + Scenario: Re-Connection via TargetUri in-process with grace period + Given a target URI "envoy://localhost:/sync.service" routed through envoy + And an option "retryGracePeriod" of type "Integer" with value "5" + And an option "retryBackoffMaxMs" of type "Integer" with value "500" + And a stable provider + And a ready event handler + And a stale event handler + And a error event handler + When a ready event was fired + When the connection is lost + Then the stale event handler should have been executed + Then the error event handler should have been executed + When the connection is restored + Then the ready event handler should have been executed diff --git a/tck-extensions/context-enrichment.feature b/tck-extensions/context-enrichment.feature new file mode 100644 index 0000000..eb506b4 --- /dev/null +++ b/tck-extensions/context-enrichment.feature @@ -0,0 +1,72 @@ +@ext-metadata @contextEnrichment +Feature: TCK extension: Context Enrichment + + # Ported from gherkin/contextEnrichment.feature. The testbed's Background "a stable + # flagd provider" moves into each scenario as the TCK's "a stable provider", because the @grace + # scenarios record options that have to come before it. + # + # The @grace scenarios set the testbed's retry timing (grace period 5s, backoff 500ms capped at + # 500ms) whatever the adoption's defaults, so the provider is back within the event timeout once + # the connection is restored. The RPC one also sets a 1000ms deadline, so an evaluation issued + # while flagd is down fails rather than outlasting the outage. + # + # The testbed's @caching is renamed @flagd-caching: the TCK reserves @caching and refuses a + # scenario that carries it. + + @in-process @rpc + Scenario: Use enriched context + Given a stable provider + And a String-flag with key "flagd-context-aware" and a default value "not" + When the flag was evaluated with details + Then the resolved details value should be "INTERNAL" + + @grace @in-process + Scenario: Use enriched context on connection error for IN-PROCESS + Given an option "retryGracePeriod" of type "Integer" with value "5" + And an option "retryBackoffMs" of type "Integer" with value "500" + And an option "retryBackoffMaxMs" of type "Integer" with value "500" + And a stable provider + And a String-flag with key "flagd-context-aware" and a default value "not" + And a stale event handler + And a ready event handler + When the flag was evaluated with details + Then the resolved details value should be "INTERNAL" + When the connection is lost + And a stale event was fired + When the flag was evaluated with details + Then the resolved details value should be "INTERNAL" + When the connection is restored + And a ready event was fired + + @grace @rpc + Scenario: Use enriched context on connection error for RPC + Given an option "deadlineMs" of type "Integer" with value "1000" + And an option "retryGracePeriod" of type "Integer" with value "5" + And an option "retryBackoffMs" of type "Integer" with value "500" + And an option "retryBackoffMaxMs" of type "Integer" with value "500" + And a stable provider + And a String-flag with key "flagd-context-aware" and a default value "not" + And a stale event handler + And a ready event handler + When the flag was evaluated with details + Then the resolved details value should be "INTERNAL" + When the connection is lost + And a stale event was fired + When the flag was evaluated with details + Then the resolved details value should be "not" + When the connection is restored + And a ready event was fired + + @rpc @flagd-caching + Scenario: Use enriched context on RPC connection will not cache the value + Given a stable provider + And a String-flag with key "flagd-context-aware" and a default value "not" + And a change event handler + And a ready event handler + When the flag was modified + And a change event was fired + And the flag was evaluated with details + Then the reason should be "TARGETING_MATCH" + # ensure that we do not cache a "TARGETING_MATCH", we should only cache evaluation with a "STATIC" reason + When the flag was evaluated with details + Then the reason should be "TARGETING_MATCH" diff --git a/tck-extensions/disabled.feature b/tck-extensions/disabled.feature new file mode 100644 index 0000000..71ba385 --- /dev/null +++ b/tck-extensions/disabled.feature @@ -0,0 +1,39 @@ +@ext-evaluation @disabled @in-process @rpc +Feature: TCK extension: Disabled flag evaluation + + # Ported from gherkin/disabled.feature. A flag with state=DISABLED resolves + # successfully with reason=DISABLED; the evaluator substitutes the caller-provided default value + # and omits the variant. Relates to: https://github.com/open-feature/flagd/issues/1965 + # + # The TCK's @disabled-flags scenarios ask the value for the four scalar flags and the reason for + # disabled-boolean-flag only, so the boolean row is left out here: the remaining rows add the + # DISABLED reason for each type and the structured flag. + + Scenario Outline: Evaluating a disabled flag returns reason DISABLED and code default + Given an option "cache" of type "CacheType" with value "disabled" + And a stable provider + And a -flag with key "" and a default value "" + When the flag was evaluated with details + Then the resolved details value should be "" + And the reason should be "DISABLED" + + Examples: + | key | type | default | + | disabled-string-flag | String | bye | + | disabled-integer-flag | Integer | 1 | + | disabled-float-flag | Float | 0.1 | + | disabled-object-flag | Object | {} | + + Scenario Outline: Flag disabled in one flag set, enabled in another + Given an option "cache" of type "CacheType" with value "disabled" + And an option "selector" of type "String" with value "" + And a stable provider + And a Boolean-flag with key "cross-flagset-flag" and a default value "false" + When the flag was evaluated with details + Then the resolved details value should be "" + And the reason should be "" + + Examples: + | selector | resolved_value | reason | + | flags/allFlags.json | false | DISABLED | + | rawflags/selector-flags.json | true | STATIC | diff --git a/tck-extensions/embed.go b/tck-extensions/embed.go new file mode 100644 index 0000000..3d765c7 --- /dev/null +++ b/tck-extensions/embed.go @@ -0,0 +1,15 @@ +// Package tckextensions carries the flagd Provider TCK extension scenarios as a +// Go module, so a Go flagd adoption can pin a revision of them in go.mod. +// +// Other languages read this directory through a git submodule. A Go module zip +// carries a submodule only as a gitlink, without its files, so Go gets the same +// bytes through this module instead. It holds no code beyond this file and has +// no dependencies. See https://github.com/open-feature/spec/issues/417. +package tckextensions + +import "embed" + +// FS holds the extension feature files, keyed by file name. +// +//go:embed *.feature +var FS embed.FS diff --git a/tck-extensions/evaluation.feature b/tck-extensions/evaluation.feature new file mode 100644 index 0000000..690bee8 --- /dev/null +++ b/tck-extensions/evaluation.feature @@ -0,0 +1,76 @@ +@ext-evaluation @rpc @in-process @file +Feature: TCK extension: flagd evaluations + + # Ported from gherkin/evaluation.feature. Asks what the TCK's evaluation.feature + # and reason.feature do not: the reason flagd reports for a zero value, a float zero value, + # zero values served by a targeting rule, and flags with no default variant. + # + # Not ported: "Resolve values" (the TCK's evaluation.feature asks it), the @deprecated + # @no-default outline, which @no-default-variant replaces, and the two @precision scenarios, + # "Resolves numeric values without loss of precision" and "Resolves an integer beyond 32 bits + # without loss of precision", which the TCK's own precision scenarios ask. + + Background: + Given an option "cache" of type "CacheType" with value "disabled" + And a stable provider + + Scenario Outline: Resolves zero value + Given a -flag with key "" and a default value "" + When the flag was evaluated with details + Then the resolved details value should be "" + And the reason should be "STATIC" + + Examples: + | key | type | default | resolved_value | + | boolean-zero-flag | Boolean | true | false | + | string-zero-flag | String | hi | | + | integer-zero-flag | Integer | 1 | 0 | + | float-zero-flag | Float | 0.1 | 0.0 | + + @targeting + Scenario Outline: Resolves zero value with targeting + Given a -flag with key "" and a default value "" + And a context containing a key "email", with type "String" and with value "ballmer@macrosoft.com" + When the flag was evaluated with details + Then the resolved details value should be "" + And the reason should be "TARGETING_MATCH" + + Examples: + | key | type | default | resolved_value | + | boolean-targeted-zero-flag | Boolean | true | false | + | string-targeted-zero-flag | String | hi | | + | integer-targeted-zero-flag | Integer | 1 | 0 | + | float-targeted-zero-flag | Float | 0.1 | 0.0 | + + @targeting + Scenario Outline: Resolves zero value with targeting using default + Given a -flag with key "" and a default value "" + And a context containing a key "email", with type "String" and with value "ballmer@none.com" + When the flag was evaluated with details + Then the resolved details value should be "" + And the reason should be "DEFAULT" + + Examples: + | key | type | default | resolved_value | + | boolean-targeted-zero-flag | Boolean | true | false | + | string-targeted-zero-flag | String | hi | | + | integer-targeted-zero-flag | Integer | 1 | 0 | + | float-targeted-zero-flag | Float | 0.1 | 0.0 | + + @no-default-variant + Scenario Outline: Resolves flag with no default variant correctly + Given a -flag with key "" and a default value "" + And a context containing a key "email", with type "String" and with value "" + When the flag was evaluated with details + Then the resolved details value should be "" + And the reason should be "" + + Examples: + | key | type | email | code_default | resolved_value | reason | + | null-default-flag | Boolean | | true | true | DEFAULT | + | null-default-flag | Boolean | | false | false | DEFAULT | + | undefined-default-flag | Integer | | 100 | 100 | DEFAULT | + | no-default-flag-null-targeting-variant | String | wozniak@orange.com | Inventor | Inventor | DEFAULT | + | no-default-flag-null-targeting-variant | String | wozniak@orange.com | Founder | Founder | DEFAULT | + | no-default-flag-null-targeting-variant | String | jobs@orange.com | CEO | CEO | TARGETING_MATCH | + | no-default-flag-undefined-targeting-variant | String | wozniak@orange.com | Retired | Retired | DEFAULT | diff --git a/tck-extensions/events.feature b/tck-extensions/events.feature new file mode 100644 index 0000000..4ac3ddf --- /dev/null +++ b/tck-extensions/events.feature @@ -0,0 +1,56 @@ +@ext-connection @rpc @in-process @file @events +Feature: TCK extension: flagd provider state changes + + # Ported from gherkin/events.feature. Not ported: "Flag change event", which the + # canonical events.feature asks in "A configuration change is signalled and applied", together + # with the changed evaluation it leaves out. + # + # These chains ask what the retry grace period decides, and they ask it by sequence rather than by + # clock: stale -> ready when the connection returns inside the grace period, stale -> error -> ready + # when the error assertion has first waited the grace period out. The testbed timed the outage + # instead (the launchpad's /restart), which made the same question depend on a 1s or 8s wall clock. + # + # Each "should have been executed" claims the record it matched and leaves the rest, so the next + # assertion waits for a newer event of its own kind while an event of another kind stays visible + # to whatever asks about it later. + # + # Timing is the testbed's "stable flagd provider" apart from the deadline, which the adoption + # chooses: a 5s grace period, and a retry backoff capped at 500ms so reconnecting does not wait + # out an exponential backoff. + + Background: + Given an option "retryGracePeriod" of type "Integer" with value "5" + And an option "retryBackoffMaxMs" of type "Integer" with value "500" + And a stable provider + + Scenario: Provider events chain ready -> error -> ready + Given a ready event handler + And a error event handler + Then the ready event handler should have been executed + When the connection is lost + Then the error event handler should have been executed + When the connection is restored + Then the ready event handler should have been executed + + @grace + Scenario: Provider events chain ready -> stale -> error -> ready + Given a ready event handler + And a error event handler + And a stale event handler + Then the ready event handler should have been executed + When the connection is lost + Then the stale event handler should have been executed + Then the error event handler should have been executed + When the connection is restored + Then the ready event handler should have been executed + + @grace + Scenario: Provider events chain ready -> stale -> ready + Given a ready event handler + And a error event handler + And a stale event handler + Then the ready event handler should have been executed + When the connection is lost + Then the stale event handler should have been executed + When the connection is restored + Then the ready event handler should have been executed with no error event before it diff --git a/tck-extensions/fractional.feature b/tck-extensions/fractional.feature new file mode 100644 index 0000000..264eb29 --- /dev/null +++ b/tck-extensions/fractional.feature @@ -0,0 +1,528 @@ +@ext-fractional @rpc @in-process @file +Feature: TCK extension: Fractional targeting + + # Ported from the gherkin/targeting.feature: every @fractional scenario, plus the + # fractional @operator-errors ones. The TCK's own @targeting scenarios do not cover the + # fractional operator. + # + # @fractional-vN tags select the bucketing version. The testbed's flagd implements v3, so an + # adopter whose evaluator does too excludes @fractional-v2. + # Untagged scenarios must hold for every version. + # The @fractional-v1 blocks are left out: no backend here implements v1 bucketing. + # + # The \u sequences in the UTF-8 outline are literal text, as Gherkin parses a table cell: + # the expected buckets hash the backslash and the hex digits, not the character they name. + # + # Where the testbed's context values are Objects or integers outside 32 bits, the TCK's context + # step cannot convert them, so those outlines use the table step with the testbed's conversion. + + Background: + Given an option "cache" of type "CacheType" with value "disabled" + And a stable provider + + @fractional + Scenario Outline: Fractional operator + Given a String-flag with key "fractional-flag" and a default value "fallback" + And a context containing a nested property with outer key "user" and inner key "name", with value "" + When the flag was evaluated with details + Then the resolved details value should be "" + + @fractional-v2 + Examples: v2 + | name | value | + | jack | hearts | + | queen | spades | + | ten | clubs | + | nine | diamonds | + | 3 | clubs | + + @fractional-v3 + Examples: v3 + | name | value | + | jack | diamonds | + | queen | diamonds | + | ten | clubs | + | nine | clubs | + | 3 | spades | + + @fractional + Scenario Outline: Fractional operator shorthand + Given a String-flag with key "fractional-flag-shorthand" and a default value "fallback" + And a context containing a targeting key with value "" + When the flag was evaluated with details + Then the resolved details value should be "" + + @fractional-v2 + Examples: v2 + | targeting key | value | + | jon@company.com | heads | + | jane@company.com | tails | + + @fractional-v3 + Examples: v3 + | targeting key | value | + | jon@company.com | tails | + | jane@company.com | tails | + + @fractional @fractional-v2 @fractional-single-entry + Scenario: Fractional operator with single entry always resolves to the only variant + Given a String-flag with key "fractional-single-entry-flag" and a default value "fallback" + And a context containing a targeting key with value "some-targeting-key" + When the flag was evaluated with details + Then the resolved details value should be "single" + + @fractional + Scenario Outline: Fractional operator with shared seed + Given a String-flag with key "fractional-flag-A-shared-seed" and a default value "fallback" + And a context containing a nested property with outer key "user" and inner key "name", with value "" + When the flag was evaluated with details + Then the resolved details value should be "" + + @fractional-v2 + Examples: v2 + | name | value | + | seven | hearts | + | eight | diamonds | + | nine | clubs | + | two | spades | + + @fractional-v3 + Examples: v3 + | name | value | + | seven | hearts | + | eight | hearts | + | nine | diamonds | + | two | diamonds | + + @fractional + Scenario Outline: Second fractional operator with shared seed + Given a String-flag with key "fractional-flag-B-shared-seed" and a default value "fallback" + And a context containing a nested property with outer key "user" and inner key "name", with value "" + When the flag was evaluated with details + Then the resolved details value should be "" + + @fractional-v2 + Examples: v2 + | name | value | + | seven | ace-of-hearts | + | eight | ace-of-diamonds | + | nine | ace-of-clubs | + | two | ace-of-spades | + + @fractional-v3 + Examples: v3 + | name | value | + | seven | ace-of-hearts | + | eight | ace-of-hearts | + | nine | ace-of-diamonds | + | two | ace-of-diamonds | + + # Hash edge-case vectors — keys chosen by brute-force search so their + # MurmurHash3-x86-32 (seed=0) falls at the six critical boundary values. + # All keys are exact 6-char MurmurHash3-x86-32 (seed=0) preimages found by exhaustive search. + # ejOoVL → hash=0 EXACT → bv(100)=0 → lower + # bY9fO- → hash=1 EXACT → bv(100)=0 → lower + # SI7p- → hash=2147483647 EXACT i32::MAX → bv(100)=49 → lower + # 6LvT0 → hash=2147483648 EXACT i32::MIN u32 → bv(100)=50 → upper + # ceQdGm → hash=4294967295 EXACT u32::MAX → bv(100)=99 → upper + @fractional @fractional-v2 + Scenario Outline: Fractional operator hash edge cases + Given a String-flag with key "fractional-hash-edge-flag" and a default value "fallback" + And a context containing a targeting key with value "" + When the flag was evaluated with details + Then the resolved details value should be "" + Examples: + | key | value | + | ejOoVL | lower | + | bY9fO- | lower | + | SI7p- | lower | + | 6LvT0 | upper | + | ceQdGm | upper | + + # Nested JSON Logic expressions as bucket variant names. + # Requires providers to support the @fractional-nested feature. + # Use -t "not @fractional-nested" to exclude during transition. + @fractional @fractional-nested + Scenario Outline: Fractional operator with nested if expression as variant name + # fractional-nested-if-flag: seed=targetingKey, bucket0=[if(tier=="premium","premium","standard"),50], bucket1=["standard",50] + # jon@company.com bv(100)=36 → bucket0; user1 bv(100)=76 → bucket1 + Given a String-flag with key "fractional-nested-if-flag" and a default value "fallback" + And a context containing a targeting key with value "" + And a context containing a key "tier", with type "String" and with value "" + When the flag was evaluated with details + Then the resolved details value should be "" + Examples: + | targetingKey | tier | value | + | jon@company.com | premium | premium | + | jon@company.com | basic | standard | + | user1 | premium | standard | + | user1 | basic | standard | + + @fractional @fractional-nested + Scenario Outline: Fractional operator with nested var expression as variant name + # fractional-nested-var-flag: seed=targetingKey, bucket0=[var("color"),50], bucket1=["blue",50] + # jon@company.com bv(100)=36 → bucket0 (resolves var "color"); user1 bv(100)=76 → bucket1 ("blue") + Given a String-flag with key "fractional-nested-var-flag" and a default value "fallback" + And a context containing a targeting key with value "" + And a context containing a key "color", with type "String" and with value "" + When the flag was evaluated with details + Then the resolved details value should be "" + Examples: + | targetingKey | color | value | + | jon@company.com | red | red | + | jon@company.com | green | green | + | user1 | red | blue | + | jon@company.com | yellow | fallback | + | jon@company.com | | fallback | + + @fractional @fractional-nested + Scenario Outline: Fractional operator with nested if expression as weight + # fractional-nested-weight-flag: seed=targetingKey, bucket0=["red",if(tier=="premium",100,0)], bucket1=["blue",10] + Given a String-flag with key "fractional-nested-weight-flag" and a default value "fallback" + And a context containing a targeting key with value "" + And a context containing a key "tier", with type "String" and with value "" + When the flag was evaluated with details + Then the resolved details value should be "" + Examples: + | targetingKey | tier | value | + | jon@company.com | premium | red | + | jon@company.com | basic | blue | + | user1 | premium | red | + | user1 | basic | blue | + + @fractional @fractional-nested + Scenario: Fractional as condition + Given a String-flag with key "fractional-as-condition-flag" and a default value "zero" + And a context containing a targeting key with value "some-targeting-key" + When the flag was evaluated with details + Then the resolved details value should be "hundreds" + + @fractional @fractional-nested + Scenario: Fractional as condition evaluates false path + Given a String-flag with key "fractional-as-condition-false-flag" and a default value "zero" + And a context containing a targeting key with value "some-targeting-key" + When the flag was evaluated with details + Then the resolved details value should be "ones" + + # fractional composes like any other operator + @fractional @fractional-nested @fractional-v3 + Scenario Outline: Fractional with null bucket as whole rule + Given a String-flag with key "fractional-null-bucket-flag" and a default value "code-default" + And a context containing a targeting key with value "" + When the flag was evaluated with details + Then the resolved details value should be "" + And the reason should be "" + And the error-code should be "" + Examples: + | targetingKey | value | reason | + | user2 | fallback | DEFAULT | + | user1 | a | TARGETING_MATCH | + + @fractional @fractional-nested @fractional-v3 + Scenario Outline: Fractional with null bucket as if condition + Given a String-flag with key "fractional-null-bucket-condition-flag" and a default value "code-default" + And a context containing a targeting key with value "" + When the flag was evaluated with details + Then the resolved details value should be "" + And the reason should be "TARGETING_MATCH" + Examples: + | targetingKey | value | + | user1 | else | + | user2 | then | + + @fractional @fractional-nested @fractional-v3 + Scenario Outline: Fractional with boolean buckets as whole rule + Given a String-flag with key "fractional-bool-bucket-flag" and a default value "code-default" + And a context containing a targeting key with value "" + When the flag was evaluated with details + Then the resolved details value should be "" + And the variant should be "" + And the reason should be "TARGETING_MATCH" + Examples: + | targetingKey | value | variant | + | user1 | on | true | + | user2 | off | false | + + @fractional @fractional-nested @fractional-v3 + Scenario Outline: Fractional result consumed by another operator + Given a String-flag with key "fractional-cat-flag" and a default value "code-default" + And a context containing a targeting key with value "" + When the flag was evaluated with details + Then the resolved details value should be "" + And the reason should be "TARGETING_MATCH" + Examples: + | targetingKey | value | + | user2 | big | + | user1 | dig | + + @fractional @fractional-nested @fractional-v3 + Scenario Outline: Fractional as bucketing input of another fractional + Given a String-flag with key "fractional-of-fractional-flag" and a default value "code-default" + And a context containing a targeting key with value "" + When the flag was evaluated with details + Then the resolved details value should be "" + And the reason should be "TARGETING_MATCH" + Examples: + | targetingKey | value | + | user2 | red | + | user3 | red | + | user1 | blue | + + @operator-errors + Scenario: fractional operator with missing bucket key falls back to default variant + Given a String-flag with key "fractional-null-bucket-key-flag" and a default value "wrong" + When the flag was evaluated with details + Then the resolved details value should be "fallback" + And the reason should be "DEFAULT" + + @operator-errors @fractional + Scenario: fractional with all-zero bucket weights falls back to default variant + Given a String-flag with key "fractional-zero-weights-flag" and a default value "wrong" + And a context containing a targeting key with value "any-user" + When the flag was evaluated with details + Then the resolved details value should be "fallback" + And the reason should be "DEFAULT" + + @operator-errors @fractional + Scenario: fractional negative bucket weight is clamped to zero + # ["one", -50] is treated as ["one", 0]; "two" gets 100% of the weight + Given a String-flag with key "fractional-negative-weight-flag" and a default value "wrong" + And a context containing a targeting key with value "any-user" + When the flag was evaluated with details + Then the resolved details value should be "two" + + @fractional @fractional-v3 + Scenario Outline: Fractional operator with basic types + Given a String-flag with key "fractional-basic-flag" and a default value "fallback" + And a context containing a key "hashing_input", with type "" and with value "" + When the flag was evaluated with details + Then the resolved details value should be "" + + Examples: v3 + | hashing_input | type | value | + | true | Boolean | bucket5 | + | false | Boolean | bucket1 | + | user1 | String | bucket22 | + | user2 | String | bucket16 | + | 123 | Integer | bucket20 | + | 456 | Integer | bucket12 | + | 1.23 | Float | bucket14 | + | 4.56 | Float | bucket18 | + | 123 | String | bucket22 | + | true | String | bucket24 | + | false | String | bucket1 | + | null | String | bucket8 | + | 1.23 | String | bucket4 | + | 0 | Integer | bucket8 | + + @fractional @fractional-v3 + Scenario Outline: Fractional operator with float mapping + Given a String-flag with key "fractional-basic-flag" and a default value "fallback" + And a context with the following keys: + | key | type | value | + | hashing_input | | | + When the flag was evaluated with details + Then the resolved details value should be "" + + Examples: v3 + | hashing_input | type | value | + | 1.0 | Float | bucket23 | + | 1 | Integer | bucket23 | + | 1.0000000000000001 | Float | bucket23 | + | -2.0 | Float | bucket12 | + | -2 | Integer | bucket12 | + | 9007199254740992.0 | Float | bucket10 | + | 9007199254740992 | Integer | bucket10 | + + @fractional @fractional-v3 + Scenario Outline: Fractional operator with zero values + Given a String-flag with key "fractional-basic-flag" and a default value "fallback" + And a context containing a key "hashing_input", with type "" and with value "" + When the flag was evaluated with details + Then the resolved details value should be "" + + Examples: v3 + | hashing_input | type | value | + | 0.0 | Float | bucket8 | + | -0.0 | Float | bucket8 | + | 0 | Integer | bucket8 | + + @fractional @fractional-v3 + Scenario Outline: Fractional operator with integer limits + Given a String-flag with key "fractional-basic-flag" and a default value "fallback" + And a context with the following keys: + | key | type | value | + | hashing_input | Integer | | + When the flag was evaluated with details + Then the resolved details value should be "" + + Examples: v3 + | hashing_input | value | + | 2147483647 | bucket10 | + | 2147483648 | bucket25 | + | -2147483648 | bucket8 | + | -2147483649 | bucket18 | + | 9007199254740991 | bucket2 | + | 23 | bucket21 | + | 24 | bucket11 | + | 255 | bucket13 | + | 256 | bucket13 | + | 65535 | bucket4 | + | 65536 | bucket11 | + | 4294967295 | bucket14 | + | 4294967296 | bucket19 | + | -24 | bucket4 | + | -25 | bucket16 | + + @fractional @fractional-v3 + Scenario Outline: Fractional operator with floats limits + Given a String-flag with key "fractional-basic-flag" and a default value "fallback" + And a context containing a key "hashing_input", with type "Float" and with value "" + When the flag was evaluated with details + Then the resolved details value should be "" + + Examples: v3 + | hashing_input | value | + | 1e20 | bucket19 | + | -1e20 | bucket25 | + | 65504.0 | bucket16 | + | -65504.0 | bucket9 | + | 0.00006103515625 | bucket22 | + | 0.000000059604644775390625 | bucket23 | + | 65505.0 | bucket17 | + | -65505.0 | bucket17 | + | 3.4028234663852886e+38 | bucket10 | + | -3.4028234663852886e+38 | bucket3 | + | 1.1754943508222875e-38 | bucket24 | + | 3.5e+38 | bucket1 | + | -3.5e+38 | bucket6 | + | 1.7976931348623157e+308 | bucket17 | + | -1.7976931348623157e+308 | bucket9 | + | 2.2250738585072014e-308 | bucket11 | + | 4.9406564584124654e-324 | bucket2 | + + @fractional @fractional-v3 + Scenario Outline: Fractional operator with map key ordering + Given a String-flag with key "fractional-basic-flag" and a default value "fallback" + And a context with the following keys: + | key | type | value | + | hashing_input | Object | | + When the flag was evaluated with details + Then the resolved details value should be "" + + Examples: v3 + | hashing_input | value | + | {"a": 1, "b": 2} | bucket23 | + | {"b": 2, "a": 1} | bucket23 | + | {"a": 1, "b": 2, "c": 3} | bucket18 | + | {"c": 3, "a": 1, "b": 2} | bucket18 | + | {"z": 1, "aa": 2} | bucket21 | + | {"aa": 2, "z": 1} | bucket21 | + | {"a": {"b": 1}, "b": 2} | bucket9 | + | {"b": 2, "a": {"b": 1}} | bucket9 | + | {"a": true, "bbb": 1, "c": "text", "dddd": 2.5} | bucket25 | + | {"c": "text", "bbb": 1, "a": true, "dddd": 2.5} | bucket25 | + | {"aaa": 1, "ÿ": 2} | bucket12 | + | {"ÿ": 2, "aaa": 1} | bucket12 | + | {"": 2, "abc": 1} | bucket25 | + | {"key": -0.0} | bucket10 | + | {"key": 0} | bucket10 | + | {"key": 0.0} | bucket10 | + | {"b": -0.0, "a": {"b": 1}, "c": 0.0} | bucket8 | + | {"c": 0.0, "b": -0.0, "a": {"b": 1}} | bucket8 | + | {"café": "façade", "b": {"résumé": {"ÿ": true}}} | bucket22 | + + @fractional @fractional-v3 + Scenario Outline: Fractional operator with advanced structures + Given a String-flag with key "fractional-basic-flag" and a default value "fallback" + And a context with the following keys: + | key | type | value | + | hashing_input | Object | | + When the flag was evaluated with details + Then the resolved details value should be "" + + Examples: v3 + | hashing_input | value | + | {} | bucket3 | + | {"a": {} } | bucket20 | + | {"b": 2, "a": {} } | bucket20 | + | {"a": {}, "b": {}} | bucket21 | + | {"a": {}, "b": []} | bucket7 | + | {"c": 0.0, "a": []} | bucket18 | + | {"c": 0.0, "b": -0.0, "a": {"b": []}} | bucket9 | + | {"a": [] } | bucket12 | + | {"a": [1, [], 3], "b": null} | bucket3 | + | {"a": ["a", "b", {}], "b": [1, null, 2]} | bucket23 | + | {"a": [1, "two", true, [], null]} | bucket23 | + | {"a": [false, 2.5, ""], "b": [], "c": [{}], "c": null} | bucket15 | + | {"x": [{"a": 1}, {"b": 2}, {"b": null}]} | bucket3 | + | {"x": [{"a": {"b": []}}] } | bucket16 | + | {"x": [{"a": {"b": null}}] } | bucket6 | + | {"x": [{"a": {"b": {}}}] } | bucket2 | + | {"": "", "": [], "": {}, "": null} | bucket25 | + | {"": {"": ""}, "": ["", {"": ""}], "": {"": {"": ""}}} | bucket11 | + | {"a": [[[[[]]]]] } | bucket15 | + | {"a": [[[[null]]]] } | bucket7 | + | {"a": {"b": {"c": {"d": {}}} } } | bucket21 | + | {"a": {"b": {"c": {"d": null}} } } | bucket19 | + + @fractional @fractional-v3 + Scenario Outline: Fractional operator with string length boundaries + Given a String-flag with key "fractional-basic-flag" and a default value "fallback" + And a context containing a key "hashing_input", with type "String" and with value "" + When the flag was evaluated with details + Then the resolved details value should be "" + + Examples: v3 + | hashing_input | value | + | | bucket4 | + | a | bucket6 | + | 12345678901234567890123 | bucket4 | + | 123456789012345678901234 | bucket14 | + | aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa | bucket18 | + | aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa | bucket7 | + + @fractional @fractional-v3 + Scenario Outline: Valid well-formed UTF-8 String consistency across different languages + Given a String-flag with key "fractional-basic-flag" and a default value "fallback" + And a context containing a key "hashing_input", with type "String" and with value "" + When the flag was evaluated with details + Then the resolved details value should be "" + + Examples: v3 + | hashing_input | value | + | あいうえおかき | bucket17 | + | あいうえおかきく | bucket16 | + | ééééééééééé | bucket16 | + | éééééééééééé | bucket19 | + | café façade résumé café | bucket9 | + | A\u0000B | bucket15 | + | e\u0301 | bucket7 | + | \uD83D\uDE00 | bucket23 | + + @fractional @fractional-v3 + Scenario Outline: Fractional operator using implicit targeting key + Given a String-flag with key "fractional-flag-shorthand" and a default value "draw" + And a context containing a targeting key with value "" + When the flag was evaluated with details + Then the resolved details value should be "" + + Examples: v3 + | targeting_key | value | + | user-1 | heads | + | user-2 | tails | + | user-3 | heads | + + @fractional @fractional-v3 + Scenario Outline: Fractional operator invalid implicit inputs + Given a String-flag with key "fractional-flag-shorthand" and a default value "fallback" + And a context containing a key "", with type "" and with value "" + When the flag was evaluated with details + Then the resolved details value should be "draw" + Then the reason should be "DEFAULT" + + Examples: v3 + | key | type | value | + | irrelevant | String | none | + | targetingKey | Integer | 12345 | + | targetingKey | Boolean | true | diff --git a/tck-extensions/go.mod b/tck-extensions/go.mod new file mode 100644 index 0000000..da1ae9e --- /dev/null +++ b/tck-extensions/go.mod @@ -0,0 +1,3 @@ +module github.com/open-feature/flagd-testbed/tck-extensions + +go 1.21 diff --git a/tck-extensions/metadata.feature b/tck-extensions/metadata.feature new file mode 100644 index 0000000..92b7ddf --- /dev/null +++ b/tck-extensions/metadata.feature @@ -0,0 +1,58 @@ +@ext-metadata @in-process @file @metadata +Feature: TCK extension: flag and flag set metadata + + # Ported from gherkin/metadata.feature. The testbed's "a stable flagd provider" + # becomes the TCK's "a stable provider", and its "a metadata flagd provider" becomes "a stable + # provider with "metadata" configuration". The two scenarios a file resolver cannot serve name a + # selector flag file and carry @metadata-provider. + + Scenario: Returns metadata + Given a stable provider + And a Boolean-flag with key "metadata-flag" and a default value "true" + When the flag was evaluated with details + Then the resolved metadata should contain + | key | metadata_type | value | + | string | String | a | + | integer | Integer | 1 | + | float | Float | 1.2 | + | boolean | Boolean | true | + + @metadata-provider + Scenario: Returns flag set metadata + Given an option "selector" of type "String" with value "rawflags/selector-flag-combined-metadata.json" + And a stable provider with "metadata" configuration + And a Boolean-flag with key "only-set-metadata-flag" and a default value "true" + When the flag was evaluated with details + Then the resolved metadata should contain + | key | metadata_type | value | + | string | String | b | + | integer | Integer | 2 | + | float | Float | 2.2 | + | boolean | Boolean | false | + | flag-set-string | String | c | + | flag-set-integer | Integer | 3 | + | flag-set-float | Float | 3.2 | + | flag-set-boolean | Boolean | false | + + @metadata-provider + Scenario: Flag metadata overwrites flag set metadata + Given an option "selector" of type "String" with value "rawflags/selector-flag-combined-metadata.json" + And a stable provider with "metadata" configuration + And a Boolean-flag with key "combined-metadata-flag" and a default value "true" + When the flag was evaluated with details + Then the resolved metadata should contain + | key | metadata_type | value | + | string | String | a | + | integer | Integer | 1 | + | float | Float | 1.2 | + | boolean | Boolean | true | + | flag-set-string | String | c | + | flag-set-integer | Integer | 3 | + | flag-set-float | Float | 3.2 | + | flag-set-boolean | Boolean | false | + + Scenario: Returns no metadata + Given a Boolean-flag with key "boolean-flag" and a default value "true" + And a stable provider + When the flag was evaluated with details + Then the resolved metadata is empty diff --git a/tck-extensions/rpc-caching.feature b/tck-extensions/rpc-caching.feature new file mode 100644 index 0000000..b37098e --- /dev/null +++ b/tck-extensions/rpc-caching.feature @@ -0,0 +1,80 @@ +@ext-connection @rpc @flagd-caching +Feature: TCK extension: Flag evaluation with caching + + # Ported from gherkin/rpc-caching.feature (RPC only). The testbed's @caching tag is + # a reserved TCK capability, which the TCK refuses on any scenario, so this feature carries + # @flagd-caching instead. + # + # Reconnect timing is the testbed's "stable flagd provider", as in events.feature. + + Background: + Given an option "cache" of type "CacheType" with value "lru" + And an option "retryGracePeriod" of type "Integer" with value "5" + And an option "retryBackoffMaxMs" of type "Integer" with value "500" + And a stable provider + + Scenario Outline: Resolves details with caching + Given a -flag with key "" and a default value "" + When the flag was evaluated with details + Then the resolved details value should be "" + And the variant should be "" + And the reason should be "STATIC" + When the flag was evaluated with details + Then the resolved details value should be "" + And the variant should be "" + And the reason should be "CACHED" + + Examples: + | key | type | default | resolved_variant | resolved_value | + | boolean-flag | Boolean | false | on | true | + | string-flag | String | bye | greeting | hi | + | integer-flag | Integer | 1 | ten | 10 | + | float-flag | Float | 0.1 | half | 0.5 | + + Scenario: Resolves Object details with caching + Given a Object-flag with key "object-flag" and a default value "{}" + When the flag was evaluated with details + Then the resolved object value should contain + | key | type | value | + | showImages | Boolean | true | + | title | String | Check out these pics! | + | imagesPerPage | Integer | 100 | + And the variant should be "template" + And the reason should be "STATIC" + When the flag was evaluated with details + Then the reason should be "CACHED" + + Scenario: Flag change event with caching + Given a String-flag with key "changing-flag" and a default value "false" + And a change event handler + When the flag was modified + And a change event was fired + And the flag was evaluated with details + Then the reason should be "STATIC" + When the flag was evaluated with details + Then the reason should be "CACHED" + When the flag was modified + And a change event was fired + And the flag was evaluated with details + Then the reason should be "STATIC" + When the flag was evaluated with details + Then the reason should be "CACHED" + + Scenario: Stale and Error stage + Given a ready event handler + And a stale event handler + And a error event handler + And a Boolean-flag with key "boolean-flag" and a default value "false" + When the flag was evaluated with details + Then the reason should be "STATIC" + When the flag was evaluated with details + Then the reason should be "CACHED" + When the connection is lost + And a stale event was fired + And the flag was evaluated with details + Then the reason should be "CACHED" + When a error event was fired + And the connection is restored + And a ready event was fired + And the flag was evaluated with details + Then the reason should be "STATIC" diff --git a/tck-extensions/selector.feature b/tck-extensions/selector.feature new file mode 100644 index 0000000..6a1508a --- /dev/null +++ b/tck-extensions/selector.feature @@ -0,0 +1,27 @@ +@ext-evaluation @in-process +Feature: TCK extension: flagd selector + + # Ported from gherkin/selector.feature. The launchpad's default configuration + # serves both flags/allFlags.json and rawflags/selector-flags.json, so the selector narrows the + # sync to the second source. + + Background: + Given an option "selector" of type "String" with value "rawflags/selector-flags.json" + And a stable provider + + Scenario Outline: Flags not found + Given a -flag with key "" and a default value "" + When the flag was evaluated with details + Then the reason should be "ERROR" + + Examples: + | key | type | default | + | boolean-flag | Boolean | false | + | string-flag | String | bye | + | integer-flag | Integer | 1 | + | float-flag | Float | 0.1 | + + Scenario: Resolve values + Given a String-flag with key "selector-flag" and a default value "foo" + When the flag was evaluated with details + Then the reason should be "STATIC" diff --git a/tck-extensions/sync-payload.feature b/tck-extensions/sync-payload.feature new file mode 100644 index 0000000..e645e6c --- /dev/null +++ b/tck-extensions/sync-payload.feature @@ -0,0 +1,40 @@ +@ext-metadata @in-process @sync-payload +Feature: TCK extension: Disabled Sync Metadata + + # Ported from gherkin/sync-payload.feature. The testbed's "a syncpayload flagd + # provider" becomes "a stable provider with "sync-payload" configuration", moved from the + # Background into each scenario so the @grace one can record options first. + # + # The @grace scenario sets the testbed's retry timing (grace period 5s, backoff 500ms capped at + # 500ms) whatever the adoption's defaults, so the provider is back within the event timeout once + # the connection is restored. + + Scenario: Use enriched context + Given a stable provider with "sync-payload" configuration + And a String-flag with key "flagd-context-aware" and a default value "not" + When the flag was evaluated with details + Then the resolved details value should be "INTERNAL" + + @grace + Scenario: Use enriched context on connection error for IN-PROCESS + Given an option "retryGracePeriod" of type "Integer" with value "5" + And an option "retryBackoffMs" of type "Integer" with value "500" + And an option "retryBackoffMaxMs" of type "Integer" with value "500" + And a stable provider with "sync-payload" configuration + And a String-flag with key "flagd-context-aware" and a default value "not" + And a stale event handler + And a ready event handler + When the flag was evaluated with details + Then the resolved details value should be "INTERNAL" + When the connection is lost + And a stale event was fired + When the flag was evaluated with details + Then the resolved details value should be "INTERNAL" + When the connection is restored + And a ready event was fired + + Scenario: Resolve value independent of context + Given a stable provider with "sync-payload" configuration + And a Boolean-flag with key "boolean-flag" and a default value "false" + When the flag was evaluated with details + Then the resolved details value should be "true" diff --git a/tck-extensions/targeting.feature b/tck-extensions/targeting.feature new file mode 100644 index 0000000..1d47525 --- /dev/null +++ b/tck-extensions/targeting.feature @@ -0,0 +1,222 @@ +@ext-targeting @rpc @in-process @file +Feature: TCK extension: Targeting rules + + # Ported from gherkin/targeting.feature: the json-evaluation scenarios the Provider + # TCK does not ask. The @fractional scenarios live in their own extension feature. + # + # Not ported: + # - "Targeting by targeting key": the TCK's evaluation.feature (@targeting) asserts both values + # and reason.feature asserts TARGETING_MATCH and DEFAULT for the same flag and keys. + # - "fractional operator with missing bucket key": a fractional scenario, despite its tags. + # + # Deviations from the testbed's wording: + # - "a stable flagd provider" collides with the TCK's "a {} provider", so the provider step is the + # TCK's "a stable provider". + # - "Time-based operations" sets the context with "a context with the following keys:", because the + # TCK's context step parses Integer as 32 bits and 4133980802 does not fit. + + Background: + Given an option "cache" of type "CacheType" with value "disabled" + And a stable provider + + # evaluator refs + Scenario Outline: Evaluator reuse + Given a String-flag with key "" and a default value "fallback" + And a context containing a key "email", with type "String" and with value "ballmer@macrosoft.com" + When the flag was evaluated with details + Then the resolved details value should be "" + Examples: + | key | value | + | some-email-targeted-flag | hi | + | some-other-email-targeted-flag | yes | + + @evaluator-refs @evaluator-refs-whitespace + Scenario Outline: Evaluator $ref resolves regardless of whitespace around the colon + Given a String-flag with key "" and a default value "fallback" + And a context containing a key "email", with type "String" and with value "" + When the flag was evaluated with details + Then the resolved details value should be "" + And the reason should be "TARGETING_MATCH" + Examples: + | key | email | value | + | ref-whitespace-compact-flag | ballmer@macrosoft.com | hi | + | ref-whitespace-space-after-colon-flag | ballmer@macrosoft.com | hi | + | ref-whitespace-space-around-colon-flag | ballmer@macrosoft.com | hi | + | ref-whitespace-compact-flag | user@example.com | bye | + | ref-whitespace-space-after-colon-flag | user@example.com | bye | + | ref-whitespace-space-around-colon-flag | user@example.com | bye | + + @evaluator-refs @non-existent-evaluator-ref + Scenario: Ref to nonexistent evaluator yields parse error + Given a String-flag with key "ref-to-nonexistent-evaluator-flag" and a default value "fallback" + When the flag was evaluated with details + Then the resolved details value should be "fallback" + And the error-code should be "PARSE_ERROR" + + @string + Scenario Outline: Substring operators + Given a String-flag with key "starts-ends-flag" and a default value "fallback" + And a context containing a key "id", with type "String" and with value "" + When the flag was evaluated with details + Then the resolved details value should be "" + Examples: + | id | value | + | abcdef | prefix | + | uvwxyz | postfix | + | abcxyz | prefix | + | lmnopq | none | + | 3 | none | + + @semver + Scenario Outline: Semantic version operator numeric comparison + Given a String-flag with key "equal-greater-lesser-version-flag" and a default value "fallback" + And a context containing a key "version", with type "String" and with value "" + When the flag was evaluated with details + Then the resolved details value should be "" + Examples: + | version | value | + | 2.0.0 | equal | + | 2.1.0 | greater | + | 1.9.0 | lesser | + | 2.0.0-alpha | lesser | + | 2.0.0.0 | none | + + @semver + Scenario Outline: Semantic version operator semantic comparison + Given a String-flag with key "major-minor-version-flag" and a default value "fallback" + And a context containing a key "version", with type "String" and with value "" + When the flag was evaluated with details + Then the resolved details value should be "" + Examples: + | version | value | + | 3.0.1 | minor | + | 3.1.0 | major | + | 4.0.0 | none | + + @semver @semver-edge-cases @semver-v-prefix + Scenario Outline: sem_ver v-prefix handling + Given a String-flag with key "semver-v-prefix-flag" and a default value "fallback" + And a context containing a key "version", with type "String" and with value "" + When the flag was evaluated with details + Then the resolved details value should be "" + Examples: + | version | value | + | 1.0.0 | match | + | v1.0.0 | match | + | V1.0.0 | match | + | 2.0.0 | no-match | + + @semver @semver-edge-cases @semver-partial-version + Scenario Outline: sem_ver partial version string handling + Given a String-flag with key "semver-partial-version-flag" and a default value "fallback" + And a context containing a key "version", with type "" and with value "" + When the flag was evaluated with details + Then the resolved details value should be "" + Examples: + | version | type | value | + | 1.5.0 | String | match | + | 1.0.0 | String | match | + | 1.0 | String | match | + | 1 | String | match | + | 1 | Integer | match | + | 1 | Float | match | + | 1.2 | Float | match | + | 2.0.0 | String | no-match | + + @semver @semver-edge-cases @semver-numeric-context + Scenario Outline: sem_ver numeric context value coercion + Given a String-flag with key "semver-numeric-context-flag" and a default value "fallback" + And a context containing a key "version", with type "" and with value "" + When the flag was evaluated with details + Then the resolved details value should be "" + Examples: + | version | type | value | + | 1.2 | Float | match | + | 1.1 | Float | no-match | + | 2 | Integer | match | + | 1 | Integer | no-match | + | 1.2 | String | match | + + @semver @semver-edge-cases @semver-build-metadata + Scenario Outline: sem_ver build metadata ignored + Given a String-flag with key "semver-build-metadata-flag" and a default value "fallback" + And a context containing a key "version", with type "String" and with value "" + When the flag was evaluated with details + Then the resolved details value should be "" + Examples: + | version | value | + | 1.0.0 | match | + | 1.0.0+other | match | + | 2.0.0 | no-match | + + Scenario Outline: Time-based operations + Given a Integer-flag with key "timestamp-flag" and a default value "0" + And a context with the following keys: + | key | type | value | + | time | Integer |