feat: Spec getExperimentFlag, trackEvent and trackExposureEvent

HARNESS.md

@@ -20,10 +20,12 @@ A harness runs an SDK's steps file for the spec runner. There is one per languag
 
 ## Finding steps
 
-- A step marker is a line comment holding either `Create the SDK client` or a step starting with
-  `When `, e.g. `When the application gets the environment flags`.
+- A step marker is a line comment starting with either `Create the SDK client` or `When `, e.g.
+  `When the application gets the environment flags`.
 - The marked function is the first function defined after the marker.
-- The marker text identifies the step, and must match the runner's text exactly.
+- The marker text identifies the step, and must match the runner's text exactly. A steps file has
+  one function per step, including the client-creating steps the runner asks for by name
+  (`Create the SDK client`, `Create the SDK client with events enabled`).
 
 Find steps at runtime where the language allows it. Otherwise, generate a dispatch table at build time
 inside the SDK's Docker build (Go does this, for example).

README.md

@@ -6,7 +6,7 @@ executed against every SDK.
 - [`features/`](features) — the spec. One `.feature` file per behaviour.
 - [`runner/`](runner) — executes the spec (pytest-bdd). Step definitions live here, written once for
   every SDK.
-- [`compose.yml`](compose.yml) — the Flagsmith API (pinned) the spec runs against.
+- [`compose.yml`](compose.yml) — the Flagsmith services (pinned) the spec runs against.
 - [`harness/`](harness) — one harness per language, running an SDK's steps file
   ([`HARNESS.md`](HARNESS.md)).
 - [`.github/workflows/run-spec.yml`](.github/workflows/run-spec.yml) — reusable workflow SDKs call.
@@ -20,9 +20,16 @@ executed against every SDK.
  │ runner │ ────────────────────────────────► │ Flagsmith │
  │        │                                   └───────────┘
  │        │  stdio  ┌──────────────────────────────┐  ▲
- │        │ ──────► │ harness ─► SDK steps file    │──┘ SDK API calls
- └────────┘         │ (SDK image, built from source)│
-                    └──────────────────────────────┘
+ │        │ ──────► │ harness ─► SDK steps file    │──┤ SDK API calls
+ │        │         │ (SDK image, built from source)│  │
+ │        │         └──────────────────────────────┘  │ events
+ │        │                                   ┌───────▼────┐
+ │        │                                   │ events API │
+ │        │                                   └───────┬────┘
+ │        │  read the delivered events                │ warehouse delivery
+ └────────┘ ────────────────────────────────► ┌───────▼───┐
+                                              │ warehouse │
+                                              └───────────┘
 ```
 
 Each scenario gets its own Flagsmith project and environment, and the steps split three ways:
@@ -36,6 +43,14 @@ Each scenario gets its own Flagsmith project and environment, and the steps spli
 The feature files are therefore the whole contract: the `When` steps in `features/` are the steps
 SDKs implement. The runner creates the scenario's SDK client just before its first `When` step.
 
+Tracking and exposure events take a longer path: an SDK sends them to Flagsmith's events API, which
+queues them for warehouse delivery to insert into the environment's warehouse.
+[`compose.yml`](compose.yml) runs that whole path, as Flagsmith's own `docker-compose.yml` does, with
+a ClickHouse standing in for the environment's warehouse. A scenario that says
+`Given the SDK sends events to Flagsmith` gets the warehouse connected to its environment — the
+events API only accepts an environment that has one — and a client with events enabled pointed at
+the events API. The runner then reads the delivered events out of the warehouse.
+
 ## Writing scenarios
 
 - Describe behaviour, not wire details: assert on what the application receives and what Flagsmith
@@ -77,6 +92,9 @@ Each release also pins the Flagsmith version in `compose.yml` that it was verifi
 
    - `Create the SDK client` receives a server-side environment key and the Flagsmith API URL, and
      returns a client created with default options.
+   - `Create the SDK client with events enabled` receives the same two, then the base URL of
+     Flagsmith's events API, and returns a client that sends events there. Only `@events`
+     scenarios use it, so an SDK without event tracking can leave it out.
    - A `When` step receives the client, then the step's placeholders in order, then its data table.
      A `<key> | value` table arrives as a map, e.g. `{"plan": "premium", "age": 30}`.
    - Values are JSON values. Pass flag and trait values through with their types intact.
@@ -151,9 +169,16 @@ uv run pytest -v
 | `FLAGSMITH_SDK_URL` | `$FLAGSMITH_URL`        | Where the SDK reaches Flagsmith                 |
 | `FLAGSMITH_PORT`    | `8000`                  | Host port for Flagsmith (`compose.yml`)         |
 | `FLAGSMITH_IMAGE`   | pinned version          | Override the Flagsmith image (`compose.yml`)    |
+| `EVENTS_SDK_URL`    | `http://events:8080`    | Where the SDK reaches the events API            |
+| `EVENTS_PORT`       | `8080`                  | Host port for the events API (`compose.yml`)    |
+| `WAREHOUSE_URL`     | `http://localhost:8123` | Where the runner reaches the warehouse          |
+| `WAREHOUSE_HOST`    | `clickhouse`            | Where Flagsmith reaches the warehouse           |
+| `CLICKHOUSE_PORT`   | `8123`                  | Host port for the warehouse (`compose.yml`)     |
 
 `SDK_COMMAND` can be any command, so you can also run a harness straight against an SDK checkout,
 without Docker, e.g. `python ../harness/python/harness.py ../../flagsmith-python-client/sdk-spec/steps.py`.
+An SDK running on the host reaches the services on the host's ports, so set
+`FLAGSMITH_SDK_URL=$FLAGSMITH_URL` and `EVENTS_SDK_URL=http://localhost:8080` with it.
 
 ## Implementing a new spec version
 

compose.yml

@@ -1,7 +1,16 @@
 # The Flagsmith API the spec runs against. The runner runs on the host and starts the
 # SDK under test in a container on this compose project's network (see README.md).
+#
+# The experimentation services mirror Flagsmith's own docker-compose.yml, so that `@events`
+# scenarios run against the real ingestion and delivery path: an SDK posts events to the
+# ingestion server, which writes them to Kafka, from where warehouse delivery inserts them
+# into the environment's warehouse. The spec provisions ClickHouse as that warehouse and
+# reads the events back out of it.
 name: sdk-spec
 
+volumes:
+  clickhouse-data:
+
 services:
   postgres:
     image: postgres:15.5-alpine
@@ -16,13 +25,17 @@ services:
 
   flagsmith:
     # The Flagsmith version a spec release is verified against.
-    image: ${FLAGSMITH_IMAGE:-docker.io/flagsmith/flagsmith:2.276.0}
+    image: ${FLAGSMITH_IMAGE:-docker.io/flagsmith/flagsmith:2.287.0}
     command: migrate-and-serve
     environment:
       DATABASE_URL: postgresql://postgres:password@postgres:5432/flagsmith
       DJANGO_ALLOWED_HOSTS: '*'
       DJANGO_SECRET_KEY: sdk-spec
       ENVIRONMENT: test
+      EXPERIMENTATION_ENABLED: 'true'
+      # Encrypts the warehouse connection's credentials; warehouse delivery decrypts them
+      # with the same value.
+      WAREHOUSE_CREDENTIALS_SECRET: &warehouse-credentials-secret sdk-spec
     ports:
       - ${FLAGSMITH_PORT:-8000}:8000
     healthcheck:
@@ -34,3 +47,101 @@ services:
       postgres:
         condition: service_healthy
 
+  kafka:
+    image: apache/kafka:3.9.1
+    environment:
+      KAFKA_NODE_ID: 1
+      KAFKA_PROCESS_ROLES: broker,controller
+      KAFKA_LISTENERS: PLAINTEXT://:9092,CONTROLLER://:9093
+      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
+      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
+      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT
+      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
+      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
+      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
+      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
+      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
+    healthcheck:
+      test: ['CMD-SHELL', '/opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092']
+      interval: 5s
+      timeout: 10s
+      retries: 30
+
+  kafka-init:
+    image: apache/kafka:3.9.1
+    entrypoint: ['/bin/sh', '-c']
+    command:
+      - |
+        for topic in flagsmith_warehouse_events external_warehouse_events external_warehouse_events_retry; do
+          /opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka:9092 --create --if-not-exists --topic $$topic
+        done
+    depends_on:
+      kafka:
+        condition: service_healthy
+
+  # The environment's warehouse: where delivered events land and the runner reads them from.
+  clickhouse:
+    image: clickhouse/clickhouse-server:25.12
+    environment:
+      CLICKHOUSE_USER: flagsmith
+      CLICKHOUSE_PASSWORD: password
+    ulimits:
+      nofile:
+        soft: 262144
+        hard: 262144
+    volumes:
+      - clickhouse-data:/var/lib/clickhouse
+    ports:
+      - ${CLICKHOUSE_PORT:-8123}:8123
+    healthcheck:
+      test: ['CMD', 'wget', '--spider', '-q', 'http://localhost:8123/ping']
+      interval: 3s
+      timeout: 3s
+      retries: 30
+
+  # Creates the events table, from the schema the pinned Flagsmith version ships.
+  clickhouse-init:
+    image: clickhouse/clickhouse-server:25.12
+    entrypoint: ['/bin/sh', '-c']
+    command:
+      - |
+        set -e
+        wget -qO /tmp/events.sql https://raw.githubusercontent.com/Flagsmith/flagsmith/v2.287.0/experimentation/clickhouse/events.sql
+        clickhouse-client --host clickhouse --user flagsmith --password password --multiquery < /tmp/events.sql
+    depends_on:
+      clickhouse:
+        condition: service_healthy
+
+  # Flagsmith's events API: where SDKs send tracking and exposure events.
+  events:
+    image: flagsmith/events-ingestion-api:0.5.1
+    environment:
+      KAFKA_BOOTSTRAP_SERVERS: kafka:9092
+      KAFKA_AUTH: none
+      KAFKA_TOPIC: flagsmith_warehouse_events
+      KAFKA_EXTERNAL_TOPIC: external_warehouse_events
+      DATABASE_URL: postgres://postgres:password@postgres:5432/flagsmith
+    ports:
+      - ${EVENTS_PORT:-8080}:8080
+    depends_on:
+      flagsmith:
+        condition: service_healthy
+      kafka-init:
+        condition: service_completed_successfully
+
+  warehouse-delivery:
+    image: flagsmith/warehouse-delivery:0.4.0
+    environment:
+      KAFKA_BOOTSTRAP_SERVERS: kafka:9092
+      KAFKA_AUTH: none
+      DATABASE_URL: postgresql://postgres:password@postgres:5432/flagsmith
+      WAREHOUSE_CREDENTIALS_SECRET: *warehouse-credentials-secret
+      # The spec asserts on events as soon as they are sent, so deliver them promptly.
+      BATCH_MAX_WAIT_MS: 1000
+    depends_on:
+      flagsmith:
+        condition: service_healthy
+      kafka-init:
+        condition: service_completed_successfully
+      clickhouse-init:
+        condition: service_completed_successfully

features/get_experiment_flag.feature

@@ -0,0 +1,50 @@
+@events
+Feature: Get an experiment flag
+  An SDK resolves a feature for an identity and records the exposure.
+
+  Background:
+    Given a Flagsmith environment
+    And the SDK sends events to Flagsmith
+
+  Scenario: An experiment flag is resolved for the identity, with its traits
+    Given the environment has the following features:
+      | feature | enabled | value   |
+      | banner  | true    | "hello" |
+    And a segment "premium_users" where trait "plan" equals "premium"
+    And segment "premium_users" overrides feature "banner" with enabled true and value "hello premium user"
+    When the application gets the experiment flag "banner" for identity "user_123" with traits:
+      | trait | value     |
+      | plan  | "premium" |
+    Then the application receives the following flags:
+      | feature | enabled | value                |
+      | banner  | true    | "hello premium user" |
+    And Flagsmith has identity "user_123" with the following traits:
+      | trait | value     |
+      | plan  | "premium" |
+
+  Scenario: An identity in a running experiment is exposed to the variant it was served
+    Given the environment has the following features:
+      | feature | enabled | value     |
+      | banner  | true    | "control" |
+    And a running experiment on feature "banner" serving variant "treatment" with value "hello treatment"
+    When the application gets the experiment flag "banner" for identity "user_123" with traits:
+      | trait | value     |
+      | plan  | "premium" |
+    Then the application receives the following flags:
+      | feature | enabled | value             |
+      | banner  | true    | "hello treatment" |
+    And Flagsmith has received the following events:
+      | event          | feature | identifier | value       | traits              |
+      | $flag_exposure | banner  | user_123   | "treatment" | {"plan": "premium"} |
+
+  Scenario: An identity outside a running experiment is not exposed
+    Given the environment has the following features:
+      | feature | enabled | value   |
+      | banner  | true    | "hello" |
+    When the application gets the experiment flag "banner" for identity "user_123" with traits:
+      | trait | value     |
+      | plan  | "premium" |
+    Then the application receives the following flags:
+      | feature | enabled | value   |
+      | banner  | true    | "hello" |
+    And Flagsmith has received no events

features/track_event.feature

@@ -0,0 +1,15 @@
+@events
+Feature: Track an event
+  An SDK sends an application's tracking events to Flagsmith.
+
+  Background:
+    Given a Flagsmith environment
+    And the SDK sends events to Flagsmith
+
+  Scenario: A tracked event reaches Flagsmith
+    When the application tracks event "checkout_completed" for identity "user_123" with value 42 and metadata:
+      | key      | value |
+      | currency | "GBP" |
+    Then Flagsmith has received the following events:
+      | event              | identifier | value | metadata            |
+      | checkout_completed | user_123   | "42"  | {"currency": "GBP"} |

features/track_exposure_event.feature

@@ -0,0 +1,15 @@
+@events
+Feature: Track an exposure event
+  An SDK records which variant of a feature an identity was shown, as a "$flag_exposure" event.
+
+  Background:
+    Given a Flagsmith environment
+    And the SDK sends events to Flagsmith
+
+  Scenario: An exposure reaches Flagsmith as a $flag_exposure event
+    When the application tracks an exposure of feature "banner" for identity "user_123" with value "control" and metadata:
+      | key           | value |
+      | experiment_id | 1     |
+    Then Flagsmith has received the following events:
+      | event          | feature | identifier | value     | metadata             |
+      | $flag_exposure | banner  | user_123   | "control" | {"experiment_id": 1} |

harness/dart/README.md

@@ -12,7 +12,7 @@ step text. At runtime, `harness.dart` calls them with `Function.apply`.
 
 - Step functions are public, top-level functions in `sdk-spec/steps.dart`, each below its step marker
   (`// Create the SDK client`, `// When …`). Import the SDK by its package name.
-- `Create the SDK client` returns the client. `When` steps take it first.
+- A `Create the SDK client…` step returns the client. `When` steps take it first.
 - Arguments arrive as decoded JSON: `String`, `int`, `double`, `bool`, `null`, and data tables as
   `Map<String, dynamic>`. Declare them with those types, e.g. `Map<String, Object?> traits`.
 - Return a JSON-encodable value, or a `Future` of one.

harness/dart/build.dart

@@ -8,7 +8,7 @@
 import 'dart:convert';
 import 'dart:io';
 
-final _marker = RegExp(r'^\s*//+\s*(When .+?|Create the SDK client)\s*$');
+final _marker = RegExp(r'^\s*//+\s*(When .+?|Create the SDK client.*?)\s*$');
 // A top-level function declaration: an optional return type, then the name and its parameters.
 final _function = RegExp(
   r'^(?:[\w$][\w$<>?,.\[\] ]*\s)?([A-Za-z_$][\w$]*)\s*(?:<[^()]*>)?\s*\(',

harness/dotnet/Harness.cs

@@ -15,7 +15,7 @@ namespace SdkSpec.Harness;
 
 static partial class Program
 {
-    [GeneratedRegex(@"^\s*//+\s*(When .+?|Create the SDK client)\s*$")]
+    [GeneratedRegex(@"^\s*//+\s*(When .+?|Create the SDK client.*?)\s*$")]
     private static partial Regex Marker();
 
     // A static method declaration: modifiers including `static`, a return type, then the name.

harness/elixir/harness.exs

@@ -5,7 +5,7 @@
 # Requires Jason, which the SDK is expected to depend on.
 
 defmodule SDKSpec.Harness do
-  @marker ~r/^\s*#+\s*(When .+?|Create the SDK client)\s*$/
+  @marker ~r/^\s*#+\s*(When .+?|Create the SDK client.*?)\s*$/
   @function ~r/^\s*def\s+([a-z_][a-zA-Z0-9_]*[?!]?)/
 
   def main([path]) do

harness/go/build.go

@@ -26,7 +26,7 @@ import (
 )
 
 var (
-	marker = regexp.MustCompile(`^//\s*(When .+?|Create the SDK client)\s*$`)
+	marker = regexp.MustCompile(`^//\s*(When .+?|Create the SDK client.*?)\s*$`)
 	module = regexp.MustCompile(`(?m)^module\s+(\S+)`)
 )
 

harness/java/Harness.java

@@ -31,7 +31,7 @@
 public class Harness {
 
   private static final Pattern MARKER =
-      Pattern.compile("^\\s*//+\\s*(When .+?|Create the SDK client)\\s*$");
+      Pattern.compile("^\\s*//+\\s*(When .+?|Create the SDK client.*?)\\s*$");
   private static final Pattern METHOD =
       Pattern.compile("^\\s*(?:[\\w@]+\\s+)*static\\s+[^=;(]+?\\s(\\w+)\\s*\\(");
   private static final Pattern PACKAGE = Pattern.compile("^\\s*package\\s+([\\w.]+)\\s*;");

harness/kotlin/Harness.kt

@@ -19,7 +19,7 @@ import java.lang.reflect.Method
 import java.lang.reflect.Modifier
 import kotlin.system.exitProcess
 
-private val MARKER = Regex("""^\s*//+\s*(When .+?|Create the SDK client)\s*$""")
+private val MARKER = Regex("""^\s*//+\s*(When .+?|Create the SDK client.*?)\s*$""")
 private val FUNCTION = Regex("""^\s*(?:[\w@]+\s+)*fun\s+(\w+)\s*\(""")
 private val PACKAGE = Regex("""^\s*package\s+([\w.]+)""")
 private val JVM_NAME = Regex("""^\s*@file:JvmName\("(\w+)"\)""")

harness/nodejs/harness.mts

@@ -11,7 +11,7 @@ import { pathToFileURL } from 'node:url';
 
 type Step = (...args: unknown[]) => unknown;
 
-const MARKER = /^\s*\/\/+\s*(When .+?|Create the SDK client)\s*$/;
+const MARKER = /^\s*\/\/+\s*(When .+?|Create the SDK client.*?)\s*$/;
 const FUNCTION =
     /^\s*(?:export\s+)?(?:async\s+)?function\s*\*?\s*(\w+)|^\s*(?:export\s+)?(?:const|let|var)\s+(\w+)\s*=/;
 

harness/php/harness.php

@@ -12,7 +12,7 @@
 
 namespace FlagsmithSdkSpec\Harness;
 
-const MARKER = '/^\s*(?:\/\/|#)\s*(When .+?|Create the SDK client)\s*$/';
+const MARKER = '/^\s*(?:\/\/|#)\s*(When .+?|Create the SDK client.*?)\s*$/';
 const FUNCTION_ = '/^\s*function\s+(\w+)\s*\(/';
 const NAMESPACE_ = '/^\s*namespace\s+([\w\\\\]+)\s*;/';
 

harness/python/harness.py

@@ -12,7 +12,7 @@
 from pathlib import Path
 from typing import Any, Callable
 
-MARKER = re.compile(r"^\s*#+\s*(When .+?|Create the SDK client)\s*$")
+MARKER = re.compile(r"^\s*#+\s*(When .+?|Create the SDK client.*?)\s*$")
 FUNCTION = re.compile(r"^\s*(?:async\s+)?def\s+(\w+)")