Flagsmith SDK behavioural spec

A language-agnostic, versioned specification of how Flagsmith SDKs behave, written in Gherkin and executed against every SDK.

features/ — the spec. One .feature file per behaviour.
runner/ — executes the spec (pytest-bdd). Step definitions live here, written once for every SDK.
compose.yml — the Flagsmith services (pinned) the spec runs against.
harness/ — one harness per language, running an SDK's steps file (HARNESS.md).
.github/workflows/run-spec.yml — reusable workflow SDKs call.
.github/workflows/compatibility.yml — runs the spec against every onboarded SDK on pull requests and nightly.

How it works

 ┌────────┐  Admin API: seed & inspect        ┌───────────┐
 │ runner │ ────────────────────────────────► │ Flagsmith │
 │        │                                   └───────────┘
 │        │  stdio  ┌──────────────────────────────┐  ▲
 │        │ ──────► │ harness ─► SDK steps file    │──┤ SDK API calls
 │        │         │ (SDK image, built from source)│  │
 │        │         └──────────────────────────────┘  │ events
 │        │                                   ┌───────▼────┐
 │        │                                   │ events API │
 │        │                                   └───────┬────┘
 │        │  read the delivered events                │ warehouse delivery
 └────────┘ ────────────────────────────────► ┌───────▼───┐
                                              │ warehouse │
                                              └───────────┘

Each scenario gets its own Flagsmith project and environment, and the steps split three ways:

Given steps set up Flagsmith through the Admin API. The runner implements them.
When steps are the application using the SDK. Every SDK implements every When step in its steps file.
Then steps check what the SDK returned and Flagsmith's resulting state. The runner implements them.

The feature files are therefore the whole contract: the When steps in features/ are the steps SDKs implement. The runner creates the scenario's SDK client just before its first When step.

Tracking and exposure events take a longer path: an SDK sends them to Flagsmith's events API, which queues them for warehouse delivery to insert into the environment's warehouse. compose.yml runs that whole path, as Flagsmith's own docker-compose.yml does, with a ClickHouse standing in for the environment's warehouse. A scenario that says Given the SDK sends events to Flagsmith gets the warehouse connected to its environment — the events API only accepts an environment that has one — and a client with events enabled pointed at the events API. The runner then reads the delivered events out of the warehouse.

Writing scenarios

Describe behaviour, not wire details: assert on what the application receives and what Flagsmith ends up with.
Table cells and value arguments are JSON literals: "text", 42, true, null.
Tag every feature with a capability (@core, …). SDKs select capabilities with a tag expression. Register new tags under markers in runner/pyproject.toml.
Prefer reusing existing steps in runner/tests/conftest.py.
A new When step is registered with one sdk_step(...) line in the runner. It is a minor version: SDKs that have not implemented it go red with "The SDK does not implement the step: …". Rewording a When step is a major version.
The compatibility matrix runs with missing-steps: xfail: scenarios using a step an SDK has not implemented yet are expected failures, reported as warnings, so a pull request adding a step stays green unless something else breaks. SDK repositories keep the default, fail.

Versioning

The spec is released as semver tags (vX.Y.Z). SDKs pin a tag, so spec changes never break an SDK until it opts in.

Each release also pins the Flagsmith version in compose.yml that it was verified against.

Onboarding an SDK

Add sdk-spec/steps.<ext> to the SDK repository: a function creating the SDK client, and one function per When step in features/. Mark each with a comment directly above it:
```
# Create the SDK client
def create_client(environment_key, api_url):
    return Flagsmith(environment_key=environment_key, api_url=api_url)


# When the application gets the environment flags
def get_environment_flags(flagsmith):
    return as_spec_flags(flagsmith.get_environment_flags())
```
- Create the SDK client receives a server-side environment key and the Flagsmith API URL, and returns a client created with default options.
- Create the SDK client with events enabled receives the same two, then the base URL of Flagsmith's events API, and returns a client that sends events there. Only @events scenarios use it, so an SDK without event tracking can leave it out.
- Create the SDK client with local evaluation receives the same two as Create the SDK client, and returns a client that evaluates flags locally, with its environment already loaded. Only @local-evaluation scenarios use it, so an SDK without local evaluation can leave it out.
- A When step receives the client, then the step's placeholders in order, then its data table. A <key> | value table arrives as a map, e.g. {"plan": "premium", "age": 30}.
- Values are JSON values. Pass flag and trait values through with their types intact.
- Steps that get flags return a list of {"feature": …, "enabled": …, "value": …}.
- Let SDK errors propagate: the scenario fails with the error.
Keep it to SDK calls. The steps file doubles as a working, CI-checked usage example.
Add sdk-spec/Dockerfile, which builds the SDK from source and runs the steps file under the language's harness. See harness/<language>/README.md. If the language has no harness yet, add one (see HARNESS.md).

Add a workflow:

name: SDK spec
on:
  pull_request:
  push:
    branches: [main]
jobs:
  sdk-spec:
    uses: Flagsmith/sdk-spec/.github/workflows/run-spec.yml@v0.3.0
    with:
      spec-ref: v0.3.0  # keep in sync with `uses:`
      language: python
      tags: core        # optional: select capabilities

Let Renovate bump both refs together:

{
  "customManagers": [
    {
      "customType": "regex",
      "managerFilePatterns": ["/^\\.github/workflows/sdk-spec\\.yml$/"],
      "matchStrings": ["spec-ref: (?<currentValue>v[\\d.]+)"],
      "depNameTemplate": "Flagsmith/sdk-spec",
      "datasourceTemplate": "github-tags"
    }
  ],
  "packageRules": [
    { "matchPackageNames": ["Flagsmith/sdk-spec"], "groupName": "SDK spec" }
  ]
}

Add the SDK to the matrix in compatibility.yml.

Running locally

# 1. Start Flagsmith
docker compose up --detach --wait

# 2. Build the SDK image, in the SDK repository
docker build -t sdk-spec-sdk -f sdk-spec/Dockerfile --build-context harness=<path to sdk-spec>/harness/python .

# 3. Run the spec
cd runner
FLAGSMITH_SDK_URL=http://flagsmith:8000 \
SDK_COMMAND="docker run -i --rm --network sdk-spec_default sdk-spec-sdk" \
uv run pytest -v

Variable	Default	Purpose
`SDK_COMMAND`	(required)	Starts the SDK under its harness
`MISSING_STEPS`	`fail`	`xfail` makes missing steps expected failures
`FLAGSMITH_URL`	`http://localhost:8000`	Where the runner reaches Flagsmith
`FLAGSMITH_SDK_URL`	`$FLAGSMITH_URL`	Where the SDK reaches Flagsmith
`FLAGSMITH_PORT`	`8000`	Host port for Flagsmith (`compose.yml`)
`FLAGSMITH_IMAGE`	pinned version	Override the Flagsmith image (`compose.yml`)
`EVENTS_SDK_URL`	`http://events:8080`	Where the SDK reaches the events API
`EVENTS_PORT`	`8080`	Host port for the events API (`compose.yml`)
`WAREHOUSE_URL`	`http://localhost:8123`	Where the runner reaches the warehouse
`WAREHOUSE_HOST`	`clickhouse`	Where Flagsmith reaches the warehouse
`CLICKHOUSE_PORT`	`8123`	Host port for the warehouse (`compose.yml`)

SDK_COMMAND can be any command, so you can also run a harness straight against an SDK checkout, without Docker, e.g. python ../harness/python/harness.py ../../flagsmith-python-client/sdk-spec/steps.py. An SDK running on the host reaches the services on the host's ports, so set FLAGSMITH_SDK_URL=$FLAGSMITH_URL and EVENTS_SDK_URL=http://localhost:8080 with it.

Implementing a new spec version

A spec version is implemented in three phases:

Spec update. A pull request to this repository is merged, and the new spec version is tagged by a maintainer (see Versioning).
Spec implementation. Renovate proposes the new version to every SDK onboarded in the compatibility matrix, as a bump pull request in the SDK's repository. A bump whose CI is red is fixed by implementing the spec in it. Fixed bumps are reviewed by the SDK's code owners and merged.
SDK releases. Maintainers release each SDK version that includes its merged bump.

A spec version is implemented once every onboarded SDK has released the new spec version.

Navigation Menu

Repository navigation

About

Repository files navigation

Flagsmith SDK behavioural spec

How it works

Writing scenarios

Versioning

Onboarding an SDK

Running locally

Implementing a new spec version

About

Releases

Packages

Contributors

Languages

Name		Name	Last commit message	Last commit date
.github		.github
features		features
harness		harness
runner		runner
.gitignore		.gitignore
AGENTS.md		AGENTS.md
CLAUDE.md		CLAUDE.md
HARNESS.md		HARNESS.md
README.md		README.md
compose.yml		compose.yml

Latest commit

History

Folders and files

Repository files navigation

Flagsmith SDK behavioural spec

How it works

Writing scenarios

Versioning

Onboarding an SDK

Running locally

Implementing a new spec version

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages