[gagantrivedi]

Specs the three experimentation methods from the issue, as a new @events capability.

What the spec now covers

Feature file	When step	SDK method
features/get_experiment_flag.feature	the application gets the experiment flag "{feature}" for identity "{identifier}" with traits:	getExperimentFlag
features/track_event.feature	the application tracks event "{event}" for identity "{identifier}" with value {value} and metadata:	trackEvent
features/track_exposure_event.feature	the application tracks an exposure of feature "{feature}" for identity "{identifier}" with value {value} and metadata:	trackExposureEvent

getExperimentFlag gets three scenarios, because it is two behaviours in one call. It
resolves the feature for the identity and persists its traits. It records an exposure
carrying the variant the identity was served, when that identity is in a running
experiment. And it records nothing when the identity is not — the behaviour that separates
it from getIdentityFlags, and the one the reference implementations go out of their way
to get right.

The events path

compose.yml brings up the experimentation services alongside Flagsmith, mirroring
Flagsmith's own docker-compose.yml, so @events scenarios run against the real path
rather than a stand-in for it:

SDK ─► events (flagsmith/events-ingestion-api) ─► kafka ─► warehouse-delivery ─► clickhouse ─► runner

• Flagsmith is pinned to 2.287.0, up from 2.276.0. That is the first release shipping
  experimentation/warehouses/clickhouse.py, the experimentation_environment_keys and
  experimentation_delivery_connections db views that ingestion and delivery read, and
  experimentation/clickhouse/events.sql. EXPERIMENTATION_ENABLED is on, and
  WAREHOUSE_CREDENTIALS_SECRET is shared with warehouse-delivery.
• Given the SDK sends events to Flagsmith connects a ClickHouse warehouse to the
  scenario's environment (POST /environments/<key>/warehouse-connections/) and asks for a
  client with events enabled. Both are needed: events are off by default in every SDK, and
  the events API only accepts an environment that has a warehouse connection.
• The runner reads delivered events out of ClickHouse (runner/sdk_spec/warehouse.py),
  keyed by the environment's API key — the same key Flagsmith reads them by in
  WarehouseConnectionViewSet.events.
• Given a running experiment on feature … serving variant … with value … creates a
  multivariate option keyed by the variant, a rollout enrolling every identity and
  allocating all of them to that option, and starts the experiment. One variant at 100%
  makes which variant an identity is served deterministic, so the exposure's value is too.

Because the events client is a separate step, Create the SDK client with events enabled,
an SDK that has not implemented it reports it missing and the scenarios behave like any
other new step: xfail in the compatibility matrix, red in SDK repositories until they add
it. Every harness's marker regex was broadened from exactly Create the SDK client to any
marker starting with it; existing steps files are unaffected, and harness/rust and
harness/swift already infer "this step creates a client" from the absence of a client
parameter, so both creation steps work there unchanged.

What a reviewer should check

• The waits. An event crosses an SDK's send timer, Kafka, and a delivery batch, so
  Flagsmith has received the following events: polls for 60s and Flagsmith has received no events waits 30s. warehouse-delivery runs with BATCH_MAX_WAIT_MS: 1000 to keep
  that bounded. A full green run takes 80s; the negative can only produce false passes,
  never false failures.
• value is asserted as text because all three reference SDKs stringify it, and
  metadata is matched on the entries the table names, because SDKs add their own and the
  ingestion server lifts sdk_version into its own column.
• clickhouse-init fetches events.sql from the raw URL pinned to v2.287.0, the way
  Flagsmith's compose does from main. Vendoring the schema here instead is an easy swap.
• EVENTS_SDK_URL defaults to http://events:8080, the compose service, rather than
  being derived from another URL. That keeps run-spec.yml unchanged, so SDK repositories
  pick events up by bumping their pinned ref. Running a harness on the host needs
  EVENTS_SDK_URL=http://localhost:8080, which the README says.

Additive for SDKs, so this is a minor version: new scenarios, a new tag, new When steps
and a new client-creating step, with no existing step reworded. The pinned Flagsmith
version moves, as a spec release's pin is expected to.

Testing

The whole stack was brought up and the spec run against flagsmith-python-client with the
three steps and Create the SDK client with events enabled added to its steps file. All
seven scenarios pass, in 80s.

To check the assertions are not passing vacuously, the same run was repeated against
deliberately broken steps files. An SDK that sends "mutant" instead of the tracked value
fails A tracked event reaches Flagsmith. One that sends an exposure for an identity
outside an experiment fails An identity outside a running experiment is not exposed. One
that resolves the flag through getIdentityFlags, recording nothing, fails An identity in a running experiment is exposed to the variant it was served. Each time, only the scenario
that should fail does.

CI confirms the stack comes up and the @core scenarios still pass against Flagsmith
2.287.0. It cannot exercise the @events assertions, because every SDK still xfails the
events client; those are covered by the local run above, and will run in CI from whichever
SDK implements the step first.

Part of #13

This pull request was opened by an AI agent (OpenHands).

[khvn26]

One question.

[flagsmithsmith]

I went ahead and added it, in 0cd8cc9 — it turned out to be straightforward, and leaving the spec's headline behaviour untested through a tag felt worse than the extra review. Drop the commit if you'd rather ship this as you reviewed it; nothing else depends on it.

Given a running experiment on feature "banner" serving variant "treatment" with value "hello treatment" creates a multivariate option keyed by the variant, a rollout enrolling every identity and allocating all of them to it, then starts the experiment. One variant at 100% means every identity is served the same one, so the exposure's value is deterministic.

  Scenario: An identity in a running experiment is exposed to the variant it was served
    Given the environment has the following features:
      | feature | enabled | value     |
      | banner  | true    | "control" |
    And a running experiment on feature "banner" serving variant "treatment" with value "hello treatment"
    When the application gets the experiment flag "banner" for identity "user_123" with traits:
      | trait | value     |
      | plan  | "premium" |
    Then the application receives the following flags:
      | feature | enabled | value             |
      | banner  | true    | "hello treatment" |
    And Flagsmith has received the following events:
      | event          | feature | identifier | value       | traits              |
      | $flag_exposure | banner  | user_123   | "treatment" | {"plan": "premium"} |

Seven scenarios pass against flagsmith-python-client in 80s. A steps file that resolves the flag through getIdentityFlags instead — right flag, no exposure — fails this scenario and only this scenario:

AssertionError: Flagsmith received []
assert 0 == 1

FAILED tests/test_spec.py::test_an_identity_in_a_running_experiment_is_exposed_to_the_variant_it_was_served
1 failed, 6 passed

Two notes from building it:

• The exposure's metadata carries experiment_id, which a feature file can't name because it's a database id. The table asserts traits instead, which is the part an SDK can get wrong.
• PATCH /projects/<id>/features/<id>/mv-options/<id>/ 500s on a partial body — MultivariateFeatureOptionSerializer.validate reads attrs["feature"] and attrs["default_percentage_allocation"] unconditionally. Nothing to do here, the runner creates the option with its key in one POST, but it may be worth an issue on Flagsmith/flagsmith.

The PR description is updated to match.

— OpenHands agent