Reentrant provider events reach client callbacks out of order #332

Problem

Applications tracking provider availability through client event handlers can receive PROVIDER_READY followed by PROVIDER_ERROR after a provider has recovered. A similar sequence delivers PROVIDER_STALE before the preceding PROVIDER_READY.

This happens when a global event handler synchronously emits another provider event. The nested event reaches client handlers before the original event does. It reproduces with openfeature-sdk 0.5.1, 0.6.5, and current main at e602d1559af71324dcc11cc45462ba47075f32f9, using Ruby 4.0.1.

Related: #300 tracks provider lifecycle event ownership, and #137 tracks broader concurrency. This reproduction uses an already initialized provider and manually emitted events, so it also occurs independently of synthetic initialization events.

Reproduction

Save this as repro.rb and run it in an environment with openfeature-sdk installed. No Datadog dependency, network requests, or background threads are needed.

require "open_feature/sdk"

class EmittingProvider < OpenFeature::SDK::Provider::InMemoryProvider
  include OpenFeature::SDK::Provider::EventEmitter
end

configuration = OpenFeature::SDK.configuration
provider = EmittingProvider.new
configuration.set_provider_and_wait(provider)
provider.emit_event(OpenFeature::SDK::ProviderEvent::PROVIDER_STALE)
client = OpenFeature::SDK::Client.new(provider: provider)
events = []

first, following = (ARGV.first == "ready_stale") ? [:READY, :STALE] : [:ERROR, :READY]
first_event = OpenFeature::SDK::ProviderEvent.const_get("PROVIDER_#{first}")
following_event = OpenFeature::SDK::ProviderEvent.const_get("PROVIDER_#{following}")
configuration.add_handler(first_event, ->(_) { provider.emit_event(following_event) })
client.add_handler(first_event, ->(_) { events << first })
client.add_handler(following_event, ->(_) { events << following })
events.clear # Ignore immediate callbacks during handler registration.

provider.emit_event(first_event)
puts "Expected: #{[first, following].inspect}; actual: #{events.inspect}"

Proposed direction

In Configuration#dispatch_provider_event, state is updated before global handlers and then client handlers run. A global handler can recursively dispatch another event before the original client callbacks finish.

Finish delivery of the current event to global and client handlers before dispatching events emitted reentrantly on the same thread. Deferred events should preserve their emission order and payload, and must be invalidated when the associated provider registration ends through shutdown or replacement. This need not impose total ordering across independent threads.

A local fix and SDK-native regression tests are in preparation. The downstream report that led to this reproduction is DataDog/dd-trace-rb#6418.