Proposal: circuit breaker support

[qualidafial]

I saw your Reddit /r/golang post a week or so ago about this package, which piqued my interest.

I took a stab at a design and tried to keep it in line with the existing API design philosophy so far as I could tell.

If you are interested, I would be happy to do the leg work myself and contribute a PR--let me know!

---

try retries a fallible operation but has no way to stop hammering a downstream that is already failing systemically. A circuit breaker needs state shared across many Do calls (and across goroutines), which doesn't fit the existing Option -> Config model, since Config is rebuilt fresh per call. It should also work identically whether the caller uses the package-level try.Do or a *try.Try client.

Proposed API design

Public API:

type CircuitState int
const (
    // Closed means the circuit is closed and calls are allowed to pass through. The circuit will
    // transition to Open after a certain number of consecutive failures.
    Closed CircuitState = iota
    
    // Open means the circuit is open and calls are not allowed to pass through. The circuit will
    // transition to HalfOpen after a certain timeout.
    Open
    
    // HalfOpen means the circuit is half-open and allows a limited number of trial calls to pass
    // through.
    HalfOpen
)

// ErrCircuitOpen is a sentinel error returned when a call is attempted while the circuit breaker
// is in the Open state.
var ErrCircuitOpen = errors.New("try: circuit breaker open")

// CircuitBreaker is a concurrency-safe circuit breaker that tracks the state of a downstream
// service, and prevents further calls when the service is failing. It transitions between three
// states: Closed, Open, and HalfOpen, based on the number of consecutive successes or failures.
//
// Circuit breakers should be shared across all calls to a particular downstream service, but not
// across different services.
type CircuitBreaker struct { /* mutex-guarded state */ }

// NewCircuitBreaker creates a new CircuitBreaker with the provided options.
func NewCircuitBreaker(opts ...CircuitBreakerOption) *CircuitBreaker

// CircuitBreakerOption is a function that configures a CircuitBreaker.
type CircuitBreakerOption func(*CircuitBreaker)

// WithFailureThreshold sets how many consecutive failures in Closed state before tripping to Open.
// Default TBD.
func WithFailureThreshold(n int) CircuitBreakerOption

// WithOpenTimeout sets how long to stay Open before allowing a trial call (HalfOpen). Default TBD.
func WithOpenTimeout(d time.Duration) CircuitBreakerOption

// WithSuccessThreshold sets how many consecutive successful requests are required to reset the
// breaker from HalfOpen to Closed. Default TBD.
func WithSuccessThreshold(n int) CircuitBreakerOption

// WithOnStateChange registers a callback that is invoked whenever the circuit breaker transitions
// between states. This can be used for observability or logging purposes.
func WithOnStateChange(func(from, to CircuitState)) CircuitBreakerOption // parallel to WithOnRetry

// WithCircuitBreaker uses the provided circuit breaker to short circuit calls when the downstream
// service is failing.
func WithCircuitBreaker(cb *CircuitBreaker) Option

Internal API:

CircuitBreaker methods share the same clock as try.Do / try.Try for time-dependent methods:

// allow returns whether a call is currently allowed based on the current state of the circuit
// breaker. Transitions from Open to HalfOpen if the timeout has elapsed. While in HalfOpen state,
// admits only one in-flight trial call at a time to avoid recreating a thundering herd against a
// downstream that's still recovering. Concurrent callers that hit allow while a trial is already in
// flight are rejected with ErrCircuitOpen until that trial completes.
func (cb *CircuitBreaker) allow(clock Clock) error

// recordSuccess records a successful call. Resets the failure count to zero. Transitions from
// HalfOpen to Closed if SuccessThreshold has been met.
func (cb *CircuitBreaker) recordSuccess(clock Clock)

// recordFailure records a failed call. Resets the success count to zero. Transitions from Closed to
// Open if FailureThreshold has been met. Transitions from HalfOpen to Open on any failure.
func (cb *CircuitBreaker) recordFailure(clock Clock)

Expected behavior

The caller constructs one CircuitBreaker per downstream dependency, and hands it to try via the WithCircuitBreaker option. This mirrors how *try.Try is already used today (one instance per service dependency, shared across call sites/goroutines).

All mutation happens behind a mutex/atomics internally; *CircuitBreaker is safe for concurrent use, same guarantee *Try already gives.

An error only counts against the breaker if try itself would have retried it:

• Permanent errors, predicate-rejected errors, exhausted per-error budgets, and a parent context that's done (via ctx.Err()) are already non-retryable and so don't count as breaker failures either.
• A per-attempt timeout from WithTimeout (child-context context.DeadlineExceeded, parent still alive) is retryable by default, so it does count as a breaker failure.
• If a caller supplies WithRetryIf, that predicate becomes the breaker's failure definition too -- one policy, no divergence between "should I retry this" and "does this count against the circuit."

The breaker transitions between states based on the number of consecutive successes or failures, and it automatically transitions from Open to HalfOpen after the specified timeout.

WithCircuitBreaker composes naturally with everything that exists today:

• try.Do(ctx, fn, try.WithCircuitBreaker(cb), try.WithAttempts(3))
• try.New(try.WithCircuitBreaker(cb), ...).Do(ctx, fn)

Integration in the retry loop

If a circuit breaker is configured:

• At the top of each attempt (just after ctx.Err() is checked), call err := cb.allow(cfg.Clock):
  • If non-nil (ErrCircuitOpen), return it immediately -- no more attempts, no backoff wait -- wrapping lastErr if one exists, same convention as cancelledErr.
  • If nil, drives Open -> HalfOpen transition automatically once OpenTimeout has elapsed.
  • Returns ErrCircuitOpen if the circuit is HalfOpen and a trial call is already in flight.
• If the attempt succeeds (err == nil), call cb.recordSuccess(cfg.Clock).
  • Resets the failure count to zero.
  • Drives HalfOpen -> Closed transition upon reaching SuccessThreshold consecutive successful attempts.
• If it fails, and the attempt is retryable (shouldRetry), call cb.recordFailure(cfg.Clock).
  • Resets the success count to zero.
  • Drives HalfOpen -> Open transition on any failure.
  • Drives Closed -> Open transition upon reaching FailureThreshold consecutive failed attempts.

Open questions

• WithAttemptsForError interaction: once a specific error's per-error budget is exhausted, shouldRetry starts returning false for it -- meaning further occurrences of that same error stop counting toward the breaker too, even though the downstream may still genuinely be failing. Acceptable, or does breaker accounting need to be independent of per-error retry budgets?
• Should RetryAfterer errors be counted against the breaker? Leaning no, because a rate limit response suggests the server is still healthy, but the client needs to slow down.
• Should WithDelayFunc be allowed to override the default backoff for trial calls in HalfOpen? Leaning no, since the point of the breaker is to avoid hammering a downstream that's still recovering. But may be worth taking the max of the two, since a downstream may have its own rate limiting that should be respected.
• Should using WithDelayFunc change whether errors are counted against the breaker? Leaning no--we can't distinguish whether an error is a rate limit or whether we're just providing a custom backoff strategy. If clients need to avoid counting rate limiting against the breaker, they should wrap errors in a RetryAfterer.
• Default success/failure thresholds and open timeout: what are reasonable defaults for a generic downstream service? Should these be NewCircuitBreaker constructor arguments instead of options?
• Is the consecutive successful/failed requests policy naive? Do we need to think about error rates and time windows?

[nodivbyzero]

This is a really solid writeup!!! Thanks for putting in the effort.

• API shape LGTM. Just make sure ErrCircuitOpen plays nicely with the existing error conventions (errors.Is, and IsPermanent returns false for it).
• RetryAfterer errors shouldn't count against the breaker. Keep the breaker and WithDelayFunc decoupled in HalfOpen. Per-error budgets tying into shouldRetry is fine for v1, just document it.
• I'd lean toward reasonable option defaults rather than required constructor args, consistent with how the rest of the API works today.
• Consecutive is fine for a first pass - we can revisit if it proves too naive in practice.

Go ahead and open a PR whenever you're ready.

[qualidafial]

I'm still wondering whether the consecutive success/failure options are naive, and have been looking at different circuit breaker frameworks to see what options they offer--specifically sony/gobreaker, Netflix Hystrix, and Resilience4j.

My goal is just to make sure that the design is future-proofed enough that we can add more options without breaking existing clients.

Closed -> Open (trip) policies:

• sony/gobreaker
  • Defaults to tripping after 5 consecutive failures
  • Configurable via ReadyToTrip(func(Counts)) bool config parameter
• Netflix Hystrix
  • Sliding time-based window stats rotating on an interval. Defaults to 10s window divided into 10 buckets (1s each)
  • Error rate above threshold (default 50%)
  • Gated by minimum number of calls (default 20) in rolling time window
• Resilience4j
  • Sliding window stats can be time-based on an interval, or count-based on a fixed number of calls. Defaults to 100 calls. If time-based, default is interpreted as 100 seconds--which is an ...interesting choice.
  • Failure rate > failure threshold % OR slow call rate > slow threshold % (tracked separately)
  • Gated by minimum number of calls in window. Defaults to 100 calls.

Open -> Half-Open policies:

• sony/gobreaker: timeout after trip (default 60s)
• Hystrix: timeout after trip (default 5s)
• Resilience4j: timeout after trip, supports exponential backoff via function

Half-Open -> Closed (reset) policies:

• sony/gobreaker: N consecutive successes 100% success rate across N
• Hystrix: 1 successful request
• Resilience4j: allow N trial calls, closes if failure / slow call rates stay below thresholds, otherwise reopens.

The big weakness of consecutive-only requests is that they have no notion of elapsed time. Consider a client that is used for an hourly cron job: if the previous job tripped the breaker, it should be automatically reset by the time the job runs again an hour later.

With that in mind, I think we need a slightly more mature design:

• Use a bounded time window with bucketed call metrics, so that stale calls naturally age out: WithSlidingTimeWindow(buckets int, bucketDuration time.Duration).
  • Default 10s window over 10 1-second buckets
• Trip the breaker on the basis of error rate rather than consecutive failures: WithFailureRateThreshold(rate double)
  • Default 0.5 (50%)
• Require a minimum number of calls before tripping: WithMinimumCalls(calls int)
  • Default 20

If in the future we need something more flexible than this, we have options:

• Expose Counts struct / interface so that trip logic can be pluggable.
• Add WithShouldTripFunc(func(Counts) bool) option. Specify that the default is to trip when total calls >= minimum calls, and error rate is above failure threshold.

[nodivbyzero]

Great research!!! I really want to go with the sliding time window approach instead of what I originally sketched, for a few concrete reasons beyond just the cron example:

• Time-decay: consecutive counters never age out. A service that fails 5 times, then goes quiet for an hour, then gets called again is still sitting right at the trip threshold with no notion that an hour has passed. A time-bucketed window naturally forgets old data.
• Rate vs. streak blindness: consecutive counting resets to zero on any single success, so a service failing 1 in 3 calls forever(a real 33% error rate. Never trips the breaker, because it never strings together enough consecutive failures. A rate-based window catches sustained partial degradation that consecutive counting is structurally blind to.
• Load sensitivity: a fixed consecutive threshold (say, 5) is too twitchy for a high-QPS service (5 failures can happen in a normal blip) and too slow for a low-QPS one (5 consecutive calls might take minutes to accumulate). Rate-based tripping with a minimum-calls gate scales better across traffic levels.
• Noise protection: the minimum-calls gate specifically guards against tripping on a tiny, statistically meaningless sample, like 2 failures out of 2 calls right after startup.

A few thoughts on the concrete design:

• Agreed on WithSlidingTimeWindow(buckets, bucketDuration) with a 10x1s default, WithFailureRateThreshold(rate) defaulting to 0.5, and WithMinimumCalls(calls) defaulting to 20. That maps pretty cleanly onto Hystrix/Resilience4j's model without dragging in their full complexity.
• For future-proofing, let's go ahead and expose the Counts struct now rather than waiting -- something like Counts{Requests, TotalFailures, ConsecutiveFailures, ConsecutiveSuccesses} (gobreaker's shape is a reasonable reference). Adding WithShouldTripFunc(func(Counts) bool) at the same time costs little and immediately gives escape-hatch flexibility, so let's ship both together instead of treating the func-based override as a v2 addition.
• Bucket rotation needs to key off cfg.Clock like everything else, so fake-clock tests stay deterministic -- worth calling out explicitly in the design doc so it doesn't get missed in review.
• One thing to pin down: does a bucket roll over lazily (only on the next allow/recordSuccess/recordFailure call) or does it need a background ticker? Lazy rollover keeps this allocation-free and avoids a goroutine per breaker which fits the library's style so far but worth confirming that's your intent too.
• For Half-Open, I'd keep it simple for now: N consecutive successes to close, any failure reopens. We can add a trial-rate-based variant later behind the same Counts/ShouldTripFunc mechanism if it turns out to be needed. No need to design that in from day one.

[qualidafial]

One thing to pin down: does a bucket roll over lazily (only on the next allow/recordSuccess/recordFailure call) or does it need a background ticker? Lazy rollover keeps this allocation-free and avoids a goroutine per breaker which fits the library's style so far but worth confirming that's your intent too.

I was thinking about this too and was leaning toward lazy rollover. This means:

• Each bucket will have to keep track of its start time so we have a reference to age out old buckets.
• Every bucket interval incurs a small time performance penalty on the first call (and any other concurrent calls), since all callers will have to wait for the mutex while we prepare the new bucket and age out the old ones. I think that complexity is worth sparing callers the responsibility of calling Close / Stop on the circuit breaker. I'll write a benchmark to quantify that impact under different loads.

[qualidafial]

Just as a follow up, I still plan to do this, but won't have time to get to this for about 3 weeks.