Skip to content

fix(api): do not log client cancellation as a sandbox create failure - #3684

Open
AdaAibaby wants to merge 1 commit into
e2b-dev:mainfrom
AdaAibaby:fix/api-client-cancel-logging
Open

AdaAibaby wants to merge 1 commit into
e2b-dev:mainfrom
AdaAibaby:fix/api-client-cancel-logging

Conversation

@AdaAibaby

@AdaAibaby AdaAibaby commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Fixes #3683.

When a client disconnects (or its SDK gives up) before the API finishes creating a sandbox, the request context is cancelled. Because the context is threaded end-to-end through placement, the in-flight create/reserve is cut and surfaces as a generic failure. Two hot paths in packages/api treated this expected context.Canceled as a server-side failure — they logged at ERROR and (in placement) incremented the node's createFails counter and excluded it for the rest of the attempt.

The access-log/metrics middleware already rewrites the HTTP status to 499 by cancel cause, yet the handler's own ERROR logs and telemetry spans still recorded these as 500, so a single client cancellation was recorded twice with two different codes. This is the same class of issue already addressed for envd (#3647) and NBD dispatch (#3274, #3348), on the API create path.

Changes

  • Placement loop (placement/placement.go): a create cut by its own context now Skips the node instead of Failing it, logs at Warn instead of ERROR, and does not exclude the node — a client disconnect says nothing about the node's health. The attempt is still counted so deadline() classifies the timeout correctly, and the loop still ends with the caller (returns deadline()). Genuine live-context failures are unchanged: they still Fail, exclude, and log at ERROR.
  • Reserve path (create_instance.go): a reservation cut by context.Canceled / context.DeadlineExceeded now logs at Warn instead of ERROR. Mirrors the existing guards in outbox/river.go, orchestrator/autoresume.go, handlers/proxy_grpc.go, and featureflags/flags.go.

Client-facing behaviour is unchanged (still 499/408 via the middleware); this only stops the duplicate ERROR logs and the createFails inflation driven by client behaviour. Scheduling was already unaffected (best-of-K scores on InProgress(), which is cleaned up by both Skip and Fail).

Tests

  • Added TestPlaceSandbox_CutCreateNotCountedAgainstNode: a create that cancels the request context mid-flight must report a timeout, keep FailsCount at zero, and drop its in-progress reservation.
  • Strengthened TestPlaceSandbox_HardFailureNotWrapped to assert a live-context failure still counts against the node (FailsCount == 1), guarding the two branches against regressing into each other.
  • go test ./internal/orchestrator/placement/... passes. go vet and golangci-lint clean on the changed packages. (The DB-backed orchestrator tests require PG_TEST_DATABASE_URL, unavailable in my env; the paths changed here have no DB dependency.)

A client disconnect (or its SDK giving up) cancels the request context,
which is threaded end-to-end through placement. Two create paths treated
the resulting context.Canceled as a server-side failure: they logged at
ERROR and, in placement, counted the node's createFails and excluded it.

- placement: a create cut by its own context now Skips (not Fails) the
  node, logs at Warn, and does not exclude it — the cut says nothing about
  the node's health. The attempt is still counted so deadline() classifies
  the timeout, and the loop still ends with the caller.
- create_instance reserve: a reservation cut by context.Canceled /
  DeadlineExceeded now logs at Warn instead of ERROR.

The access-log/metrics middleware already maps these requests to 499/408
by cancel cause, so client-facing behaviour is unchanged; this only stops
the duplicate ERROR logs and inflated createFails driven by client
behaviour.

Fixes e2b-dev#3683

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

api: client cancellation during sandbox placement/reserve logged at ERROR and counted as create failure

2 participants