You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
api/shared: context.Canceled classified as client-closed (499) inconsistently and misattributes server-initiated cancellation #3685
The HTTP middleware decides a request was "client closed" (499) purely from errors.Is(cancelCause, context.Canceled). This has two problems.
1. Server-initiated cancellation is misattributed to the client
context.Canceled does not mean "the client closed the connection" — it only
means some context in the chain was cancelled. The API server roots every
request at a base context (packages/api/main.go:260-266, BaseContext: func(net.Listener) context.Context { return ctx }) whose cancel() fires on process teardown (main.go:273-274). When that happens,
in-flight requests see plain context.Canceled — not ErrRequestTimeout —
and are logged/recorded as 499 client closed even though the server
cancelled them. During a shutdown/drain window this mislabels our own
teardown as client behaviour.
The timeout path already has a dedicated cause (ErrRequestTimeout, mapped to
408 in timeout.go); server-initiated cancellation has none, so it falls into
the context.Canceled catch-all.
2. The 499 reclassification is applied inconsistently
The 499 rewrite happens in three middleware sites, all keyed off the same context.Canceled check, but only on the summary surfaces:
Handler-level error reporting that runs insidec.Next() is never
reconciled with this. telemetry.ReportErrorByCode with a 5xx code calls ReportCriticalError (packages/shared/pkg/telemetry/tracing.go:94-99, 64-80),
which emits an ERROR-level zap log and a span.RecordError exception event
with the original 5xx framing. The middleware later flips the span status to
499, but the ERROR log and the recorded exception event remain. So one
client/shutdown cancellation surfaces as a 499 on the summary surfaces and a
5xx ERROR + exception event underneath.
(#3683 / #3684 fixed the sandbox-create paths by not logging cancellation at
ERROR in the first place. This issue is the general case: the middleware layer
should classify cancellation consistently rather than relying on every handler
to special-case it.)
Expected behavior
Distinguish server-initiated cancellation (base-context cancel, drain) from
client disconnect, the way ErrRequestTimeout is already distinguished —
e.g. a dedicated cancel cause set when the server cancels, so it is not
reported as 499 client-closed.
Apply the cancellation classification consistently: if a request is
reclassified to 499, the handler-level ERROR log and the span exception
event for the same request should not simultaneously present it as an
unqualified 5xx.
Impact
Shutdown/drain windows inflate "client closed" (499) counts with the
server own teardown.
5xx ERROR logs and span exception events are emitted for requests the
summary metrics/spans count as 499, double-counting cancellations and
muddying 5xx-based alerting.
Activity