Skip to content

api/shared: context.Canceled classified as client-closed (499) inconsistently and misattributes server-initiated cancellation #3685

Open

Description

@AdaAibaby

Describe the bug

The HTTP middleware decides a request was "client closed" (499) purely from
errors.Is(cancelCause, context.Canceled). This has two problems.

1. Server-initiated cancellation is misattributed to the client

context.Canceled does not mean "the client closed the connection" — it only
means some context in the chain was cancelled. The API server roots every
request at a base context (packages/api/main.go:260-266,
BaseContext: func(net.Listener) context.Context { return ctx }) whose
cancel() fires on process teardown (main.go:273-274). When that happens,
in-flight requests see plain context.Canceled — not ErrRequestTimeout —
and are logged/recorded as 499 client closed even though the server
cancelled them. During a shutdown/drain window this mislabels our own
teardown as client behaviour.

The timeout path already has a dedicated cause (ErrRequestTimeout, mapped to
408 in timeout.go); server-initiated cancellation has none, so it falls into
the context.Canceled catch-all.

2. The 499 reclassification is applied inconsistently

The 499 rewrite happens in three middleware sites, all keyed off the same
context.Canceled check, but only on the summary surfaces:

  • access log: packages/shared/pkg/middleware/logging.go:87-88
  • metrics: packages/shared/pkg/middleware/otel/metrics/middleware.go:89-91
  • span status: packages/shared/pkg/middleware/otel/tracing/middleware.go:127-135

Handler-level error reporting that runs inside c.Next() is never
reconciled with this. telemetry.ReportErrorByCode with a 5xx code calls
ReportCriticalError (packages/shared/pkg/telemetry/tracing.go:94-99, 64-80),
which emits an ERROR-level zap log and a span.RecordError exception event
with the original 5xx framing. The middleware later flips the span status to
499, but the ERROR log and the recorded exception event remain. So one
client/shutdown cancellation surfaces as a 499 on the summary surfaces and a
5xx ERROR + exception event underneath.

(#3683 / #3684 fixed the sandbox-create paths by not logging cancellation at
ERROR in the first place. This issue is the general case: the middleware layer
should classify cancellation consistently rather than relying on every handler
to special-case it.)

Expected behavior

  • Distinguish server-initiated cancellation (base-context cancel, drain) from
    client disconnect, the way ErrRequestTimeout is already distinguished —
    e.g. a dedicated cancel cause set when the server cancels, so it is not
    reported as 499 client-closed.
  • Apply the cancellation classification consistently: if a request is
    reclassified to 499, the handler-level ERROR log and the span exception
    event for the same request should not simultaneously present it as an
    unqualified 5xx.

Impact

  • Shutdown/drain windows inflate "client closed" (499) counts with the
    server own teardown.
  • 5xx ERROR logs and span exception events are emitted for requests the
    summary metrics/spans count as 499, double-counting cancellations and
    muddying 5xx-based alerting.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions