Skip to content

Hangs on HTTP requests bodies larger than one tsnet MTU #11

Description

@myoung34

Summary

ts-unplug reverse-proxies fine for small requests, but any request whose body
exceeds roughly one tsnet path-MTU (~1200 bytes of HTTP body) hangs forever
— the
proxy never returns response headers and the client eventually times out. Small
requests through the exact same proxy, tsnet node, and backend succeed in <1 s.

This makes ts-unplug unusable as a proxy for real API clients. Concretely, it
cannot proxy an LLM gateway for the pi coding agent (and
by extension the Anthropic/OpenAI SDKs): every chat request carries a system prompt
plus tool definitions (>1.2 KB), so every request hangs. From the user's side
pi just sits on "Working…" and then reports Operation aborted (its client-side
timeout firing), having received zero bytes.

Environment

  • ts-unplug from github.com/tailscale/ts-plug@v0.0.0-20260627053921-1937455808f0/cmd/ts-unplug/ts-unplug.go
  • Built with go1.26.8, embeds tailscale.com@v1.94.2
  • Host: WSL2 (5.15.167.4-microsoft-standard-WSL2, Ubuntu 24.04), userspace tsnet node
  • tsnet reports a 1360-byte peer MTU to the backend and upgrades to a direct
    connection: magicsock: disco: node ... now using <backend>:16573 mtu=1360
  • Invocation: ts-unplug -dir <dir> -port 8899 <backend-tailscale-ip>

Reproduction

Point ts-unplug at any tailnet HTTP backend and send POST bodies of increasing
size. Everything below goes to the same backend; only the request-body size varies.

probe() {
  pad=$(head -c "$1" /dev/zero | tr '\0' x)
  # build a JSON body of a known size
  body=$(printf '{"pad":"%s"}' "$pad")
  printf 'body=%dB  ->  ' "${#body}"
  timeout 12 curl -s -o /dev/null -w '%{http_code}\n' \
    -H 'content-type: application/json' \
    --data "$body" \
    http://127.0.0.1:8899/<any-endpoint> || echo TIMEOUT
}
for n in 900 1000 1100 1200 1300 1400 2000 4000; do probe "$n"; done

Observed (through ts-unplug)

body= 1064B  ->  200
body= 1164B  ->  200
body= 1214B  ->  TIMEOUT   <-- hangs, no response headers ever arrive
body= 1264B  ->  TIMEOUT
body= 1364B  ->  TIMEOUT
body= 2174B  ->  TIMEOUT
body= 4174B  ->  TIMEOUT

The cliff sits at ~1.2 KB of body — i.e. once the request no longer fits in a
single tsnet MTU (1360) segment.

Control: same large bodies straight to the backend (bypassing ts-unplug)

Sending the identical large bodies directly to the backend over the host's own
Tailscale (kernel WireGuard) path succeeds every time:

body=  1674B  ->  200  0.99s
body=  2174B  ->  200  0.76s
body=  4174B  ->  200  0.69s
body= 20000B  ->  200  0.93s

So the backend and the tailnet are fine; the failure is exclusively in
ts-unplug's userspace tsnet reverse-proxy path
for multi-segment request bodies.

Real-world impact (the pi case)

pi's Anthropic-style request captured off the wire is POST /v1/messages?beta=true
with a 22,835-byte body (system prompt + 4 tool schemas). Through ts-unplug
the proxy logs the inbound request but never produces a response:

REQ  POST /v1/messages          <- request reaches the proxy
(no RESP — upstream never returns headers)
http: proxy error: context canceled   <- 30 s later, client gives up

The same request sent directly to the backend returns a normal streaming
text/event-stream 200 in ~1 s. Stripping the body down to <1.2 KB is the only way
to get a response through ts-unplug, which is obviously not viable for a real
client.

Likely cause

cmd/ts-unplug/ts-unplug.go builds a stock
httputil.NewSingleHostReverseProxy over a tsnet.Server. The hang correlates
exactly with the tsnet peer MTU (1360). This looks like a path-MTU / segmentation
issue in the userspace (gVisor/netstack) datapath: requests that fit in one segment
pass; anything that must be split stalls. tsnet logs also show
magicsock: peermtu: updating peer MTU status ... failed ... disabling.

What I tried

  • Forcing identity content-encoding (ruled out response gzip as the cause).
  • TS_DEBUG_MTU=1280 — could not get a clean run to confirm; a documented, working
    way to clamp the tsnet MTU (or fix PMTUD) from ts-unplug would help.

Ask

  1. Confirm whether this is a tsnet PMTU/segmentation bug or a ts-unplug datapath
    configuration gap.
  2. Expose a flag/env to clamp the tsnet MTU (e.g. 1280) from ts-unplug, and/or fix
    large-body handling so it works out of the box.

Happy to provide full -debug-tsnet netmap/packet-filter captures on request.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions