api-proxy

A lightweight service registry and streaming relay for internal microservices. Services self-register on startup and send a heartbeat every 60 seconds. Callers relay requests through the proxy without knowing the upstream IP or port.

Service  ──register──►  api-proxy registry
Caller   ──relay─────►  api-proxy  ──stream──►  Service

The proxy picks a healthy instance, injects the stored x-api-key, echoes or mints an x-request-id, and streams the upstream response back byte-for-byte. It also transparently forwards WebSocket and other HTTP Upgrade connections (see WebSocket / upgrade forwarding). Instances that stop heartbeating are evicted after 90 seconds.

By default a service must be directly reachable from the proxy (shared VPC, Fly private networking, same LAN). For a service behind NAT with no inbound port at all, run the client in tunnel mode instead — it dials out to the proxy, so nothing needs to reach it.

How it compares

api-proxy sits between a few categories of tool and borrows a bit from each — but the combination (registry + streaming proxy + optional NAT tunnel, one small binary, one control plane) is the point.

Tool	Self-registering registry	Streaming HTTP/WS proxy	NAT tunnel	Footprint
api-proxy	✅ heartbeat + auto-evict	✅ subdomain-routed	✅ optional	one Rust binary
nginx / Traefik	❌ needs external discovery	✅	❌	mature, but you wire discovery yourself
Consul + Envoy	✅	via Envoy	❌	full service mesh, several moving parts
frp / bore	❌	❌ raw TCP, no HTTP routing	✅	tunnel-only, no service semantics
ngrok / Cloudflare Tunnel	❌	✅	✅ hosted	SaaS — not self-hosted, subscription

Rough rule of thumb: reach for nginx/Traefik if you already have service discovery solved and just need a router; reach for Consul+Envoy if you need a real mesh (retries, circuit breaking, mTLS everywhere); reach for frp/bore/ngrok if all you need is a raw tunnel. Reach for api-proxy when you want registry + routing + (optionally) NAT tunneling without standing up multiple systems — most useful for small fleets of internal services, GPU workers that come and go, or dev/staging environments where the alternative is hand-maintained IP lists.

Quick start

# Start the proxy (auth disabled). Subdomains are resolved against --base-domain.
cargo run --bin proxy server --base-domain proxy.localhost

# Register a service from another terminal
curl -s -X POST http://proxy.localhost:8080/proxy \
  -H 'content-type: application/json' \
  -d '{"jsonrpc":"2.0","method":"register","params":{"service":"my-svc","ip":"127.0.0.1","port":9000},"id":1}'

# Relay a request through the proxy (streaming) — service is the subdomain, not the path
curl -s http://my-svc.proxy.localhost:8080/health
curl -s -X POST http://my-svc.proxy.localhost:8080/rpc \
  -H 'content-type: application/json' \
  -d '{"key":"value"}'

my-svc.proxy.localhost resolves to 127.0.0.1 out of the box on most systems (any *.localhost name does per RFC 6761). In production, point a wildcard DNS record (*.your-domain.com) and a wildcard TLS cert at the proxy, and pass the apex as --base-domain your-domain.com.

Routes

Method	Path	Description
GET	/proxy/health	Proxy liveness check — {"ok": true}
POST	/proxy	JSON-RPC 2.0 control plane (register, list, query)
ANY	/*path	Streaming relay to a healthy instance, service picked by Host subdomain (HTTP + WebSocket/upgrade)

Reserved paths: /proxy and /proxy/health are matched before the relay wildcard on every host, including service subdomains. If an upstream service exposes a route at exactly /proxy or /proxy/health, it is not reachable through the relay — rename the upstream route or mount it under a different path.

Host required: a request whose Host header isn't <label>.<base_domain> (exactly one label, matching a registered service or not) gets 400 Bad Request before any registry lookup happens.

API — JSON-RPC 2.0 (POST /proxy)

The control-plane uses standard JSON-RPC 2.0:

{ "jsonrpc": "2.0", "method": "<method>", "params": { ... }, "id": 1 }

Responses follow the same envelope with either a result or an error field.

register

Called by services on startup and every 60 s as a heartbeat.

{
  "service": "my-svc",
  "ip":      "10.0.1.5",
  "port":    8080,
  "api_key": "optional-key-forwarded-to-upstream"
}

• Re-registering the same ip:port updates last_seen (heartbeat) without consuming a slot.
• Maximum 10 distinct instances per service name.
• Requires x-proxy-auth header when PROXY_SECRET is set (see Auth).
• A service name cannot be registered as both push and tunnel — the second kind is rejected.

register_tunnel

Called by tunnel-mode clients on startup and every 60 s as a heartbeat. Unlike register, no ip/port is stored — the data path comes from the raw tunnel socket pool the client builds separately by dialing the proxy's tunnel port.

{
  "service": "my-svc",
  "client_id": "a1b2c3d4-..."
}

• Re-registering the same client_id updates last_seen without consuming a slot.
• Maximum 10 distinct tunnel clients per service name.
• Requires x-proxy-auth header when PROXY_SECRET is set.

list_services

Returns all registered services and their instances.

{}

get_service

Returns instances for a single service.

{ "service": "my-svc" }

Relay (<service>.<base_domain>/*path)

Relay calls go directly to the upstream instance — no JSON-RPC envelope. The service is selected by the request's Host header, not the path. The proxy:

1. Reads service from the Host header (<service>.<base_domain>) — 400 if it doesn't match
2. Looks up healthy instances registered under service
3. Picks one (random, or pinned by X-Routing-Key)
4. Forwards the request with the original HTTP method, body, and query string
5. Streams the upstream response back byte-for-byte (no buffering)

Relay requests do not require x-proxy-auth. The path and query string are forwarded verbatim, so GET my-svc.proxy.example.com/items?page=2&limit=50 reaches the upstream as GET /items?page=2&limit=50.

Sticky sessions (X-Routing-Key)

Pass X-Routing-Key: <key> to pin all requests with the same key to one instance. The affinity entry has a 10-minute sliding TTL. If the pinned instance is evicted the proxy falls back to a random healthy instance transparently.

Use a stable job ID as the key so that submit → poll → result calls all reach the same GPU:

KEY=$(uuidgen)
# Submit
curl -s -X POST http://tts-rs.proxy.example.com/rpc \
  -H "x-routing-key: $KEY" \
  -H 'content-type: application/json' \
  -d '{"jsonrpc":"2.0","method":"tts.submit","params":{...},"id":1}'

# Poll (same instance)
curl -s -X POST http://tts-rs.proxy.example.com/rpc \
  -H "x-routing-key: $KEY" \
  -H 'content-type: application/json' \
  -d '{"jsonrpc":"2.0","method":"tts.status","params":{"job_id":"..."},"id":2}'

WebSocket / upgrade forwarding

The relay transparently proxies WebSocket connections and any other HTTP Upgrade-based protocol. No extra configuration or separate route is needed — the proxy detects the Connection: Upgrade handshake on a normal relay request and switches to byte-transparent mode, exactly like Traefik or nginx.

Client ──WS handshake──►  api-proxy  ──WS handshake──►  Service
       ◄──101 Switching Protocols──             ◄──101──
       ◄═══════════ bidirectional frames bridged byte-for-byte ═══════════►

How it works:

1. The relay detects the upgrade (Connection: upgrade + Upgrade: header).
2. It dials the selected instance over raw TCP and replays the handshake (path, query string, and Sec-WebSocket-* headers preserved untouched).
3. On 101 Switching Protocols from the upstream, it bridges the two sockets in both directions until either side closes.

Notes:

• Sticky routing applies. Use X-Routing-Key to pin a long-lived socket to one instance (WebSocket connections are inherently sticky for their lifetime).
• No idle timeout is imposed on an upgraded connection — the relay's request timeout applies only to ordinary request/response calls, so quiet WebSocket sessions are not killed.
• Auth follows the same rule as other relay calls: no x-proxy-auth required.

Connect through the proxy just like any other relay call — service comes from the subdomain:

ws://my-svc.proxy.example.com/socket?token=abc

Response headers

Header	Description
x-instance	ip:port for a push instance, tunnel:<client_id> for a tunnel instance
x-request-id	Echoed from client, or a newly minted UUID

Response headers are added on ordinary relay calls. On an upgraded (WebSocket) connection the upstream's 101 response is returned verbatim, so x-instance is not injected.

Auth

When PROXY_SECRET is set, all control-plane methods require an x-proxy-auth header. Relay calls never require auth.

key    = SHA-256(PROXY_SECRET)
tag    = HMAC-SHA-256(key, timestamp_u64_le_bytes)
header = x-proxy-auth: {unix_timestamp}:{hex_tag}

Timestamps within ±30 s of the server clock are accepted (replay protection). The built-in client (proxy client) computes this header automatically.

Client mode

api-proxy also ships a client binary that registers the current machine and heartbeats:

proxy client \
  --to      https://your-proxy.example.com \
  --service my-svc \
  --port    8080

The client detects its public IP via the CONTAINER_IP environment variable (preferred) or falls back to querying https://api.ipify.org.

Tunnel mode (NAT traversal)

Push mode (above) requires the proxy to dial the client's ip:port directly — fine on a shared network, but it can't reach a client behind NAT with no port-forward. Tunnel mode inverts the connection: the client dials out to the proxy's tunnel port and keeps re-supplying fresh sockets, so nothing ever needs to reach the client.

tunnel client  ──dial out──►  proxy tunnel port (raw TCP, HMAC-authenticated)
                                        │
Caller  ──relay (Host: my-svc.domain)──►  proxy  ──pulls a pooled socket──►  tunnel client  ──►  local service

# Server: bind a second port for tunnel clients (in addition to --port)
proxy server --base-domain proxy.example.com --tunnel-port 8081

# Client: dial out instead of registering an ip:port
proxy client \
  --to           https://proxy.example.com \
  --service      my-svc \
  --port         8080 \
  --mode         tunnel \
  --tunnel-port  8081

Relay calls work exactly the same as push mode (http://my-svc.proxy.example.com/...) — the proxy picks the transport transparently based on how the service registered.

How it works:

1. The client authenticates and dials the tunnel port, sending <service>:<client_id>:<ts>:<hmac-hex>\n.
2. The proxy holds that socket in a per-client pool. The client immediately wires it to its local service and reconnects the instant it closes, so the pool keeps refilling.
3. On a relay request, the proxy pulls a pooled socket instead of dialing out, and relays over it exactly like the WebSocket-upgrade path — plain HTTP and WebSocket both work.
4. A service name can have several tunnel clients registered (distinct --client-ids); the proxy picks one at random per request, giving load distribution across them.

Notes:

• A single client process only holds one tunnel connection at a time by default (dial, bridge, reconnect is sequential). Raise --tunnel-concurrency (default 8) to run more reconnect loops in parallel if you expect concurrent traffic.
• No sticky routing (X-Routing-Key) across tunnel instances yet — affinity pins by ip:port, which tunnel sockets don't have.
• The tunnel port is plaintext TCP, not TLS — the HMAC handshake proves the connection is authorized, not that it's encrypted. Acceptable on Fly's private networking; if the tunnel port itself crosses an untrusted network, put it behind your own TLS.
• A dead tunnel client is evicted the same way as push instances — 90 s after its last register_tunnel heartbeat, same as HEARTBEAT_TIMEOUT_SECS.

Configuration

Env var / flag	Default	Description
PORT / --port	8080	Port the server listens on
PROXY_SECRET / --secret	(none)	HMAC secret; omit to disable auth
PROXY_BASE_DOMAIN / --base-domain	(required)	Base domain for subdomain routing (server mode)
PROXY_TUNNEL_PORT / --tunnel-port	8081	Raw TCP port tunnel clients dial (server + tunnel-mode client)
SERVICE_NAME / --service	tts-rs	Service name for client mode
PROXY_CLIENT_MODE / --mode	push	push or tunnel — see Tunnel mode
TUNNEL_CLIENT_ID / --client-id	(random)	Stable id for a tunnel client; random UUID if unset
TUNNEL_CONCURRENCY / --tunnel-concurrency	8	Concurrent reconnect loops (tunnel mode only)
CONTAINER_IP	(none)	Pre-set public IP; skips ipify lookup in client mode
API_KEY / --api-key	(none)	API key stored server-side and injected on relay (push mode)
PROXY_SERVER / --to	(required)	Proxy URL for client mode

Docker

docker build -t api-proxy .
docker run \
  -e PROXY_SECRET=mysecret \
  -e PROXY_BASE_DOMAIN=proxy.example.com \
  -p 8080:8080 -p 8081:8081 \
  api-proxy

The image exposes both the HTTP port (8080) and the tunnel port (8081, see Tunnel mode); map both if you plan to use tunnel clients.

Fly.io deployment

# Edit fly.toml — change app name, region, and PROXY_BASE_DOMAIN first
fly deploy
fly secrets set PROXY_SECRET=<strong-secret>

The fly.toml in this repo is a working example, including the raw-TCP tunnel port block. Change app and primary_region before deploying. For subdomain routing you'll also need a wildcard DNS record and TLS cert pointed at the app:

fly certs add "*.proxy.example.com"
# then add the CNAME/A record fly certs show tells you to

The tunnel port (8081) is plaintext TCP passthrough with no Fly-managed TLS — this is intentional (see Tunnel mode), no extra cert setup needed for it.

Known limitations

• In-memory registry — the registry is not persisted; a proxy restart clears all registrations. Services re-register within one heartbeat interval (60 s).
• Single node — no built-in clustering; run one proxy instance per deployment.
• Request body limit — the request body streams through unbuffered (no path buffers it in memory), but a request declaring Content-Length over 2 MiB is rejected with 413 before dialing forward. A body sent chunked, with no Content-Length, is not size-checked.
• No connection reuse to backends — every relay call dials a fresh TCP connection to the chosen instance rather than pooling keep-alive connections. Simpler, one relay code path for everything (plain, WebSocket, tunnel); revisit if connection-setup overhead matters for a high-request-rate service.
• Relay forwards request headers verbatim — no default content-type is injected if the caller didn't set one, and a body on a GET/DELETE/HEAD is forwarded rather than dropped.
• Wildcard DNS + TLS required — service routing is entirely subdomain-based, so you need a *.base_domain DNS record and a matching wildcard (or per-service) TLS cert in production. There is no path-based fallback.
• No sticky routing across tunnel instances — see Tunnel mode.
• No raw TCP/UDP tunneling — tunnel mode relays HTTP/1 (including WebSocket upgrades) only; arbitrary non-HTTP protocols (e.g. SSH) aren't supported.

License

MIT — see LICENSE.