Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 12 additions & 38 deletions .github/workflows/wan-benchmark.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,17 +2,6 @@
name: wan-benchmark

'on':
pull_request:
paths:
- 'crates/yip-bench/**'
- 'bin/yipd/**'
- 'crates/yip-transport/**'
- 'crates/yip-wire/**'
- '!crates/yip-bench/RESULTS.md'
- '!crates/yip-bench/wan/results/**'
- '!assets/**'
- '!**.md'
- '!docs/**'
push:
branches: [main]
paths:
Expand Down Expand Up @@ -184,33 +173,18 @@ jobs:
python3 crates/yip-bench/generate-graphs.py
git add crates/yip-bench/RESULTS.md crates/yip-bench/wan/results/ assets/bench-*.svg README.md
if ! git diff --staged --quiet; then
if [ "${{ github.event_name }}" = "pull_request" ]; then
REF="${{ github.event.pull_request.head.ref }}"
echo "Fetching latest $REF from remote..."
git fetch origin "$REF"
# Stash our newly generated benchmark artifacts
git stash push -m "wan-benchmarks" crates/yip-bench/RESULTS.md crates/yip-bench/wan/results/ assets/bench-*.svg README.md
git checkout "$REF"
git stash pop || true
python3 crates/yip-bench/generate-graphs.py
git add crates/yip-bench/RESULTS.md crates/yip-bench/wan/results/ assets/bench-*.svg README.md
git commit -m "bench(wan): record automated WAN benchmark results [skip ci]"
echo "Pushing benchmark results directly to PR branch $REF..."
git push origin "$REF"
else
git commit -m "bench(wan): record automated WAN benchmark results [skip ci]"
echo "Pushing benchmark results to main..."
if ! git push origin main 2>/dev/null; then
echo "::warning::Direct push to main was blocked by branch protection rules. Creating results PR instead."
BRANCH="automated/wan-bench-$(date +%Y%m%d-%H%M%S)"
git checkout -b "$BRANCH"
git push origin "$BRANCH"
gh pr create \
--title "bench(wan): record automated WAN benchmark results" \
--body "Automated real-world WAN benchmark run results across multi-region testing." \
--base main \
--head "$BRANCH" || true
fi
git commit -m "bench(wan): record automated WAN benchmark results [skip ci]"
echo "Pushing benchmark results to main..."
if ! git push origin main 2>/dev/null; then
echo "::warning::Direct push to main was blocked by branch protection rules. Creating results PR instead."
BRANCH="automated/wan-bench-$(date +%Y%m%d-%H%M%S)"
git checkout -b "$BRANCH"
git push origin "$BRANCH"
gh pr create \
--title "bench(wan): record automated WAN benchmark results" \
--body "Automated real-world WAN benchmark run results across multi-region testing." \
--base main \
--head "$BRANCH" || true
fi
fi

Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ All benchmark charts are generated deterministically from verified testbed data.

### Key Benchmark Findings

- **33× to 98× Higher Throughput Under Loss**: On real-world multi-region cloud WANs (AWS `us-east-1` ↔ `us-west-2`, ~60.6ms RTT), `yip` maintains **16.6 Mbit/s** at 10% packet loss, whereas WireGuard and OpenVPN collapse to **0.5 Mbit/s**.
- **23× to 138× Higher Throughput Under Loss**: On real-world multi-region cloud WANs (AWS `us-east-1` ↔ `us-west-2`, ~55–60ms RTT), `yip` maintains **14.0 Mbit/s** at 10% packet loss (vs. WireGuard collapsing to **0.6 Mbit/s**, a 23× advantage) and **221.3 Mbit/s** at 3% loss (vs. WireGuard's **1.6 Mbit/s**, a 138× advantage).
- **Flat ~62ms P99 Tail Latency Under Loss**: Userspace systematic Cauchy RS-FEC reconstructs lost symbols in-flight without waiting for retransmission, suppressing head-of-line blocking and keeping P99 tail latency flat at ~62ms up to 10% link loss (vs. WireGuard and OpenVPN spiking to 180–235ms+ due to ARQ/TCP retransmission timeouts).
- **0% Effective Loss (100% Recovery)**: Systematic Cauchy RS-FEC reconstructs missing symbols within the receive window, preserving a flat 0% ping loss profile up to 10% injected channel drop.
- **40.27 Gbps Line-Rate Multi-Core Scaling**: Lock-free SPSC ring sharding scales near-linearly from 8.68 Gbps on 1 core to **40.27 Gbps @ 3.93 Mpps** on 8 cores via AF_XDP zero-copy driver.
Expand Down Expand Up @@ -138,7 +138,7 @@ Full reproduction instructions, test scripts, and historical logs are in [`crate
| Feature | `yip` | WireGuard | Tailscale | OpenVPN |
| :--- | :---: | :---: | :---: | :---: |
| **Loss Recovery** | **Systematic RS-FEC (GF(256))** | Retransmit Only | Retransmit Only | Retransmit Only |
| **Throughput @ 5% Loss** | **76% Retained (800 Mbps)** | 5.5% (Collapses) | 5.5% (Collapses) | < 5% (Collapses) |
| **Throughput @ 5% Loss (10G Netns)** | **76% Retained (800 Mbps)** | 5.5% (Collapses) | 5.5% (Collapses) | < 5% (Collapses) |
| **Max Throughput** | **40+ Gbps (AF_XDP)** | In-kernel (~25–30 Gbps) | Userspace WireGuard | Userspace OpenVPN |
| **Anti-DPI Transports** | **Xray-REALITY TLS 1.3 / uTLS** | None (Plain UDP) | None (Plain WireGuard) | Obfs4 / None |
| **DPI Oracle Testing** | **4 Engines (nDPI, Zeek, etc.)** | None | None | None |
Expand Down
Binary file modified assets/social-card.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
6 changes: 3 additions & 3 deletions assets/social-card.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
12 changes: 6 additions & 6 deletions assets/wire-format.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
20 changes: 10 additions & 10 deletions crates/yip-bench/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -416,22 +416,22 @@ Under network impairment (simulated with `tc netem` on the WAN egress), yip's pr
| injected loss | baseline (no tunnel) | yip | WireGuard | OpenVPN |
|---|--:|--:|--:|--:|
| 0% | 0% | 0% | 0% | 0% |
| 1% | 0% | 0% | 0% | 0% |
| 3% | 5% | 1% | 3% | 3% |
| 5% | 7% | 1% | 4% | 5% |
| 10% | 12% | 0% | 11% | 6% |
| 1% | 1% | 0% | 1% | 1% |
| 3% | 5% | 0% | 2% | 3% |
| 5% | 4% | 0% | 4% | 5% |
| 10% | 11% | 0% | 14% | 10% |

**TCP Goodput Under Loss (Mbit/s):**

| injected loss | baseline (no tunnel) | yip | WireGuard | OpenVPN |
|---|--:|--:|--:|--:|
| 0% | 420.5 | 156.8 | 420.8 | 18.0 |
| 1% | 3.0 | 11.3 | 3.2 | 2.7 |
| 3% | 1.7 | 117.6 | 1.2 | 1.6 |
| 5% | 1.6 | 81.5 | 1.0 | 0.9 |
| 10% | 0.6 | 16.6 | 0.5 | 0.5 |
| 0% | 370.4 | 149.7 | 407.7 | 23.7 |
| 1% | 3.7 | 245.7 | 1.9 | 2.4 |
| 3% | 2.1 | 221.3 | 1.6 | 1.2 |
| 5% | 1.5 | 67.0 | 0.8 | 0.7 |
| 10% | 0.8 | 14.0 | 0.6 | 0.7 |

At 10% packet loss, WireGuard and OpenVPN collapse to ~0.5 Mbit/s due to TCP window backoff, while yip sustains **16.6 Mbit/s** (~33× WireGuard) with effective end-to-end loss held to 0%.
At 10% packet loss, WireGuard and OpenVPN collapse to ~0.6 Mbit/s due to TCP window backoff, while yip sustains **14.0 Mbit/s** (~23× WireGuard) with effective end-to-end loss held to 0%. Across 1% to 10% loss, yip delivers **23× to 138× higher throughput** than WireGuard.

### Reproducing WAN benchmarks

Expand Down
20 changes: 16 additions & 4 deletions crates/yip-bench/generate-graphs.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,10 @@
COLOR_IO_URING = "#34d399" # Emerald Green
COLOR_BASELINE = "#64748b" # Slate Gray

DEFAULT_WAN_HISTORY = Path(__file__).resolve().parent / "wan" / "results" / "history.csv"
DEFAULT_SHARDING_SCALE = Path(__file__).resolve().parent / "results" / "sharding-scale.csv"
DEFAULT_LATENCY_JITTER = Path(__file__).resolve().parent / "results" / "latency-jitter.csv"


class SvgCanvas:
def __init__(self, width: int = 840, height: int = 480):
Expand Down Expand Up @@ -295,11 +299,13 @@ def parse_latency_jitter(path: Path):
}


def generate_throughput_loss_chart(data: dict) -> str:
def generate_throughput_loss_chart(data: dict | None = None) -> str:
"""
Generates assets/bench-throughput-loss.svg
Logarithmic scale for TCP Goodput (Mbit/s).
"""
if data is None:
data = parse_wan_history(DEFAULT_WAN_HISTORY)
canvas = SvgCanvas(width=840, height=480)

# Header
Expand Down Expand Up @@ -481,11 +487,13 @@ def val_to_y(val: float) -> float:
return canvas.render()


def generate_fec_recovery_chart(data: dict) -> str:
def generate_fec_recovery_chart(data: dict | None = None) -> str:
"""
Generates assets/bench-fec-recovery.svg
Linear scale for Effective Packet Loss % (0% to 15%).
"""
if data is None:
data = parse_wan_history(DEFAULT_WAN_HISTORY)
canvas = SvgCanvas(width=840, height=480)

# Header
Expand Down Expand Up @@ -631,11 +639,13 @@ def val_to_y(val: float) -> float:
return canvas.render()


def generate_multicore_scaling_chart(data: dict) -> str:
def generate_multicore_scaling_chart(data: dict | None = None) -> str:
"""
Generates assets/bench-multicore-scaling.svg
Linear scale for Multi-Stream TCP Throughput (Gbps).
"""
if data is None:
data = parse_sharding_scale(DEFAULT_SHARDING_SCALE)
canvas = SvgCanvas(width=840, height=480)

# Header
Expand Down Expand Up @@ -770,12 +780,14 @@ def val_to_y(val: float) -> float:
return canvas.render()


def generate_bench_latency_jitter(data: dict) -> str:
def generate_bench_latency_jitter(data: dict | None = None) -> str:
"""
Generates assets/bench-latency-jitter.svg
Linear scale for Round-Trip P99 Tail Latency (0 to 240 ms) across packet loss (0% to 10%).
Visualizes RS-FEC head-of-line blocking elimination vs ARQ retransmission timeouts.
"""
if data is None:
data = parse_latency_jitter(DEFAULT_LATENCY_JITTER)
canvas = SvgCanvas(width=840, height=480)

# Header
Expand Down
6 changes: 3 additions & 3 deletions crates/yip-bench/generate-social-card.py
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ def build_social_card_svg() -> str:

# Graph title
lines.append(f' <text x="24" y="32" font-family="{FONT_SANS}" font-size="14" font-weight="700" fill="#f0f6fc">Throughput Under Network Packet Loss (AWS us-east-1 ↔ us-west-2, 60ms RTT)</text>')
lines.append(f' <text x="{gw - 24}" y="32" font-family="{FONT_MONO}" font-size="12" font-weight="700" fill="#fb923c" text-anchor="end">33× to 98× Higher Goodput at 3%–10% Loss</text>')
lines.append(f' <text x="{gw - 24}" y="32" font-family="{FONT_MONO}" font-size="12" font-weight="700" fill="#fb923c" text-anchor="end">23× to 138× Higher Goodput at 3%–10% Loss</text>')

# Simple stylised curves - ending at x=900 so labels at x=925 fit inside 1120 card
# x: 0% (x=80), 1% (x=280), 3% (x=490), 5% (x=700), 10% (x=900)
Expand All @@ -129,8 +129,8 @@ def build_social_card_svg() -> str:
lines.append(f' <polyline points="{yip_str}" fill="none" stroke="#f97316" stroke-width="3.5" stroke-linecap="round" stroke-linejoin="round" />')

# Labels at end of curves (comfortably inside card)
lines.append(f' <text x="918" y="144" font-family="{FONT_MONO}" font-size="11.5" font-weight="700" fill="#f97316">yip (16.6 Mbit/s)</text>')
lines.append(f' <text x="918" y="194" font-family="{FONT_MONO}" font-size="11" font-weight="600" fill="#38bdf8">WireGuard (0.5 Mbit/s)</text>')
lines.append(f' <text x="918" y="144" font-family="{FONT_MONO}" font-size="11.5" font-weight="700" fill="#f97316">yip (14.0 Mbit/s)</text>')
lines.append(f' <text x="918" y="194" font-family="{FONT_MONO}" font-size="11" font-weight="600" fill="#38bdf8">WireGuard (0.6 Mbit/s)</text>')

# X-axis markers
for x_pos, loss_lbl in [(80, "0% Loss"), (280, "1%"), (490, "3%"), (700, "5%"), (900, "10% Loss")]:
Expand Down
Loading
Loading