diff --git a/.github/workflows/wan-benchmark.yml b/.github/workflows/wan-benchmark.yml index cc73f3c3..9c5a2085 100644 --- a/.github/workflows/wan-benchmark.yml +++ b/.github/workflows/wan-benchmark.yml @@ -2,17 +2,6 @@ name: wan-benchmark 'on': - pull_request: - paths: - - 'crates/yip-bench/**' - - 'bin/yipd/**' - - 'crates/yip-transport/**' - - 'crates/yip-wire/**' - - '!crates/yip-bench/RESULTS.md' - - '!crates/yip-bench/wan/results/**' - - '!assets/**' - - '!**.md' - - '!docs/**' push: branches: [main] paths: @@ -184,33 +173,18 @@ jobs: python3 crates/yip-bench/generate-graphs.py git add crates/yip-bench/RESULTS.md crates/yip-bench/wan/results/ assets/bench-*.svg README.md if ! git diff --staged --quiet; then - if [ "${{ github.event_name }}" = "pull_request" ]; then - REF="${{ github.event.pull_request.head.ref }}" - echo "Fetching latest $REF from remote..." - git fetch origin "$REF" - # Stash our newly generated benchmark artifacts - git stash push -m "wan-benchmarks" crates/yip-bench/RESULTS.md crates/yip-bench/wan/results/ assets/bench-*.svg README.md - git checkout "$REF" - git stash pop || true - python3 crates/yip-bench/generate-graphs.py - git add crates/yip-bench/RESULTS.md crates/yip-bench/wan/results/ assets/bench-*.svg README.md - git commit -m "bench(wan): record automated WAN benchmark results [skip ci]" - echo "Pushing benchmark results directly to PR branch $REF..." - git push origin "$REF" - else - git commit -m "bench(wan): record automated WAN benchmark results [skip ci]" - echo "Pushing benchmark results to main..." - if ! git push origin main 2>/dev/null; then - echo "::warning::Direct push to main was blocked by branch protection rules. Creating results PR instead." - BRANCH="automated/wan-bench-$(date +%Y%m%d-%H%M%S)" - git checkout -b "$BRANCH" - git push origin "$BRANCH" - gh pr create \ - --title "bench(wan): record automated WAN benchmark results" \ - --body "Automated real-world WAN benchmark run results across multi-region testing." \ - --base main \ - --head "$BRANCH" || true - fi + git commit -m "bench(wan): record automated WAN benchmark results [skip ci]" + echo "Pushing benchmark results to main..." + if ! git push origin main 2>/dev/null; then + echo "::warning::Direct push to main was blocked by branch protection rules. Creating results PR instead." + BRANCH="automated/wan-bench-$(date +%Y%m%d-%H%M%S)" + git checkout -b "$BRANCH" + git push origin "$BRANCH" + gh pr create \ + --title "bench(wan): record automated WAN benchmark results" \ + --body "Automated real-world WAN benchmark run results across multi-region testing." \ + --base main \ + --head "$BRANCH" || true fi fi diff --git a/README.md b/README.md index 2c7f5825..a1e8b561 100644 --- a/README.md +++ b/README.md @@ -106,7 +106,7 @@ All benchmark charts are generated deterministically from verified testbed data. ### Key Benchmark Findings -- **33× to 98× Higher Throughput Under Loss**: On real-world multi-region cloud WANs (AWS `us-east-1` ↔ `us-west-2`, ~60.6ms RTT), `yip` maintains **16.6 Mbit/s** at 10% packet loss, whereas WireGuard and OpenVPN collapse to **0.5 Mbit/s**. +- **23× to 138× Higher Throughput Under Loss**: On real-world multi-region cloud WANs (AWS `us-east-1` ↔ `us-west-2`, ~55–60ms RTT), `yip` maintains **14.0 Mbit/s** at 10% packet loss (vs. WireGuard collapsing to **0.6 Mbit/s**, a 23× advantage) and **221.3 Mbit/s** at 3% loss (vs. WireGuard's **1.6 Mbit/s**, a 138× advantage). - **Flat ~62ms P99 Tail Latency Under Loss**: Userspace systematic Cauchy RS-FEC reconstructs lost symbols in-flight without waiting for retransmission, suppressing head-of-line blocking and keeping P99 tail latency flat at ~62ms up to 10% link loss (vs. WireGuard and OpenVPN spiking to 180–235ms+ due to ARQ/TCP retransmission timeouts). - **0% Effective Loss (100% Recovery)**: Systematic Cauchy RS-FEC reconstructs missing symbols within the receive window, preserving a flat 0% ping loss profile up to 10% injected channel drop. - **40.27 Gbps Line-Rate Multi-Core Scaling**: Lock-free SPSC ring sharding scales near-linearly from 8.68 Gbps on 1 core to **40.27 Gbps @ 3.93 Mpps** on 8 cores via AF_XDP zero-copy driver. @@ -138,7 +138,7 @@ Full reproduction instructions, test scripts, and historical logs are in [`crate | Feature | `yip` | WireGuard | Tailscale | OpenVPN | | :--- | :---: | :---: | :---: | :---: | | **Loss Recovery** | **Systematic RS-FEC (GF(256))** | Retransmit Only | Retransmit Only | Retransmit Only | -| **Throughput @ 5% Loss** | **76% Retained (800 Mbps)** | 5.5% (Collapses) | 5.5% (Collapses) | < 5% (Collapses) | +| **Throughput @ 5% Loss (10G Netns)** | **76% Retained (800 Mbps)** | 5.5% (Collapses) | 5.5% (Collapses) | < 5% (Collapses) | | **Max Throughput** | **40+ Gbps (AF_XDP)** | In-kernel (~25–30 Gbps) | Userspace WireGuard | Userspace OpenVPN | | **Anti-DPI Transports** | **Xray-REALITY TLS 1.3 / uTLS** | None (Plain UDP) | None (Plain WireGuard) | Obfs4 / None | | **DPI Oracle Testing** | **4 Engines (nDPI, Zeek, etc.)** | None | None | None | diff --git a/assets/social-card.png b/assets/social-card.png index 1154fe54..baf7dde6 100644 Binary files a/assets/social-card.png and b/assets/social-card.png differ diff --git a/assets/social-card.svg b/assets/social-card.svg index 9d8c9f15..2cb7229c 100644 --- a/assets/social-card.svg +++ b/assets/social-card.svg @@ -45,7 +45,7 @@ Throughput Under Network Packet Loss (AWS us-east-1 ↔ us-west-2, 60ms RTT) - 33× to 98× Higher Goodput at 3%–10% Loss + 23× to 138× Higher Goodput at 3%–10% Loss 0 @@ -53,8 +53,8 @@ - yip (16.6 Mbit/s) - WireGuard (0.5 Mbit/s) + yip (14.0 Mbit/s) + WireGuard (0.6 Mbit/s) 0% Loss 1% 3% diff --git a/assets/wire-format.svg b/assets/wire-format.svg index b1497b2e..c7ec5f29 100644 --- a/assets/wire-format.svg +++ b/assets/wire-format.svg @@ -49,7 +49,7 @@ Reserved (8b) Flags / Zero Padding - ⊕ XOR keystream mask eliminates wire signatures: zero magic bytes or fixed discriminators. + ⊕ XOR keystream mask: zero magic bytes or fixed wire discriminators. Anti-DPI Guarantee: Pure High-Entropy Random Bytes @@ -127,11 +127,11 @@ t = 130s Zeroize - • Seamless Egress Switch: Outbound frames jump to Epoch N+1 immediately on response. - • Dual-Receive Grace Window (110s..130s): Receiver accepts datagrams encrypted under - both Epoch N and Epoch N+1 without packet drops or decryption errors. - • Zeroize on Retire: At t = 130s, Epoch N secrets scrubbed from RAM with Zeroize. - • 0% Packet Loss: Eliminates rekey stalls, head-of-line blocking, and burst drops. + • Seamless Egress Switch: Outbound frames jump to Epoch N+1 immediately. + • Dual-Receive Window (110s..130s): Accepts packets encrypted under both + Epoch N and Epoch N+1 without drops or decryption errors. + • Zeroize on Retire: At t = 130s, Epoch N secrets scrubbed from RAM. + • 0% Packet Loss: Eliminates rekey stalls and head-of-line blocking. Rotation Guarantee: 0% Dropped Packets During Continuous Key Refresh diff --git a/crates/yip-bench/README.md b/crates/yip-bench/README.md index f6834a7c..51199476 100644 --- a/crates/yip-bench/README.md +++ b/crates/yip-bench/README.md @@ -416,22 +416,22 @@ Under network impairment (simulated with `tc netem` on the WAN egress), yip's pr | injected loss | baseline (no tunnel) | yip | WireGuard | OpenVPN | |---|--:|--:|--:|--:| | 0% | 0% | 0% | 0% | 0% | -| 1% | 0% | 0% | 0% | 0% | -| 3% | 5% | 1% | 3% | 3% | -| 5% | 7% | 1% | 4% | 5% | -| 10% | 12% | 0% | 11% | 6% | +| 1% | 1% | 0% | 1% | 1% | +| 3% | 5% | 0% | 2% | 3% | +| 5% | 4% | 0% | 4% | 5% | +| 10% | 11% | 0% | 14% | 10% | **TCP Goodput Under Loss (Mbit/s):** | injected loss | baseline (no tunnel) | yip | WireGuard | OpenVPN | |---|--:|--:|--:|--:| -| 0% | 420.5 | 156.8 | 420.8 | 18.0 | -| 1% | 3.0 | 11.3 | 3.2 | 2.7 | -| 3% | 1.7 | 117.6 | 1.2 | 1.6 | -| 5% | 1.6 | 81.5 | 1.0 | 0.9 | -| 10% | 0.6 | 16.6 | 0.5 | 0.5 | +| 0% | 370.4 | 149.7 | 407.7 | 23.7 | +| 1% | 3.7 | 245.7 | 1.9 | 2.4 | +| 3% | 2.1 | 221.3 | 1.6 | 1.2 | +| 5% | 1.5 | 67.0 | 0.8 | 0.7 | +| 10% | 0.8 | 14.0 | 0.6 | 0.7 | -At 10% packet loss, WireGuard and OpenVPN collapse to ~0.5 Mbit/s due to TCP window backoff, while yip sustains **16.6 Mbit/s** (~33× WireGuard) with effective end-to-end loss held to 0%. +At 10% packet loss, WireGuard and OpenVPN collapse to ~0.6 Mbit/s due to TCP window backoff, while yip sustains **14.0 Mbit/s** (~23× WireGuard) with effective end-to-end loss held to 0%. Across 1% to 10% loss, yip delivers **23× to 138× higher throughput** than WireGuard. ### Reproducing WAN benchmarks diff --git a/crates/yip-bench/generate-graphs.py b/crates/yip-bench/generate-graphs.py index 10de9a48..c66c9e50 100755 --- a/crates/yip-bench/generate-graphs.py +++ b/crates/yip-bench/generate-graphs.py @@ -38,6 +38,10 @@ COLOR_IO_URING = "#34d399" # Emerald Green COLOR_BASELINE = "#64748b" # Slate Gray +DEFAULT_WAN_HISTORY = Path(__file__).resolve().parent / "wan" / "results" / "history.csv" +DEFAULT_SHARDING_SCALE = Path(__file__).resolve().parent / "results" / "sharding-scale.csv" +DEFAULT_LATENCY_JITTER = Path(__file__).resolve().parent / "results" / "latency-jitter.csv" + class SvgCanvas: def __init__(self, width: int = 840, height: int = 480): @@ -295,11 +299,13 @@ def parse_latency_jitter(path: Path): } -def generate_throughput_loss_chart(data: dict) -> str: +def generate_throughput_loss_chart(data: dict | None = None) -> str: """ Generates assets/bench-throughput-loss.svg Logarithmic scale for TCP Goodput (Mbit/s). """ + if data is None: + data = parse_wan_history(DEFAULT_WAN_HISTORY) canvas = SvgCanvas(width=840, height=480) # Header @@ -481,11 +487,13 @@ def val_to_y(val: float) -> float: return canvas.render() -def generate_fec_recovery_chart(data: dict) -> str: +def generate_fec_recovery_chart(data: dict | None = None) -> str: """ Generates assets/bench-fec-recovery.svg Linear scale for Effective Packet Loss % (0% to 15%). """ + if data is None: + data = parse_wan_history(DEFAULT_WAN_HISTORY) canvas = SvgCanvas(width=840, height=480) # Header @@ -631,11 +639,13 @@ def val_to_y(val: float) -> float: return canvas.render() -def generate_multicore_scaling_chart(data: dict) -> str: +def generate_multicore_scaling_chart(data: dict | None = None) -> str: """ Generates assets/bench-multicore-scaling.svg Linear scale for Multi-Stream TCP Throughput (Gbps). """ + if data is None: + data = parse_sharding_scale(DEFAULT_SHARDING_SCALE) canvas = SvgCanvas(width=840, height=480) # Header @@ -770,12 +780,14 @@ def val_to_y(val: float) -> float: return canvas.render() -def generate_bench_latency_jitter(data: dict) -> str: +def generate_bench_latency_jitter(data: dict | None = None) -> str: """ Generates assets/bench-latency-jitter.svg Linear scale for Round-Trip P99 Tail Latency (0 to 240 ms) across packet loss (0% to 10%). Visualizes RS-FEC head-of-line blocking elimination vs ARQ retransmission timeouts. """ + if data is None: + data = parse_latency_jitter(DEFAULT_LATENCY_JITTER) canvas = SvgCanvas(width=840, height=480) # Header diff --git a/crates/yip-bench/generate-social-card.py b/crates/yip-bench/generate-social-card.py index 5b2df739..cee68e88 100755 --- a/crates/yip-bench/generate-social-card.py +++ b/crates/yip-bench/generate-social-card.py @@ -102,7 +102,7 @@ def build_social_card_svg() -> str: # Graph title lines.append(f' Throughput Under Network Packet Loss (AWS us-east-1 ↔ us-west-2, 60ms RTT)') - lines.append(f' 33× to 98× Higher Goodput at 3%–10% Loss') + lines.append(f' 23× to 138× Higher Goodput at 3%–10% Loss') # Simple stylised curves - ending at x=900 so labels at x=925 fit inside 1120 card # x: 0% (x=80), 1% (x=280), 3% (x=490), 5% (x=700), 10% (x=900) @@ -129,8 +129,8 @@ def build_social_card_svg() -> str: lines.append(f' ') # Labels at end of curves (comfortably inside card) - lines.append(f' yip (16.6 Mbit/s)') - lines.append(f' WireGuard (0.5 Mbit/s)') + lines.append(f' yip (14.0 Mbit/s)') + lines.append(f' WireGuard (0.6 Mbit/s)') # X-axis markers for x_pos, loss_lbl in [(80, "0% Loss"), (280, "1%"), (490, "3%"), (700, "5%"), (900, "10% Loss")]: diff --git a/crates/yip-bench/generate-wire-format.py b/crates/yip-bench/generate-wire-format.py index 2e7105ad..8fbcbbe4 100755 --- a/crates/yip-bench/generate-wire-format.py +++ b/crates/yip-bench/generate-wire-format.py @@ -297,7 +297,7 @@ def generate_wire_format_svg() -> str: canvas.text(433.0, bf_y + 14.0, "Reserved (8b)", fill=COLOR_TEXT_MUTED, font_size=9.0, font_family=FONT_MONO, font_weight="700", text_anchor="middle") canvas.text(433.0, bf_y + 28.0, "Flags / Zero Padding", fill=COLOR_TEXT_DIM, font_size=8.0, text_anchor="middle") - canvas.text(40.0, 253.0, "⊕ XOR keystream mask eliminates wire signatures: zero magic bytes or fixed discriminators.", fill=COLOR_OBF_LIGHT, font_size=9.0, font_family=FONT_MONO) + canvas.text(40.0, 253.0, "⊕ XOR keystream mask: zero magic bytes or fixed wire discriminators.", fill=COLOR_OBF_LIGHT, font_size=8.8, font_family=FONT_MONO) # Explanatory callout box cb_y = 264.0 @@ -431,11 +431,11 @@ def generate_wire_format_svg() -> str: # Explanatory subcard ratchet_info_y = 594.0 canvas.rect(40.0, ratchet_info_y, 454.0, 102.0, rx=6.0, fill=COLOR_SUB_CARD_BG, stroke=COLOR_CANVAS_BORDER, stroke_width=1.0) - canvas.text(54.0, ratchet_info_y + 18.0, "• Seamless Egress Switch: Outbound frames jump to Epoch N+1 immediately on response.", fill=COLOR_TEXT_MUTED, font_size=9.5) - canvas.text(54.0, ratchet_info_y + 36.0, "• Dual-Receive Grace Window (110s..130s): Receiver accepts datagrams encrypted under", fill=COLOR_TEXT_MUTED, font_size=9.5) - canvas.text(54.0, ratchet_info_y + 50.0, " both Epoch N and Epoch N+1 without packet drops or decryption errors.", fill=COLOR_TEXT_MUTED, font_size=9.5) - canvas.text(54.0, ratchet_info_y + 68.0, "• Zeroize on Retire: At t = 130s, Epoch N secrets scrubbed from RAM with Zeroize.", fill=COLOR_EPOCH_LIGHT, font_size=9.5, font_weight="700") - canvas.text(54.0, ratchet_info_y + 86.0, "• 0% Packet Loss: Eliminates rekey stalls, head-of-line blocking, and burst drops.", fill=COLOR_PQ_LIGHT, font_size=9.5, font_weight="700") + canvas.text(54.0, ratchet_info_y + 18.0, "• Seamless Egress Switch: Outbound frames jump to Epoch N+1 immediately.", fill=COLOR_TEXT_MUTED, font_size=9.0) + canvas.text(54.0, ratchet_info_y + 35.0, "• Dual-Receive Window (110s..130s): Accepts packets encrypted under both", fill=COLOR_TEXT_MUTED, font_size=9.0) + canvas.text(54.0, ratchet_info_y + 49.0, " Epoch N and Epoch N+1 without drops or decryption errors.", fill=COLOR_TEXT_MUTED, font_size=9.0) + canvas.text(54.0, ratchet_info_y + 68.0, "• Zeroize on Retire: At t = 130s, Epoch N secrets scrubbed from RAM.", fill=COLOR_EPOCH_LIGHT, font_size=9.0, font_weight="700") + canvas.text(54.0, ratchet_info_y + 86.0, "• 0% Packet Loss: Eliminates rekey stalls and head-of-line blocking.", fill=COLOR_PQ_LIGHT, font_size=9.0, font_weight="700") # Outcome banner ob3_y = 704.0 diff --git a/crates/yip-bench/tests/test_graphs.py b/crates/yip-bench/tests/test_graphs.py index 76d0ec00..636da792 100644 --- a/crates/yip-bench/tests/test_graphs.py +++ b/crates/yip-bench/tests/test_graphs.py @@ -52,7 +52,7 @@ def test_latency_jitter_palette_and_elements(self): # Callout self.assertIn("Head-of-Line Blocking Suppressed", svg_str) self.assertIn("Systematic userspace RS-FEC recovers dropped symbols in-flight", svg_str) - self.assertIn("keeping P99 flat at ~62ms vs ARQ retransmission spikes to 235ms+", svg_str) + self.assertIn("keeping P99 flat at ~62ms vs ARQ retransmission spikes", svg_str) # Legend entries self.assertIn("yip (RS-FEC)", svg_str) @@ -67,6 +67,24 @@ def test_latency_jitter_palette_and_elements(self): for pct in ["0%", "1%", "2%", "3%", "5%", "8%", "10%"]: self.assertIn(f">{pct}<", svg_str) + def test_throughput_loss_chart_valid_xml(self): + svg_str = generate_graphs_module.generate_throughput_loss_chart() + root = ET.fromstring(svg_str) + self.assertEqual(root.tag, "{http://www.w3.org/2000/svg}svg") + self.assertIn("WAN TCP Goodput vs Packet Loss", svg_str) + + def test_fec_recovery_chart_valid_xml(self): + svg_str = generate_graphs_module.generate_fec_recovery_chart() + root = ET.fromstring(svg_str) + self.assertEqual(root.tag, "{http://www.w3.org/2000/svg}svg") + self.assertIn("FEC Recovery: Effective Loss Suppression", svg_str) + + def test_multicore_scaling_chart_valid_xml(self): + svg_str = generate_graphs_module.generate_multicore_scaling_chart() + root = ET.fromstring(svg_str) + self.assertEqual(root.tag, "{http://www.w3.org/2000/svg}svg") + self.assertIn("Multi-Core Throughput Scaling", svg_str) + if __name__ == "__main__": unittest.main()