Skip to content

fix(bench): stabilize benchmark CI workflow, fix SVG overflows, and harmonize docs - #168

Merged
vxfemboy merged 2 commits into
mainfrom
fix/ci-deny-and-mutants
Oct 26, 2026
Merged

vxfemboy merged 2 commits into
mainfrom
fix/ci-deny-and-mutants

Conversation

@vxfemboy

Copy link
Copy Markdown
Member

Summary

This PR resolves the root causes of volatile/flapping benchmark numbers across PRs, fixes SVG rendering defects, repairs failing benchmark unit tests, and harmonizes documentation with the latest verified dataset:

  1. Stop Benchmark Graph Flapping in CI:

    • Removed pull_request auto-commit trigger from .github/workflows/wan-benchmark.yml.
    • Prevents GitHub Actions from spinning up single-shot 12-second tests on burstable 1-vCPU AWS t3.small instances on every PR and force-pushing transient, fluctuating cloud numbers back to PR branches.
    • Preserves manual workflow_dispatch and scheduled runs for controlled, reproducible testing.
  2. Fix SVG Overflows & Formatting Defects:

    • crates/yip-bench/generate-wire-format.py:
      • Shortened the Panel A anti-DPI line (⊕ XOR keystream mask: zero magic bytes or fixed wire discriminators.) so it stays cleanly within the 510px card margin instead of breaching out to 520px+.
      • Re-flowed and tightened the Panel C 120s Epoch Ratchet bullets (Zeroize on Retire and 0% Packet Loss), eliminating a 60px overflow beyond the right border.
      • Deterministically regenerated assets/wire-format.svg.
  3. Repair Broken Benchmark Unit Tests:

    • crates/yip-bench/generate-graphs.py:
      • Added default path fallbacks to all chart generator functions (data=None) so they can be invoked standalone without raising TypeError.
    • crates/yip-bench/tests/test_graphs.py:
      • Fixed callout string assertions and added test methods for all 4 generated SVGs.
      • pytest crates/yip-bench/tests/ now passes 16/16 tests cleanly.
  4. Harmonize Documentation & Showcase Assets:

    • Aligned README.md, crates/yip-bench/README.md, and generate-social-card.py to cite the latest verified commit (19a499c):
      • 23× to 138× Higher Goodput Under Loss (14.0 Mbit/s @ 10% loss vs WireGuard's 0.6 Mbit/s [23×]; 221.3 Mbit/s @ 3% loss vs WireGuard's 1.6 Mbit/s [138×]).
      • Disambiguated the 800 Mbps (76% retained) metric as the 10G Netns / Multi-Core LAN testbed so it is not confused with 1-vCPU public WAN numbers.

Verification

  • python3 crates/yip-bench/generate-graphs.py --check: PASS
  • python3 crates/yip-bench/generate-wire-format.py --check: PASS
  • pytest crates/yip-bench/tests/: 16/16 PASS
  • bash crates/yip-bench/tests/test-generate-graphs.sh: ALL PASS
  • bash crates/yip-bench/tests/test-generate-wire-format.sh: ALL PASS
  • cargo fmt --check: PASS
  • cargo clippy --workspace --all-targets -- -D warnings: PASS
  • cargo shear: PASS
  • cargo test --workspace: ALL PASS (443+ tests)

…onize documentation

- Remove pull_request auto-commit trigger from wan-benchmark.yml to prevent CI bot from mutating PR branches with noisy burstable VPS runs
- Fix wire-format.svg text overflows on Panel A (mask line) and Panel C (epoch ratchet bullets)
- Add data fallback in generate-graphs.py and repair pytest test_graphs.py (all 16 tests passing)
- Harmonize README, crates/yip-bench/README.md, and social-card to authoritative dataset (23x-138x loss advantage, 10G netns tag)
@vxfemboy
vxfemboy temporarily deployed to benchmark-approval October 26, 2026 17:43 — with GitHub Actions Inactive
@vxfemboy
vxfemboy temporarily deployed to benchmark-approval October 26, 2026 17:43 — with GitHub Actions Inactive
@vxfemboy
vxfemboy temporarily deployed to benchmark-approval October 26, 2026 17:45 — with GitHub Actions Inactive
@vxfemboy
vxfemboy temporarily deployed to benchmark-approval October 26, 2026 17:45 — with GitHub Actions Inactive
@vxfemboy
vxfemboy merged commit df4c164 into main Oct 26, 2026
11 checks passed
@vxfemboy
vxfemboy deleted the fix/ci-deny-and-mutants branch October 26, 2026 17:46

This branch was previously deployed

1 inactive deployment
benchmark-approval — b88be2a2 Deployed Oct 26, 2026 by vxfemboy via benchmarks #60
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant