|
| 1 | +# Native v3 packaged benchmarks |
| 2 | + |
| 3 | +`make bench` is opt-in. It compiles an optimized C++20 server outside the source |
| 4 | +and build trees using the selected installed public headers and library. Independent |
| 5 | +Python clients run in separate processes. TLS-on requires all sixteen cells: |
| 6 | +shared/static × managed/external × H1 plaintext, H1 TLS, H2 TLS and H3 QUIC. |
| 7 | +TLS-off runs four plaintext cells and checks typed `not_supported` with no bound |
| 8 | +port for encrypted protocols. Unavailable protocols never count as measurements. |
| 9 | + |
| 10 | +The measurement clients require Python 3 with hyper-h2 **4.4.1** and aioquic |
| 11 | +**1.3.0**. These are test dependencies only. Missing clients fail preflight. |
| 12 | +Select an OpenSSL 3.5 LTS provider explicitly. For example, from a fresh checkout: |
| 13 | + |
| 14 | +```sh |
| 15 | +bash bootstrap |
| 16 | +python3 -m venv /tmp/native-benchmark-clients |
| 17 | +/tmp/native-benchmark-clients/bin/pip install h2==4.4.1 aioquic==1.3.0 |
| 18 | +mkdir -p build/native-on |
| 19 | +cd build/native-on |
| 20 | +CXX=clang++-18 CC=clang-18 CXXFLAGS='-O2 -DNDEBUG' \ |
| 21 | + V3_TLS_CFLAGS="-I$PROVIDER/include" \ |
| 22 | + V3_TLS_LIBS="-L$PROVIDER/lib -lssl -lcrypto" \ |
| 23 | + ../../configure --enable-v3-tls=yes --disable-examples --prefix="$PREFIX" |
| 24 | +make -j4 |
| 25 | +make install |
| 26 | +make bench BENCH_PREFIX="$PREFIX" BENCH_PROVIDER_PREFIX="$PROVIDER" \ |
| 27 | + BENCH_TLS=yes BENCH_OUTPUT="$NEW_OUTPUT" \ |
| 28 | + BENCH_PYTHON_COMMAND=/tmp/native-benchmark-clients/bin/python |
| 29 | +``` |
| 30 | + |
| 31 | +Repeat with a distinct fresh build/prefix/output and `--enable-v3-tls=no`, |
| 32 | +`BENCH_TLS=no`. Library and consumer optimization flags are recorded. The runner |
| 33 | +requires the final optimization option to be `-O2` or `-O3`, and rejects |
| 34 | +sanitizer flags/symbols and allocator preload. |
| 35 | +Sanitizer measurements are correctness evidence and cannot enter comparisons. |
| 36 | +The direct entry point provides `--source`, `--build`, `--prefix`, |
| 37 | +`--provider-prefix`, `--tls`, `--compiler`, `--client-python` and `--output`. |
| 38 | + |
| 39 | +The full workload has 4096 measured requests, 1024 response bytes and four |
| 40 | +server workers. H1/H2 concurrency is 8; H3 concurrency is 4. H1 warmup is 256. |
| 41 | +H1 uses eight persistent connections; |
| 42 | +H2 uses 64 connection epochs, each retaining one authenticated connection for |
| 43 | +eight warmup requests and 64 measured requests with eight concurrent streams: |
| 44 | +**512 actual warmup requests** and 4096 measured requests. The public default |
| 45 | +limits each H2 connection to 128 stream openings per second. Epochs stay below |
| 46 | +that allowance without changing production policy or adding artificial pacing. |
| 47 | +H3 has sixteen live-request seats and bounded lifetime stream records, so its |
| 48 | +full workload uses |
| 49 | +1024 connection epochs, each with four concurrent warmup requests followed by |
| 50 | +four measured streams: **4096 actual warmup requests** and 4096 measured requests. |
| 51 | +Every H3 epoch performs verified connection setup, completes warmup, reuses that |
| 52 | +connection for measurement, then closes it. The normal per-connection semantic |
| 53 | +data pool is capped at 393216 bytes; each active request reserves body and |
| 54 | +response buffers plus metadata. Concurrency 8 can produce H3_EXCESSIVE_LOAD |
| 55 | +refusals; concurrency 4 stays within the default admission capacity. This |
| 56 | +benchmark changes no production resource policy. Each epoch uses a new retained QUIC connection. |
| 57 | +The full cell deadline is 300 seconds, including setup and shutdown of client |
| 58 | +connections. Server startup and orderly shutdown each have an additional |
| 59 | +15-second bound. `--deadline` permits 1..600 seconds; other bounded overrides are |
| 60 | +`--warmup`/`--requests` (1..65536), `--concurrency` (1..8), `--h3-concurrency` (1..4, capped by |
| 61 | +`--concurrency`), `--payload` (1..16384) and `--workers` (1..32). H2/H3 epoch warmup follows concurrency, independent of the |
| 62 | +H1 warmup option. Overrides appear in the receipt; any deviation from defaults |
| 63 | +is labelled a reduced/custom workload rather than full acceptance evidence. |
| 64 | + |
| 65 | +Timestamps are integer `time.monotonic_ns()` nanoseconds with clock resolution |
| 66 | +recorded. A request starts immediately before dispatch and ends at complete |
| 67 | +response receipt. Clients verify protocol negotiation, certificate trust and |
| 68 | +`a.example`, status 200, exact payload and run/request identity. Setup and all |
| 69 | +warmup samples are retained separately. Each measurement window starts after |
| 70 | +successful warmup and ends at its last measured completion. Duration is the sum |
| 71 | +of nonoverlapping measured windows in seconds; throughput is successful measured |
| 72 | +completions divided by that sum, in requests/second. For H2/H3 this excludes each |
| 73 | +epoch's handshake, warmup and close, and is **not whole-process wall throughput**. |
| 74 | +Whole-process wall seconds are also recorded. Differing protocol concurrency |
| 75 | +and connection reuse settings make these observations unsuitable for a direct |
| 76 | +protocol performance ranking. Latency percentiles use every |
| 77 | +successful measured sample without trimming: sort durations, select one-based |
| 78 | +rank `ceil(N*p/100)`, and display p50/p95/p99 in milliseconds. Raw request IDs, |
| 79 | +epoch membership and timestamps permit independent recomputation. |
| 80 | + |
| 81 | +Each attempt must use a new output directory. Failures remain there. Receipts |
| 82 | +reference artifacts by absolute path, byte count and SHA-256, including raw samples, |
| 83 | +server observations, commands, executable, depfiles and link maps. Keep the |
| 84 | +recorded external consumer directory and package/provider/client installations |
| 85 | +alongside the receipt. Deleting or changing these invalidates verification. The |
| 86 | +server independently counts successful warmup/measured response writes and errors |
| 87 | +after orderly shutdown. Client exit zero alone is insufficient. |
| 88 | + |
| 89 | +`validate(path, expected_identity, expected_run_id)` is available from |
| 90 | +`scripts/run-v3-native-benchmarks.py` for release adapter integration. Offline |
| 91 | +verification checks current source/package/build/workload identities and every |
| 92 | +artifact, then recomputes samples/statistics and compares server counts. The |
| 93 | +expected TLS identity selects the required matrix, and every exported cell metric |
| 94 | +must agree with raw observations in the declared units. Required consumer source, |
| 95 | +local header, executable, link map, command, copied client/fixture and client-input |
| 96 | +records must remain present and consistent; installed headers are bound by the |
| 97 | +expected package identity. A caller |
| 98 | +requiring new execution must obtain the fresh runner's run ID; revalidating an old |
| 99 | +receipt is integrity verification, not a new benchmark. CLI `--validate RECEIPT` |
| 100 | +accepts `--expected-run-id`. Hashes establish integrity against recorded inputs; |
| 101 | +fresh execution binds accepted evidence to the current run. They are not |
| 102 | +cryptographic attestation against an attacker controlling the producer and all |
| 103 | +artifacts. Synthetic parsing fixtures cannot substitute for a live attempt. |
| 104 | + |
| 105 | +PRD §3.9 and the repair roadmap specify no absolute native throughput/latency |
| 106 | +minimum or comparable regression baseline. Historical `test/PERFORMANCE.md` |
| 107 | +v1/v2 numbers and unrelated 10×/2× claims do not apply. Acceptance requires full |
| 108 | +successful bounded measurements, exact counts, finite positive throughput, |
| 109 | +ordered finite percentiles, zero unexpected errors, and unchanged TASK-180 |
| 110 | +resource gates: three output/upload saturation/reset/refill cycles, stalled |
| 111 | +observations >=0.99 seconds, CPU/wall <=0.2 of one core and warm RSS growth |
| 112 | +<=1,048,576 bytes. Run the existing normal `native_protocol_memory`, |
| 113 | +`resource_limits`, installed public H2/H3 drivers and 90-second resource probe |
| 114 | +separately; benchmark throughput does not replace their proof. Private resource |
| 115 | +fixtures remain supporting evidence. Nonlocal platform checks belong to CI/v3 PR. |
0 commit comments