test: add configurable NeMo Relay latency benchmark

.gitattributes

@@ -2,3 +2,4 @@
 # SPDX-License-Identifier: Apache-2.0
 
 examples/python-grpc-worker-plugin/nemo_relay_python_grpc_worker_example/worker.py text eol=lf
+*.cmd text eol=crlf

docs/reference/performance.mdx

@@ -29,6 +29,50 @@ Use these practices when applying the concept in application or integration code
 - Use execution intercepts when you need to wrap real execution and sanitize guardrails when you only need to change emitted observability payloads.
 - Use binding-native typed wrappers and codecs when provider payload conversion would otherwise be repeated at many call sites.
 
+## Latency Benchmark
+
+Use the opt-in latency benchmark to measure the local latency that Relay
+adds around Codex and Claude Code traffic. The harness uses deterministic
+loopback OpenAI and Anthropic providers so network and model-service latency do
+not hide Relay's contribution.
+
+Run the benchmark with the following command:
+
+```bash
+just latency-benchmark
+```
+
+The command builds the release CLI. Its default configuration measures the
+following paths:
+
+- Direct requests to the mock provider.
+- Relay with no exporter, which isolates the managed gateway pipeline.
+- Relay with the ATOF file exporter.
+- Relay with the OTLP exporter sending to a local OpenTelemetry HTTP receiver.
+- Full `nemo-relay hook-forward` subprocesses for Codex and Claude Code.
+- Cold Relay process startup through gateway readiness.
+
+Gateway scenarios cover OpenAI Responses and Anthropic Messages, buffered and
+streaming responses, multiple request payload sizes, and multiple concurrency
+levels.
+
+For buffered calls, inspect total latency. For streaming calls, inspect both
+time to the first content delta and total stream time. The report includes
+p50, p95, and p99 paired latency differences and a bootstrap 95% confidence
+interval for the median.
+
+For setup, CLI options, configuration files, middleware variants, storage,
+output paths, and troubleshooting, refer to the
+[latency benchmark run guide](https://github.com/NVIDIA/NeMo-Relay/blob/main/scripts/latency_benchmark/README.md).
+
+Treat results as environment-specific. Record the commit, release build,
+hardware, operating system, workload sizes, and sample counts when sharing a
+number. Prefer added milliseconds over percentages: a small absolute increase
+can look disproportionately large when the direct loopback baseline is much
+faster than a real model call. Use real Codex or Claude Code runs as an
+end-to-end validation, not as the primary gateway measurement, because host
+startup and scheduling add unrelated variance.
+
 ## Related Topics
 
 Use these links to continue into adjacent concepts and workflows.

justfile

@@ -1169,6 +1169,28 @@ test-codex-plugin-e2e:
 test-claude-plugin-e2e:
     ./scripts/test-claude-plugin-e2e.sh
 
+# Opt-in: builds the release CLI and runs configurable local latency suites.
+[positional-arguments]
+latency-benchmark *benchmark_args:
+    #!/usr/bin/env bash
+    set -euo pipefail
+    result_dir={{ quote(output_dir) }}
+    result_dir="${result_dir:-target/benchmark-results}"
+    for argument in "$@"; do
+        if [[ "$argument" == "-h" || "$argument" == "--help" ]]; then
+            exec uv run --locked python -m scripts.latency_benchmark.src "$@"
+        fi
+    done
+    cargo build --locked --release -p nemo-relay-cli
+    uv run --locked python -m scripts.latency_benchmark.src \
+        --relay-bin target/release/nemo-relay \
+        --output "$result_dir/nemo-relay-latency-report.json" \
+        "$@"
+
+# Runs the fast latency benchmark fixture tests without building Relay.
+test-latency-benchmark:
+    uv run --locked python -m pytest scripts/latency_benchmark/tests
+
 # --set [output_dir=<path>] [ci=true|false]
 test-rust:
     #!/usr/bin/env bash

scripts/README.md

@@ -25,6 +25,42 @@ These checks exercise installed coding-agent clients and are intentionally outsi
 - `just test-codex-plugin-e2e`
 - `just test-claude-plugin-e2e`
 
+## Latency Benchmark
+
+Run `just latency-benchmark` to build the release CLI and compare
+direct provider requests with Relay's minimal, ATOF file exporter, and OTLP
+exporter configurations. The benchmark also measures full hook subprocess and
+cold gateway startup time. It writes structured results under
+`target/benchmark-results/nemo-relay-latency-report.json` and
+`target/benchmark-results/nemo-relay-latency-report.html` by default and is
+intentionally outside regular CI. Large ATOF output goes to a
+`nemo-relay-latency-*` directory in the operating system's temporary location
+and is removed after a normal run.
+
+Run `just test-latency-benchmark` to execute the fixture's fast unit tests
+without building Relay.
+
+The defaults live in `scripts/latency_benchmark/config/default.toml`. Supply a
+partial TOML file with `--config`, or override individual values on the command
+line. For example, this runs only a small OpenAI gateway matrix:
+
+```bash
+just latency-benchmark \
+  --tests gateway \
+  --providers openai \
+  --payload-sizes 4096 \
+  --concurrency 1 \
+  --samples 10
+```
+
+Run `just latency-benchmark --help` to list all overrides without
+building Relay. The three selectable suites are `gateway`, `hooks`, and
+`startup`. Each run writes machine-readable JSON and a self-contained HTML
+report with graphs. Add repeatable `--middleware NAME=PATH` options to measure
+custom Relay plugin configurations alongside the three default variants. Refer
+to [`latency_benchmark/README.md`](latency_benchmark/README.md) for the complete
+human-facing run guide.
+
 ## Internal Layout
 
 - `docs/`: Fern reference-generation, migration cleanup, and `docs-website` branch sync helpers. Generated API reference output under `docs/reference/api/*-library-reference/` is ignored and recreated by `just docs`.