Skip to content

Commit fdea5d3

Browse files
LeeCampbellclaude
andcommitted
docs: add benchmark segmentation guidance to prevent agent timeouts
Full benchmark suites (3 classes × 3 runtimes) exceed the 30-minute agent iteration timeout. Document segmentation strategies so agents run benchmarks by class, by runtime, or both. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
1 parent 9cb7362 commit fdea5d3

2 files changed

Lines changed: 21 additions & 3 deletions

File tree

spec/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ This file provides guidance on where to find standards and specifications for th
4141
| encoding, LEB128, DEFLATE, compression | [Histogram Encoding](./tech-standards/histogram-encoding.md) |
4242
| log format, V2, persistence | [Histogram Encoding](./tech-standards/histogram-encoding.md) |
4343
| xUnit, test, FluentAssertions | [Testing Standards](./tech-standards/testing-standards.md) |
44-
| benchmark, performance, allocation, BenchmarkDotNet | [Testing Standards](./tech-standards/testing-standards.md) |
44+
| benchmark, performance, allocation, BenchmarkDotNet, timeout | [Testing Standards](./tech-standards/testing-standards.md) |
4545
| naming convention, XML docs, style | [Coding Standards](./tech-standards/coding-standards.md) |
4646
| build, NuGet, AppVeyor, CI/CD | [Build System](./tech-standards/build-system.md) |
4747
| milestone, issue, PR, GitHub | [GitHub CLI Reference](./tech-standards/github.md) |

spec/tech-standards/testing-standards.md

Lines changed: 20 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -224,12 +224,30 @@ Both levels of benchmark are required because:
224224

225225
### Running Benchmarks
226226

227+
> **Timeout warning:** A full benchmark run (all classes, all runtimes) can take **over 30 minutes**.
228+
> Automated agents have a 30-minute iteration timeout and **must not** attempt a full suite in a single run.
229+
> Always segment benchmark runs as described below.
230+
231+
**Segmentation strategies** (pick one or combine):
232+
233+
- **By benchmark class** — run one category at a time (e.g. encoding, recording, leading-zero-count)
234+
- **By runtime** — restrict to a single target framework per run (e.g. `net8.0` only)
235+
- **By filter** — use `--filter` to select specific benchmark methods
236+
227237
```bash
228238
# Build in Release mode (required)
229239
dotnet build HdrHistogram.Benchmarking/ -c Release
230240

231-
# Run specific benchmarks
232-
dotnet run -c Release --project HdrHistogram.Benchmarking/ -- --filter '*ClassName*'
241+
# Run a SINGLE benchmark class (recommended for agents)
242+
dotnet run -c Release --project HdrHistogram.Benchmarking/ -- --filter '*Recording32BitBenchmark*'
243+
dotnet run -c Release --project HdrHistogram.Benchmarking/ -- --filter '*LeadingZeroCount64BitBenchmark*'
244+
245+
# Run benchmarks for a SINGLE runtime only
246+
dotnet run -c Release --project HdrHistogram.Benchmarking/ -- --filter '*ClassName*' --runtimes net8.0
247+
dotnet run -c Release --project HdrHistogram.Benchmarking/ -- --filter '*ClassName*' --runtimes net9.0
248+
249+
# Combine both: one class, one runtime (fastest, safest for agents)
250+
dotnet run -c Release --project HdrHistogram.Benchmarking/ -- --filter '*Recording32BitBenchmark*' --runtimes net8.0
233251

234252
# Export results as JSON for comparison
235253
dotnet run -c Release --project HdrHistogram.Benchmarking/ -- --filter '*ClassName*' --exporters json

0 commit comments

Comments
 (0)