Add a BenchmarkDotNet suite for solve hot paths - #104
Draft
HowardvanRooijen wants to merge 1 commit into
Draft
HowardvanRooijen wants to merge 1 commit into
HowardvanRooijen wants to merge 1 commit into
Conversation
A new Z3.Linq.Benchmarks project, following the endjin BenchmarkDotNet guidelines, to find performance and memory hot paths in the library and confirm an optimisation helped. SolveBenchmarks runs end-to-end Solve/Optimize over six environment shapes - flat symbols, a value tuple, a nested object, a collection, a wider symbol set, and an optimisation - each against a theorem Z3 decides in microseconds, so the time and the MemoryDiagnoser Allocated column are dominated by the library's own per-solve work (reflection env-building, expression translation, marshalling) rather than the solver. Nothing is cached across solves today, so those numbers are what an optimisation can move. A first dry run already shows the shape of it: the collection case allocates ~2.3x the flat-symbol baseline, and the nested, wide and optimise cases ~2x - the marshalling loop, recursive marshalling and per-solve reflection. The project is not packable, suppresses CS1591 like the other non-library projects, and is wired into the solution and central package management. BenchmarkDotNet.Artifacts is already gitignored. The README documents the run commands, the optimise/re-measure loop, and JSON export for bdna regression tracking. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PipoHZJsgydrV3JC3fQN6Q
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a
Z3.Linq.Benchmarksproject - a BenchmarkDotNet suite,following the endjin BenchmarkDotNet guidelines, for finding performance and memory hot paths in
the library and confirming an optimisation actually helped. It measures; it changes no library
code.
What it measures, and why it's the right thing
SolveBenchmarksruns end-to-endSolve/Optimizeover six environment shapes, each against atheorem Z3 decides in microseconds. That is deliberate: with the solver's own work negligible,
the time and - more usefully - the
MemoryDiagnoserAllocatedcolumn are dominated by thelibrary's per-solve work: building the environment by reflection, translating the constraint
expression trees, and marshalling the model back onto the CLR members. Nothing is cached across
solves today, so re-solving the same theorem re-does all of it - which is exactly what these numbers
can be used to change.
ScalarSymbols(baseline)Symbols<int,int>ValueTupleNestedObjectCollectionWideSymbolsSymbols<int,int,int,int,int>)OptimizeEach theorem is built once in
[GlobalSetup]; the benchmark measures only the solve, and returnsthe result so the JIT cannot eliminate the call.
It already points somewhere
A first
--job dryrun (allocation is meaningful even in Dry; the DryMeanis not) - relative tothe flat-symbol baseline:
ValueTupleScalarSymbolsNestedObjectOptimizeWideSymbolsCollectionThe collection path allocates over twice the flat baseline, and the nested/wide/optimise shapes
around twice - the marshalling loop (
ArrayList+ToArray, boxing), the recursive marshalling,and the per-solve reflection. That is the map an optimisation would follow; this PR just draws it.
The loop it enables
Allocated/Meancolumns say which shape to look at.GetProperties/GetCustomAttributesreflection, boxing of value-type members).RatioandAlloc Ratioagainst the baseline show the movement.JSON export plus the
bdnatool tracks regressions across commits, as the README and the endjinguidelines describe.
Wiring
solutions/Z3.Linq.Benchmarks/(Exe,IsPackable=false), referencingZ3.Linq.BenchmarkDotNet0.15.8 added to central package management; the project references itversion-less like every other package here.
Z3.Linq.slnx, so it builds with the solution (validated) but - being a plain Exe, nota test project - is not run by
dotnet test.CS1591suppressed as on the test/examples/demo projects; a[Benchmark]method is a toolsurface, not an API.
BenchmarkDotNet.Artifacts/was already in.gitignore; no artifacts are committed.Verification
dotnet build solutions/Z3.Linq.slnx -c Release- clean,TreatWarningsAsErrorsanddocumentation generation on
dotnet run -c Release --project solutions/Z3.Linq.Benchmarks -- --filter '*' --job dryexecuted all six benchmarks and produced the memory table above
./build.ps1 -Configuration Release- 46 tasks, 0 errors, 0 warnings, with the new project inthe build
Release note
No library or public-API change - a developer tool that ships with the source, not the package.
Releases remain on hold under #60 regardless.
🤖 Generated with Claude Code