Repository navigation
Benchmark harness, figures and panel data in bench/ - #32
Merged
Merged
Conversation
The measurement harness (measure, aggregate, run and monitor, the catalog and its tiers, the held-out sizes) and the wrappers of the vendor libraries move into bench/. Machine profiles are templates here (no host names); private profiles go into bench/machines.local/ (not versioned) and take precedence over the public ones with the same id. A profile with a host runs over ssh. The ducc0 competitor is optional: without the built binding (ducc-ffi/build.sh against a ducc0 source tree) it is left out with a warning. bench/Project.toml instantiates the harness with the checked-out package. Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
A job above max_load (or a busy GPU) is shown as waiting with its reason and is checked again every 30 s. --force still starts at once. Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
m1max and m1gpu are one physical machine: they share the group "mac", so a single monitor call runs the CPU jobs first and the GPU jobs after them. Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
…in the speed-up figure Work in progress, local only: the panel data is not final and bench/overview.jl still names the private hosts (to be removed before any push). Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
ab/bench_pr.jl and ab/bench_mt.jl time single cases for one package version, so two checkouts can be compared with interleaved rounds. panels.toml and the panel tables are the inputs of the two figure scripts. Co-Authored-By: Claude Haiku 5.5 <noreply@anthropic.com>
Member
Author
|
@oscardssmith: This was sitting in a branch.... It explains the benchmarks I do when comparing changes. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Moves the benchmark harness into
bench/, so the measurements behind the benchmark figures can be reproduced from the repository.bench/harness/: measurement (one process per thread count), aggregation, the catalog with its tiers, held-out sizes (heldout.txt, drawn once with a fixed seed), and the run and monitor scripts. Machine profiles are templates with no host names; private profiles go intobench/machines.local/, which is not versioned.bench/overview/run_kinds.jl: library setups. The wrappers for VkFFT, cuFFT and ducc0 are inbench/vkfft,bench/cufft,bench/ducc-ffi(sources and build scripts only; ducc0 is not part of the repository).bench/ab/: single-case A/B scripts for comparing two versions with interleaved rounds.bench/panels.tomlandbench/results/panel-*.csv: inputs of the two figure scripts (fig_speedup.jl,fig_standing.jl). The speed-up figure now shows held-out sizes as outlines; the standing figure shows all cases, smooth and non-smooth lengths.bench/Project.tomlandbench/envs/metal/: environments for the CPU harness and the Metal GPU profile.Nothing in the package source changes.
Notes
🤖 Generated with Claude Code