spack-benchmark

Spack extension that measures how long the concretizer takes. spack solve-benchmark runs suites of concretization benchmarks in isolated processes, records each sample in a result file, and compares two result files with a statistical test.

Installation

$ unset SPACK_PYTHON  # ensure Spack uses the Python from the virtual environment
$ python3 -m venv .venv
$ source .venv/bin/activate
$ pip install "git+https://github.com/spack/spack-benchmark#egg=spack_solver_benchmark"

Quick start

List the suites:

$ spack solve-benchmark list
compilers        solves with 1 to 8 compilers offered for reuse
duplicates       solves under each duplicates strategy
env-roots        RADIUSS packages solved as one environment, per solve kind
propagation      variant propagation
radiuss          RADIUSS packages, one solve each
reuse            solves with 0 to 1900 specs of the E4S stack offered for reuse
static-analysis  solves with static analysis off and on, under the requirements of E4S

Run one, with 3 samples per benchmark and 4 samples at a time:

$ spack solve-benchmark run radiuss -r 3 -n 4 -o radiuss.json
==> Warming up suite radiuss...
==> suite radiuss: builtin@14d4cfe7a8195d714b09d67dd2a104c4cd2ea867, linux-ubuntu24.04-alderlake, 2 compilers from lockfile
==> clingo 5.5.0, Python 3.14.5
==>    14.9s  [  1%]   2.0 GB  ok                ascent [kind=together, roots=1] (seed 2820513866)
==>    13.9s  [  2%]   2.0 GB  ok                mfem [kind=together, roots=1] (seed 3231483282)
...

Each line shows a sample: its time, the peak memory of its process, its status, the benchmark, and the seed used to shuffle its problem.

Print a markdown report on the result file:

$ spack solve-benchmark report radiuss.json

Workflows

A change to Spack

Run the suite with each Spack, and read recipes from the same checkout of spack-packages:

$ ~/spack-develop/bin/spack solve-benchmark run radiuss -r 5 --builtin ~/spack-packages -o before.json
$ ~/spack-pr/bin/spack solve-benchmark run radiuss -r 5 --builtin ~/spack-packages -o after.json
$ spack solve-benchmark compare before.json after.json -o plot.png

compare exits with status 1 if a benchmark changed solution.

A change to spack-packages

Run the suite with the same Spack on each checkout of spack-packages, and pass --packages-change to compare:

$ spack solve-benchmark run radiuss -r 5 --builtin ~/spack-packages -o before.json
$ spack solve-benchmark run radiuss -r 5 --builtin ~/pr-checkout -o after.json
$ spack solve-benchmark compare before.json after.json --packages-change

Both files must then come from the same Spack commit, in a git checkout with no uncommitted changes. The builtin commits may differ. Changed recipes change the hashes of the specs that use them, so hash changes are listed but do not make compare fail.

A change to the .lp files

Dump the problems of a suite once, then replay them before and after the change, so setup does not run again:

$ spack solve-benchmark run radiuss --dump problems -o dump.json
...
==> wrote the problems of 34 benchmarks to problems
$ spack solve-benchmark replay problems -r 5 -o before.json
$ $EDITOR lib/spack/spack/solver/concretize.lp
$ spack solve-benchmark replay problems -r 5 -o after.json
$ spack solve-benchmark compare before.json after.json

See Replaying dumped problems for what is dumped and what a replay checks.

Reproducing a slow sample

Rerun the benchmark with the seed printed on the line of the slow sample:

$ spack solve-benchmark run radiuss --filter '^mfem ' --seed 3231483282 -o again.json

If the sample was dumped, replay --dumped-seed --filter NAME solves the dumped problem with the same order of facts, without running setup.

How it works

What a run solves

Runs are hermetic: a benchmark solves the same problem on every machine with the same platform, operating system, microarchitecture and spack-packages commit. Each worker reads only:

• Spack's defaults configuration scope;
• the configuration scope of the suite, which sets the target granularity;
• the configuration scope of the benchmark, if it has one;
• a scope of the run, with the builtin checkout, an empty store, and a misc cache shared by the workers of the run.

The site, system and user configuration of the machine are ignored. The compilers come from a lockfile in the suite, and are offered for reuse in every solve.

Samples

Each sample solves a benchmark once, in its own process. It first shuffles the facts of its problem, since the time clingo takes depends on their order, and records the seed of the shuffle. A sample ends with one of these statuses:

• ok;
• timeout, if it ran longer than the timeout of its benchmark;
• invariant-failed, if its result does not satisfy an invariant of the benchmark;
• error, if the solve failed.

run and replay exit with status 1 if any sample is not ok. At the end, run warns about each benchmark whose ok samples found more than one solution: these are ties between optimal solutions.

Result files

Each sample is appended to the result file as soon as it completes. A result file has one JSON record per line, and the first record is the metadata of the run:

• the suite, the host arch, and the configuration scopes;
• the builtin commit, and whether its checkout had uncommitted changes;
• the Spack commit, and whether its checkout had uncommitted changes;
• the compilers in the lockfile;
• the versions of clingo and Python.

run fails if the builtin commit changes while it runs, or if a clean checkout gets uncommitted changes.

Each following record is a sample, with:

• its benchmark, series, parameters and configuration scope;
• its seed and status;
• the time spent in each phase of each solve;
• the clingo statistics of each solve: atoms, rules, bodies, vars, constraints, choices, conflicts and restarts;
• the hashes of the concrete roots;
• its profile, if it was profiled;
• the peak resident memory of its process, in bytes.

A sample has one solve per round for when_possible, and one per root for separately.

Profiling

run --profile adds one profiled sample per benchmark to the result file. A thread of the worker reads the Python stack of the solve every 10 ms, and attributes to it the time since the previous reading. Time spent in clingo shows up as _run_clingo. Profiled samples are left out of the timings of compare and report.

report lists the functions with the most own time, with the share of the sampled time in which each is the innermost frame (own), or anywhere in the stack (inclusive). Frames are written as function (file:line), with the path of the file relative to its sys.path entry and the first line of the function, e.g. reorder_flags (spack/solver/asp.py:3182). --focus PATTERN restricts the profile to the stacks with a frame matching a regular expression, so --focus '^reorder_flags ' selects the post-processing of the flags of a solve.

Replaying dumped problems

replay is the other command writing a result file. run --dump DIR writes to a new directory, for the first sample of each benchmark, the problem of each of its solves and the names of the control files clingo loaded. The statements of a problem are stored in the order setup emitted them.

replay loads, grounds and solves each problem with the control files of the Spack running it, and records the time of each of those phases. Each replayed sample shuffles the problems with its seed, as a sample of run does, or with the seed of the dumped sample of its benchmark with --dumped-seed. The dumped benchmarks keep their timeouts.

A replayed sample is error if its problem is unsatisfiable or its best model has errors. Invariants are not checked and no hashes are recorded, since no specs are built from the models, so compare reports no hash changes or ties for replays. A control file added to the solver after the dump is not loaded, unless an #include in a loaded file names it.

Reporting on one file

report prints:

• the median share of each phase in the time of each benchmark, with the time outside the phases as other;
• for each series, the exponent x of a fit of the median time of each phase to k^x, k being the parameter of the series;
• the median clingo statistics of each benchmark;
• the benchmarks whose ok samples have more than one solution;
• if the file has profiled samples, their profile, see Profiling.

--filter PATTERN restricts the report to the benchmarks whose name matches a regular expression.

Comparing two files

compare refuses two files:

• from different suites or host archs;
• run on different builtin commits, or on a builtin checkout with uncommitted changes;
• that timed different phases, such as a replay and a run;
• replayed from different dumps.

With --packages-change, the second condition is on the Spack commit instead, and a Spack outside a git checkout is refused too, see A change to spack-packages.

It then prints, in markdown:

1. the Spack and builtin commits and the clingo and Python versions of each file, with a warning if the clingo or Python versions differ;
2. the benchmarks whose status changed;
3. the benchmarks whose solution changed, and the ties in each file;
4. for each phase, the median time of each benchmark in both files, and whether the phase improved or regressed: two one-sided Wilcoxon tests on the ratios of the medians, at a 5% significance level.

The plot shows the median time of each phase. Benchmarks of a series that differ in one numeric parameter share a panel, with time against that parameter: env-roots has one panel per kind, with time against the number of roots.

Writing a suite

A suite is a directory under src/spack_solver_benchmark/suites/:

suites/radiuss/
  suite.py         defines SUITE
  config/          configuration scope of the suite
  compilers/       spack.yaml and spack.lock of the compilers offered for reuse
  reuse/           optional, spack.yaml and spack.lock of the specs a benchmark can reuse

config/ and compilers/ can be symlinks to those of another suite, as in propagation, which shares both with radiuss.

Benchmarks

suite.py creates the suite and adds benchmarks to it:

from spack_solver_benchmark.suite import Suite, no_duplicates, satisfies

SUITE = Suite(description="RADIUSS packages, one solve each", platforms=["linux"])
SUITE.add(
    name="mfem",
    specs=["mfem"],
    kind="together",
    timeout=120,
    invariants=[satisfies("mfem ^hypre")],
)

The specs of a benchmark are solved as the roots of an environment, as kind prescribes:

• together solves them all at once, as unify: true does;
• when_possible solves them in rounds, as unify: when_possible does;
• separately solves them one at a time, as unify: false does, but not in parallel.

The kind and the number of roots are parameters of every benchmark.

Series

A series is a loop adding benchmarks with the same name and different parameters. Parameters are recorded in each sample and appended to the name of the benchmark:

• compilers=k offers only the first k compilers of the lockfile;
• reuse=k offers the first k reusable specs of the reuse lockfile, dependencies first, so that the specs offered include their dependencies;
• other parameters go in params.

A name such as mfem [kind=together, roots=1, compilers=4] lets --filter 'compilers=(1|8)' select benchmarks by parameter.

for k in (1, 2, 4, 8):
    SUITE.add(
        name="mfem", specs=["mfem"], kind="together", compilers=k, invariants=[no_duplicates()]
    )

config is a configuration scope of the benchmark, which takes precedence over the scope of the suite. It is recorded in each sample but is not a parameter, so a series varying it names the value in params:

for strategy in ("none", "minimal", "full"):
    SUITE.add(
        name="mfem",
        specs=["mfem"],
        kind="together",
        params={"duplicates": strategy},
        config={"concretizer": {"duplicates": {"strategy": strategy}}},
    )

Invariants

• satisfies(constraint) requires every concrete root to satisfy a constraint;
• no_duplicates(allow=[...]) requires each package to appear at most once among the link and run dependencies of a root;
• reused(constraint) requires every root to be reused when a spec offered from the reuse lockfile satisfies the constraint.

An invariant is a function taking a Solved tuple, whose roots are the concrete roots and whose reuse_set holds the specs offered from the reuse lockfile, and returning an error message or None.

Lockfiles

regenerate concretizes the lockfiles of a suite again, with the configuration of the suite and the recipes of the builtin checkout (or of --builtin PATH):

$ spack solve-benchmark regenerate reuse
==> Concretizing the lockfiles of suite reuse with builtin@14d4cfe7a8195d714b09d67dd2a104c4cd2ea867...
==> wrote compilers/spack.lock, 2 roots, 38 specs
==> wrote reuse/spack.lock, 279 roots, 1975 specs

• The reuse environment is concretized with the compilers of the new compiler lockfile offered for reuse.
• Compilers on the host are never detected, so the compiler environment has to declare the compiler that builds the others as an external. Its roots must not be externals, since the suite scope does not declare them.
• The lockfiles are written only if every environment concretizes, through the symlinks of the suite if there are any.

The specs in the lockfiles remain reusable when the builtin checkout moves, so they only need regenerating to change their content.

Tests

$ spack python -m pytest tests

The tests in tests/test_run.py run a small suite end to end, reading recipes from the builtin repository of the Spack running them.