Fix cabal issue 7557: don’t run benchmarks in parallel or with the build

[ulysses4ever]

Disclaimer: this was created with Claude Code (Opus 5.5, ultra code first and then medium). It went through a couple of design iterations with me.

---

Fixes #7557: cabal bench should not run benchmarks in parallel.

With -j, every node of the install plan runs in parallel, and with per-component builds each benchmark suite is its own node. So cabal bench could run several benchmarks at the same time, from the same package or across a project, and also while other components were still being compiled. Both skew benchmark results.

Change. cabal bench now builds everything first and only then runs the benchmarks, one at a time:

• The bench phase of a package (buildAndRegisterUnpackedPackage) no longer runs its benchmarks, but records them in a DeferredBenchmarks queue threaded from rebuildTargets.
• Once InstallPlan.execute has finished (so all builds, and downloads, are done), rebuildTargets runs the recorded benchmarks one at a time, in plan order (InstallPlan.executionOrder), and records their failures in the BuildOutcomes, exactly as before.
• Building stays parallel and -j defaults are unchanged. No new concurrency primitive is needed. See Note [Running benchmarks].

Behaviour changes to be aware of

• Benchmarks start only once everything is built, so the first benchmark result comes later than before.
• Output order changes even with -j1: builds now come before benchmark runs. The golden outputs of NewBuild/CmdBench/MultipleBenchmarks and Regression/T5309 were re-accepted; both diffs are pure reorderings (same lines).
• Without --keep-going, no benchmark is run if any package fails to build (previously, benchmarks that happened to be built before the failure could run), and no further benchmark is run after one fails. With --keep-going, the benchmarks of all successfully built packages run, one at a time.
• For a legacy whole-package node (e.g. build-type: Custom) with both a library and a benchmark, its dependents may now be built before its benchmark runs, and a failing benchmark no longer marks them as DependentFailed.
• Separate cabal processes are not serialised.
• prs: 0000 in changelog.d/issue-7557.md is a placeholder until there is an upstream PR number.

History. The first commit serialised benchmark runs with a lock; review pointed out that builds could still overlap a running benchmark, so the second commit replaces the lock with deferring benchmarks until everything is built. Squashing on merge is fine.

Testing (Linux, GHC 9.4.7)

• New test cabal-testsuite/PackageTests/NewBuild/CmdBench/Sequential: v2-bench -j3 all over three benchmarks in two packages (one benchmark depends on a library). Each benchmark fails if another one runs at the same time, and the test checks that no Configuring/Preprocessing/Building/Compiling/Linking line appears after the first Benchmark …: RUNNING.... It fails 3/3 on master (benchmarks overlap) and 3/3 with only the first, lock-based commit (a benchmark is still being linked while another runs), and passes 3/3 with this PR.
• Related cabal-testsuite tests pass (22): NewBuild/CmdBench/*, NewBuild/CmdTest/*, NewBuild/T3460, NewBuild/CmdBuild/OnlyConfigure, TestChangeDir, Target, Regression/T5309, AutogenModules/MainIsBench, AllowNewer, AllowOlder, Freeze/*, ShowBuildInfo/Complex/single, GHCJS/BuildRunner. No other golden file in cabal-testsuite has a build step after a benchmark run.
• cabal-install:test:integration-tests2: 29/30 pass; the one failure is a haddock test that needs boot-library .haddock files missing from Ubuntu's GHC package, unrelated to this change.
• Manually checked: a failing benchmark with and without --keep-going, a build failure with and without --keep-going, and --build-timings.
• fourmolu 0.12 and typos are clean; hlint 3.8 reports no hints (CI uses 3.10).

QA Notes

In a project with several benchmark suites (e.g. in two packages, one of whose benchmarks depends on a library), run cabal bench all -j4 from a clean state:

• All build output (Configuring, Building, [n of m] Compiling, Linking) should appear before the first Benchmark <name>: RUNNING....
• Each benchmark should print Benchmark <name>: FINISH (or ERROR) before the next one starts, in the order of the build plan.

Make one benchmark exit with failure: without --keep-going the following benchmarks are not run; with --keep-going they are, and the failure is reported at the end. Make one component fail to compile: without --keep-going no benchmark is run; with it, the benchmarks of the components that built are run.

---

Template Α: This PR modifies behaviour or interface

• Patches conform to the coding conventions.
• Any changes that could be relevant to users have been recorded in the changelog.
  • Is the change significant? If so, remember to add significance: significant in the changelog file.
• The documentation has been updated, if necessary.
• Manual QA notes have been included.
• Tests have been added.

🤖 Generated with Claude Code

https://claude.ai/code/session_016nzdeKCYFDCkWsyM6etgkL

[ulysses4ever]

@Bodigrim are you interested in approving? I thought I'd ask since you left a comment...

[Bodigrim]

Thanks for working on this!

[Mikolaj]

Would the "significant" changelog flag make sense? Maybe a release note snippet? This is sure to break peoples workflows, often for good.

[ulysses4ever]

@Mikolaj, agreed on significance, added. Also expanded the changelog entry with the user-visible consequences, which should be helpful for release-notes readers. Let me know if you have specific suggestions for it though.

Thanks for the comments, I think I addressed all of them by basically accepting all of your suggestions and resolved the comments threads (I hope I wasn't too eager to do it, let me know!).