[eterceros]

Summary

Follow-up to #2248 (merged), refs #2247. Measurements taken after the merge showed that two parts of the benchmark build job added there do not pay off: the per-leg feature narrowing and the actions/cache of the release build. The real bottleneck was CARGO_BUILD_JOBS: 2 on 8-CPU runners. This PR builds one binary with every backend, compiles it with 8 jobs, and drops the cache. 5 files, 2 commits (the second only corrects wording and figures after an Opus 5.5 review).

What this PR changes (before → after)

#	Area	Before (main @ 6f62064)	After (this PR)	Evidence
1	Cargo features of the benchmark binary	Derived per selected leg by build_features.py (for example R4,mongodb); backend=core builds the full set	Always R4,sqlite,postgres,mongodb,elasticsearch	Narrower cold builds took 10 min 47 s to 11 min 56 s against 13 min 22 s to 14 min 22 s for the full set (about 1-3.5 min saved), and the binary differed per dispatch: a build without sqlite has no bulk-export sidecar for mongodb-based servers (crates/hfs/src/main.rs); the benchmark itself loads data with $bulk-submit, so it is not affected. Restoring another feature set's cache saved 43 s
2	Number of compiles	One cargo build in the build job; every leg downloads hfs-bench-binary-<run id>	Unchanged: still exactly one cargo build in the workflow, now always the same binary	grep -n 'cargo build' on the workflow: one command (line 365), the other two hits are comments
3	CARGO_BUILD_JOBS of the build job	2 (workflow default, about 2 of 8 CPUs used)	8, for the build job only (needs: setup kept)	Cold build: 13 min 04 s and 14 min 22 s with 2 jobs, 7 min 03 s with 4, 4 min 40 s with 8; MemAvailable fell by 7.2-7.4 GiB with 4 and with 8 jobs (2 jobs not measured)
4	Release build cache (actions/cache)	Restore and save, one entry per feature set	Removed	Hit from another runner: 12 min 08 s in total (167 s restore plus 9 min 09 s rebuilding 8 workspace crates); a miss paid 126-292 s to save; each entry used 0.8-1.8 GB of the 10 GB repository quota
5	build_features.py and its test	Present	Removed	147 tests remain (154 before)
6	README section "Build job" and "~13-minute" comments	Described the cache and a 49.8 s hit	Measured table, why the cache was dropped; comments say "~5-minute"	-

Evidence

1. Compile time against CARGO_BUILD_JOBS

Dispatches of backend=sqlite tests=prewarm on throwaway branches (tmp/bench-steve-jobs4, tmp/bench-steve-jobs8, identical except the job count and a time and memory sampler around cargo build). All runners report 8 CPUs.

Jobs	State of target/	Wall time	CPU used	Memory drop	Run
2 (before)	cold	13 min 22 s, 13 min 30 s, 14 min 22 s	-	not measured	40322445009, 40335047310, 40385370213 (all 485 crates)
4	cold	7 min 03 s	388 %	7.4 GiB	40521626800 (github-agent6)
8	cold	4 min 40 s	650 %	7.2 GiB	40521631125 (github-agent1)
cargo default (8)	cold	4 min 33 s	-	-	40507218293 (github-agent1)
4	warm	1 min 56 s	288 %	2.7 GiB	40523191815 (github-agent7)
8	warm	37 s	455 %	1.7 GiB	40523196753 (github-agent6)

"Memory drop" is the fall of MemAvailable during the build; the largest single process was about 4.5 GiB.

The two warm rows are not comparable with the cold rows: the 4-job run started from an older build on its runner and the 8-job run reused the target/ of the 4-job cold run. A runner keeps target/ until another workflow checks out into the same workspace with clean: true (about 30 self-hosted jobs in 17 workflow files do), so a build can still start cold.

2. Why the cache was dropped

Run	What it showed
40415455483	R4,sqlite cold build 10 min 47 s, then 176 s to save a 1.05 GB entry
40507070678	full set, partial hit from another feature set's cache (github-agent7): 150 s restore, 12 min 34 s compiling 83 crates, 272 s to save
40510258406	exact-key hit on a different runner (github-agent8): 167 s restore of 1,827 MB, then cargo rebuilt 8 workspace crates in 9 min 09 s because the fresh checkout's file times are newer than the restored outputs; job total 12 min 08 s
40417473144	the 49.8 s hit quoted in #2248 ran on the runner that had saved that cache three minutes earlier, so its own target/ was still warm: it is not typical

Correction to #2248: its description and the README said a cache hit restores in about 100 s and rebuilds 2 crates in 49.8 s; that is the single sample in the last row. The cache was only ever judged with CARGO_BUILD_JOBS=2; with 8 jobs a cold build (4 min 40 s) is already shorter than a typical hit (12 min 08 s).

3. Checks

• python3 -m unittest discover on .github/scripts/fhir-bench with an empty HOME: 147 tests, OK, 1 skipped (the real-data smoke test).
• The workflow YAML parses; resolve-matrix.sh passes bash -n; no reference to build_features, bench-build, actions/cache or a cache warmer remains.
• The final commit was dispatched as-is (backend=sqlite tests=prewarm, run 40524593116): "Build HFS binary" succeeded on github-agent8 with CARGO_BUILD_JOBS: 8, 83 s for the build step and 6 s for the upload. That runner's target/ was warm (3 crates compiled), so this run confirms the job works, not the cold time.

Known limits

• Each timing is a single run on one runner; cold times for the same build ranged from 4 min 33 s to 14 min 22 s depending on the job count, the runner and its load.
• Memory was sampled with MemAvailable and /usr/bin/time -v; it was not measured while another job shares the host.
• The benchmark legs are unchanged: only the build job and its documentation change.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!