ci(bench): compile once with the full backend set and 8 build jobs

.github/scripts/fhir-bench/README.md

@@ -55,9 +55,22 @@ Outputs: `run-time.md` and `run-timing.json`. `compare_legs.py --timing run-time
 
 ## Build job
 
-`build_features.py '<matrix json>'` derives the `cargo build --features` list from the selected legs: `R4` always, plus only the backends needed (`sqlite`, `postgres`, `mongodb`; `elasticsearch` for any `*-elasticsearch` leg), in fixed order. The workflow builds `helios-hfs` in release mode with `--no-default-features --features <list>`.
+The workflow compiles **once**: the `build` job runs the only `cargo build` of the workflow and uploads `target/release/hfs`; every leg downloads that binary and picks its backend at runtime through `HFS_STORAGE_BACKEND`. The binary is always built with the full feature set `R4,sqlite,postgres,mongodb,elasticsearch` (FHIR R4 only). Building only the backends of the selected legs saved just about 1-3.5 min on a cold build (measured with 2 jobs) and made the binary differ between dispatches: a build without `sqlite` has no bulk-export job store for mongodb-based servers (the benchmark itself does not call `$export`).
 
-Rust cache: `actions/cache@v5` stores the cargo registry/git data and `target/release/{deps,build,.fingerprint}`. Key = runner OS + feature list + `Cargo.lock` hash, with restore-keys falling back to the same features, then any bench cache. The first run per key is cold.
+The `build` job sets `CARGO_BUILD_JOBS: 8`; every other job keeps the workflow default of 2. The runners have 8 CPUs and the compile is CPU-bound. Measured with `backend=sqlite tests=prewarm` dispatches on throwaway branches, full feature set, wall time of `cargo build --release`:
+
+| `CARGO_BUILD_JOBS` | `target/` | Wall time | CPU used | Drop in available memory | Largest process | Run |
+|---|---|---:|---:|---:|---:|---|
+| 2 (before) | cold | 13 min 22 s, 13 min 30 s, 14 min 22 s | - | - | - | runs 40322445009, 40335047310, 40385370213 (cargo's own timing, all 485 crates) |
+| 4 | cold | 7 min 03 s | 388 % | 7.4 GiB | 4.5 GiB | run 40521626800 (github-agent6) |
+| 8 | cold | 4 min 40 s | 650 % | 7.2 GiB | 4.5 GiB | run 40521631125 (github-agent1) |
+| cargo default (8) | cold | 4 min 33 s | - | - | - | run 40507218293 (github-agent1, cache-warming self-check) |
+| 4 | warm | 1 min 56 s | 288 % | 2.7 GiB | 2.9 GiB | run 40523191815 (github-agent7) |
+| 8 | warm | 37 s | 455 % | 1.7 GiB | 2.5 GiB | run 40523196753 (github-agent6) |
+
+The two warm rows are not cold numbers and not comparable with each other: the 4-job run started from an older build on its runner, the 8-job run reused the `target/` of the 4-job cold run. A runner keeps its `target/` until another workflow checks out into the same workspace with the default `clean: true` (about 30 self-hosted jobs in 17 workflow files do, counted by their checkout step), which is why a full-set build on `github-agent3` can still start cold.
+
+There is no build cache. An `actions/cache` of the release build was tried and dropped: on an exact-key hit from a different runner the restore took 167 s for 1,827 MB and cargo still rebuilt eight workspace crates in 9 min 09 s (12 min 08 s in total, run 40510258406), because the fresh checkout's file times are newer than the restored outputs; a miss paid 126-292 s to save; and every entry used 0.8-1.8 GB of the repository's 10 GB cache quota. The cache was only judged with 2 jobs; a hit with 8 jobs was not measured. The 49.8 s hit quoted earlier (run 40417473144) ran on the runner that had saved that cache minutes before and is not typical.
 
 ## Docker host
 

.github/scripts/fhir-bench/resolve-matrix.sh

@@ -2,7 +2,7 @@
 #
 # Resolve the `backend` workflow_dispatch input into a job matrix for
 # fhir-benchmark.yml, and validate the free-text tuning inputs in one place
-# before the ~13-minute build runs — once, not once per leg; its single
+# before the ~5-minute build runs — once, not once per leg; its single
 # binary serves every leg in the resolved matrix.
 #
 # Called from: the `setup` job's "Resolve matrix from backend input" step.
@@ -150,7 +150,7 @@ fi
 # HFS_MONGO_POOL from the same raw input (env:
 # IN_HFS_MONGO_MAX_CONNECTIONS) without re-checking it, trusting
 # this step to have already rejected a bad value before the
-# ~13-minute build runs on every leg, including sqlite/postgres.
+# ~5-minute build runs on every leg, including sqlite/postgres.
 HFS_MONGO_POOL="${IN_HFS_MONGO_MAX_CONNECTIONS:-32}"
 # At most 4 digits (matched BEFORE the -gt 1000 test below): an
 # overlong digit string handed straight to `[ -gt ]` risks "value

.github/workflows/fhir-benchmark.yml

@@ -249,7 +249,7 @@ jobs:
   # max_parallel > 1 (clamped to 1..2 below — the shared 4-CPU Docker
   # host). Include order = intended run order: bare backends first. Also
   # validates the free-text tuning inputs here, in one place, before the
-  # ~13-minute build.
+  # ~5-minute build.
   # ────────────────────────────────────────────────────────────────────────────
   setup:
     name: Resolve backend matrix
@@ -300,8 +300,8 @@ jobs:
         run: bash "$GITHUB_WORKSPACE/.fhir-bench-setup/.github/scripts/fhir-bench/resolve-matrix.sh"
 
   # ────────────────────────────────────────────────────────────────────────────
-  # Compile one release `hfs` binary with the backends the selected legs need
-  # (sqlite, postgres, mongodb, elasticsearch) linked in — the backend is
+  # Compile one release `hfs` binary with all four backends (sqlite, postgres,
+  # mongodb, elasticsearch) always linked in — the backend is
   # selected at runtime via HFS_STORAGE_BACKEND, so a single build serves every
   # matrix leg. `ui` is dropped so the benchmark exercises the FHIR API only.
   # ────────────────────────────────────────────────────────────────────────────
@@ -310,6 +310,12 @@ jobs:
     needs: setup
     runs-on: [self-hosted, Linux]
     timeout-minutes: 120
+    env:
+      # Raised for this job only (the workflow default of 2 applies elsewhere): the
+      # runners have 8 CPUs and the compile is CPU-bound. Cold builds took 13-15 min
+      # with 2 jobs, 7 min with 4 and 4 min 40 s with 8; 4 and 8 jobs lowered
+      # MemAvailable by the same 7.2-7.4 GiB (2 jobs not measured); see the "Build job" section of .github/scripts/fhir-bench/README.md.
+      CARGO_BUILD_JOBS: 8
 
     steps:
       - name: Checkout
@@ -318,7 +324,6 @@ jobs:
           clean: false
 
       - name: Install Rust toolchain
-        id: rust
         uses: dtolnay/rust-toolchain@stable
         with:
           toolchain: stable
@@ -346,60 +351,20 @@ jobs:
       # mongodb pulls in mongocrypt-sys (native libmongocrypt, needs cmake);
       # these runners already build it via ci.yml's --all-features. s3 is not
       # compiled: no s3 leg is benchmarked (see header).
-      # Compile only the backends the selected legs need (FHIR R4 only, always).
-      # build_features.py maps the setup job's matrix to the feature list:
-      # sqlite -> sqlite, postgres -> postgres, mongodb -> mongodb,
-      # <x>-elasticsearch -> x + elasticsearch; core/all gives the full set.
-      - name: Resolve build features
-        id: features
-        env:
-          MATRIX: ${{ needs.setup.outputs.matrix }}
-        run: |
-          FEATURES=$(python3 .github/scripts/fhir-bench/build_features.py "$MATRIX")
-          echo "features=$FEATURES" >> "$GITHUB_OUTPUT"
-          # actions/cache keys cannot contain commas (run 40415264411 failed with
-          # "Key Validation Error"), so the cache key uses the list joined with +.
-          echo "key=${FEATURES//,/+}" >> "$GITHUB_OUTPUT"
-          echo "Building with features: $FEATURES"
-
-      # Rust cache. The repo has no cache action; its convention is a warm
-      # target/ kept on the self-hosted runner (checkout clean: false plus
-      # .github/actions/cargo-target-gc). That never helped this job: ci.yml
-      # builds the debug profile, so the release profile starts cold on every
-      # runner that has not built it (485 crates, 14 min). actions/cache is the
-      # first-party action (pinned by major tag like the others). Key = lockfile
-      # + feature list (different features compile different crates); the
-      # restore-keys fallbacks reuse a close cache and cargo rebuilds only the
-      # difference. Saved after a successful build on any branch (keys are
-      # immutable, so no clash); a branch reads its own and main's caches.
-      # Only target/release/{deps,build,.fingerprint} are cached, not
-      # incremental/ or the linked binary. The first run per key is still cold.
-      # Key includes the rustc version (steps.rust.outputs.cachekey) so a toolchain
-      # bump gets a fresh key. This job does not run cargo-target-gc and release
-      # deps/ can be 2-3.5 GB, so key churn can evict other entries.
-      - name: Cache Rust build (release)
-        uses: actions/cache@v5
-        with:
-          path: |
-            ${{ env.CARGO_HOME }}/registry/index
-            ${{ env.CARGO_HOME }}/registry/cache
-            ${{ env.CARGO_HOME }}/git/db
-            target/release/deps
-            target/release/build
-            target/release/.fingerprint
-          key: bench-release-${{ runner.os }}-${{ steps.features.outputs.key }}-${{ steps.rust.outputs.cachekey }}-${{ hashFiles('Cargo.lock') }}
-          restore-keys: |
-            bench-release-${{ runner.os }}-${{ steps.features.outputs.key }}-${{ steps.rust.outputs.cachekey }}-
-            bench-release-${{ runner.os }}-${{ steps.features.outputs.key }}-
-            bench-release-${{ runner.os }}-
-
+      # The binary is ALWAYS built with the full feature set (FHIR R4 only), and this
+      # is the only `cargo build` of the workflow: every leg downloads the artifact
+      # and picks its backend at runtime via HFS_STORAGE_BACKEND. Narrowing the set
+      # per leg saved only about 1-3.5 min on a cold build and made the binary differ
+      # between dispatches (a build without `sqlite` has no bulk-export job store
+      # for mongodb-based servers), so it was dropped; an actions/cache of the
+      # release build was dropped too (a hit from another runner still rebuilt the
+      # workspace crates and a miss paid 2-5 min to save). `ui` is left out so the
+      # benchmark exercises the FHIR API only.
       - name: Build release binary
-        env:
-          FEATURES: ${{ steps.features.outputs.features }}
         run: |
           cargo build --release -p helios-hfs \
             --no-default-features \
-            --features "$FEATURES"
+            --features "R4,sqlite,postgres,mongodb,elasticsearch"
 
       - name: Upload binary
         uses: actions/upload-artifact@v7
@@ -1128,7 +1093,7 @@ jobs:
           # HFS silently falls back to 10 on a non-numeric value
           # (mongodb/backend.rs, config.max_connections). Already validated
           # in setup's "Resolve matrix from backend input" (same regex, plus
-          # the 1..1000 range check) before the ~13-minute build — no re-check
+          # the 1..1000 range check) before the ~5-minute build — no re-check
           # here, since re-validating the raw input here too (as this step
           # used to) is redundant on every leg, including sqlite/postgres.
           HFS_MONGO_POOL="${IN_HFS_MONGO_MAX_CONNECTIONS:-32}"