From c61849e859f05b3644c1924868da31bdcf2845d2 Mon Sep 17 00:00:00 2001 From: Edson Terceros Date: Tue, 27 Oct 2026 14:47:48 +0000 Subject: [PATCH 1/2] ci(bench): compile once with the full backend set and 8 build jobs Follow-up to #2248. - Always build the full feature set (R4,sqlite,postgres,mongodb,elasticsearch): narrowing it per leg saved only 2-4 min on a cold build and produced a different binary per dispatch. Remove build_features.py and its test. - Raise CARGO_BUILD_JOBS to 8 for the build job only. The runners have 8 CPUs and the compile is CPU-bound; measured cold, 2 jobs took 13-15 min, 4 jobs 7 min 03 s and 8 jobs 4 min 40 s with the same peak memory. - Drop the actions/cache step added in #2248: on a hit from another runner cargo still rebuilt the workspace crates (12 min 08 s in total), a restore cost 99-167 s and a miss paid 126-277 s to save. - The workflow still has a single cargo build, in the build job; every leg downloads its artifact. Keep needs: setup on the build job and update the comments that quoted a ~13-minute build. - README: rewrite the Build job section with the measurements. Co-Authored-By: Claude Sonnet 5.5 --- .github/scripts/fhir-bench/README.md | 17 ++++- .github/scripts/fhir-bench/build_features.py | 46 ------------ .github/scripts/fhir-bench/resolve-matrix.sh | 4 +- .../scripts/fhir-bench/test_build_features.py | 45 ------------ .github/workflows/fhir-benchmark.yml | 71 +++++-------------- 5 files changed, 35 insertions(+), 148 deletions(-) delete mode 100644 .github/scripts/fhir-bench/build_features.py delete mode 100644 .github/scripts/fhir-bench/test_build_features.py diff --git a/.github/scripts/fhir-bench/README.md b/.github/scripts/fhir-bench/README.md index 327735b56..180cdcc43 100644 --- a/.github/scripts/fhir-bench/README.md +++ b/.github/scripts/fhir-bench/README.md @@ -55,9 +55,22 @@ Outputs: `run-time.md` and `run-timing.json`. `compare_legs.py --timing run-time ## Build job -`build_features.py ''` derives the `cargo build --features` list from the selected legs: `R4` always, plus only the backends needed (`sqlite`, `postgres`, `mongodb`; `elasticsearch` for any `*-elasticsearch` leg), in fixed order. The workflow builds `helios-hfs` in release mode with `--no-default-features --features `. +The workflow compiles **once**: the `build` job runs the only `cargo build` of the workflow and uploads `target/release/hfs`; every leg downloads that binary and picks its backend at runtime through `HFS_STORAGE_BACKEND`. The binary is always built with the full feature set `R4,sqlite,postgres,mongodb,elasticsearch` (FHIR R4 only). Building only the backends of the selected legs saved just 2-4 min on a cold build (measured with 2 jobs) and made the binary differ between dispatches: a build without `sqlite` has no bulk-export job store for mongodb-based servers (the benchmark itself does not call `$export`). -Rust cache: `actions/cache@v5` stores the cargo registry/git data and `target/release/{deps,build,.fingerprint}`. Key = runner OS + feature list + `Cargo.lock` hash, with restore-keys falling back to the same features, then any bench cache. The first run per key is cold. +The `build` job sets `CARGO_BUILD_JOBS: 8`; every other job keeps the workflow default of 2. The runners have 8 CPUs and the compile is CPU-bound. Measured with `backend=sqlite tests=prewarm` dispatches on throwaway branches, full feature set, wall time of `cargo build --release`: + +| `CARGO_BUILD_JOBS` | `target/` | Wall time | CPU used | Drop in available memory | Largest process | Run | +|---|---|---:|---:|---:|---:|---| +| 2 (before) | cold | 13 min 04 s, 14 min 22 s | - | - | - | runs 40353889556, 40385370213 (cargo's own timing) | +| 4 | cold | 7 min 03 s | 388 % | 7.4 GiB | 4.7 GB | run 40521626800 (github-agent6) | +| 8 | cold | 4 min 40 s | 650 % | 7.2 GiB | 4.7 GB | run 40521631125 (github-agent1) | +| cargo default (8) | cold | 4 min 33 s | - | - | - | run 40507218293 (github-agent1, cache-warming self-check) | +| 4 | warm | 1 min 56 s | 288 % | 2.7 GiB | 3.1 GB | run 40523191815 (github-agent7) | +| 8 | warm | 37 s | 455 % | 1.7 GiB | 2.6 GB | run 40523196753 (github-agent6) | + +The two warm rows are not cold numbers and not comparable with each other: the 4-job run started from an older build on its runner, the 8-job run reused the `target/` of the 4-job cold run. A runner keeps its `target/` until another workflow checks out into the same workspace with the default `clean: true` (about 30 self-hosted jobs in 17 workflow files do, counted by their checkout step), which is why two builds on `github-agent3` three hours apart were both cold. + +There is no build cache. An `actions/cache` of the release build was tried and dropped: on an exact-key hit from a different runner the restore took 167 s for 1,827 MB and cargo still rebuilt eight workspace crates in 9 min 09 s (12 min 08 s in total, run 40510258406), because the fresh checkout's file times are newer than the restored outputs; a miss paid 126-277 s to save; and every entry used 0.8-1.8 GB of the repository's 10 GB cache quota. The cache was only judged with 2 jobs; a hit with 8 jobs was not measured. The 49.8 s hit quoted earlier (run 40417473144) ran on the runner that had saved that cache minutes before and is not typical. ## Docker host diff --git a/.github/scripts/fhir-bench/build_features.py b/.github/scripts/fhir-bench/build_features.py deleted file mode 100644 index de52947d4..000000000 --- a/.github/scripts/fhir-bench/build_features.py +++ /dev/null @@ -1,46 +0,0 @@ -#!/usr/bin/env python3 -"""Derive the `cargo build --features` list from the resolved benchmark matrix. - -Usage: build_features.py '' (or the JSON on stdin) - -The matrix is resolve-matrix.sh's output, {"include":[{"backend":"postgres-elasticsearch",...}]}. -Mapping: sqlite -> sqlite; postgres -> postgres; mongodb -> mongodb; --elasticsearch -> x plus elasticsearch. FHIR R4 is always on (no other -FHIR version is ever compiled). Output order is fixed so the same leg set -always yields the same string (it is also part of the build cache key). -""" -import json -import sys - -PRIMARIES = ("sqlite", "postgres", "mongodb") -ORDER = ("R4", "sqlite", "postgres", "mongodb", "elasticsearch") - - -def features_for(matrix): - feats = {"R4"} - legs = matrix.get("include", []) - if not legs: - raise ValueError("matrix has no legs") - for leg in legs: - name = leg["backend"] - base, sep, es = name.partition("-") - if base not in PRIMARIES or (sep and es != "elasticsearch"): - raise ValueError(f"unknown backend leg {name!r}") - feats.add(base) - if sep: - feats.add("elasticsearch") - return ",".join(f for f in ORDER if f in feats) - - -def main(argv): - raw = argv[1] if len(argv) > 1 else sys.stdin.read() - try: - print(features_for(json.loads(raw))) - except (ValueError, KeyError, TypeError) as e: - print(f"::error::build_features: {e}", file=sys.stderr) - return 1 - return 0 - - -if __name__ == "__main__": - sys.exit(main(sys.argv)) diff --git a/.github/scripts/fhir-bench/resolve-matrix.sh b/.github/scripts/fhir-bench/resolve-matrix.sh index 26c8f94b0..a74fed609 100755 --- a/.github/scripts/fhir-bench/resolve-matrix.sh +++ b/.github/scripts/fhir-bench/resolve-matrix.sh @@ -2,7 +2,7 @@ # # Resolve the `backend` workflow_dispatch input into a job matrix for # fhir-benchmark.yml, and validate the free-text tuning inputs in one place -# before the ~13-minute build runs — once, not once per leg; its single +# before the ~5-minute build runs — once, not once per leg; its single # binary serves every leg in the resolved matrix. # # Called from: the `setup` job's "Resolve matrix from backend input" step. @@ -150,7 +150,7 @@ fi # HFS_MONGO_POOL from the same raw input (env: # IN_HFS_MONGO_MAX_CONNECTIONS) without re-checking it, trusting # this step to have already rejected a bad value before the -# ~13-minute build runs on every leg, including sqlite/postgres. +# ~5-minute build runs on every leg, including sqlite/postgres. HFS_MONGO_POOL="${IN_HFS_MONGO_MAX_CONNECTIONS:-32}" # At most 4 digits (matched BEFORE the -gt 1000 test below): an # overlong digit string handed straight to `[ -gt ]` risks "value diff --git a/.github/scripts/fhir-bench/test_build_features.py b/.github/scripts/fhir-bench/test_build_features.py deleted file mode 100644 index 7f62ea02e..000000000 --- a/.github/scripts/fhir-bench/test_build_features.py +++ /dev/null @@ -1,45 +0,0 @@ -import json -import unittest - -from build_features import features_for - - -def m(*legs): - return {"include": [{"backend": b, "port": 1} for b in legs]} - - -class BuildFeatures(unittest.TestCase): - def test_single_legs(self): - self.assertEqual(features_for(m("sqlite")), "R4,sqlite") - self.assertEqual(features_for(m("postgres")), "R4,postgres") - self.assertEqual(features_for(m("mongodb")), "R4,mongodb") - - def test_composite_legs(self): - self.assertEqual(features_for(m("sqlite-elasticsearch")), "R4,sqlite,elasticsearch") - self.assertEqual(features_for(m("postgres-elasticsearch")), "R4,postgres,elasticsearch") - self.assertEqual(features_for(m("mongodb-elasticsearch")), "R4,mongodb,elasticsearch") - - def test_core_is_todays_list(self): - legs = ["sqlite", "postgres", "mongodb", "sqlite-elasticsearch", - "postgres-elasticsearch", "mongodb-elasticsearch"] - self.assertEqual(features_for(m(*legs)), "R4,sqlite,postgres,mongodb,elasticsearch") - - def test_order_independent(self): - self.assertEqual(features_for(m("postgres", "sqlite")), "R4,sqlite,postgres") - - def test_elasticsearch_group(self): - legs = ["sqlite-elasticsearch", "postgres-elasticsearch", "mongodb-elasticsearch"] - self.assertEqual(features_for(m(*legs)), "R4,sqlite,postgres,mongodb,elasticsearch") - - def test_errors(self): - with self.assertRaises(ValueError): - features_for(m("redis")) - with self.assertRaises(ValueError): - features_for({"include": []}) - - def test_json_roundtrip(self): - self.assertEqual(features_for(json.loads(json.dumps(m("sqlite")))), "R4,sqlite") - - -if __name__ == "__main__": - unittest.main() diff --git a/.github/workflows/fhir-benchmark.yml b/.github/workflows/fhir-benchmark.yml index 7bd998cb0..c185c2a81 100644 --- a/.github/workflows/fhir-benchmark.yml +++ b/.github/workflows/fhir-benchmark.yml @@ -249,7 +249,7 @@ jobs: # max_parallel > 1 (clamped to 1..2 below — the shared 4-CPU Docker # host). Include order = intended run order: bare backends first. Also # validates the free-text tuning inputs here, in one place, before the - # ~13-minute build. + # ~5-minute build. # ──────────────────────────────────────────────────────────────────────────── setup: name: Resolve backend matrix @@ -310,6 +310,12 @@ jobs: needs: setup runs-on: [self-hosted, Linux] timeout-minutes: 120 + env: + # Raised for this job only (the workflow default of 2 applies elsewhere): the + # runners have 8 CPUs and the compile is CPU-bound. Cold builds took 13-15 min + # with 2 jobs, 7 min with 4 and 4 min 40 s with 8 at the same ~7.5 GB peak + # memory; see the "Build job" section of .github/scripts/fhir-bench/README.md. + CARGO_BUILD_JOBS: 8 steps: - name: Checkout @@ -318,7 +324,6 @@ jobs: clean: false - name: Install Rust toolchain - id: rust uses: dtolnay/rust-toolchain@stable with: toolchain: stable @@ -346,60 +351,20 @@ jobs: # mongodb pulls in mongocrypt-sys (native libmongocrypt, needs cmake); # these runners already build it via ci.yml's --all-features. s3 is not # compiled: no s3 leg is benchmarked (see header). - # Compile only the backends the selected legs need (FHIR R4 only, always). - # build_features.py maps the setup job's matrix to the feature list: - # sqlite -> sqlite, postgres -> postgres, mongodb -> mongodb, - # -elasticsearch -> x + elasticsearch; core/all gives the full set. - - name: Resolve build features - id: features - env: - MATRIX: ${{ needs.setup.outputs.matrix }} - run: | - FEATURES=$(python3 .github/scripts/fhir-bench/build_features.py "$MATRIX") - echo "features=$FEATURES" >> "$GITHUB_OUTPUT" - # actions/cache keys cannot contain commas (run 40415264411 failed with - # "Key Validation Error"), so the cache key uses the list joined with +. - echo "key=${FEATURES//,/+}" >> "$GITHUB_OUTPUT" - echo "Building with features: $FEATURES" - - # Rust cache. The repo has no cache action; its convention is a warm - # target/ kept on the self-hosted runner (checkout clean: false plus - # .github/actions/cargo-target-gc). That never helped this job: ci.yml - # builds the debug profile, so the release profile starts cold on every - # runner that has not built it (485 crates, 14 min). actions/cache is the - # first-party action (pinned by major tag like the others). Key = lockfile - # + feature list (different features compile different crates); the - # restore-keys fallbacks reuse a close cache and cargo rebuilds only the - # difference. Saved after a successful build on any branch (keys are - # immutable, so no clash); a branch reads its own and main's caches. - # Only target/release/{deps,build,.fingerprint} are cached, not - # incremental/ or the linked binary. The first run per key is still cold. - # Key includes the rustc version (steps.rust.outputs.cachekey) so a toolchain - # bump gets a fresh key. This job does not run cargo-target-gc and release - # deps/ can be 2-3.5 GB, so key churn can evict other entries. - - name: Cache Rust build (release) - uses: actions/cache@v5 - with: - path: | - ${{ env.CARGO_HOME }}/registry/index - ${{ env.CARGO_HOME }}/registry/cache - ${{ env.CARGO_HOME }}/git/db - target/release/deps - target/release/build - target/release/.fingerprint - key: bench-release-${{ runner.os }}-${{ steps.features.outputs.key }}-${{ steps.rust.outputs.cachekey }}-${{ hashFiles('Cargo.lock') }} - restore-keys: | - bench-release-${{ runner.os }}-${{ steps.features.outputs.key }}-${{ steps.rust.outputs.cachekey }}- - bench-release-${{ runner.os }}-${{ steps.features.outputs.key }}- - bench-release-${{ runner.os }}- - + # The binary is ALWAYS built with the full feature set (FHIR R4 only), and this + # is the only `cargo build` of the workflow: every leg downloads the artifact + # and picks its backend at runtime via HFS_STORAGE_BACKEND. Narrowing the set + # per leg saved only 2-4 min on a cold build and made the binary differ + # between dispatches (a build without `sqlite` has no bulk-export job store + # for mongodb-based servers), so it was dropped; an actions/cache of the + # release build was dropped too (a hit from another runner still rebuilt the + # workspace crates and a miss paid 2-5 min to save). `ui` is left out so the + # benchmark exercises the FHIR API only. - name: Build release binary - env: - FEATURES: ${{ steps.features.outputs.features }} run: | cargo build --release -p helios-hfs \ --no-default-features \ - --features "$FEATURES" + --features "R4,sqlite,postgres,mongodb,elasticsearch" - name: Upload binary uses: actions/upload-artifact@v7 @@ -1128,7 +1093,7 @@ jobs: # HFS silently falls back to 10 on a non-numeric value # (mongodb/backend.rs, config.max_connections). Already validated # in setup's "Resolve matrix from backend input" (same regex, plus - # the 1..1000 range check) before the ~13-minute build — no re-check + # the 1..1000 range check) before the ~5-minute build — no re-check # here, since re-validating the raw input here too (as this step # used to) is redundant on every leg, including sqlite/postgres. HFS_MONGO_POOL="${IN_HFS_MONGO_MAX_CONNECTIONS:-32}" From 5dcd4b57923420aedd66b1eb07db8151b7d85371 Mon Sep 17 00:00:00 2001 From: Edson Terceros Date: Tue, 27 Oct 2026 15:05:34 +0000 Subject: [PATCH 2/2] docs(bench): correct build-job timings and wording after review Cold ranges, memory wording (MemAvailable drop, not a peak), save times, header comment now says all four backends are always linked in. Co-Authored-By: Claude Sonnet 5.5 --- .github/scripts/fhir-bench/README.md | 16 ++++++++-------- .github/workflows/fhir-benchmark.yml | 10 +++++----- 2 files changed, 13 insertions(+), 13 deletions(-) diff --git a/.github/scripts/fhir-bench/README.md b/.github/scripts/fhir-bench/README.md index 180cdcc43..0bf093640 100644 --- a/.github/scripts/fhir-bench/README.md +++ b/.github/scripts/fhir-bench/README.md @@ -55,22 +55,22 @@ Outputs: `run-time.md` and `run-timing.json`. `compare_legs.py --timing run-time ## Build job -The workflow compiles **once**: the `build` job runs the only `cargo build` of the workflow and uploads `target/release/hfs`; every leg downloads that binary and picks its backend at runtime through `HFS_STORAGE_BACKEND`. The binary is always built with the full feature set `R4,sqlite,postgres,mongodb,elasticsearch` (FHIR R4 only). Building only the backends of the selected legs saved just 2-4 min on a cold build (measured with 2 jobs) and made the binary differ between dispatches: a build without `sqlite` has no bulk-export job store for mongodb-based servers (the benchmark itself does not call `$export`). +The workflow compiles **once**: the `build` job runs the only `cargo build` of the workflow and uploads `target/release/hfs`; every leg downloads that binary and picks its backend at runtime through `HFS_STORAGE_BACKEND`. The binary is always built with the full feature set `R4,sqlite,postgres,mongodb,elasticsearch` (FHIR R4 only). Building only the backends of the selected legs saved just about 1-3.5 min on a cold build (measured with 2 jobs) and made the binary differ between dispatches: a build without `sqlite` has no bulk-export job store for mongodb-based servers (the benchmark itself does not call `$export`). The `build` job sets `CARGO_BUILD_JOBS: 8`; every other job keeps the workflow default of 2. The runners have 8 CPUs and the compile is CPU-bound. Measured with `backend=sqlite tests=prewarm` dispatches on throwaway branches, full feature set, wall time of `cargo build --release`: | `CARGO_BUILD_JOBS` | `target/` | Wall time | CPU used | Drop in available memory | Largest process | Run | |---|---|---:|---:|---:|---:|---| -| 2 (before) | cold | 13 min 04 s, 14 min 22 s | - | - | - | runs 40353889556, 40385370213 (cargo's own timing) | -| 4 | cold | 7 min 03 s | 388 % | 7.4 GiB | 4.7 GB | run 40521626800 (github-agent6) | -| 8 | cold | 4 min 40 s | 650 % | 7.2 GiB | 4.7 GB | run 40521631125 (github-agent1) | +| 2 (before) | cold | 13 min 22 s, 13 min 30 s, 14 min 22 s | - | - | - | runs 40322445009, 40335047310, 40385370213 (cargo's own timing, all 485 crates) | +| 4 | cold | 7 min 03 s | 388 % | 7.4 GiB | 4.5 GiB | run 40521626800 (github-agent6) | +| 8 | cold | 4 min 40 s | 650 % | 7.2 GiB | 4.5 GiB | run 40521631125 (github-agent1) | | cargo default (8) | cold | 4 min 33 s | - | - | - | run 40507218293 (github-agent1, cache-warming self-check) | -| 4 | warm | 1 min 56 s | 288 % | 2.7 GiB | 3.1 GB | run 40523191815 (github-agent7) | -| 8 | warm | 37 s | 455 % | 1.7 GiB | 2.6 GB | run 40523196753 (github-agent6) | +| 4 | warm | 1 min 56 s | 288 % | 2.7 GiB | 2.9 GiB | run 40523191815 (github-agent7) | +| 8 | warm | 37 s | 455 % | 1.7 GiB | 2.5 GiB | run 40523196753 (github-agent6) | -The two warm rows are not cold numbers and not comparable with each other: the 4-job run started from an older build on its runner, the 8-job run reused the `target/` of the 4-job cold run. A runner keeps its `target/` until another workflow checks out into the same workspace with the default `clean: true` (about 30 self-hosted jobs in 17 workflow files do, counted by their checkout step), which is why two builds on `github-agent3` three hours apart were both cold. +The two warm rows are not cold numbers and not comparable with each other: the 4-job run started from an older build on its runner, the 8-job run reused the `target/` of the 4-job cold run. A runner keeps its `target/` until another workflow checks out into the same workspace with the default `clean: true` (about 30 self-hosted jobs in 17 workflow files do, counted by their checkout step), which is why a full-set build on `github-agent3` can still start cold. -There is no build cache. An `actions/cache` of the release build was tried and dropped: on an exact-key hit from a different runner the restore took 167 s for 1,827 MB and cargo still rebuilt eight workspace crates in 9 min 09 s (12 min 08 s in total, run 40510258406), because the fresh checkout's file times are newer than the restored outputs; a miss paid 126-277 s to save; and every entry used 0.8-1.8 GB of the repository's 10 GB cache quota. The cache was only judged with 2 jobs; a hit with 8 jobs was not measured. The 49.8 s hit quoted earlier (run 40417473144) ran on the runner that had saved that cache minutes before and is not typical. +There is no build cache. An `actions/cache` of the release build was tried and dropped: on an exact-key hit from a different runner the restore took 167 s for 1,827 MB and cargo still rebuilt eight workspace crates in 9 min 09 s (12 min 08 s in total, run 40510258406), because the fresh checkout's file times are newer than the restored outputs; a miss paid 126-292 s to save; and every entry used 0.8-1.8 GB of the repository's 10 GB cache quota. The cache was only judged with 2 jobs; a hit with 8 jobs was not measured. The 49.8 s hit quoted earlier (run 40417473144) ran on the runner that had saved that cache minutes before and is not typical. ## Docker host diff --git a/.github/workflows/fhir-benchmark.yml b/.github/workflows/fhir-benchmark.yml index c185c2a81..3a6a7a118 100644 --- a/.github/workflows/fhir-benchmark.yml +++ b/.github/workflows/fhir-benchmark.yml @@ -300,8 +300,8 @@ jobs: run: bash "$GITHUB_WORKSPACE/.fhir-bench-setup/.github/scripts/fhir-bench/resolve-matrix.sh" # ──────────────────────────────────────────────────────────────────────────── - # Compile one release `hfs` binary with the backends the selected legs need - # (sqlite, postgres, mongodb, elasticsearch) linked in — the backend is + # Compile one release `hfs` binary with all four backends (sqlite, postgres, + # mongodb, elasticsearch) always linked in — the backend is # selected at runtime via HFS_STORAGE_BACKEND, so a single build serves every # matrix leg. `ui` is dropped so the benchmark exercises the FHIR API only. # ──────────────────────────────────────────────────────────────────────────── @@ -313,8 +313,8 @@ jobs: env: # Raised for this job only (the workflow default of 2 applies elsewhere): the # runners have 8 CPUs and the compile is CPU-bound. Cold builds took 13-15 min - # with 2 jobs, 7 min with 4 and 4 min 40 s with 8 at the same ~7.5 GB peak - # memory; see the "Build job" section of .github/scripts/fhir-bench/README.md. + # with 2 jobs, 7 min with 4 and 4 min 40 s with 8; 4 and 8 jobs lowered + # MemAvailable by the same 7.2-7.4 GiB (2 jobs not measured); see the "Build job" section of .github/scripts/fhir-bench/README.md. CARGO_BUILD_JOBS: 8 steps: @@ -354,7 +354,7 @@ jobs: # The binary is ALWAYS built with the full feature set (FHIR R4 only), and this # is the only `cargo build` of the workflow: every leg downloads the artifact # and picks its backend at runtime via HFS_STORAGE_BACKEND. Narrowing the set - # per leg saved only 2-4 min on a cold build and made the binary differ + # per leg saved only about 1-3.5 min on a cold build and made the binary differ # between dispatches (a build without `sqlite` has no bulk-export job store # for mongodb-based servers), so it was dropped; an actions/cache of the # release build was dropped too (a hit from another runner still rebuilt the