[eterceros]

Summary

Closes #2247. One PR for the FHIR Server Benchmark (.github/workflows/fhir-benchmark.yml, .github/scripts/fhir-bench/): the five requests from the benchmark review (drop the $reindex timing, explicit indexing mode per leg, an indexing-time row, a $sql-run suite, a readable report with history), plus what was found while doing it (a server bug, a waiting-time block, a faster build, a test workflow). 12 commits, 30 files, +5,321 / -751.

What this PR adds

#	Area	Before (main @ b790972)	After (this PR)	Main files
1	$reindex timing	Opt-in indexing suite (not part of all) and a reindex row; up to ~5 min per leg	Suite, row, helpers and tests deleted; tests=indexing is skipped with a log line. Time until searchable is what the Bulk Import rows measure	fhir-benchmark.yml, suite-lib.sh, summary_backends.py, compare_legs.py
2	Indexing mode per leg	Not set: every leg used the server default (deferred), including the *-elasticsearch legs	Set in bulk-import.sh. sqlite, postgres, mongodb: deferred. Every *-elasticsearch leg: inline with HFS_BULK_SUBMIT_FILE_CONCURRENCY=4, HFS_BULK_SUBMIT_BATCH_SIZE=1000, HFS_ELASTICSEARCH_BULK_CONCURRENCY=4. One per-backend table in crates/rest/README.md; other docs point to it	bulk-import.sh, crates/rest/README.md
3	Indexing time	search-index rebuild row: rebuild seconds only	↳ search indexing (seconds): rebuild seconds on deferred legs, N s (= ingest) on inline legs	compare_legs.py
4	$sql-run	Not measured	New HFS-owned k6 suite sql-run (10 VUs x 1 min, read-only, between search and insert): five views on the biggest resource types (Observation x3, Claim, ExplanationOfBenefit; not Patient). Rows for requests/s, rows/s, p95, errors, plus a per-view table in each leg summary	k6/sql-run.js, compare_legs.py, summary_backends.py
5a	Import rows	Batch/Transaction Import, POST / Bundles, Bulk Import, $bulk-submit	Named after the two UI processes (/ui/batch, /ui/bulk-import); Bulk Import row added to each leg's Results table	compare_legs.py, fhir-benchmark.yml
5b	Charts	None	Collapsed ### Charts section appended after the unchanged tables, in the comparison and in each leg (two charts per row, one colour per leg)	md_charts.py
5c	HTML report	None	Single-file dark HTML report (summary, validity, KPIs, timeline with an All view, batch and bulk phases, sql-run views, trend, full data, health, glossary), uploaded with archive: false and linked on the last line of the summary	report_html.py, report_template.html
5d	Run history	None	Last 5 runs of the branch plus the latest main run; old and new row labels joined; trend data kept as an artifact	history.py
6	Waiting vs working	Only wall-clock: a queued run looks like a slow run	Run time block under the report header: total, waiting for resources, working, and sequencing behind earlier legs; tile and timeline rows in the HTML	run_timing.py
7	Server bug	GET /$sql-run?_format=ndjson with Accept-Encoding: gzip answered 200 with an empty body	Fixed (.fuse() on the streamed body) with a regression test; the same pattern in the bulk-submit fetcher is fused too	crates/rest/src/handlers/sof/run.rs, crates/rest/src/bulk_submit_fetcher.rs
8	Build job	Always the 4 backends, no cache: 14 min 22 s cold, 485 crates	Feature list derived from the selected legs (always R4, only the needed backends) and a release-profile cache. Cold 10-12 min; with a cache hit the build took 49.8 s	build_features.py, fhir-benchmark.yml
9	Tests of the helper scripts	Not run in CI	Path-filtered workflow runs the 154 unit tests and a syntax gate on pull requests that touch the benchmark	fhir-bench-tests.yml
10	Stale text	"4-CPU / 11 GB Docker host" in 7 places; conflicting indexing advice	No hard-coded host memory (the capacity gate measured MemTotal: 20000 MB); one source of truth for the indexing mode	workflows, scripts, READMEs

Evidence

1. Which CI run validates what

All runs are workflow_dispatch runs on throwaway branches of this work (tmp/bench-steve-*), except the reference. The final tree of this PR equals the tree of 4eb61b6 (the 12 commits here were cherry-picked onto main @ b790972 without changes); earlier runs ran earlier states of the same branch.

Claim	Run	Commit	Result
Reference (before)	39657850695	main @ e0bd5cb	6 legs, all suites
Inline recipe, rows, charts, HTML on the three Elasticsearch legs	40385370213	ece668e	success, 2 h 05 min, all suites; HTML artifact 93 KB
sql-run first run: found the gzip bug	40322445009	2aedc18	success overall, sql-run 99.9 % errors (empty gzip body)
sql-run with a client-side workaround	40335047310	a4f41e7	10,070 requests, 0 failed, 160 req/s
sql-run with the server fix, gzip requested	40353889556	bb8688b	7,347 requests, 0 failed
Run time block on real runs	computed on 40353889556 (total 53 min 26 s, waiting 11 s) and 40385370213 (2 h 05 min, waiting 14 s); rendered in 40405097906	83a7a27	success (the block was added after run 40385370213, so no full benchmark run on this branch shows it yet)
Charts, HTML and history render on 1 and 6 legs	40379259996, 40382243831	throwaway render test	success; reviewed on the run page
Tests workflow	40411764455	97b2f3f (push-triggered copy; the real trigger is the pull_request path filter)	success, Python 3.12 runner
Cache key bug found	40415264411	fb5f926	failed at the cache step: actions/cache keys cannot contain commas; fixed in 4eb61b6
Cache: cold build, then hit	40415455483, 40417473144	4eb61b6	success; see section 5
Other feature sets build, link and start	40488873857, 40488876795, 40488881064	4eb61b6	success; see section 5

2. Results on the three Elasticsearch legs (run 40385370213, all suites)

Suite	sqlite+ES	postgres+ES	mongodb+ES
Batch/Transaction Import (resources/s)	6,166	5,645	4,521 (marked ‡: 4 failed checks)
Bulk Import until searchable (resources/s)	3,260	7,570	4,788
↳ search indexing	500 s (= ingest)	215 s (= ingest)	341 s (= ingest)
crud (req/s)	696	817	769 (incomplete corpus)
search (req/s)	927	688	697 (incomplete corpus)
sql-run (req/s, 0 failed)	136	137	102
sql-run (rows/s)	356,135	357,493	266,289
sql-run p95 (ms)	122.8	155.4	214.7
insert (creates/s)	258	261	263

The shared Docker host was not quiet during this run (two other CI runs in progress when it started), so absolute crud/search numbers are indicative. mongodb+ES imported 996/1000 bundles in the Batch import (4 failed checks), the same count as in the reference run; the standalone mongodb leg imported 989/1000 there.

3. Bulk Import until searchable, before and after

Leg	Reference run 39657850695 (older main, deferred)	This branch (inline recipe)
sqlite+ES	220 s ingest + 313 s rebuild = 533 s	500 s (run 40385370213)
postgres+ES	315 s + 321 s = 636 s	195 s, 210 s, 215 s (runs 40322445009, 40335047310, 40385370213)
mongodb+ES	645 s + 350 s = 996 s	341 s (run 40385370213)

Caveats: the reference run is main @ e0bd5cb, 143 commits (73 without merges) before b790972, and deferred, so the gap mixes the recipe with code changes; the controlled comparison is section 4. sqlite+ES shows almost no gain in CI: its SQLite primary ingests one file at a time whatever FILE_CONCURRENCY says. Run 40353889556 (postgres+ES) took 463 s + a 27 s rebuild and was marked by the report: three bulk-index pages timed out at 30 s on a loaded host and 758 resources were rebuilt afterwards.

4. Controlled experiment for the recipe (local, one server binary)

133,527-resource slice, seconds until searchable, one run per cell unless several are listed, same binary (main @ e0bd5cb) for every arm, 0 errors in all arms:

Arm	Settings	sqlite+ES	postgres+ES	mongodb+ES
B	inline, server defaults	82.1	80.1	82.1
C (chosen)	inline, 4 files, batch 1000, ES bulk concurrency 4	24.0, 18.0	24.1, 24.1	32.2, 52.1
D	C + index queue 64 / 8 workers	20.1, 24.0	18.2, 132.2, 64.2	26.1
F	C + 1 s refresh interval (with wait_for)	46.0	50.1	50.1
G	D + WRITE_REFRESH=false + 1 s refresh	22.0	22.2	24.1
E	earlier #941 recipe	-	-	40.2
A	deferred (reference only)	93.4	-	-

C was chosen over D because D showed no repeatable gain and was unstable on postgres+ES (18 / 132 / 64 s). Run-to-run spread is large (for example 18-52 s for the same settings), so only differences above about 15 % count. Memory with C on postgres+ES: 2.15 GB peak RSS at 133k resources, 2.97 GB at 489k (203.5 s). Standalone backends: deferred was 26-27 % faster than inline on postgres and sqlite, and a tie on mongodb (same slice).

5. Build job: feature sets and cache

Run	Features	Build step	Cache
reference (run 40385370213)	R4,sqlite,postgres,mongodb,elasticsearch (fixed)	14 min 22 s, 485 crates	none
40415455483	R4,sqlite	10 min 47 s	miss, saved ~1.05 GB
40417473144	R4,sqlite	49.8 s (2 crates rebuilt)	hit, restored in 99 s
40488873857	R4,mongodb	10 min 04 s	restore-key fallback to the R4+sqlite cache, saved its own
40488876795	R4,postgres,elasticsearch	11 min 55 s	miss, saved
40488881064	R4,sqlite,elasticsearch	11 min 56 s	miss, saved

Each of the last three linked a binary and ran a prewarm leg successfully (mongodb, postgres-elasticsearch, sqlite-elasticsearch). A cold build of a narrower set is only moderately faster than the old fixed set; the large gain is the cache. backend=core and all still produce the full feature list.

6. Local verification

• python3 -m unittest discover on .github/scripts/fhir-bench with an empty HOME: 154 tests, OK, 1 skipped (the real-data smoke test, which needs FHIR_BENCH_REAL_RUN_DIR).
• cargo test -p helios-rest --test http_compression: 13 passed, including the new gzip test, which failed with Unfold must not be polled after it returned Ready(None) before the fix.
• cargo check -p helios-hfs --no-default-features --features ... for all six backend sets: R4,sqlite 95 s, R4,postgres 95 s, R4,mongodb 102 s, R4,sqlite,elasticsearch 62 s, R4,postgres,elasticsearch 64 s, R4,mongodb,elasticsearch 75 s, no warnings (warm target; indicative).
• build_features.py run on the real output of resolve-matrix.sh for every backend option.
• Chart and report generators were mutation-tested on real artifacts: changing an input number (a throughput, an ingest time, a leg name, adding a seventh leg, dropping sql-run) changed exactly the matching bars and labels; no value from the reference run is hard-coded.
• cargo fmt --check on the touched Rust files; bash -n on every script; both workflow files parse.

7. Why one run fell to half (run 40353889556)

Five other CI jobs (Test Rust, Code Coverage) used the same Docker host during its crud and search suites: containers on the host went from 11 to 34 and free memory from 15.5 GB to 3.2 GB. Elasticsearch's own indexing time per document doubled (0.906 ms against 0.476 and 0.494 in the two quiet runs) and crud fell by the same factor (530 against 961 req/s), while Postgres stayed at 0.055-0.060 ms per call in all runs. The new Run time block does not measure this kind of contention; it measures queueing and the capacity gate wait only.

Known limits

• Measurements are single runs on a Docker host shared with 18 other workflows; compare runs only after checking what else was running.
• The report history restarts for renamed rows (the old index rebuild row is kept separate because it measures a different quantity).
• The HTML has no "Search by query shape" section in CI: the shape step must stay after the artifact upload (two past runner losses, documented in the workflow).
• A mongodb or mongodb-elasticsearch dispatch builds without the sqlite feature, so the bulk-export sidecar is off in that binary (the benchmark does not use it).
• sql-run is placed after search only for tests=all; with an explicit list it runs where listed.
• The cache is limited by the repository's 10 GB cache quota; the first run per key is cold. The recipe values are intentionally not workflow form inputs.
• Not covered by a full benchmark run on this branch: the Run time block (checked on real runs offline and in a render test).

Related issues

Closes #2247. Related, not closed here: #1899, #1957 and #2042 (about the $reindex indexing suite removed here), #1953 (host contention marking), #1892 and #1902 (the insert and bulk-import suites).

Where to look

• Run page of the three-leg run, summary and artifacts: https://github.com/HeliosSoftware/hfs/actions/runs/40385370213
• Per-backend indexing table: crates/rest/README.md, section "$bulk-submit indexing mode per storage backend"
• Benchmark script overview: .github/scripts/fhir-bench/README.md

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!