fhir-benchmark: explicit per-leg indexing mode, sql-run suite, and a visual report with run history #2247

Summary

Make the FHIR Server Benchmark (.github/workflows/fhir-benchmark.yml, .github/scripts/fhir-bench/) say clearly how each leg indexes, measure SQL-on-FHIR ($sql-run), stop timing $reindex, and make the run report readable (UI-named import rows, collapsed charts, a standalone HTML report and a run history). Delivered as one task and one PR.

Motivation

• The bulk-import legs ran with whatever the server defaulted to: every *-elasticsearch leg ingested with deferred indexing, and nobody could tell from the report which mode a number came from. Elasticsearch needs to index while it ingests, and the defaults were 3-4x slower than a tuned configuration.
• $reindex is not something users run to load data, and the indexing time that matters (load until searchable) was not shown per leg.
• $sql-run had no performance coverage.
• Reports could not be read without the workflow file open: row names did not match the two UI import processes (/ui/batch, /ui/bulk-import), there were no charts, no run history, and a run that waited for resources looked like a slow run.

Acceptance criteria

1. Remove the $reindex timing. The opt-in indexing suite and its row are deleted (recoverable from git history).
2. Explicit indexing mode per leg, documented once.
   • sqlite, postgres, mongodb: deferred (HFS_BULK_SUBMIT_DEFER_INDEXING=true).
   • every *-elasticsearch leg: inline with the ingest, HFS_BULK_SUBMIT_FILE_CONCURRENCY=4, HFS_BULK_SUBMIT_BATCH_SIZE=1000, HFS_ELASTICSEARCH_BULK_CONCURRENCY=4.
   • one per-backend table in crates/rest/README.md; other docs point to it. The recipe values are not workflow form inputs.
3. Indexing time row. ↳ search indexing (seconds) under Bulk Import; on Elasticsearch legs it reads N s (= ingest).
4. sql-run suite. HFS-owned k6 script, five views on the biggest resource types (Observation x3, Claim, ExplanationOfBenefit; not Patient), requests/s, rows/s and per-view latency; runs after search and before insert.
5. Report.
   • Batch/Transaction Import and Bulk Import rows named after the two UI processes.
   • Collapsed Mermaid charts after the unchanged tables (compare job and each leg).
   • Standalone dark HTML report (artifact, archive: false) linked from the last line of the summary.
   • Run history (last runs of the branch plus the latest main run), old and new row labels joined.
   • A "Run time" block: total, waiting for resources, working, and sequencing behind earlier legs.
6. Everything found on the way is part of the same PR (see below).

Found while doing it (same PR)

• Server bug: GET /$sql-run?_format=ndjson with Accept-Encoding: gzip answered 200 with an empty body (futures::stream::unfold panics when the compression layer polls it again after the end). Fixed with .fuse() in crates/rest/src/handlers/sof/run.rs (channel_body_response), regression test in crates/rest/tests/http_compression.rs. The same pattern in bulk_submit_fetcher.rs (read through tokio-util's StreamReader, which re-polls after EOF) is fused too.
• The text "4 CPU / 11 GB Docker host" was stale: the capacity gate of run 40385370213 measured MemTotal: 20000 MB.
• The build job compiled all four backends cold every run (14 min 22 s, 485 crates, no cache). It now builds only the backends of the selected legs (always R4) and restores a Rust release cache.
• A path-filtered workflow runs the 154 helper-script unit tests on pull requests that touch them.

Evidence

Elasticsearch recipe, 133,527-resource slice, one run per cell, seconds until searchable (run-to-run spread is large, e.g. 18-52 s for the same settings):

Standalone backends: deferred was 26-27% faster than inline on postgres and sqlite, a tie on mongodb.

CI, three Elasticsearch legs, run 40385370213 (2 h 05 min), Bulk Import until searchable versus the reference run 39657850695:

The reference run is 143 commits (73 without merges) older than the code of this branch, so the gap mixes the recipe with code changes. sqlite+ES shows almost no gain (its SQLite primary ingests one file at a time whatever FILE_CONCURRENCY says).

sql-run (run 40335047310): 10,070 requests, 0 failed, 160 requests/s, 419,202 rows/s.

Build: cold R4,sqlite in 10 min 47 s; with the cache restored, 49.8 s (run 40417473144).

A run that overlapped five other CI jobs on the shared Docker host (run 40353889556) had Elasticsearch's own per-document indexing time double (0.906 vs about 0.48 ms) and crud/search throughput fall by the same factor, while Postgres stayed at about 0.06 ms per call.

Related issues

#1899, #1957 and #2042 are about the $reindex indexing suite that this removes. #1953 (host contention is only marked on postgres crud) and #1892 / #1902 (the insert and bulk-import suites) touch the same files. Closing or updating them is left to the maintainers.

Scope / out of scope

• Out of scope: other FHIR versions (the build stays R4), new workflow form inputs for the recipe values, a dedicated Docker host for the benchmark, the "Search by query shape" section in the HTML (the shape step must stay after the artifact upload: two past runner losses).
• Known limits: single-run measurements; the history restarts for renamed rows; the "waiting for resources" figure does not measure host contention.