Manual test pass: MongoDB backend #940

Activity

1. smunini

   assigned

   dougc95

   on Sep 4, 2026
2. smunini

   added

   testing

   on Sep 4, 2026
3. dougc95 commented on Sep 10, 2026

   

   dougc95

   on Sep 10, 2026

   Contributor

   More actions

   mongodb row — complete

   Tester: @dougc95 · Windows 11 Pro 10.0.26200 / x86_64 (MSVC) · 2026-09-06 → 2026-09-10
   Commits: 6080c4cb8 (T0–T4), then 0e201b577 after a mid-pass rebuild (T5–T9)

   Backend	T0 Build	T1 Start	T2 Batch/Tx	T3 Bulk import	T4 Search	T5 Bulk export	T6 ViewDefinition	T7 SQL export	T8 Subscription	T9 Dashboard
   mongodb	✅	✅	✅	❌	❌	❌	✅	❌	❌	✅

   Five pass, five fail. Of the five failures, three trace to already-reported causes, and one (T8) is fixed upstream in a commit that merged hours after the build under test.

   Environment deviations

   Build	Explicit feature list, not --all-features — that flag also enables headless and produces a binary with no /ui (#975, fixed after this build)
   MongoDB	mongo:7.0 --replSet rs0. Standalone makes transaction Bundles return 501, so §4's infrastructure cannot pass T2 (#997)
   Corpus	Reduced + split: 9 types in 36 files, 11,197,644 resources, FILE_CONCURRENCY=16
   Rebuild	The repo moved to 0e201b577 on 2026-09-09; T4 was re-verified against the new binary with 0 differences

   Timings

   • T3 import: 19 h 55 m for 11,198,033 resources — ~156 rec/s average, degrading from 300 to 96 as indexes outgrew the cache (perf(mongodb): $bulk-submit runs at ~60 resources/s per file — 23 documents and ~250 index key insertions per resource, 55 GB of indexes against a 7 GB cache #1000, since fixed)
   • T4: 84 queries in 33 min
   • T7 finished in: 4 s for vd-ndjson — 11,705 rows

   Issues filed from this row

   #
   #997	Matrix: transaction Bundles need a replica set, §4 starts standalone
   #998	Submission never reaches a terminal state; the import re-ingests forever
   #999	Multi-parameter searches 500 with distinct too big
   #1000	$bulk-submit ~60 rec/s per file — since fixed
   #1001	TransientTransactionError entries dropped without retry
   #1002	Composite and uri :below return 400
   #1003	Results header shows page size as the result count
   #1004	Matrix corrections (now twelve, across T0–T9)
   #1040	_sort by a search parameter scans the whole resource type — new
   #1041	A bulk export whose lease expires restarts from scratch forever — new

   Evidence comments were added to #870, #958, #968, #969, #1008, #1009, #1010, #1014. Roughly half of what this pass found was already tracked upstream, so every finding went through a duplicate check across 400 issues before anything was filed.

   The one pattern worth naming

   Three of the open 🔴 findings — #999, #1040 and #1041 — are the same failure in different clothes: an operation whose cost is proportional to the whole collection rather than to the requested result. One materialises every matching id before intersecting; one sorts every resource of a type before filtering to the 165 that matched; one sweeps every resource until its lease expires, then starts over.

   The counter-evidence is just as consistent. Work sized to the answer is fast on this backend: 11,705 rows exported in 4 s, 50 rows drawn from 7.7 M Observations in 12 ms, a warm Condition?patient=PID in 1.2 s. They are likely to want the same remedy — push the selective predicate down before materialising.

   Two things that will cost the next tester time

   Cache temperature changes results, not just timings. Indexes are ~55 GB against a 7.14 GB WiredTiger cache. Condition?patient=PID returns nothing in 120 s on a cold server and 1.2 s warm — same binary, same data. It also means a 500 and a timeout are the same failure: warm, the query reaches the 16 MB distinct cap and fails fast; cold, it never gets there. Our original tally of "5 × 500 and 10 × timeout" was one failure mode, not two. Do not compare a first pass after a restart.

   Several matrix expectations are wrong, in both directions. Chained and _has are marked N/A and work. Bulk $export is marked N/A in three places and works. Parquet date → string is correct per the SoF ballot's type table (#569), not the defect the doc implies. Row 5.1's "3 files" only holds under 1,000 resources per type. T9's "~30 s" to see the failure chip is really ~5 minutes, because it counts exhausted deliveries rather than connection failures. All twelve corrections are on #1004.

   Deliverables

   • Full report, all ten steps with per-step detail: https://claude.ai/code/artifact/4e28eb88-c333-43cc-92cb-fac4dc16b025
   • T4-api.tsv (84 queries with counts, status, timings) and T5-api.tsv
   • Bulk export manifest; $sql-export result (note: a Parameters, not a Bulk Data manifest)
   • Output samples in NDJSON, CSV, Parquet and the ZIP from Download all resources
   • hfs-mongodb.log — 13 server starts

   Screenshots are attached below.
4. dougc95 commented on Oct 19, 2026

   

   dougc95

   on Oct 19, 2026

   Last edited by dougc95

   Contributor

   More actions

   mongodb row: complete (bounded pass, bulk_1k)

   Tester: @dougc95. Claude Code agents ran the pass, driving the web UI with headless Playwright.
   Environment: Linux VPS vmi3636660 (Ubuntu, kernel 6.8.0-147-generic, x86_64, 12 cores, 47 GB RAM) with Docker. There is no Rust toolchain on the host, so the build and the server ran in the hfs-rust-dev image. The UI was driven by Playwright 1.49.1 with headless Chromium 131.
   Date: 2026-10-19. The build ran 05:34Z → 14:10Z, and T1–T9 ran 14:01Z → 18:56Z.
   Commit: d3976b4bd (v0.2.5). One hfs binary was used for every step: sha256 de774a4f839f1fe0061c753b7f8840f7e749d9863efe667ee08a113a2b81aee8, 989,906,640 bytes.

   Backend (HFS_STORAGE_BACKEND)	T0 Build	T1 Start	T2 Batch / Transaction	T3 Bulk import	T4 Search types	T5 Bulk export	T6 ViewDefinition	T7 SQL export (VD / query / view)	T8 Subscription	T9 Activity dashboard
   mongodb	pass	pass	pass	pass	pass	pass	pass	pass	pass	pass

   BOUNDED PASS. T3 imported the Synthea bulk_1k corpus, as this issue's T3 line asks. That is the Aidbox archive of 1,000 transaction Bundles, converted to 50 NDJSON files covering 24 resource types: 1,630,685 resources and 1,000 patients. The 18,955,865-resource fhir2 corpus of matrix §7 was not used.

   Every value the matrix ties to fhir2 was recomputed from the loaded NDJSON and adapted. That covers the anchor PID 7d24f7a0-… (Parker433), the 11,705 patients, the 7,699,987 Observations, the 24-file manifest, and the T4/T5/T7 counts. The anchor patient here is Jamaal Logan Pacocha 12fc97d7-73ba-8fbf-0f4b-daaabdef0b4c.

   This row therefore says nothing about fhir2-scale limits. No search matched more than 300,000 ids (#1896), and no SQL Query dependency exceeded 1,000,000 rows (#1473).

   Ten pass, zero fail, no new product issue.

   One T1 sub-check failed as written: "sidebar shows MongoDB". A second, independent run confirmed that this comes from a stale matrix line and is not an HFS fault (see Failures). The other deviations from the written text are matrix errors, listed under Matrix corrections.

   Cell notes

   • T0. cargo build --workspace --all-features --release exited 0 (Finished release profile [optimized] target(s) in 424m 02s, BUILD_EXIT=0 at 14:10:06Z), and hfs --help prints usage.

     • T1 started at 14:01:20Z, about 9 min before the build exited, while the last crate (pysof) was still compiling. The issue owner chose this. Every binary was already built, and the hfs binary was not touched afterwards: its mtime is 12:12:34Z and its sha256 was unchanged at report time.
     • git status --short was empty after the build.

   • T1. Every check in the matrix's pass criteria (L340-344) passed:

     • /health returned 200 after 21 s.
     • /metadata reports fhirVersion 4.0.1, and fhirVersion=5.0 returns 5.0.0.
     • The log line Initializing MongoDB backend url=mongodb://localhost:27017/?replicaSet=rs0&directConnection=true database=helios appears, with no ERROR and no panic.
     • /ui renders. The dashboard reads 1.4k (the exact figure is 1,377), and the clinical rail is at 0.

     The L337 annotation "sidebar shows the backend" fails as written, because the HFS UI has no backend label.

   • T2. All 12 sub-checks passed in the UI.

     • 6.1 was rejected with 400 "Conditional reference 'Organization?identifier=…|25c85bcc-…' matches no existing resource" and wrote nothing.
     • 6.2 created 9 + 8.
     • 6.3 answered 662 created, HTTP 200, in 14.8 s.

   • T3. All 18 expected-pass sub-checks passed.

     • 7.5 step 5 (Elasticsearch) is N/A on this row.
     • 7.6 was run: the re-upload was rejected with "… matches more than one resource", and the Larkin query still returns 1.

   • T4. 4.13 is N/A: 501 "Feature 'full-text search not available' is not implemented.", as §1 marks it.

     • 87 queries were typed into the Resources QUERY box (rows 4.1–4.29): 86 pass and 1 is N/A, with 0 failures, no 422, no 5xx and no timeout. The count on screen equalled the Bundle.total received on all 87.
     • Composite search passes in 4.8 and 4.21.
     • POST _search (from the issue checklist) was checked through the API, because the Resources page only sends GET. It answered 200 with total 5, the same ids as the GET.

   • T5. All 20 sub-checks passed. 9.5 (S3 output backend) is N/A on this row.

     • These were checked through the API by design, because the Export page has no format selector: the three _outputFormat spellings (202), text/csv (400 "unsupported _outputFormat 'text/csv'"), and patient- and group-level $export.

   • T6. All 17 sub-checks passed.

     • The issue checklist asks for an API-created ViewDefinition (201) and $sql-run in json/csv/ndjson/parquet (200, 1,001 rows each). Those were done through the API by design.

   • T7. All 26 sub-checks passed. 11.7 (S3 sink) is N/A.

     • All 12 cells of the subject-kind × format grid were exported from /ui/sql/export.
     • Every file was parsed in its own format and compared row by row with an offline model of the corpus: 0 mismatches.

   • T8. All 8 sub-checks passed: the handshake, requested → active, 3 notifications with Bearer manual-token, no notification for a Condition, and delivery after the receiver restarted.

     • $status was read through the API (events-since-subscription-start 6).
     • $events (issue checklist, not in the matrix) was not run.

   • T9. All 12 sub-checks passed.

     • The Error chip appeared 15 min 06 s after the upload, as MANUAL_TESTING_MATRIX.md: five expectations that no longer match HFS (T9 Error chip timing since #2024, T5 file counts and Retry, T4 4.10) #2141 item 1 describes.
     • Reactivation worked through the editor.
     • Rehydration after a restart passed, with 0 × Failed to persist subscription status transition.
     • The disabled-engine notice passed, and the page works without JavaScript.

   Environment deviations

   Build	The command was CARGO_BUILD_JOBS=2 cargo build --workspace --all-features --release, run in Docker image hfs-rust-dev:1776 (rustc/cargo 1.99.0, Python 3.11.2). --release was added because matrix L143 omits it. Attempt 1 used 4 jobs and was killed (see Timings).
   T0 / T1 overlap	T1 started at 14:01:20Z, before BUILD_EXIT=0 at 14:10:06Z, with only pysof left compiling. The hfs binary that was used is unchanged.
   Server	HFS ran in container hfs940-server (--network host, worktree mounted read-only at /work), with stdout appended to hfs-mongodb.log. It was not started with ./target/release/hfs | tee.
   Environment	The matrix common block, plus HFS_BULK_SUBMIT_OUTPUT_DIR=$WORK/submit (keeps the read-only checkout clean) and HFS_REQUEST_TIMEOUT=600. Deferred indexing was left at its default (on).
   MongoDB	mongo:7.0 (mongod 7.0.43) as a single-node replica set rs0 with host networking, bound to 127.0.0.1 and ::1, on a named volume, with --wiredTigerCacheSizeGB 8 because the host is shared.
   Corpus	bulk_1k (Aidbox archive, sha256 7844b5f8…5eaf5f, 418,110,093 bytes), converted to 50 NDJSON parts plus a manifest. It was served by nginx 1.31.7 with keep-alive on localhost:8000. The whole-file byte check gave 50 ok and 0 TRUNC.
   Anchor	Jamaal Logan Pacocha 12fc97d7-… (male, 2008-07-17, Haverhill) replaced Parker433 / 7d24f7a0-…. Every count came from an offline model of the loaded NDJSON.
   UI	Headless Chromium drove the real /ui. Uploads used the page's own file input, and each step has screenshots. Every check done through an endpoint instead of the UI is marked as such above.
   Receiver	webhook.py was used, with the same JSON lines plus a timestamp, instead of the §12.1 heredoc.
   Host	The host was shared with a parallel workflow (#1776: cargo and testcontainers). During T3, none of its containers appeared and the load stayed at or below 2.2 on 12 cores.
   Restarts	Four restarts were deliberate: T7 11.3 at 18:01:16Z, T9 13.5 at 18:49:34Z, and T9 13.6 with subscriptions off (18:50:34Z) and then restored (18:53:17Z). There was none between T3 Submit and the completion of the rebuild. In total there were 5 server starts.

   Timings

   • T0 build.
     • Attempt 1 (4 jobs): 05:34:18Z → 07:05:08Z, 1 h 30 m 50 s, then failed. rustc for helios-fhir was killed with signal: 9, SIGKILL, probably OOM.
     • Attempt 2 (2 jobs): from 07:05:50Z to the hfs binary at 12:12:34Z took 5 h 06 m 44 s. BUILD_EXIT=0 came at 14:10:06Z, 7 h 04 m 16 s after the start (cargo reports 424m 02s).
     • From the first start to exit 0: 8 h 35 m 48 s.
   • T3 (bounded, bulk_1k). 1,000 transaction Bundles, converted to 50 NDJSON files, gave 1,630,685 resources.
     • Ingest: 00:52:03. Submit at 14:19:12Z, Processing finished at 2026-10-19T15:11:14.624Z: 522 res/s.
     • Deferred rebuild: 01:12:18. Job aba79678-7564-4abf-94c7-6c033b9c7425 ran 15:10:46.678Z → 16:23:04.382Z: completed, errorCount 0, 1,631,364 resources, 31,203,895 index entries, 376 res/s.
     • Searchable: 02:03:52 after Submit (219 res/s).
     • Database: helios dataSize 20,933,311,989; storageSize 4,061,364,224; indexSize 11,022,086,144; objects 37,759,601.
   • T4. The 87 count queries took 6 min 29 s in one UI session (16:38:35Z → 16:45:04Z). Most answered in 0.3–0.7 s. The slow ones, cold then warm:
     • 4.11 _has:Observation took 50.5 s, then 19.2 s warm.
     • 4.8 composite took 32.4 s, then 4.8 s warm.
     • 4.5.0, 4.5.2, 4.2.3, 4.3.1 and 4.21.5 took 5.9–7.2 s.
     • The whole of T4 took about 26 min.
   • T5. Bounded export cards finished in 0m 06s–0m 07s. 5.2 one-patient took 0m 22s. The all-types export (16 files) took 1m 48s with JS off and 0m 34s with JS on.
   • T7.
     • 7.a vd-ndjson (1,001 rows): finished in 0m 05s on the card, which is the 5 s poll. The server's exportStartTime → exportEndTime was 0.18 s.
     • SQL Query exports took 0m 26s on the card and 22.2–24.5 s on the server, because they materialize observation_flat's 689,186 rows.
     • The full observation_flat export took 25.97 s and produced 2 files (500,000 + 189,186 rows).
     • The tall_female_patients preview with a value took 20–38 s.
   • T9. From the upload with the receiver down to consecutive_failures=3 and the Error chip took 15 min 06 s: 3 notifications × 10 attempts, with 30 Connection failed warnings.

   Failures

   No product failure, no new issue. The one sub-check that failed as written:

   Check	Request / action	Observed	Expected	Verdict
   T1 5.ui	GET /ui, /ui/resources and 10 more /ui pages, cold and warm, in Chromium and with curl. Read the sidebar text.	The sidebar holds the brand "Helios FHIR Server / hfs v0.2.5", the nav groups and the "FHIR R4" selector. No page contains mongo or backend (curl -s $HFS/ui | grep -ci mongo → 0).	L337: "dashboard renders; sidebar shows the backend and FHIR version"	Matrix error, reproduced. crates/ui/templates/layouts/base.html has never had a backend field: git log -S backend on it is empty, and L337 dates from the matrix's first commit b216565e6. The backend was confirmed from the startup log instead. Evidence was added to #2141.

   Verified fixed since the September row (6080c4cb8 / 0e201b577)

   Finding	September	Now, on d3976b4bd
   #975: --all-features drops /ui	Built with an explicit feature list	The --all-features binary serves /ui (200).
   #997: transactions need a replica set	§4 was standalone	§4 uses a replica set. 6.1 answers 400 naming the conditional reference; 6.3 creates 662 in 14.8 s.
   #998: submission never terminal (T3 ❌)	Re-ingested forever	Completed, Output files 24, Error files 0, "processing finished cleanly (24 outputs)".
   #1000: $bulk-submit ~60 rec/s per file	19 h 55 m for 11.2 M resources, about 156 rec/s	522 res/s ingest and 219 res/s to searchable on 1.63 M resources (a different corpus size).
   #1001: TransientTransactionError entries dropped	Short type counts	All 24 types are exact on the rail and by _summary=count.
   #999: distinct too big (T4 ❌)	Multi-parameter searches gave 500s and timeouts	Every multi-parameter row answers 200 with the expected count.
   #1002 / #1206: composite search, uri :below	400	4.7 :below returns 1. 4.8 composite returns 13,858, equal to 4.5. The 4.21 composite returns 3.
   #1003: header shows the page size	Wrong totals	The count on screen equals Bundle.total on 87/87 queries.
   #1040: _sort scans the whole type	Slow	4.12, 4.22 and 4.27 sort correctly, each in 2.6 s or less.
   #1041: export lease restarts forever (T5 ❌)	Never finished	Every T5 export meant to complete did so (6–108 s); cancel-me cancelled and bad-group failed, as designed. Lease expiry itself was not provoked.
   #1008: SQL Query/View exports hang (T7 ❌)	Writing 0 of 1	The full 3×4 grid completed; SQL Query exports take 26 s.
   #1009: _typeFilter		5.3 returns only the anchor's 1 active Condition.
   #1010: missing Group		5.13 fails at once: "kick-off answered 404: Could not find the resource 'Group/does-not-exist'."
   #1014: ViewDefinition save prompt		10.3 shows "This view definition still has 1 error. Save it anyway?"
   #969: progress counter goes backwards		N Resources written never decreased in 100 curl samples and 11 browser samples.
   #870: batch POST id collisions	Overwritten	Ids are server-assigned; the 4.20 rows give 2 each (the T2 copy plus the corpus copy).
   #958: Resources table		The Sort dropdown and the _elements=status column work.
   T8 (❌ in September; fixed upstream after that build)		8/8 pass, including queued delivery after the receiver restarts.

   Still open: confirmed or not reached by this row

   • MANUAL_TESTING_MATRIX.md: five expectations that no longer match HFS (T9 Error chip timing since #2024, T5 file counts and Retry, T4 4.10) #2141 (matrix: five stale expectations): all five were confirmed on this row. Item 1: the Error chip took 15 min 06 s. Item 2: 5.4 gave 2 files for 1,001 lines. Item 3: files are named <Type>-0001.ndjson. Item 4: Retry creates a new card, so Delete is needed twice. Item 5: rows carry no subject.display.
   • bulk-submit: expose the deferred search-index rebuild (outstanding, job id, outcome) to API clients #1919 (deferred rebuild not exposed to API clients): still true. The job id came only from the deferred reindex generation started … job_id= log line; the /ui banner does not show it either.
   • mongodb search: every search collects all matching ids before paging, so a 20-row page costs the whole match set and fails with 422 above 300,000 matches #1896 (MongoDB search collects every id; 422 above 300,000 matches): not reached. The largest searched match set was 16,448. GET /Observation?_summary=count over 689,186 answered 200 in 1.2 s with no 422.
   • sof(sql-query): a depends-on ViewDefinition is materialized in full before the query's WHERE, so any query over a view past HFS_SOF_SQLQUERY_MAX_SOURCE_ROWS_PER_VD fails with row limit exceeded #1473 (SQL Query dependency row cap): not reached. observation_flat has 689,186 rows, below the 1,000,000 cap.
   • Deferred reindex after bulk fast-load does not survive a restart on PostgreSQL and MongoDB: the #1125 index_pending marker is implemented only by the SQLite job store #1283 (deferred rebuild lost on restart): not exercised. There was no restart during the rebuild.
   • MongoDB: WriteConflict aborts exhaust the 3-attempt bundle retry under 20 concurrent imports — 477/1000 benchmark bundles fail with 503 #1776 (WriteConflict under 20 concurrent imports): not exercised.

   Observations (not failures; seen once, not independently verified, nothing filed)

   1. Search and history Bundles are served as Content-Type: application/json, while reads and /metadata are served as application/fhir+json; fhirVersion=4.0. This covers GET /Patient?_id=…, GET /Patient/<id>/_history and POST /Patient/_search. The $export status manifest is also application/json. No existing issue was found; rest: PATCH never negotiates the response format, and conditional PATCH has no route #863 covers PATCH only.
   2. Cancelling a SQL export (7.e, 17:58:50Z) logs ERROR helios_rest::error: internal error while processing request error.detail=View execution was cancelled. It is the only ERROR line in the log that is not an expected 501. The UI shows Cancelled correctly. Related closed issues: sof: deterministic order, bounded memory and cancellable runs on every SQL-on-FHIR runner #1623, sof: stop database work when a run is abandoned or times out #1864.
   3. On short SQL export jobs, the card's finished in is the 5 s poll interval rather than the run time: 0m 05s–0m 07s on the card against 0.12–0.66 s on the server.
   4. The Resources results header reads "1 results", from queries-results-total = { $count } results (crates/ui/locales/en/main.ftl:492). ui(export): the 'N running' header on Exports and SQL Exports stays stale after a job finishes; singular counts use plural words #1583 (closed) fixed the same wording on the Exports pages.
   5. The first start on an empty database logs 3 × WARN … LockTimeout … helios.search_index … running it again while the generation-4 index build runs. Seeding then finished with failed=0.
   6. The cold/warm search timings listed under Timings (4.11, 4.8).
   7. The ViewDefinition editor's key completion tags resource as required but not status, while the guided form marks status required. The lint models status as optional on purpose (crates/sof/src/lint.rs:400-414).
   8. The T3 ingest (3,122 s) and rebuild (4,338 s) are well above the CI mongodb figures cited for perf(mongodb): $bulk-submit runs at ~60 resources/s per file — 23 documents and ~250 index key insertions per resource, 55 GB of indexes against a 7 GB cache #1000 (about 650–690 s ingest and 600–1,450 s rebuild). The hardware differs, the mongod cache was capped at 8 GB, and no T3 time criterion exists.

   Not run

   • N/A on this row: 7.5 step 5 (Elasticsearch counts), 9.5 (S3 output backend), 11.7 (S3 sink), and 4.13 _content (501).
   • T8: $events, which is on the issue checklist but not in the matrix procedure.
   • T9:
     • Recovery inside the retry window; only reactivation after the window was exercised.
     • Row order under the Sort menu; only one Subscription exists.
     • Idle > 0, which needs a websocket subscription.
     • HFS_SUBSCRIPTION_REHYDRATE=false.
   • T5: cancelling an export that had made real progress. 5.5 was cancelled at "Waiting for the first status report…".
   • T4: a Saved query created in the UI, because the UI has no save control. One was seeded through PATCH /_user/settings and removed afterwards.
   • Abort of an in-flight manifest (bulk-submit: Abort cannot stop an in-flight manifest, and a failed abort is invisible #968), and every fhir2-scale expectation (this is a bounded pass).

   Matrix corrections

   There are 38 new entries against d3976b4bd, beyond the five already in #2141 (all five confirmed here). The full list with line numbers is on the evidence page. The main ones:

   • L337: no backend label in the sidebar.
   • L343: 1,372 SearchParameters, not 1,374.
   • L143: the build command lacks --release.
   • L148-150 and L1435: --all-features skips the R6 download, and the tree stayed clean.
   • L659: the rebuild banner has no job id.
   • L688: there is no Patient stat card.
   • L761, L801, L823 and L845: there is no Open in New Tab link.
   • L809: 4.14 does not account for the seeded _summary=true row.
   • L774, L861, L1244 and L1296: Edit raw no longer exists.
   • L872-874: download pills are on the detail page.
   • L1095-1099: 11.2 step 4 shows a waiting notice and a Parameters field.
   • L1176-1178: 3 of 4, not 2 of 4.
   • L1189: the 11.6 preview wording.
   • L1317-1320: T9 Delivered is the event count in the webhook log.
   • L1343-1345: Sent and Delivered reset to 0 after a restart.

   These are added to #2141.

   Evidence

   All the attachments are on one page: https://claude.ai/artifact/GMuvgtEWESzwZb1JWLhN25. It shows each file's contents in tables, charts and log views; the files themselves are not downloadable from it.

   • Summary: the filled-in matrix row.
   • Screenshots: /ui after the import, the T3 submission in its Completed state, the T2 Per-Action Outcomes for the transaction, SQL Exports with completed cards, and /ui/subscriptions after T9 step 3 and in the error state.
   • T0: a timeline of both build attempts and the end of each build log (attempt 2 BUILD_EXIT=0, attempt 1 the SIGKILL).
   • T3: import and rebuild progress, counts per resource type, the database size, the submission log, the whole-file byte check and the timing line.
   • T4: the 87 queries (count, status, cold/warm ms), each with the URL it actually sent.
   • T5: the system-, patient- and group-level manifests, and the contents of the ZIP from Download All Resources.
   • T7: card time vs server time per job, the per-file parse results, both result Parameters (not Bulk Data manifests), and the first rows of one output file per subject kind × format.
   • Server log: hfs-mongodb.log (1,164 lines, 5 server starts, 3 ERROR lines: 2 expected 501s and observation 2), with every WARN and ERROR grouped and a searchable view of all lines.
   • Matrix corrections: all 38 entries with line numbers.
5. dougc95 commented on Oct 27, 2026

   

   dougc95

   on Oct 27, 2026

   Contributor

   More actions

   Closing

   The 2026-10-19 pass on v0.2.5 (d3976b4bd) passes all ten cells of the mongodb row: see the results comment. The evidence, screenshots included, is on one page.

   This was a bounded pass on bulk_1k (1,630,685 resources), so it says nothing about fhir2-scale limits. T8's $events was not run.

   Still open:

   • MANUAL_TESTING_MATRIX.md: five expectations that no longer match HFS (T9 Error chip timing since #2024, T5 file counts and Retry, T4 4.10) #2141: matrix corrections, including the 38 entries from this row.
   • bulk-submit: expose the deferred search-index rebuild (outstanding, job id, outcome) to API clients #1919: the deferred search-index rebuild is not exposed to API clients.
   • mongodb search: every search collects all matching ids before paging, so a 20-row page costs the whole match set and fails with 422 above 300,000 matches #1896: MongoDB search above 300,000 matches. This corpus never reached it.
6. dougc95

   closed this as completedon Oct 27, 2026