Skip to content

Reduce Loom ingestion time and inventory writes - #216

Open
matthewpeterkort wants to merge 16 commits into
mainfrom
perf/ingest
Open

matthewpeterkort wants to merge 16 commits into
mainfrom
perf/ingest

Conversation

@matthewpeterkort

@matthewpeterkort matthewpeterkort commented Nov 18, 2026 •

Copy link
Copy Markdown
Collaborator

Ingest performance

The latest ArangoDB sanity load took 34.36 s on the valid META_SMALL subset, down from an 86.06 s median across three runs of the prior write path (about 60% less wall time). The direct-inventory change alone took 53.73 s. These loads used eight workers and eight writers. The earlier ingestion round measured 240.99 s → 94.62 s in a separate comparison.

  • Walk decoded FHIR values for semantic discovery, encode vertices and edges once, and load files concurrently within shared worker and writer budgets. Advance checkpoints in input order.
  • Write the 941 final semantic inventory entries directly. The previous write path wrote 559,281 contributions before materializing the same groups. Source copy re-keys entries, uploads replace entries for the selected resource type, and retained backfill still uses contributions.
  • On fresh databases, reduce the index count from 11 to 3 per FHIR resource collection and on fhir_edge. Keep auth in the generation indexes and use native edge indexes for walks. Copy pages use the primary index, and the relationship count query is covered by the edge index. Existing databases retain any older indexes.
  • Emit request bytes, latency, writer-permit waits, and finalization timings. Batch and writer defaults are unchanged.

Verification

  • Normalized documents matched across 370,048 vertices and 756,130 edges. Of the 941 inventory entries, five examples differed under byte ordering. Each was verified as the byte-order minimum of its group. All other fields matched.
  • Full Go tests, Go comment lint, and the integration suite passed with its documented external-case exclusions. Focused Arango checks covered source copy and replacement, index plans, and typed preview behavior.
  • The 34.36 s result is a single sanity run. The final interleaved benchmark and real-data ClickHouse publication acceptance were skipped.

Ingest performance plan · Write-path plan · Arango index contract

Remove the BMEG schema modules and generic ingestion fallback. Generate
the model, validators, extractors, and compiler metadata from the pinned
HL7 R5 package and Loom-owned edge and validation rules. Preserve the
configured resource allowlist and existing publication schema identity.

Verified generation is reproducible. The generated model, validators,
and edge extractors match main byte for byte. Across 70,000 META_SMALL
records, accept/reject counts and canonical edge document digests match.
The Go suite, Go comment lint, and isolated locked-cohort Docker and
Chrome acceptance passed. Acceptance run f39518fab9172bd5 was cleaned up.

Same-model adversarial reviews covered runtime schema membership,
publication and uploads, and generator metadata. Regression tests cover
the local and absolute JSON Pointer references and nested anyOf rules
found in review. The repeated-reference metadata fix passed the existing
Explorer negative test and an independent regeneration.

Reviewed runtime diff SHA-256:
bbf96193e229bb18630dde707901c52469997564056e43171e86aed9f70db6a4
Reviewed generated metadata SHA-256:
3bf5a764c523a323af0b91f10595d96c343349aae789712e92eed5c819055adf
Check in the official JSON Schema and selected StructureDefinitions. Derive
supported models and forward reference edges from those files. Remove the
legacy graph schema, compatibility snapshot, and runtime schema settings.

Compile the pinned primitive patterns into direct lexical checks. Five
single-core runs over 48,000 shared accepted records measured 7.67 us per
record median for direct checks versus 16.70 us for regex checks. Both
variants emitted 96,557 edges and matched canonical edge digests. The full
70,000-record fixture rejects 20,000 DocumentReferences with numeric
Attachment.size and 2,000 ImagingStudies with whitespace in Coding.system.

New publications record the official input digest. Cross-schema partial
updates require a reload so old edges cannot be copied into a new generation.

Validated with the Go suite, direct pattern tests, compiler conformance,
Go comment lint, and isolated Docker and Chrome acceptance run
6e428a0ee3da3756. Same-model adversarial review covered primitive pattern
equivalence, FHIR edges and metadata, and schema identity migration.
Final metadata SHA-256:
485fbfd84469f7777414eed51ece8f41687458bc054f3bcf16b2af1711fc8156
Check calendar dates, base64 content, and signed integer64 strings. Normalize legacy numeric integer64 values at ingest while preserving strict Coding.system validation and the conforming decode path.

Verification: generator, model, FHIR schema, and ingest tests; Go comment lint; read-only primitive and compatibility reviews.
Update Sonic to v1.15.3 and its loader to v0.5.2 for native Go 1.27 support. Keep the existing error coverage under Go 1.27 vet and apply the new gofmt indentation.

Verification: root and client Go suites passed on Go 1.27.4, including formatting and vet gates. Native Sonic decode verified on Darwin arm64.
Go 1.27 changes the DEFLATE output embedded by oapi-codegen. Verified both blobs decode to identical OpenAPI JSON. FHIR and OpenAPI regeneration, route ownership checks, and API tests pass.
Copied the four vertex-namespace files from the operator's in-progress fhir/own-generation worktree without editing that worktree. Focused generated row and generation identity tests, full Go suite with loopback, and Go comment lint passed. Reviewed with the Phase 0 snapshot; no verified findings in the vertex change.
Phase 0. Measure default per-file ingestion with a counting inserter and capture documents for normalized vertex, edge, and real-AQL inventory comparisons. Route the production loader and test harness through the same sequential scheduling entry point so Phase 3 is measurable.

B0 META_SMALL digest ee1d460a880f10a96d5f10a54d863aabd9aafa56f89a3f2b7507e7653d839d0a. Go 1.27.4. 28.15 s median (27.75–28.68 s), 13,145 rows/s, 370,048 rows, 756,130 edges, 2,739,454 inventory docs. The exact real-Arango load exited 1 after 210.90 s with 5,920 ImagingStudy validation errors and no activation.

Focused Go packages, full make test-go with loopback access, and make lint-go-comments passed. B0 reviewed patch SHA256 808fb942b1c85b65b29e4eb296acd5140904c46d0859154e5034939ae86e01a0 with three gpt-6.1-sol low reviewers. Resolved the capture identity false pass by running the production materializer; no verified findings remain.
Phase 1. Share traversal between the byte walker and decoded-value walker. Preserve ordered facts, observations, numeric example encoding, and source evidence. Coverage census keeps its JSON boundary because current coverage calls supply only a map.

B1 preliminary META_SMALL run: 27.58 s, 13,418 rows/s, field_profile 146.54 worker-s and decode 22.72 worker-s. The full capture matched B0 for normalized edges and vertices and sorted raw inventory inputs. Focused packages, full Go suite, and Go comment lint passed. The ordered-array counterexample ruled out the existing fingerprint as a semantic cache key; the optional 1b cache was not retained or timed.

Reviewed patch SHA256 4c579a05492bfb99242d1e47aff6a31d0f5788f0cdca2fa73378568fc295bf73 with three gpt-6.1-sol low roles. Reproduced and fixed scalar and projected numeric format drift. Typed nil parity fixed. Differential coverage includes 270 rows across 112 fixtures and the full B0 capture. No verified findings remain.
Phase 4. Generated typed edge documents carry auth, generation, and source-file identity to a single encoding. Relationship keys come from typed edges, with the raw path retained for custom builders. The public ExtractEdges wrapper remains.

B2 preliminary META_SMALL run: 20.09 s, 18,421 rows/s, field_profile 117.08 worker-s, decode 17.63 worker-s. All normalized capture output matches B0. Generation, package tests, full Go suite, comment lint, and focused real-Arango source copy/key collision checks passed. Corpus verified 375,968 rows in all eight auth/generation/source combinations.

Three gpt-6.1-sol low roles reviewed patch SHA256 22b13e52f316d706c4a03f84ef0baa6e9874c05425312a1ac42435ac227f1d99. Custom fallback ownership and stale relationship key issues were fixed and regression-checked. Corrected a baseline Arango test that still expected reverse edge documents. No verified findings remain.
Phase 2. Coalesce exact AQL reducer input vectors, sum population, and combine truncation. Leave distinct MIN/MAX inputs for Arango's collation and preserve retained backfill. Use bounded spill runs only above the memory budget, stable file-relative identities, and source-key replacement on every batch. No inventory schema bump because completed builds always materialize the same indexed entry format.

B3 preliminary META_SMALL: 28.53 s, 12,968 rows/s, 559,281 inventory docs and 631.44 MB, versus 2,739,454 docs and 3,160.18 MB at B0. Vertices, edges, and real AQL materialized inventory entries all match B0. Focused packages, full Go suite, Go comment lint, make test-integration, real indexed inventory equality, and aggregated source-copy tests passed.

Exact integrated patch SHA256 5d85b5fef1c59daa2a5f5cbab8354bf1ebffd47c07316cfc7b564a2329a80c08 reviewed by three gpt-6.1-sol low roles. Confirmed and resolved repeated decode and descriptor exhaustion. Verified 257 runs under ulimit -n 128. No unresolved findings.
Phase 3. Schedule larger files first, share dynamic row and writer permits, and advance semantic inventory checkpoints immediately over the completed input prefix. Preserve serial mode, row-error ordering, independent catalog retention, and per-file progress. Single-file resource and upload entry points stay sequential.

B4 preliminary META_SMALL: 29.70 s, 12,458 rows/s, 559,281 inventory docs / 631.44 MB, 756,130 edges. Normalized vertices, edges, and materialized inventory match B0. Focused and full Go checks, comment lint, make test-integration, and race-enabled live Arango serial/concurrent, prefix, fail-fast, and cancellation checks passed.

Final patch SHA256 eb29ffd935ce4b04b2bddd2e0789b839fb7d62f4753b17d17d89b348c5c60dc9. Same-model gpt-6.1-sol low reviewers checked checkpoint ordering, bounded concurrency, final output, and retention. Restored an independent generation merge cache and verified preservation of fields at low retention. No unresolved findings.
Record request body bytes, request duration and writer permit wait by
collection class. Emit timings for the five generation finalization stages.
Preserve the write-path benchmark plan with the instrumentation.

Verified with focused Go tests, make test-go, make lint-go-comments and
make test-integration on owned Arango and ClickHouse. The integration
driver passed with its explicit external-case exclusions. The first run
failed its source identity check and was rerun against the frozen tree.

Interrogate used three same-model gpt-6.1-sol reviewers at low effort,
covering request accounting under shared permits, cancellation and raw API
compatibility, and finalization ordering. No verified unresolved findings.
Reviewed workers/client diff SHA-256 a1e0fad2b53c6e753951ab40e06811211f46f19e99dc80eec1b7492a35b830fe.
Final orchestration blob 15afc36.
Integration source fingerprint 7139ae7fd2a24cbc73f5da409eb4f3265315b846901edd3b4fd333d2bcb35d52.

The instrumented C0 measurements follow the commit and are recorded
under outputs/ingest-perf/C0.
Reduce file observations to final inventory entries with bounded spill
merging. Merge repeated resource types across files, publish entries at
the current index version, and re-key them for copy and replacement.
Keep retained contribution collection and backfill behavior.

C0 median 86.06 s (82.72–86.68 s). Corrected direct-write capture
55.15 s; final watched load 54.79 s, with a stalled Docker-stats sampler
recorded as a runner failure and capture recovered from the same DB.
Normalized hash and count equality across 14 vertex/edge collections.
941 entries match except five byte-order examples, all verified as group
members and byte minima. Both snapshots have 370048 vertices and 756130
edges. File inventory writes drop from 559281 raw entries to 941 final rows.

Passed focused reducer, spill, copy, paging, selection and retained tests,
make test-go, lint-go-comments, and make test-integration with its
documented external-case exclusions. Source copy E2E passes after updating
its obsolete physical raw-row assertion to compare re-keyed final rows.
Full integration fingerprint before the targeted test-only update:
06ef082d199c54dbdb44f1a5ede0b28ce458c6608b93090ff225b8c225b9b3ac.

Interrogate panel: three gpt-6.1-sol reviewers at low effort, same model.
Reviewed staged tree b7b398b including the
test-only change, with lifecycle/spill re-review on f5deb645648ac61df3e7a02d27e293a06ff8d2b3.
Reproduced and fixed unbounded open spill descriptors and late spill
creation after writer failure. Both reproducers pass on the fixes.
No unresolved verified findings.
Use the two generation resource indexes and one auth-leading edge
index with stored source_file. Keep native edge traversal indexes;
remove redundant per-endpoint persistent indexes from new bootstrap.
Hint the primary index for copy pagination and project constant scopes
from relationship rebuild without reading edge documents. Use stored
auth and type in typed preview indexes while preserving type filtering
and legacy index compatibility. Record the index usage contract.

Prior same-host Arango sanity: 34.36 s with these ingest changes after
53.73 s on committed Track A, 86.06 s median C0. All 370048 vertices,
756130 edges and 941 entries matched the previous capture contract
with the allowed five example differences. Tested highest-degree walks
10.99/69.59 ms p95 compared with 10.53/76.61 ms in the equivalent C0 DB.
The benchmark was measured before final preview-only fixes. Durable
post-restart measurements follow.

Passed full Go, comment-lint, focused Arango, and the integrated suite
with its documented external-case exclusions. The earlier standalone
projection run lacked fixture collections; the driver seeds them.

Three same-model gpt-6.1-sol low-effort interrogate reviewers covered
bootstrap, relationship/copy behavior and preview compatibility. Fixed
confirmed lost root resourceType predicate, then rechecked live and
with the original focused probe. The final test/formatting delta was
reviewed separately. No unresolved verified findings in reviewed scope.
Final staged tree d6201f9.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant