Repository navigation
Reduce Loom ingestion time and inventory writes - #216
Open
matthewpeterkort wants to merge 16 commits into
Open
matthewpeterkort wants to merge 16 commits into
matthewpeterkort wants to merge 16 commits into
Conversation
Remove the BMEG schema modules and generic ingestion fallback. Generate the model, validators, extractors, and compiler metadata from the pinned HL7 R5 package and Loom-owned edge and validation rules. Preserve the configured resource allowlist and existing publication schema identity. Verified generation is reproducible. The generated model, validators, and edge extractors match main byte for byte. Across 70,000 META_SMALL records, accept/reject counts and canonical edge document digests match. The Go suite, Go comment lint, and isolated locked-cohort Docker and Chrome acceptance passed. Acceptance run f39518fab9172bd5 was cleaned up. Same-model adversarial reviews covered runtime schema membership, publication and uploads, and generator metadata. Regression tests cover the local and absolute JSON Pointer references and nested anyOf rules found in review. The repeated-reference metadata fix passed the existing Explorer negative test and an independent regeneration. Reviewed runtime diff SHA-256: bbf96193e229bb18630dde707901c52469997564056e43171e86aed9f70db6a4 Reviewed generated metadata SHA-256: 3bf5a764c523a323af0b91f10595d96c343349aae789712e92eed5c819055adf
Check in the official JSON Schema and selected StructureDefinitions. Derive supported models and forward reference edges from those files. Remove the legacy graph schema, compatibility snapshot, and runtime schema settings. Compile the pinned primitive patterns into direct lexical checks. Five single-core runs over 48,000 shared accepted records measured 7.67 us per record median for direct checks versus 16.70 us for regex checks. Both variants emitted 96,557 edges and matched canonical edge digests. The full 70,000-record fixture rejects 20,000 DocumentReferences with numeric Attachment.size and 2,000 ImagingStudies with whitespace in Coding.system. New publications record the official input digest. Cross-schema partial updates require a reload so old edges cannot be copied into a new generation. Validated with the Go suite, direct pattern tests, compiler conformance, Go comment lint, and isolated Docker and Chrome acceptance run 6e428a0ee3da3756. Same-model adversarial review covered primitive pattern equivalence, FHIR edges and metadata, and schema identity migration. Final metadata SHA-256: 485fbfd84469f7777414eed51ece8f41687458bc054f3bcf16b2af1711fc8156
Check calendar dates, base64 content, and signed integer64 strings. Normalize legacy numeric integer64 values at ingest while preserving strict Coding.system validation and the conforming decode path. Verification: generator, model, FHIR schema, and ingest tests; Go comment lint; read-only primitive and compatibility reviews.
Update Sonic to v1.15.3 and its loader to v0.5.2 for native Go 1.27 support. Keep the existing error coverage under Go 1.27 vet and apply the new gofmt indentation. Verification: root and client Go suites passed on Go 1.27.4, including formatting and vet gates. Native Sonic decode verified on Darwin arm64.
Go 1.27 changes the DEFLATE output embedded by oapi-codegen. Verified both blobs decode to identical OpenAPI JSON. FHIR and OpenAPI regeneration, route ownership checks, and API tests pass.
Copied the four vertex-namespace files from the operator's in-progress fhir/own-generation worktree without editing that worktree. Focused generated row and generation identity tests, full Go suite with loopback, and Go comment lint passed. Reviewed with the Phase 0 snapshot; no verified findings in the vertex change.
Phase 0. Measure default per-file ingestion with a counting inserter and capture documents for normalized vertex, edge, and real-AQL inventory comparisons. Route the production loader and test harness through the same sequential scheduling entry point so Phase 3 is measurable. B0 META_SMALL digest ee1d460a880f10a96d5f10a54d863aabd9aafa56f89a3f2b7507e7653d839d0a. Go 1.27.4. 28.15 s median (27.75–28.68 s), 13,145 rows/s, 370,048 rows, 756,130 edges, 2,739,454 inventory docs. The exact real-Arango load exited 1 after 210.90 s with 5,920 ImagingStudy validation errors and no activation. Focused Go packages, full make test-go with loopback access, and make lint-go-comments passed. B0 reviewed patch SHA256 808fb942b1c85b65b29e4eb296acd5140904c46d0859154e5034939ae86e01a0 with three gpt-6.1-sol low reviewers. Resolved the capture identity false pass by running the production materializer; no verified findings remain.
Phase 1. Share traversal between the byte walker and decoded-value walker. Preserve ordered facts, observations, numeric example encoding, and source evidence. Coverage census keeps its JSON boundary because current coverage calls supply only a map. B1 preliminary META_SMALL run: 27.58 s, 13,418 rows/s, field_profile 146.54 worker-s and decode 22.72 worker-s. The full capture matched B0 for normalized edges and vertices and sorted raw inventory inputs. Focused packages, full Go suite, and Go comment lint passed. The ordered-array counterexample ruled out the existing fingerprint as a semantic cache key; the optional 1b cache was not retained or timed. Reviewed patch SHA256 4c579a05492bfb99242d1e47aff6a31d0f5788f0cdca2fa73378568fc295bf73 with three gpt-6.1-sol low roles. Reproduced and fixed scalar and projected numeric format drift. Typed nil parity fixed. Differential coverage includes 270 rows across 112 fixtures and the full B0 capture. No verified findings remain.
Phase 4. Generated typed edge documents carry auth, generation, and source-file identity to a single encoding. Relationship keys come from typed edges, with the raw path retained for custom builders. The public ExtractEdges wrapper remains. B2 preliminary META_SMALL run: 20.09 s, 18,421 rows/s, field_profile 117.08 worker-s, decode 17.63 worker-s. All normalized capture output matches B0. Generation, package tests, full Go suite, comment lint, and focused real-Arango source copy/key collision checks passed. Corpus verified 375,968 rows in all eight auth/generation/source combinations. Three gpt-6.1-sol low roles reviewed patch SHA256 22b13e52f316d706c4a03f84ef0baa6e9874c05425312a1ac42435ac227f1d99. Custom fallback ownership and stale relationship key issues were fixed and regression-checked. Corrected a baseline Arango test that still expected reverse edge documents. No verified findings remain.
Phase 2. Coalesce exact AQL reducer input vectors, sum population, and combine truncation. Leave distinct MIN/MAX inputs for Arango's collation and preserve retained backfill. Use bounded spill runs only above the memory budget, stable file-relative identities, and source-key replacement on every batch. No inventory schema bump because completed builds always materialize the same indexed entry format. B3 preliminary META_SMALL: 28.53 s, 12,968 rows/s, 559,281 inventory docs and 631.44 MB, versus 2,739,454 docs and 3,160.18 MB at B0. Vertices, edges, and real AQL materialized inventory entries all match B0. Focused packages, full Go suite, Go comment lint, make test-integration, real indexed inventory equality, and aggregated source-copy tests passed. Exact integrated patch SHA256 5d85b5fef1c59daa2a5f5cbab8354bf1ebffd47c07316cfc7b564a2329a80c08 reviewed by three gpt-6.1-sol low roles. Confirmed and resolved repeated decode and descriptor exhaustion. Verified 257 runs under ulimit -n 128. No unresolved findings.
Phase 3. Schedule larger files first, share dynamic row and writer permits, and advance semantic inventory checkpoints immediately over the completed input prefix. Preserve serial mode, row-error ordering, independent catalog retention, and per-file progress. Single-file resource and upload entry points stay sequential. B4 preliminary META_SMALL: 29.70 s, 12,458 rows/s, 559,281 inventory docs / 631.44 MB, 756,130 edges. Normalized vertices, edges, and materialized inventory match B0. Focused and full Go checks, comment lint, make test-integration, and race-enabled live Arango serial/concurrent, prefix, fail-fast, and cancellation checks passed. Final patch SHA256 eb29ffd935ce4b04b2bddd2e0789b839fb7d62f4753b17d17d89b348c5c60dc9. Same-model gpt-6.1-sol low reviewers checked checkpoint ordering, bounded concurrency, final output, and retention. Restored an independent generation merge cache and verified preservation of fields at low retention. No unresolved findings.
Record request body bytes, request duration and writer permit wait by collection class. Emit timings for the five generation finalization stages. Preserve the write-path benchmark plan with the instrumentation. Verified with focused Go tests, make test-go, make lint-go-comments and make test-integration on owned Arango and ClickHouse. The integration driver passed with its explicit external-case exclusions. The first run failed its source identity check and was rerun against the frozen tree. Interrogate used three same-model gpt-6.1-sol reviewers at low effort, covering request accounting under shared permits, cancellation and raw API compatibility, and finalization ordering. No verified unresolved findings. Reviewed workers/client diff SHA-256 a1e0fad2b53c6e753951ab40e06811211f46f19e99dc80eec1b7492a35b830fe. Final orchestration blob 15afc36. Integration source fingerprint 7139ae7fd2a24cbc73f5da409eb4f3265315b846901edd3b4fd333d2bcb35d52. The instrumented C0 measurements follow the commit and are recorded under outputs/ingest-perf/C0.
Reduce file observations to final inventory entries with bounded spill merging. Merge repeated resource types across files, publish entries at the current index version, and re-key them for copy and replacement. Keep retained contribution collection and backfill behavior. C0 median 86.06 s (82.72–86.68 s). Corrected direct-write capture 55.15 s; final watched load 54.79 s, with a stalled Docker-stats sampler recorded as a runner failure and capture recovered from the same DB. Normalized hash and count equality across 14 vertex/edge collections. 941 entries match except five byte-order examples, all verified as group members and byte minima. Both snapshots have 370048 vertices and 756130 edges. File inventory writes drop from 559281 raw entries to 941 final rows. Passed focused reducer, spill, copy, paging, selection and retained tests, make test-go, lint-go-comments, and make test-integration with its documented external-case exclusions. Source copy E2E passes after updating its obsolete physical raw-row assertion to compare re-keyed final rows. Full integration fingerprint before the targeted test-only update: 06ef082d199c54dbdb44f1a5ede0b28ce458c6608b93090ff225b8c225b9b3ac. Interrogate panel: three gpt-6.1-sol reviewers at low effort, same model. Reviewed staged tree b7b398b including the test-only change, with lifecycle/spill re-review on f5deb645648ac61df3e7a02d27e293a06ff8d2b3. Reproduced and fixed unbounded open spill descriptors and late spill creation after writer failure. Both reproducers pass on the fixes. No unresolved verified findings.
Use the two generation resource indexes and one auth-leading edge index with stored source_file. Keep native edge traversal indexes; remove redundant per-endpoint persistent indexes from new bootstrap. Hint the primary index for copy pagination and project constant scopes from relationship rebuild without reading edge documents. Use stored auth and type in typed preview indexes while preserving type filtering and legacy index compatibility. Record the index usage contract. Prior same-host Arango sanity: 34.36 s with these ingest changes after 53.73 s on committed Track A, 86.06 s median C0. All 370048 vertices, 756130 edges and 941 entries matched the previous capture contract with the allowed five example differences. Tested highest-degree walks 10.99/69.59 ms p95 compared with 10.53/76.61 ms in the equivalent C0 DB. The benchmark was measured before final preview-only fixes. Durable post-restart measurements follow. Passed full Go, comment-lint, focused Arango, and the integrated suite with its documented external-case exclusions. The earlier standalone projection run lacked fixture collections; the driver seeds them. Three same-model gpt-6.1-sol low-effort interrogate reviewers covered bootstrap, relationship/copy behavior and preview compatibility. Fixed confirmed lost root resourceType predicate, then rechecked live and with the original focused probe. The final test/formatting delta was reviewed separately. No unresolved verified findings in reviewed scope. Final staged tree d6201f9.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ingest performance
The latest ArangoDB sanity load took 34.36 s on the valid META_SMALL subset, down from an 86.06 s median across three runs of the prior write path (about 60% less wall time). The direct-inventory change alone took 53.73 s. These loads used eight workers and eight writers. The earlier ingestion round measured 240.99 s → 94.62 s in a separate comparison.
fhir_edge. Keep auth in the generation indexes and use native edge indexes for walks. Copy pages use the primary index, and the relationship count query is covered by the edge index. Existing databases retain any older indexes.Verification
Ingest performance plan · Write-path plan · Arango index contract