Repository navigation
Perf/ingest r4 - #217
Open
matthewpeterkort wants to merge 7 commits into
Open
Perf/ingest r4#217matthewpeterkort wants to merge 7 commits into
matthewpeterkort wants to merge 7 commits into
Conversation
Reuse bounded, worker-local fact and scope buffers and clear row references before reuse. Group semantic inventory contributions by their exact four fields, preserving the sort order used by existing spill runs. On 370,048 matched valid META_SMALL rows with default GC and eight workers, five interleaved runs cut median CPU-floor wall from 17.03 s to 12.44 s, allocated bytes from 92.51 GB to 37.72 GB, and peak RSS from 1.146 GB to 1.075 GB. Eight-writer Arango loads went from 32.48 s to 29.16 s. Matched 2,739,454 ordered semantic emissions and the stored vertices, edges, and inventory. Thirty capped field-catalog sample sets varied; all differing values were verified in their corresponding payloads. Verified make test-go, make lint-go-comments, make test-integration, focused race tests, and native verify-loom fixture publication to ClickHouse. The integration suite reports its external fixture checks as skipped. Review: three gpt-6.1-sol reviewers, no unresolved findings. Reviewed base c35c92a and patch SHA256 c591b191027913bd6fa9ce8970dbd3f33b885ec5127d91d04cc92b2651b4d649.
Look up extension URL owners at dotted boundaries in ascending depth. A scan-oracle test checks the same ordered chain for nested extensions, empty URL values, and owners that merely share a string prefix. Include the Round 4 measurement and verification plan. Three no-write samples per source on ResearchStudy moved from 18.83 s median (18.81–19.10) to 1.15 s (1.12–1.15). Full CDA no-write medians moved from 50.52 s to 34.52 s. The 429 ResearchStudy rows and all CDA row, edge, and error counts match in both binaries. Verified focused catalog tests, make test-go, make lint-go-comments, and make test-integration on the isolated B7 + Item 1 snapshot. Reviewed b7a35f5 + SHA-256 342aa0bd41ae66ab65a555e7f4546385d2c9aa5bee05d01e60d697381f06cd29 with three gpt-6.1-sol low-effort pstack:interrogate reviewers (same model). No unresolved findings.
Reuse declared elements, required names, and choice groups across walks. The bounded cache keys on immutable elements and exact required names. Initialize the flattened fields once per known resource type. Unknown input names never enter the cache. Three no-write samples per candidate on CDA. Scope caching moved the median from 34.52 s and 137.02 GB allocated to 26.40 s and 88.66 GB. Resource-field initialization then moved it to 26.08 s and 84.63 GB. On META_SMALL, the cumulative median moved from 10.08 s and 37.63 GB to 7.65 s and 23.75 GB. CDA allocations exceeded the baseline spread. Validated scope diagnostic order and parent-specific requirements, concurrent field publication, and unknown name retention under -race. make test-go, make lint-go-comments, and make test-integration passed on the exact production and test snapshot. Reviewed b7a35f5 + cumulative diff SHA-256 bf3967a1f557f8454f7fe77d7dd324323181f615b2966b1e2e97493a09028c04 with three same-model gpt-6.1-sol low-effort pstack:interrogate reviewers. Confirmed unbounded negative-cache retention was fixed, verified using the 10,000-name reproducer. No unresolved findings.
Send up to 256 lines or 1 MiB per group and take one shared worker permit per group. Preserve the physical row identity and cancel between rows, including when progress callbacks cancel during a batch. Three no-write runs per candidate. CDA median 26.08 to 22.74 s and loader CPU 212.26 to 187.22 s. META_SMALL median 7.65 to 6.68 s. CDA allocated bytes rose from 84.63 to 84.87 GB, while peak RSS fell from 1.90 to 1.81 GB. Wall gain exceeded baseline spread. Focused row/error/checkpoint tests and batch cancellation race test passed. make test-go, make lint-go-comments, and make test-integration passed on the exact final source and test snapshot. Reviewed b7a35f5 + cumulative diff SHA-256 47e02b38f57c5f458c21f8e57e926281f78ffcbb1ee8fc3a3dee569ceb88fd69 with three same-model gpt-6.1-sol low-effort pstack:interrogate reviewers. No unresolved findings.
Share the default between the CLI and the loader, while preserving explicit positive queue-size overrides. Fresh CDA, one load per setting after line batching. The CLI default of 8 batches used 2.28 GB peak loader RSS and took 96.35 s. Two batches used 2.01 GB and took 91.05 s. Four used 2.15 GB and took 90.73 s. Two batches saved 271 MB against the measured CLI default, with wall time inside the 17.98 s baseline spread. The CLI parse and normalization tests verify the default and an explicit override. make test-go, make lint-go-comments, and make test-integration passed. Reviewed exact base 8ed76e9 + diff SHA-256 64ca215328eb7f864f384bd9bb5b73a1ed98bbd4b3893ce749402e922bbcf45a using three same-model gpt-6.1-sol low-effort pstack:interrogate reviewers. No unresolved findings.
CDA observed 131.70 s baseline median versus 109.63 s compact-key load. Edge request bytes fell from 2.606 GB to 2.192 GB. Preserve the persisted identity format for existing and resumed generations. Live Go/AQL parity and manifest lifecycle tests passed. Confirmed mismatched identity CAS mutated state before the fix; fixed transition and activation reject it before writes. Go, lint, and integration gates passed. Interrogate receipt: b7a35f5 + f477e3fbc9ed914d7f46ed345d94eb69b18669b19278c146c9be3353eb6d9116, three same-model gpt-6.1-sol low reviews, no unresolved findings.
Integrate the local Builder follow-up on PR #217. Remove source rules from the recipe, verify saved legacy digests before converting drafts, and return stale editors to the selected node. Add the optional publication transport used by the Calypr Funnel integration. Verified 1,601 UI tests, affected Explorer and migration Go suites, Go comment lint, and the scoped migration against disposable Docker Arango. Reviewed migration, removal contract and Builder UI separately. Staged diff sha256 bb85a056a73d76dd9bb4018aa0aa0543ccec09e3841d06ec8c7b5d6123e24278.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.