Skip to content

Perf/ingest r4 - #217

Open
matthewpeterkort wants to merge 7 commits into
perf/ingestfrom
perf/ingest-r4
Open

matthewpeterkort wants to merge 7 commits into
perf/ingestfrom
perf/ingest-r4

Conversation

@matthewpeterkort

Copy link
Copy Markdown
Collaborator

No description provided.

Reuse bounded, worker-local fact and scope buffers and clear row references
before reuse. Group semantic inventory contributions by their exact four
fields, preserving the sort order used by existing spill runs.

On 370,048 matched valid META_SMALL rows with default GC and eight workers,
five interleaved runs cut median CPU-floor wall from 17.03 s to 12.44 s,
allocated bytes from 92.51 GB to 37.72 GB, and peak RSS from 1.146 GB to
1.075 GB. Eight-writer Arango loads went from 32.48 s to 29.16 s.

Matched 2,739,454 ordered semantic emissions and the stored vertices,
edges, and inventory. Thirty capped field-catalog sample sets varied;
all differing values were verified in their corresponding payloads.

Verified make test-go, make lint-go-comments, make test-integration, focused
race tests, and native verify-loom fixture publication to ClickHouse.
The integration suite reports its external fixture checks as skipped.

Review: three gpt-6.1-sol reviewers, no unresolved findings. Reviewed base
c35c92a and patch SHA256
c591b191027913bd6fa9ce8970dbd3f33b885ec5127d91d04cc92b2651b4d649.
Look up extension URL owners at dotted boundaries in ascending depth.
A scan-oracle test checks the same ordered chain for nested extensions,
empty URL values, and owners that merely share a string prefix. Include
the Round 4 measurement and verification plan.

Three no-write samples per source on ResearchStudy moved from 18.83 s
median (18.81–19.10) to 1.15 s (1.12–1.15). Full CDA no-write medians
moved from 50.52 s to 34.52 s. The 429 ResearchStudy rows and all CDA
row, edge, and error counts match in both binaries.

Verified focused catalog tests, make test-go, make lint-go-comments,
and make test-integration on the isolated B7 + Item 1 snapshot.

Reviewed b7a35f5 + SHA-256
342aa0bd41ae66ab65a555e7f4546385d2c9aa5bee05d01e60d697381f06cd29
with three gpt-6.1-sol low-effort pstack:interrogate reviewers
(same model). No unresolved findings.
Reuse declared elements, required names, and choice groups across walks.
The bounded cache keys on immutable elements and exact required names.
Initialize the flattened fields once per known resource type. Unknown
input names never enter the cache.

Three no-write samples per candidate on CDA. Scope caching moved the
median from 34.52 s and 137.02 GB allocated to 26.40 s and 88.66 GB.
Resource-field initialization then moved it to 26.08 s and 84.63 GB.
On META_SMALL, the cumulative median moved from 10.08 s and 37.63 GB
to 7.65 s and 23.75 GB. CDA allocations exceeded the baseline spread.

Validated scope diagnostic order and parent-specific requirements,
concurrent field publication, and unknown name retention under -race.
make test-go, make lint-go-comments, and make test-integration passed
on the exact production and test snapshot.

Reviewed b7a35f5 + cumulative diff SHA-256
bf3967a1f557f8454f7fe77d7dd324323181f615b2966b1e2e97493a09028c04
with three same-model gpt-6.1-sol low-effort pstack:interrogate reviewers.
Confirmed unbounded negative-cache retention was fixed, verified using
the 10,000-name reproducer. No unresolved findings.
Send up to 256 lines or 1 MiB per group and take one shared worker
permit per group. Preserve the physical row identity and cancel between
rows, including when progress callbacks cancel during a batch.

Three no-write runs per candidate. CDA median 26.08 to 22.74 s and
loader CPU 212.26 to 187.22 s. META_SMALL median 7.65 to 6.68 s.
CDA allocated bytes rose from 84.63 to 84.87 GB, while peak RSS
fell from 1.90 to 1.81 GB. Wall gain exceeded baseline spread.

Focused row/error/checkpoint tests and batch cancellation race test
passed. make test-go, make lint-go-comments, and make test-integration
passed on the exact final source and test snapshot.

Reviewed b7a35f5 + cumulative diff SHA-256
47e02b38f57c5f458c21f8e57e926281f78ffcbb1ee8fc3a3dee569ceb88fd69
with three same-model gpt-6.1-sol low-effort pstack:interrogate reviewers.
No unresolved findings.
Share the default between the CLI and the loader, while preserving
explicit positive queue-size overrides.

Fresh CDA, one load per setting after line batching. The CLI default
of 8 batches used 2.28 GB peak loader RSS and took 96.35 s. Two
batches used 2.01 GB and took 91.05 s. Four used 2.15 GB and took
90.73 s. Two batches saved 271 MB against the measured CLI default,
with wall time inside the 17.98 s baseline spread.

The CLI parse and normalization tests verify the default and an
explicit override. make test-go, make lint-go-comments, and make
test-integration passed.

Reviewed exact base 8ed76e9 + diff SHA-256
64ca215328eb7f864f384bd9bb5b73a1ed98bbd4b3893ce749402e922bbcf45a
using three same-model gpt-6.1-sol low-effort pstack:interrogate
reviewers. No unresolved findings.
CDA observed 131.70 s baseline median versus 109.63 s compact-key load. Edge request bytes fell from 2.606 GB to 2.192 GB. Preserve the persisted identity format for existing and resumed generations.

Live Go/AQL parity and manifest lifecycle tests passed. Confirmed mismatched identity CAS mutated state before the fix; fixed transition and activation reject it before writes. Go, lint, and integration gates passed.

Interrogate receipt: b7a35f5 + f477e3fbc9ed914d7f46ed345d94eb69b18669b19278c146c9be3353eb6d9116, three same-model gpt-6.1-sol low reviews, no unresolved findings.
Integrate the local Builder follow-up on PR #217. Remove source rules from the recipe, verify saved legacy digests before converting drafts, and return stale editors to the selected node. Add the optional publication transport used by the Calypr Funnel integration.

Verified 1,601 UI tests, affected Explorer and migration Go suites, Go comment lint, and the scoped migration against disposable Docker Arango. Reviewed migration, removal contract and Builder UI separately. Staged diff sha256 bb85a056a73d76dd9bb4018aa0aa0543ccec09e3841d06ec8c7b5d6123e24278.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant