Spike: lazy streaming SELECT cursor for JS bindings#1
Draft
rybesh wants to merge 1 commit into
Draft
Conversation
Add Store.querySolutions(query, options) returning a QuerySolutions cursor that holds the QuerySolutionIter alive instead of draining it into an Array, plus QuerySolutions.nextBatch(count) / .variables. Lets the WASM bindings stream SELECT results in bounded batches rather than materializing the whole result set (the dual of the output-side memory ceiling tracked downstream in dkglab/graph-store#50). Key enabler: on_store() snapshots storage rather than borrowing &Store, so QueryResults/QuerySolutionIter are 'static and can live in a wasm-bindgen struct across JS calls with no self-referential borrow. The snapshot is MVCC-isolated, so an open cursor is unaffected by a concurrent load/clear (and pins that snapshot until closed). Spike only: minimal option parsing (base_iri), SELECT-only, no TS typings, no reuse of query()'s option handling. Verified by building real wasm and streaming a SELECT in batches in Node (lazy batching, stable exhaustion, snapshot isolation across a mid-iteration DELETE). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Spike — not for merge as-is
De-risks a lazy streaming SELECT cursor for the WASM/JS bindings, so the
worker side of graph-store can pull SELECT results in bounded batches instead
of materializing the entire result set into one
Array<Map>(or oneserialized string). This is the WASM-binding-side enabler for the output
memory ceiling tracked in dkglab/graph-store#50.
What it adds (
js/src/store.rs, +79/-1)Instead of the eager
for solution in solutions { array.push(...) }drain,the iterator is handed to a
#[wasm_bindgen]cursor struct and pulledcountrows at a time.Why it works (the bit worth reviewing)
BoundPreparedSparqlQuery::on_store()snapshots storage rather thanborrowing
&Store:so
execute()yieldsQueryResults<'static>→QuerySolutionIter<'static>.Being
'static, the iterator stores in a wasm-bindgen struct and is pulledacross separate JS calls with no self-referential borrow and no
self_cell/
ouroboros. The snapshot is MVCC-isolated, so an open cursor is unaffectedby a concurrent
load/clear— and pins that snapshot in the WASM-32 heapuntil
free().Verification
Built real wasm (
build_package.py, debug) and ran in Node:Snapshot isolation — deleting all triples mid-iteration (
DELETE WHERE { ?s ?p ?o },store size → 0) still yielded all 5 original rows from the open cursor.
Known gaps before this is mergeable
query()'s full option parsing (default_graph/named_graphs/
base_iri) instead of the trimmed base_iri-only path.QueryTriplestwin for CONSTRUCT/DESCRIBE (QueryResults::Graph).Storeclass isskip_typescript, soquerySolutionsneeds adding by hand alongside the generatedQuerySolutionsclass).js/test/store.test.tscases for batching + exhaustion.stream: trueoption on
query().you can iterate or serialize.
🤖 Generated with Claude Code