Skip to content

Spike: lazy streaming SELECT cursor for JS bindings#1

Draft
rybesh wants to merge 1 commit into
mainfrom
spike/lazy-cursor
Draft

Spike: lazy streaming SELECT cursor for JS bindings#1
rybesh wants to merge 1 commit into
mainfrom
spike/lazy-cursor

Conversation

@rybesh

@rybesh rybesh commented Jun 11, 2026

Copy link
Copy Markdown
Member

Spike — not for merge as-is

De-risks a lazy streaming SELECT cursor for the WASM/JS bindings, so the
worker side of graph-store can pull SELECT results in bounded batches instead
of materializing the entire result set into one Array<Map> (or one
serialized string). This is the WASM-binding-side enabler for the output
memory ceiling tracked in dkglab/graph-store#50.

What it adds (js/src/store.rs, +79/-1)

// Store
querySolutions(query: string, options?): QuerySolutions   // SELECT only (spike)

// QuerySolutions cursor
readonly variables: string[]            // available up front, before iterating
nextBatch(count: number): Array<Map>    // empty Array == exhausted
free() / [Symbol.dispose]()             // wasm-bindgen generated; releases the snapshot

Instead of the eager for solution in solutions { array.push(...) } drain,
the iterator is handed to a #[wasm_bindgen] cursor struct and pulled
count rows at a time.

Why it works (the bit worth reviewing)

BoundPreparedSparqlQuery::on_store() snapshots storage rather than
borrowing &Store:

pub fn on_store(self, store: &Store) -> BoundPreparedSparqlQuery<'static> {
    let reader = store.storage().snapshot();   // owns a snapshot
    ...
}

so execute() yields QueryResults<'static>QuerySolutionIter<'static>.
Being 'static, the iterator stores in a wasm-bindgen struct and is pulled
across separate JS calls with no self-referential borrow and no self_cell
/ ouroboros. The snapshot is MVCC-isolated, so an open cursor is unaffected
by a concurrent load/clear — and pins that snapshot in the WASM-32 heap
until free().

Verification

Built real wasm (build_package.py, debug) and ran in Node:

variables: [ 's', 'p', 'o' ]
batch 1: 3 rows … batch 2: 3 rows … batch 3: 1 rows   # 7 rows, lazy, resumable
DONE: 7 rows over 3 batches
post-exhaustion nextBatch length: 0                    # stable exhaustion
CONSTRUCT rejected as expected

Snapshot isolation — deleting all triples mid-iteration (DELETE WHERE { ?s ?p ?o },
store size → 0) still yielded all 5 original rows from the open cursor.

Known gaps before this is mergeable

  • Fold into query()'s full option parsing (default_graph / named_graphs
    / base_iri) instead of the trimmed base_iri-only path.
  • QueryTriples twin for CONSTRUCT/DESCRIBE (QueryResults::Graph).
  • Hand-written TS typings (the Store class is skip_typescript, so
    querySolutions needs adding by hand alongside the generated
    QuerySolutions class).
  • js/test/store.test.ts cases for batching + exhaustion.
  • Decide opt-in shape: a distinct method (as here) vs. a stream: true
    option on query().
  • Possible upstream: Turn query result to application/sparqlresult+json oxigraph/oxigraph#627 already wants a results object
    you can iterate or serialize.

🤖 Generated with Claude Code

Add Store.querySolutions(query, options) returning a QuerySolutions cursor
that holds the QuerySolutionIter alive instead of draining it into an Array,
plus QuerySolutions.nextBatch(count) / .variables. Lets the WASM bindings
stream SELECT results in bounded batches rather than materializing the whole
result set (the dual of the output-side memory ceiling tracked downstream in
dkglab/graph-store#50).

Key enabler: on_store() snapshots storage rather than borrowing &Store, so
QueryResults/QuerySolutionIter are 'static and can live in a wasm-bindgen
struct across JS calls with no self-referential borrow. The snapshot is
MVCC-isolated, so an open cursor is unaffected by a concurrent load/clear
(and pins that snapshot until closed).

Spike only: minimal option parsing (base_iri), SELECT-only, no TS typings,
no reuse of query()'s option handling. Verified by building real wasm and
streaming a SELECT in batches in Node (lazy batching, stable exhaustion,
snapshot isolation across a mid-iteration DELETE).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant