Skip to content

Fix traversal cycles, both() filter, statement reuse; WAL default; unmask CI - #19

Merged
michaeloboyle merged 8 commits into
mainfrom
feature/10-traversal-correctness-perf
Nov 3, 2026
Merged

michaeloboyle merged 8 commits into
mainfrom
feature/10-traversal-correctness-perf

Conversation

@michaeloboyle

@michaeloboyle michaeloboyle commented Nov 2, 2026 •

Copy link
Copy Markdown
Owner

Summary

Fixes the defects and the CI mask found by a scale benchmark (1k to 100k nodes) and a Neo4j comparison.

Evidence

Before (main a7f9ea7) After
Full suite 2 suites crash (heap OOM / timeout), CI green via || true 424 passed of 424, ~3s
New regression specs, red first 2-cycle hang; both() returned 2 extra nodes; 199 prepares; journal delete 10 of 10 pass
traverse depth 3 at 1k nodes (median) 2.67ms (raw SQLite + hydrate: 0.83ms) 0.415ms (raw: 0.384ms)
autocommit createNode, median of 5 1,491/s internal SSD, 1,202/s external 4,405/s internal SSD, 21,277/s external

No existing test file was changed. The new specs are in tests/unit/TraversalQuery-regressions.test.ts.

#18 ruling

TraversalQuery › traversal from a non-existent node contradicted Database.test.ts:746. The owner ruled on 2026-11-02 to keep the error, so that spec now expects Start node with ID … not found (f4116be). This is the only existing test changed in this PR, and it changed by owner ruling.

Behaviour changes to review

  1. traverse(x).out('E').toArray() with no maxDepth() now goes one hop per step, or down to minDepth() if that is deeper. Before, it walked without bound: on DAGs it returned every reachable node, and on cycles it never returned. On a 5-node chain, main returned c1 to c4 and this branch returns c1; minDepth(2) returned c2 to c4 and now returns c2. Callers that relied on deep walks must add .maxDepth(n). Documented in docs/API.md › toArray().
  2. toPaths() keeps its old unbounded default (it terminates thanks to its cycle detection) and ignores minDepth, so toPaths() and toArray() now default differently.
  3. Without unique(), results grow exponentially with depth on dense graphs (complete 12-node graph, maxDepth(5): 64,471 results in 121ms). That is what "once per distinct path" means; unique() is the reachability mode.
  4. minDepth(n) greater than maxDepth(m) returns [] without an error.
  5. Without unique(), the start node is no longer returned when a cycle leads back to it.
  6. Opening an existing file database converts it to WAL permanently. -wal/-shm side files appear, and a later open with { wal: false } still reports wal. WAL is unsafe on network filesystems; use { wal: false } there.

Pre-PR verification

Adversarial verification returned PARTIAL twice. The first pass found one regression, minDepth(n) without maxDepth returning [], which broke the documented example. That is fixed in 37d57c5, test first. Re-verification of 37d57c5 confirmed the fix: on the chain, minDepth(2) returned c2..c4 on main, [] before the fix and c2 now, and 8 edge cases were checked.

Also confirmed:

  • every new spec failed on main first
  • the 2 remaining failures reproduce on main
  • no existing test file changed
  • shortestPath/allPaths/toPaths results match main
  • the statement cache handles closed and reopened connections correctly
  • build and lint pass (0 errors)

PARTIAL rather than PASS because the local second-opinion judge model was unstable (0.43 to 0.86 on the same evidence, then a timeout). That is noise in the checker, recorded here rather than counted as a pass. The README benchmark figures were not re-run by the verifier.

Not in this PR

Closes #10
Closes #11
Closes #12
Closes #13
Closes #14
Closes #15
Closes #17
Closes #18

🤖 Generated with Claude Code

https://claude.ai/code/session_01UkPPxXPqpZDv12PSugDzm7

michaeloboyle and others added 6 commits November 2, 2026 14:59
…ement reuse, WAL default

Refs #10 #11 #12 #13

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UkPPxXPqpZDv12PSugDzm7
…atements

- toArray(): without unique(), never revisit a node on the current path;
  without maxDepth(), follow each defined step once (integration specs);
  the start node is never returned or filtered
- both(edge, nodeType): apply the type filter to both UNION branches
- cache prepared statements per connection; head index instead of
  queue.shift(); unique() marks on enqueue; toPaths uses a parent chain

Closes #10
Closes #11
Closes #12

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UkPPxXPqpZDv12PSugDzm7
Opt out with { wal: false }. In-memory and readonly databases unchanged.

Closes #13

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UkPPxXPqpZDv12PSugDzm7
Closes #15

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UkPPxXPqpZDv12PSugDzm7
Closes #14

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UkPPxXPqpZDv12PSugDzm7
Without this, minDepth(2) with one step returned [] and broke the documented
example. Found by pre-PR verification.

Refs #10

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UkPPxXPqpZDv12PSugDzm7
Copilot AI balanced review requested due to automatic review settings November 2, 2026 20:41

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Copilot was unable to complete this review at the selected tier due to model policy or plan restrictions.

🟡 Changes recommended

A moderate, active issue remains because TraversalQuery.ts has inconsistent filter semantics between toPaths() and toArray(), potentially blocking traversal when the start node fails the predicate.

1 open issue
What changed in this PR

This pull request fixes traversal correctness issues (cycle termination, both() filtering), improves traversal performance via statement reuse and queue/path optimizations, switches file-backed SQLite DBs to WAL-by-default (opt-out supported), and updates CI/docs/benchmarks so failures are no longer masked.

Changes:

  • Fix traversal termination on cycles; adjust default toArray() traversal behavior when maxDepth() isn’t provided; improve traversal efficiency (statement caching, O(1) dequeue, path parent-chain).
  • Fix both(edgeType, nodeType) so nodeType filtering applies across both UNION branches.
  • Default file DBs to journal_mode=WAL with synchronous=NORMAL (opt-out via { wal: false }); add regression tests + scale benchmark; unmask CI failures.

Key review feedback to address:

  • toPaths() applies .filter(...) to the start node while toArray() skips it, creating inconsistent filter semantics across traversal APIs. (moderate)
  • Regression test monkey-patches raw.prepare without try/finally, risking leaked state across tests on assertion failure. (moderate)
  • Minor nits: README hardcoded Jest counts may go stale; consider avoiding exact pass/fail totals. Also consider not retaining prev back-references in unique() mode to reduce memory retention. (nits)
File Summary
tests/​unit/​TraversalQuery-regressions.test.ts Adds regression coverage for cycle termination, both() nodeType filtering, statement reuse, and WAL defaults.
src/​types/​index.ts Exposes DatabaseOptions.wal?: boolean to support WAL opt-out.
src/​query/​TraversalQuery.ts Implements cycle-safe traversal semantics, fixes both() filtering, improves traversal performance, and caches prepared statements.
src/​core/​Database.ts Defaults file-backed DBs to WAL + synchronous=NORMAL (unless opted out / readonly / in-memory).
README.md Updates test/benchmark documentation and reported scale results.
docs/​API.md Documents updated toArray() depth/duplicates behavior and cycle termination expectations.
benchmarks/​scale-benchmark.js Adds a reproducible 1k–100k scale benchmark script.
.github/​workflows/​ci.yml Removes `

This pull request requires changes.

Review effort: Balanced

Comment on lines 339 to 342
// Apply filter
if (this.filterPredicate && !this.filterPredicate(node)) {
continue;
}
michaeloboyle and others added 2 commits November 2, 2026 16:47
transaction() issued a bare BEGIN, so mergeNode()/mergeEdge() (which use
transaction() internally) failed inside a caller's transaction. A nested
call now opens a SAVEPOINT; commit releases it, rollback undoes only the
inner work and the outer transaction continues.

Closes #17

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UkPPxXPqpZDv12PSugDzm7
Two specs contradicted each other. Owner ruled to keep the existing
behaviour (throw, Database.test.ts:746), so the TraversalQuery spec that
expected [] now expects the same error.

Closes #18

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UkPPxXPqpZDv12PSugDzm7
@michaeloboyle
michaeloboyle merged commit 942835e into main Nov 3, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants