Skip to content

impr(store_volatile): collapse to a single ordered_set, LMDB-shaped#887

Open
shyba wants to merge 1 commit intoedgefrom
impr/volatile_ordered_edge
Open

impr(store_volatile): collapse to a single ordered_set, LMDB-shaped#887
shyba wants to merge 1 commit intoedgefrom
impr/volatile_ordered_edge

Conversation

@shyba
Copy link
Copy Markdown

@shyba shyba commented Apr 24, 2026

alternative to #879 -- instead of a helper indexer, change the main table to ordered set

What

Replaces current layout with a single ETS ordered_set, LMDB-shaped: {Path, {raw,V} | {link,T} | group} where list/3 is a prefix range scan with subtree-skip; group membership is no longer materialised.

Why

The bag-indexed design (PR #879) removed the expensive sets:add_element ordset-copy, but ets:bag insert scans the same-key chain for dedup — single-parent-heavy writes were also still slow.

A single ordered_set makes insert O(log N) unconditionally, removes the cross-table consistency complexity entirely, and matches LMDB/FS semantics (group is a marker, children are discovered by range scan). Tradeoff: point reads drop from O(1) hash to O(log N) tree; in practice the suite shows reads holding within noise.

How

  • One ordered_set per store instance, keyed on path; values {raw,V} / {link,T} / group.
  • list/3 is ets:next from <<Path, $/>>, accumulating the first segment of each descendant into a #{} map. After each immediate child it jumps past that subtree via <<Prefix, Name, $0>> (one byte past $/), so deep groups don't pay per-descendant cost.
  • Root listing seeds from ets:first and skips the root marker — handles keys whose first byte sorts before $/ (e.g. base64url's -).
  • write/3 / link/3 collapse to a single ets:insert/2; ancestors are walked once and each gets insert_new of group.
  • make_group/3 is idempotent: leaves an existing group alone, converts a raw/link to group otherwise.
  • Subtree overwrite uses ets:select_delete with a binary_part prefix guard — no recursion.
  • list_path/3 returns {error, not_found} for an empty root so a chained store fallback isn't masked by {ok, []}.

No change to external semantics. read/3, type/3, resolve/3, link-of-link, group-of-links all preserved.

Measured impact

hb_store benchmark suite, scale=0.5 (50 000 records, 1 KiB values):

test pre-index bag-indexed (#879) this PR
key write 2 231 /s 3 711 /s 519 k /s
key read 664k /s 637k /s 554 k /s
list write 9 961 /s 64 955 /s 291 k /s
list ops 523k /s 389k /s 210 k /s
message write 905 /s 1 363 /s 1 779 /s
message read 2 792 /s 4 351 /s 4 558 /s

Single-parent write goes from 1.66× (bag-indexed) to ~233× vs pre-index, eliminating the O(N²) tail. List ops regress from 523k/s to 210k/s without re-introducing O(N²) writes;

Tests

15 cases, all isolated via hb_test_utils:test_store/2:

  • max_ttl_test — periodic reset.
  • Overwrite invariants: overwrite_link_to_raw_test, overwrite_group_to_raw_test, overwrite_group_to_link_test, implicit_group_conversion_test.
  • Listing semantics: list_test, list_dedup_test, list_with_link_test, list_root_test, list_root_includes_pre_slash_keys_test (regression: --prefixed keys), list_deep_subtree_test (subtree-skip), list_large_flat_group_test (10k siblings, dedup performance), prefixed_sibling_no_duplicate_test (interleaved sibling between a child and its subtree), sibling_with_zero_suffix_test ($0 jump-token boundary).
  • empty_root_reports_not_found_test — store-chain fallback regression.

@shyba shyba force-pushed the impr/volatile_ordered_edge branch from 9afa65a to 29fe785 Compare April 28, 2026 22:33
@shyba shyba marked this pull request as ready for review April 29, 2026 19:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant