impr(store_volatile): collapse to a single ordered_set, LMDB-shaped#887
Open
impr(store_volatile): collapse to a single ordered_set, LMDB-shaped#887
Conversation
9afa65a to
29fe785
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
alternative to #879 -- instead of a helper indexer, change the main table to ordered set
What
Replaces current layout with a single ETS
ordered_set, LMDB-shaped:{Path, {raw,V} | {link,T} | group}wherelist/3is a prefix range scan with subtree-skip; group membership is no longer materialised.Why
The bag-indexed design (PR #879) removed the expensive
sets:add_elementordset-copy, butets:baginsert scans the same-key chain for dedup — single-parent-heavy writes were also still slow.A single
ordered_setmakes insert O(log N) unconditionally, removes the cross-table consistency complexity entirely, and matches LMDB/FS semantics (group is a marker, children are discovered by range scan). Tradeoff: point reads drop from O(1) hash to O(log N) tree; in practice the suite shows reads holding within noise.How
ordered_setper store instance, keyed on path; values{raw,V}/{link,T}/group.list/3isets:nextfrom<<Path, $/>>, accumulating the first segment of each descendant into a#{}map. After each immediate child it jumps past that subtree via<<Prefix, Name, $0>>(one byte past$/), so deep groups don't pay per-descendant cost.ets:firstand skips the root marker — handles keys whose first byte sorts before$/(e.g. base64url's-).write/3/link/3collapse to a singleets:insert/2; ancestors are walked once and each getsinsert_newofgroup.make_group/3is idempotent: leaves an existinggroupalone, converts a raw/link togroupotherwise.ets:select_deletewith abinary_partprefix guard — no recursion.list_path/3returns{error, not_found}for an empty root so a chained store fallback isn't masked by{ok, []}.No change to external semantics.
read/3,type/3,resolve/3, link-of-link, group-of-links all preserved.Measured impact
hb_storebenchmark suite, scale=0.5 (50 000 records, 1 KiB values):Single-parent write goes from 1.66× (bag-indexed) to ~233× vs pre-index, eliminating the O(N²) tail. List ops regress from 523k/s to 210k/s without re-introducing O(N²) writes;
Tests
15 cases, all isolated via
hb_test_utils:test_store/2:max_ttl_test— periodic reset.overwrite_link_to_raw_test,overwrite_group_to_raw_test,overwrite_group_to_link_test,implicit_group_conversion_test.list_test,list_dedup_test,list_with_link_test,list_root_test,list_root_includes_pre_slash_keys_test(regression:--prefixed keys),list_deep_subtree_test(subtree-skip),list_large_flat_group_test(10k siblings, dedup performance),prefixed_sibling_no_duplicate_test(interleaved sibling between a child and its subtree),sibling_with_zero_suffix_test($0jump-token boundary).empty_root_reports_not_found_test— store-chain fallback regression.