perf(storage): SCAN hot-plane pages are true O(COUNT) — hash-range segment walk (#368) - #380
Merged
Merged
Conversation
…gment walk (#368) Completes the second half of #368. PR #379 made the SCAN cursor a position in stable hash space (fixing the stable-key guarantee); this change makes each page's hot-plane cost independent of keyspace size. The cursor hash becomes the DashTable's own fixed-seed xxh64 key hash truncated to its top 48 bits. Because the extendible-hashing directory is indexed by the hash's top `global_depth` bits, ascending directory order IS ascending hash order and directory entry d covers exactly the hash range [d << (64-D), (d+1) << (64-D)) — segments are range-partitioned in hash space. The new `DashTable::hash_page` starts at the segment covering the cursor and walks segments in ascending range order, stopping at the first segment boundary once COUNT entries are collected: later segments can only hold larger hashes, so the page is complete without touching the rest of the table. A segment with local_depth < global_depth occupies a contiguous directory run, so alias-dedup is a consecutive store-index comparison. Split/merge/directory-doubling between pages stays safe by construction: the cursor is a position in hash space, and structural churn only changes WHICH segment covers that position, never the set of keys at or above it. Equal-hash groups can never straddle a page (equal top-48 bits => same segment; pages are whole-segment granular), which preserves the PR #379 collision boundary rule unchanged. `Database::scan_hot_page` wraps the walk (maps 64-bit table hashes to the 48-bit cursor space via >> 16, judges liveness from the entry during iteration), and `scan_core` feeds its selection heap from that pruned page instead of a full `iter_live_keys` walk. The pathological all-one-hash second pass now rescans only the already-fetched page. The cold plane keeps its filtered in-RAM index walk (bounded by spilled-key count); ordered cold-side paging remains a follow-up. Red/green coverage: new DashTable unit tests (empty-table terminal page, alive-filter + more-flag, 4000-key drain in (hash, key) order under forced split churn between pages) plus the existing six SCAN semantics tests (churn stability, exact duplicate-free drain, MATCH paging, cold-plane enumeration x2, COUNT-as-hint) all green on the new path. Verified on Linux VM (4 shards, real server): 500/500 keys drained in 12 pages, MATCH key:1* = 111/111; scaling proof — 500 SCAN pages at COUNT 100 take 0.10s on a 10k-key DB and 0.11s on a 1,000,000-key DB (flat per-page cost across a 100x keyspace; the previous full-walk design re-hashed every live key on every page). SCAN cursors from before this change are invalidated (cursor hash function changed); cursors are documented as ephemeral — restart scans at 0. Multi-shard composite packing (upper 16 shard | lower 48 per-shard) is unchanged. Refs #368 author: Tin Dang <tindang.ht97@gmail.com>
Qodo reviews are paused for this user.Troubleshooting steps vary by plan Learn more → On a Teams plan? Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center? |
📝 WalkthroughWalkthroughChangesSCAN hot-plane paging
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant Client
participant scan_core
participant Database
participant DashTable
Client->>scan_core: SCAN cursor and COUNT
scan_core->>Database: scan_hot_page(cursor, count, now_ms)
Database->>DashTable: hash_page(from_h48, want, alive)
DashTable-->>Database: sorted live entries and more
Database-->>scan_core: 48-bit hot candidates
scan_core-->>Client: selected keys and next cursor
Possibly related issues
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
src/command/key.rs (2)
1108-1128: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winTerminate instead of overflowing the 48-bit cursor.
When
h_last == (1 << 48) - 1,h_last + 1no longer fits the cursor slot;from_h48 << 16then wraps to zero and can restart the scan.Proposed fix
- next_cursor = h_last + 1; + const MAX_HASH48: u64 = (1_u64 << 48) - 1; + next_cursor = if h_last == MAX_HASH48 { + 0 + } else { + h_last + 1 + };🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/command/key.rs` around lines 1108 - 1128, Update the cursor advancement in the scan flow around h_last and next_cursor to terminate when h_last reaches the maximum 48-bit value, rather than computing h_last + 1. Preserve the existing extra-member collection and only assign the incremented cursor for values below that boundary, preventing from_h48 conversion from wrapping and restarting the scan.
1015-1169: 📐 Maintainability & Code Quality | 🟠 Major | 🏗️ Heavy liftSplit these oversized Rust modules before adding further SCAN logic.
src/command/key.rs#L1015-L1169: move SCAN parsing, paging, and tests into a directory submodule.src/storage/db.rs#L1443-L1473: split database read/enumeration operations into a focused module.As per coding guidelines, “No single Rust file should exceed 1500 lines,” command-group files should become directory modules, and read/write implementations above 1000 lines should be split.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/command/key.rs` around lines 1015 - 1169, Split the oversized SCAN implementation in src/command/key.rs (lines 1015-1169) into a directory submodule containing SCAN parsing, paging, and tests, preserving the existing scan_hash48 and scan_core behavior; also move the database read/enumeration operations in src/storage/db.rs (lines 1443-1473) into a focused module, updating module declarations and call sites as needed without changing behavior.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/storage/dashtable/mod.rs`:
- Around line 570-580: Bound the segment traversal independently of the alive(k,
v) filter so expired or rejected entries cannot force scanning the entire
keyspace for one page. In the scan path around finish_page and the segment
iteration, stop after examining the requested COUNT of candidate entries,
preserve the last examined hash as the continuation hash, and return it even
when no entries qualify. Add a regression test covering all-rejected and
mostly-expired tables while preserving normal pagination behavior.
---
Outside diff comments:
In `@src/command/key.rs`:
- Around line 1108-1128: Update the cursor advancement in the scan flow around
h_last and next_cursor to terminate when h_last reaches the maximum 48-bit
value, rather than computing h_last + 1. Preserve the existing extra-member
collection and only assign the incremented cursor for values below that
boundary, preventing from_h48 conversion from wrapping and restarting the scan.
- Around line 1015-1169: Split the oversized SCAN implementation in
src/command/key.rs (lines 1015-1169) into a directory submodule containing SCAN
parsing, paging, and tests, preserving the existing scan_hash48 and scan_core
behavior; also move the database read/enumeration operations in
src/storage/db.rs (lines 1443-1473) into a focused module, updating module
declarations and call sites as needed without changing behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 76464815-5a49-4ebf-9e34-ed619b527756
📒 Files selected for processing (4)
CHANGELOG.mdsrc/command/key.rssrc/storage/dashtable/mod.rssrc/storage/db.rs
Comment on lines
+570
to
+580
| if out.len() >= want { | ||
| // Enough collected and at least one unvisited segment | ||
| // remains; everything in it hashes above what we have. | ||
| return (Self::finish_page(out), true); | ||
| } | ||
| let seg = self.segments.get(store_idx); | ||
| for (k, v) in seg.iter_occupied() { | ||
| let h = hash_key(k.as_ref()); | ||
| if h >= from_hash && alive(k, v) { | ||
| out.push((h, k.clone())); | ||
| } |
There was a problem hiding this comment.
🚀 Performance & Scalability | 🟠 Major | 🏗️ Heavy lift
Preserve O(COUNT) when alive rejects entries.
The stop condition counts only qualifying entries. An expired-heavy table can therefore visit every segment before producing a small page, leaving SCAN page cost O(keyspace). Bound traversal independently of alive, return a continuation hash, and add an all-rejected/mostly-expired regression test.
Also applies to: 727-750
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/storage/dashtable/mod.rs` around lines 570 - 580, Bound the segment
traversal independently of the alive(k, v) filter so expired or rejected entries
cannot force scanning the entire keyspace for one page. In the scan path around
finish_page and the segment iteration, stop after examining the requested COUNT
of candidate entries, preserve the last examined hash as the continuation hash,
and return it even when no entries qualify. Add a regression test covering
all-rejected and mostly-expired tables while preserving normal pagination
behavior.
…th invariant Adversarial-review follow-ups on the O(COUNT) page walk: - SCAN/SCAN-readonly now mask the parsed cursor to the low 48 bits. Legitimate resumed cursors always fit (multi-shard composites are unpacked by coordinate_scan before dispatch), but at --shards 1 a client-supplied out-of-range cursor (e.g. a composite cursor replayed from a multi-shard server, or garbage) previously filtered out every key — h48 is always < 2^48 — and returned an empty page with cursor 0: a false "scan complete" on a non-empty keyspace. Pre-existing behavior (same comparison since PR #379), closed here because this PR owns the cursor-semantics surface. Red/green: scan_out_of_range_cursor_clamps_to_hash_space. - debug_assert in Database::scan_hot_page that the DashTable directory depth is <= 48 (new DashTable::directory_depth accessor). The h64→h48 bridge and the "equal-h48 group never straddles a page" guarantee are load-bearing on this; depth > 48 needs a 2^48-entry directory — unreachable in practice, but now asserted rather than assumed. author: Tin Dang <tindang.ht97@gmail.com>
Contributor
Author
|
Adversarial review (independent agent, 7 attack vectors): no defects confirmed — range-partition bit arithmetic verified by hand-trace incl. multi-level directory doubling; |
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
src/storage/dashtable/mod.rs (1)
542-602: 🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy liftEnforce
wantas a hard page-size limit.The check at Line [581] runs only when entering a new segment, while Lines [587-591] append every qualifying entry from the current segment. A page with
want = 1can therefore return an entire segment’s contents, violating the documented “up towant” contract and inflating SCAN page work. Add a regression test with a smallwant, then truncate or otherwise page segment candidates while preserving correct continuation semantics.Also applies to: 738-761
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/storage/dashtable/mod.rs` around lines 542 - 602, Enforce want as a hard maximum in hash_page by limiting qualifying entries appended from each segment, rather than checking only between segments. Preserve continuation semantics: return true whenever collection stops with unvisited candidates, and ensure subsequent calls can resume without skipping entries; add a regression test using a small want that verifies page length and continuation behavior.
♻️ Duplicate comments (1)
src/storage/dashtable/mod.rs (1)
581-591: 🚀 Performance & Scalability | 🟠 MajorPreviously reported alive-filter traversal issue remains unresolved.
When
aliverejects most or all entries,out.len()never reacheswant, so this method scans every remaining segment before returning. That keeps expired-heavy SCAN pages proportional to total keyspace rather than page size. Bound traversal independently of qualifying results and preserve continuation progress for rejected entries.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/storage/dashtable/mod.rs` around lines 581 - 591, Update the scan traversal around the out.len() limit and alive filtering so it bounds work by visited entries or segments rather than requiring want qualifying results, while still advancing continuation state past rejected entries. Preserve returning qualifying entries and the existing finish_page behavior, ensuring expired-heavy scans do not traverse the entire remaining keyspace.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@src/storage/dashtable/mod.rs`:
- Around line 542-602: Enforce want as a hard maximum in hash_page by limiting
qualifying entries appended from each segment, rather than checking only between
segments. Preserve continuation semantics: return true whenever collection stops
with unvisited candidates, and ensure subsequent calls can resume without
skipping entries; add a regression test using a small want that verifies page
length and continuation behavior.
---
Duplicate comments:
In `@src/storage/dashtable/mod.rs`:
- Around line 581-591: Update the scan traversal around the out.len() limit and
alive filtering so it bounds work by visited entries or segments rather than
requiring want qualifying results, while still advancing continuation state past
rejected entries. Preserve returning qualifying entries and the existing
finish_page behavior, ensuring expired-heavy scans do not traverse the entire
remaining keyspace.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 5b58a02b-96f4-40cd-87cd-c5c30cd8346b
📒 Files selected for processing (3)
src/command/key.rssrc/storage/dashtable/mod.rssrc/storage/db.rs
🚧 Files skipped from review as they are similar to previous changes (2)
- src/storage/db.rs
- src/command/key.rs
pilotspacex-byte
added a commit
that referenced
this pull request
Jul 17, 2026
…#368) (#382) * perf(storage): SCAN cold-plane range resume — hash-ordered cold index (#368) Completes the two-plane O(COUNT) SCAN page walk. PR #380 made the hot plane a hash-range segment walk; this change gives the cold plane (spilled keys under disk-offload) the same property. ColdIndex's primary map becomes a BTreeMap keyed by (scan_hash48(key), key) — the identical 48-bit cursor hash the hot plane uses (xxh64 top 48 bits). The new `ColdIndex::range_from` + `Database::cold_only_keys_from` seek to the cursor in O(log n) and yield ascending live cold-only candidates; scan_core takes the first COUNT (ascending order makes them exactly the smallest cold candidates) instead of filtering every spilled key on every page. The pathological all-one-hash branch range-scans just the equal-hash group. The cold-only liveness predicate (TTL from the cached ColdLocation + hot-shadow check) is extracted into one shared helper so the full-walk (KEYS/RANDOMKEY) and range-resume paths cannot diverge on the two-plane partition invariant (#364). Trade-offs, deliberate: - lookup/remove pay an O(log n) tree descent instead of an O(1) hash probe via an equal-hash range probe. These sit on disk-read-through, promotion, DEL-of-cold-key, and sweep paths where the descent is noise next to the I/O they front; no command hot path touches them. - The ordered map REPLACES the hash map rather than sidecar-ing it: no duplicate per-entry Bytes, per-entry RAM stays comparable at the G2 tens-of-millions-of-entries scale, and there is no dual-structure drift risk — every mutation still funnels through ColdIndex's own methods (map stays private). - BTreeMap cannot borrow-match a (u64, &[u8]) probe against a (u64, Bytes) key, so remove_raw finds the owned key via the equal-hash range (Bytes clone = refcount bump, no data copy). Public ColdIndex API, recovery rebuild, sweep semantics, and the file_refs reclamation spine are unchanged. sweep_expired's batch-cap comment updated: iteration is now deterministic (hash, key) order; capped-out entries are still never permanently skipped because reclaimed entries stop occupying batch slots. Red/green coverage: range_from hash-order/resume/exactly-once (200 entries, paged), equal-hash-probe lookup/remove independence, SCAN paged drain across planes exactly-once (50 hot + 40 cold + 10 both-planes at COUNT 7), cold spill churn between pages keeps stable keys. All existing cold_enumeration, cold_index, and SCAN semantics tests green on the new path. Refs #368 author: Tin Dang <tindang.ht97@gmail.com> * fix(command): SCAN top-of-hash-space cursor terminates instead of restarting Review follow-up on #382: the pathological all-one-hash branch advanced the cursor to h_last + 1; at the exact 48-bit maximum that value (2^48) is masked back to 0 by the cursor clamp on the next call — an infinite scan restart. Nothing can hash above the max, so the scan is complete: return cursor 0. Not practically constructible (needs a full page of keys at the exact max hash); documented in-line, no test. author: Tin Dang <tindang.ht97@gmail.com> --------- Co-authored-by: Tin Dang <tindang.ht97@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Completes the second half of #368. PR #379 made the SCAN cursor a position in stable hash space (stable-key guarantee); this PR makes each page's hot-plane cost independent of keyspace size.
DashTable::hash_page: starts at the segment covering the cursor, walks segments in ascending range order, stops at the first segment boundary once COUNT entries are collected. Alias-dedup is a consecutive store-index comparison (alocal_depth < global_depthsegment occupies a contiguous directory run).Database::scan_hot_pagewraps the walk;scan_corefeeds its selection heap from the pruned page instead of a fulliter_live_keyswalk. Cold plane keeps its filtered in-RAM index walk (follow-up in SCAN cursor is a positional index over a per-call re-sort — no stable-key guarantee, and O(hot+cold) re-sort per page #368).Test plan
(hash, key)order under forced split churn between pages-D warningson default +runtime-tokio,jemallocMATCH key:1*= 111/111Refs #368
Summary by CodeRabbit
Performance
(hash, key)ordering, even during concurrent inserts.Documentation
0.