perf(btreemap): add insert_many for batched insertion#440
Open
hpeebles wants to merge 5 commits into
Open
Conversation
`insert` writes the target node to stable memory on every call, and when
a node splits it writes the new sibling and the parent too. Inserting a
run of keys therefore rewrites the same nodes repeatedly: building a
10k-entry map one key at a time performs 15,976 node writes across just
1,997 distinct nodes.
`insert_many` keeps the current root-to-leaf path in memory and writes a
node out only once the input has moved past it, bringing those same 10k
keys down to 4,032 writes. The header is likewise written once per call
rather than once per key.
map.insert_many((0..1_000_000).map(|i| (i, compute(i))));
The iterator is consumed lazily and never collected, so the memory cost
is one tree path however many pairs are supplied.
## Ordering
Coalescing pays off when consecutive keys land in the same node, so keys
should be supplied in sorted order. Direction is irrelevant: a
descending run unwinds the path leftwards instead of rightwards and
produces an identical write count.
Out-of-order keys stay correct rather than corrupting the tree. Each
level of the held path records the key range its node owns at both ends,
so a key arriving early unwinds to an ancestor that covers it, as far as
the root if necessary. Tracking the lower bound as well as the upper one
costs a comparison per key.
## Measurements
canbench, 10k u64 pairs each:
ascending, empty map 334.19M -> 76.91M instructions (4.3x)
descending, empty map 386.65M -> 96.43M instructions (4.0x)
unordered, caller sorts 356.43M -> 86.45M instructions (4.1x)
unordered, fed as-is 356.43M -> 341.07M instructions (1.0x)
ascending run into 100k 294.07M -> 49.06M instructions (6.0x)
The sort in the third row is inside the measured section. Unordered
input fed straight in forfeits the coalescing, but is not slower than
the insert loop it replaces.
## Implementation notes
Holding the whole path rather than only the leaf is what makes this
work. Two thirds of the writes in a split-heavy workload are the sibling
and the parent, and a held parent absorbs many child splits before it is
written; holding only the leaf coalesces 2x rather than 4x.
Node splits are handled in place while the path is held, except in two
cases that fall back to the ordinary insert path: a full root leaf, and
a parent with no room for a promoted median. Those fire 277 times per
10k ascending keys, about one insert in 36. A fallback flushes the path,
so nodes it had buffered are written again when the batch re-descends
into them; that accounts for the whole gap between the 4,032 writes
above and the 1,997 that writing each dirty node exactly once would
take. Handling the split cascade in place would close it, at the cost of
considerably more intricate code in the riskiest part of the tree, so it
is left as a possible follow-up.
The batch machinery lives in `src/btreemap/bulk_insert.rs`, leaving
`BTreeMap::insert_many` in `btreemap.rs` as a thin wrapper over it.
Dropping `BulkInsert` writes back whatever it still holds, which is how
a batch is completed: the drop is load-bearing on every call.
`insert` is unchanged; its body moves into `insert_serialized` so the
fallback can reuse it with an already-serialized value. All 252
pre-existing benchmarks are unchanged to within noise.
## Testing
Differential proptests and unit tests build the same map with repeated
`insert` and with `insert_many`, and require the results to be
indistinguishable: across node cache sizes 0/1/16, the V1,
V1-migrated-to-V2 and V2 layouts, unbounded values large enough to spill
onto overflow pages, duplicate and overwriting batches, and batch sizes
chosen to hit both fallbacks. Ordering is covered by ascending,
descending, zigzag, sawtooth and shuffled inputs. Allocator chunk counts
return to zero after draining a map built by `insert_many`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
|
|
|
|
|
A leaf split promotes a median into its parent, so a full parent used to end the batch: the path was flushed and the ordinary insert path grew the tree instead. Every node the path had buffered was then written again on the way back down, which cost roughly half of the writes `insert_many` was saving. The split now cascades. `split_leaf` walks up to the topmost full level, grows a new root if the whole path is full, then splits top-down: each split leaves its node half empty, so the level below always has room for the median it promotes. Writing each dirty node exactly once is now reached rather than approached — 10k ascending keys perform 1,997 writes, matching the number of distinct nodes in the resulting tree, down from 4,032. An `insert` loop over the same keys performs 15,976. ascending, empty map 77.17M -> 35.88M instructions (4.3x -> 9.3x) unordered, caller sorts 86.71M -> 45.42M instructions (4.1x -> 7.8x) descending, empty map 96.58M -> 52.53M instructions (4.0x -> 7.4x) Nothing else moves: 262 benchmarks, 0 regressed, 3 improved. The workloads that barely split — inserting into gaps in an existing map, pure overwrites, unordered input — are unchanged, which is where the saving was expected to come from. The levels below a split keep the bounds they already hold: the separators either side of them survive, landing in one half or the other. Only their index among the parent's children moves, by the number of children that stayed with the left half. Removing the fallback also removes the last call from `BulkInsert` back into `BTreeMap::insert`. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
insertwrites the target node to stable memory on every call, and when a node splits it writes the new sibling and the parent too. Inserting a run of keys therefore rewrites the same nodes repeatedly: building a 10k-entry map one key at a time performs 15,976 node writes across just 1,997 distinct nodes.insert_manykeeps the current root-to-leaf path in memory and saves a modified node to stable memory only once the input has moved past it. The same 10k keys come down to 1,997 writes — exactly the number of distinct nodes, so every node is written once and none is rewritten. The header is likewise written once per call rather than once per key.The iterator is consumed lazily and never collected, so the memory cost is one tree path however many pairs are supplied.
BTreeSet::insert_manydelegates to it.Ordering
Coalescing pays off when consecutive keys land in the same node, so keys should be supplied in sorted order. Direction is irrelevant: a descending run unwinds the path leftwards instead of rightwards and produces an identical write count.
Out-of-order keys stay correct rather than corrupting the tree. Each level of the held path records the key range its node owns at both ends, so a key arriving early unwinds to an ancestor that covers it, as far as the root if necessary. Tracking the lower bound as well as the upper one costs a comparison per key.
Those recorded ranges are also what lets the fast path skip re-searching the ancestors it is still holding. A key stored in an ancestor is one of that ancestor's separators, and the separators are exactly the exclusive bounds handed down to the children, so a key that any held node covers cannot be sitting in a node above it.
Measurements
canbench. The first five rows insert 10k
u64pairs; the last overwrites 2k existing keys whose 1 KiB values live on V2 overflow pages.insertloopinsert_manyThe sort in the third row is inside the measured section. Unordered input fed straight in forfeits the coalescing, but is not slower than the insert loop it replaces.
The overwrite row gains from a second effect.
inserthas to read the displaced value back in order to return it, which for a value on overflow pages means reading it out of stable memory;insert_manyreturns nothing, so the newNode::set_valuereplaces the value without loading the old one. Measured separately, coalescing alone accounts for 74.29M -> 14.64M and skipping the discarded reads takes it the rest of the way to 10.08M.Implementation notes
Holding the whole path rather than only the leaf is what makes this work. Two thirds of the writes in a split-heavy workload are the sibling and the parent, and a held parent absorbs many child splits before it is written; holding only the leaf coalesces 2x rather than 9x.
Splits are handled entirely in place. A leaf split promotes a median into its parent, which may itself be full, so the split cascades:
split_leafwalks up to the topmost full level, grows a new root if the whole path is full, then splits top-down — each split leaves its node half empty, so the level below always has room for the median it promotes. The levels below a split keep the bounds they already hold, since the separators either side of them survive and simply land in one half or the other; only their index among the parent's children moves, by the number of children that stayed with the left half.The batch machinery lives in
src/btreemap/bulk_insert.rs, leavingBTreeMap::insert_manyinbtreemap.rsas a thin wrapper over it. DroppingBulkInsertwrites back whatever it still holds, which is how a batch is completed: the drop is load-bearing on every call.insertis untouched — the diff tobtreemap.rsis purely additive.BulkInsertcarries the value unserialized and encodes it at the point of use. The new benchmarks name their plain-loop baselinesinsert_loop_*so they read as the counterparts ofinsert_many_*. All 252 pre-existing benchmarks are unchanged to within noise.Public API added
BTreeMap::insert_manyBTreeSet::insert_manyTesting
Differential proptests and unit tests build the same map with repeated
insertand withinsert_many, and require the results to be indistinguishable: across node cache sizes 0/1/16, the V1, V1-migrated-to-V2 and V2 layouts, unbounded values large enough to spill onto overflow pages, duplicate and overwriting batches, and batch sizes chosen to hit the cascade at the node capacity boundaries. Ordering is covered by ascending, descending, zigzag, sawtooth and shuffled inputs. Batches large enough to grow the root repeatedly are drained back to empty afterwards, so a mislinked child or a stale index would show up as chunks the allocator never gets back.🤖 Generated with Claude Code