Stop buffering doc-values updates in heap during merges - #16570
Open
jimczi wants to merge 6 commits into
Open
Conversation
Stage 1 (no behavior change): validate that the carry-over accumulated in heap by ReadersAndUpdates.mergingDVUpdates for the whole merge duration is redundant with the source segments on-disk state plus residual pending updates. An assertion in commitMergedDeletesAndUpdates recomputes the carry-over from disk and checks it matches. Adds a deterministic test that resolves numeric and binary updates while a merge is paused mid-flight.
Stage 2: add buildMappedDVUpdatesFromDisk, which reconstructs the mapped carry-over (field -> delGen -> updates) purely from each source segment on-disk state plus residual pending updates, with no per-merge heap retention. The flushed bulk is read back via a current-vs-baseline diff and collapsed into one packet at minResidualDelGen-1 (or completedDelGen); residual updates keep their real delGens. Since flushing is a monotonic prefix by completedDelGen, that slot is always below all residual/still-running gens and above minGen, preserving newest-wins order. The assertion now checks this reconstruction is effect- equivalent to the mergingDVUpdates path; production still consumes mergingDVUpdates.
Stage 3: commitMergedDeletesAndUpdates now consumes buildMappedDVUpdatesFromDisk instead of the heap-retained mergingDVUpdates. The source loop keeps only the hard-delete carry-over and minGen; the merging path is retained solely as the assertion oracle (buildMappedDVUpdatesFromMerging). The cross-check compares the resulting merged value per doc with a baseline fallback, tolerating no-op updates and resets that mergingDVUpdates carries but the disk reconstruction correctly omits.
Delete ReadersAndUpdates.mergingDVUpdates and all of its plumbing (the addDVUpdate copy-branch, isMerging, setIsMerging, getMergingDVUpdates, dropMergingUpdates, and the getReaderForMerge pending-copy). The doc-values- update merge carry-over is reconstructed at commit from the source segments on-disk state plus their residual pending updates, so no resolved update packets are retained in heap for the whole merge. This removes memory that grew with update volume times merge duration and was unaccounted by IndexWriter#ramBytesUsed. The residual updates are typed from the packet itself rather than the source reader, so an update to a field not yet written to that segment (e.g. a soft- deletes field on its first update) is still carried over. Adds TestMergeCarryOverFromDisk covering numeric, binary, a >maxOverlays fold, and a soft-delete resolved while a segment is merged. Drops the obsolete isMerging assertions from TestReaderPool.
jimczi
changed the title
jimczi
changed the title
ChrisHegarty
approved these changes
Aug 28, 2026
ChrisHegarty
left a comment
ChrisHegarty
left a comment
Contributor
There was a problem hiding this comment.
Left a few minor comments, but otherwise LGTM.
jimczi
force-pushed
the
remove-merging-dv-updates
branch
from
August 28, 2026 14:15
9133a49 to
036c8b4
Compare
jimczi
force-pushed
the
remove-merging-dv-updates
branch
from
August 28, 2026 14:21
036c8b4 to
73a8d47
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
While a segment is being merged, doc-values updates that resolve on it are kept in an in-memory buffer (
ReadersAndUpdates.mergingDVUpdates) so they can be carried over to the merged segment. Unlike deletes, which buffer only doc ids, numeric and binary doc-values updates buffer their actual values (the longs and byte strings). This buffer therefore grows with both the number of updates and their size, is retained for the entire duration of a merge, and is not counted byIndexWriter#ramBytesUsed.This removes that buffer. The carry-over is instead reconstructed at merge commit from what the source segments already persist: their on-disk doc-values, diffed against the merge-reader baseline, plus the still-pending updates that haven't been written yet.
The trade-off is some extra doc-values reads when a merge commits, but these are bounded by the merge itself. Behavior is unchanged. Builds on the incremental doc-values updates in #16418.