Backport incremental doc-values updates to 10.x (#16418) - #16554
Open
jimczi wants to merge 3 commits into
Open
Conversation
|
|
||
| static final int VERSION_CURRENT = VERSION_86; | ||
| /** The version that records per-field incremental doc-values overlay generations. */ | ||
| public static final int VERSION_10_6 = 11; |
Contributor
There was a problem hiding this comment.
No issue here, but for the main branch, maybe we can rename the constant field, from VERSION_11 to VERSION_10_6 - after this PR is merged?
Contributor
There was a problem hiding this comment.
I came here to say exactly that, +1
When you update a doc-values field today, Lucene rewrites the whole column for that field, even if you only touched a few docs. So a tiny change writes a lot, and it gets worse as the segment grows. The idea is simple: when an update only sets values (no unset), write just the changed docs as a sparse "delta" generation, and stack the deltas on top of the base column at read time, newest wins. Updating a field becomes `O(changed docs)` instead of `O(column)`. Deltas would pile up, so there's a small lifecycle per field: - every flush writes a delta with only the changed docs - too many deltas -> fold them into one sparse generation - deltas end up covering the whole column -> fold back to a single dense column - a merge flattens everything back to a normal column Numeric and binary only, set-only (a value unset falls back to the current dense rewrite). No doc-values format change, the deltas use the codec's existing sparse encoding. An old Lucene can't read the overlay, so this rolls the segments format version (`VERSION_11_0`), the same way past segments-file changes did (like the SegmentCommitInfo id in 8.6): every commit written by this version uses it, and an older reader rejects the index up front with `IndexFormatTooNewException` instead of reading a delta as if it was the whole column. Reading a newer index with an older Lucene isn't in the contract anyway (Lucene is forward-only), so nothing is lost. On `main` the feature is on by default; pass `IndexWriterConfig#setMaxDocValuesOverlays(0)` for the classic full-column rewrite (0 = off, N = keep up to N overlays before a fold). A 10.x backport would be opt-in via that default. (cherry picked from commit 75eddfa)
Cover the soft-deletes field riding the overlay fold lifecycle, the addIndexes(Directory) copy-as-is carry-over, and an explicit checkIndex on the deep sparse-fold-over-dense-base path. (cherry picked from commit c026d00)
updateBinaryDocValue lacked the index-sort guard that updateNumericDocValue and the varargs updateDocValues already have. An index sort field is never binary, so the check goes before the doc-values-type validation, which would otherwise mask it with a less clear type-mismatch error.
jimczi
force-pushed
the
backport_16418_branch_10x
branch
from
August 27, 2026 09:02
de5413e to
960181a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of #16418 (incremental doc-values updates) to
branch_10x.Opening this mostly to confirm we're ok backporting it to 10.x, since it touches the segments file format.
What it does
A set-only doc-values update (NUMERIC or BINARY) is written as a sparse "delta" overlay holding just the updated docs, layered over the base column at read time instead of rewriting the whole column. Turns per-update write amplification from O(column) into O(updated docs).
Unlike on main, it's opt-in on 10.x (
DEFAULT_MAX_DOC_VALUES_OVERLAYS = 0), since it's a new feature landing in a minor. Enable withIndexWriterConfig#setMaxDocValuesOverlays(n).The thing worth confirming for 10.x
The segments file format gets a new version,
SegmentInfos.VERSION_10_6, written unconditionally on every commit (same as main). So any index written by 10.6+ is rejected by older 10.x readers withIndexFormatTooNewException, whether or not it uses the feature.I kept the bump unconditional to match main rather than gating it on overlay presence. It follows the forward-only policy (older Lucene never promises to read a newer index) and matches the
VERSION_86precedent, which was itself introduced in 8.6. Flagging it since a format bump for every index in a minor is the part most worth a second opinion.Differences from the main commit
DEFAULT_MAX_DOC_VALUES_OVERLAYS = 0instead of 16).VERSION_11_0renamed toVERSION_10_6(same value, named for the release it lands in).SegmentDocValuesProducer#getSkipperkeepsthrows IOExceptionhere, sinceDocValuesProducer#getSkipperstill declares it on 10.x (main dropped it in an unrelated change).Otherwise a straight cherry-pick of 75eddfa, plus the follow-up coverage from #16550 (soft deletes over the overlay, addIndexes(Directory) carry-over, and an explicit checkIndex on the deep fold path).
Plan is to let it bake on main a bit longer before this merges, so no rush here, just want the backport signed off.