Skip to content

Store doc values disi data into a separate file - #16540

Draft
sgup432 wants to merge 3 commits into
apache:mainfrom
sgup432:disi_file_change
Draft

Store doc values disi data into a separate file#16540
sgup432 wants to merge 3 commits into
apache:mainfrom
sgup432:disi_file_change

Conversation

@sgup432

@sgup432 sgup432 commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Description

This moves out the doc values disi data (used to store sparse data) into a separate file so that we can prefetch and keep it hot in the cache if needed.

Closes - #16298

@costin costin left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 on moving IndexedISI into its own file.

A few things stood out to me:

The writeDISI method buffers the entire IndexedDISI blob via BBDO while the old code wrote DISI directly, without any buffering. For typical fields this is fine however for large segments, the bitmap could end up to a few MB; wanted to check if this was a conscious tradeoff and if there's any data assessing the impact.

I noticed there are no new tests for the TLV extensibility mechanism; the existing doc-values tests exercise the new write path but the key design feature (skipping unknown TLV tags) has zero coverage. It would be good to have an explicit bwc test verifying pre-VERSION_DISI_EXTENSIBLE_FILE segments are read correctly when disiData is null for example.

Lastly, the skip index file uses FileTypeHint.INDEX but this PR uses FileTypeHint.DATA. Was this an intentional choice ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants