Store doc values disi data into a separate file - #16540
Conversation
costin
left a comment
There was a problem hiding this comment.
+1 on moving IndexedISI into its own file.
A few things stood out to me:
The writeDISI method buffers the entire IndexedDISI blob via BBDO while the old code wrote DISI directly, without any buffering. For typical fields this is fine however for large segments, the bitmap could end up to a few MB; wanted to check if this was a conscious tradeoff and if there's any data assessing the impact.
I noticed there are no new tests for the TLV extensibility mechanism; the existing doc-values tests exercise the new write path but the key design feature (skipping unknown TLV tags) has zero coverage. It would be good to have an explicit bwc test verifying pre-VERSION_DISI_EXTENSIBLE_FILE segments are read correctly when disiData is null for example.
Lastly, the skip index file uses FileTypeHint.INDEX but this PR uses FileTypeHint.DATA. Was this an intentional choice ?
Description
This moves out the doc values disi data (used to store sparse data) into a separate file so that we can prefetch and keep it hot in the cache if needed.
Closes - #16298