Repository navigation
Conversation
chc__col_read_string doubles str.data while rows arrive & keeps the final capacity, so a decoded block retains up to twice its string bytes: a column holding 257 bytes keeps a 512 byte buffer, one holding 64 MiB + 1 keeps 128 MiB. Shrink the buffer once the column is complete. A refused shrink keeps the larger buffer.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
chc__col_read_string doubles str.data as rows arrive and keeps the final capacity. A decoded block can be held long after the read, and it holds up to twice its string bytes. A column with 257 bytes of strings keeps a 512-byte buffer, and one with 64 MiB + 1 byte keeps 128 MiB.
Once the column is complete, the buffer is now shrunk to the loaded size. If the allocator refuses the shrink, the larger buffer stays and the read still succeeds. An all-empty column keeps its 256-byte seed, because some allocators treat a 0-byte realloc as free.
test_string_exact_size decodes two rows of 200 and 57 bytes and checks for a 257-byte buffer. On main it gets 512.