Skip to content

Trim String column data to its loaded size - #37

Open
iliaal wants to merge 1 commit into
ClickHouse:mainfrom
iliaal:ch-string-shrink
Open

iliaal wants to merge 1 commit into
ClickHouse:mainfrom
iliaal:ch-string-shrink

Conversation

@iliaal

@iliaal iliaal commented Nov 21, 2026

Copy link
Copy Markdown
Contributor

chc__col_read_string doubles str.data as rows arrive and keeps the final capacity. A decoded block can be held long after the read, and it holds up to twice its string bytes. A column with 257 bytes of strings keeps a 512-byte buffer, and one with 64 MiB + 1 byte keeps 128 MiB.

Once the column is complete, the buffer is now shrunk to the loaded size. If the allocator refuses the shrink, the larger buffer stays and the read still succeeds. An all-empty column keeps its 256-byte seed, because some allocators treat a 0-byte realloc as free.

test_string_exact_size decodes two rows of 200 and 57 bytes and checks for a 257-byte buffer. On main it gets 512.

chc__col_read_string doubles str.data while rows arrive & keeps the final
capacity, so a decoded block retains up to twice its string bytes: a
column holding 257 bytes keeps a 512 byte buffer, one holding 64 MiB + 1
keeps 128 MiB. Shrink the buffer once the column is complete. A refused
shrink keeps the larger buffer.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant