Repository navigation
Conversation
…types A dictionary batch is decoded with the value type of the first field that uses its id. A second field with the same id and another value type then viewed those buffers as its own type and read past them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pedrobtz
force-pushed
the
fix-ipc-shared-dictionary-types
branch
from
October 22, 2026 15:27
fcbeb55 to
af99348
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ArrowIpcDictionariesInit()supports several fields sharing one dictionary id. It makes oneArrowIpcDictionaryper unique id and decodes its batches with the value type of the first field that has that id. Nothing checks that the other fields with the id have the same value type. If they don't,ArrowIpcDecoderWalkSetArrayView()views the decoded dictionary with the second field's type.For a
stringdictionary read by a field whose values arelarge_string, the 4-byte offsets are read as 8-byte ones. ASan reports a heap-buffer-overflow inArrowArrayViewValidateDefault().ArrowIpcArrayStreamReaderhits this on a schema message with two such fields, followed by a dictionary batch and a record batch.ArrowIpcDictionariesInit()now returnsEINVALwhen two fields share an id but not a value type. The comparison covers formats, children and nested dictionaries, not names, flags or metadata. Shared dictionaries with the same value type work as before.The new
NanoarrowIpcSharedDictionaryValueTypestest initialises dictionaries for two fields with id 0, first bothstring, thenstringandlarge_string. It expects success and thenEINVAL. Without the fix the second case succeeds. I also read the stream that crashed throughArrowIpcArrayStreamReader. It now fails with the new message instead of reading out of bounds.Found by fuzzing the IPC reader in the zuarrow R package, which vendors nanoarrow.
ctestpasses in a plain Debug build and with ASan and UBSan (410 tests each). The changed files are formatted with the pinned clang-format 16.0.6.🤖 Generated with Claude Code