Repository navigation
Conversation
init_byte_mappings treated bytes 161..255 as identity code points. GPT-2's
bytes_to_unicode() excludes 173 (U+00AD, soft hyphen) from that range and
assigns it the next free code point after the other non-printables
(U+0143). Any UTF-8 sequence containing byte 0xAD -- e.g. 歯 (E6 AD AF) --
therefore tokenized to <byte> <unk> <byte> instead of the reference token,
and decoded to U+FFFD.
The fix must exclude 173 from BOTH the identity byte list and the
identity branch of the assignment loop (the branch is selected by byte
value, so moving the byte between lists alone has no effect).
Verified: the resulting 256-entry mapping equals HF GPT2
bytes_to_unicode() byte-for-byte; cactus_tokenize("歯") -> [33687] as in
the reference tokenizer.
Refs cactus-compute#803 (second item).
Signed-off-by: Rasaboun <40967731+Rasaboun@users.noreply.github.com>
Rasaboun
changed the title
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #806. (Found alongside #803, whose Python-binding half is #804.)
What
BPETokenizer::init_byte_mappingstreats bytes 161..255 as identity code points. GPT-2 byte-level BPE (bytes_to_unicode()) excludes 173 (U+00AD, soft hyphen) from that range and assigns it the next free code point after the other non-printables (U+0143). This PR excludes 173 from both the identity byte list and the identity branch of the assignment loop — the branch is selected by byte value, so moving the byte between lists alone has no effect (that was my first, ineffective attempt; documented so nobody repeats it).Symptom
Any UTF-8 sequence containing byte 0xAD, e.g.
歯(E6 AD AF), tokenizes to<byte> <unk> <byte>instead of the reference token and decodes to U+FFFD. With LiquidAI/LFM2.5-230M:歯[672, 0, 617][33687]歯医者[672, 0, 617, 8773, 4046][33687, 8773, 4046]明日の朝9時に歯医者に電話することを思い出させてVerification
GPT2Tokenizerbytes_to_unicode()byte-for-byte (checked against an exact replica of the loop's control flow).apple/build.sh(device, simulator, macOS);cactus_tokenizevia the macOS framework gives the table above; a React Native app on the iOS simulator round-trips歯intact in tool-call arguments where it previously produced�.No other files touched; no performance impact (mapping is built once).