BPE byte-level mapping: byte 0xAD mapped to identity instead of U+0143 (GPT-2 bytes_to_unicode) — CJK chars containing 0xAD tokenize to <unk> #806

Summary

BPETokenizer::init_byte_mappings (cactus-engine/src/bpe.cpp) maps bytes 161..255 to identity code points. GPT-2 byte-level BPE (bytes_to_unicode()) excludes byte 173 (U+00AD, soft hyphen) from that range and assigns it the next free code point after the other non-printables (U+0143). Any UTF-8 sequence containing byte 0xAD therefore tokenizes differently from the reference tokenizer and decodes to U+FFFD.

Repro (LiquidAI/LFM2.5-230M bundle, cactus_tokenize / Cactus.tokenize in the RN bridge)

Only characters containing byte 0xAD are affected — everything else in the sentence tokenizes identically to HF. Decode side of the same table: cactus run on the stock lfm2.5-230m-cq4 bundle echoes 歯医者 as �丈医者. Observed end-to-end in a React Native app: a tool-call argument 歯医者に電話する came back as �Reception医者に電話する.

Root cause

for (int i = 161; i <= 255; ++i) bytes.push_back(i);          // includes 173
...
} else if (byte >= 161 && byte <= 255) {                      // identity branch also catches 173

GPT-2: identity ranges are 33..126, 161..172, 174..255; all other bytes (0..32, 127..160, 173) get 256+n in increasing byte order → 173 → U+0143.

Fix

Exclude 173 from both the identity list and the identity branch condition (the branch is selected by byte value, so moving the byte between lists alone is a no-op), and append 173 to remaining_bytes after 160. PR: #805. Verified: the resulting 256-entry mapping equals HF GPT2Tokenizer.bytes_to_unicode() byte-for-byte; cactus_tokenize("歯") → [33687] after the change.

Related: #803 (Python-binding ensure_ascii escaping — a separate, input-path bug; both were found while running a multilingual tool-calling model on cactus).

Environment: cactus v2.1.0 / main 09cb35a, macOS ARM64 + iOS simulator.