Non-ASCII input corrupted: Python binding escapes to \uXXXX (ensure_ascii) and the engine never decodes it #803

Summary

Non-ASCII input (accented Latin, Japanese, Korean, …) is corrupted before it reaches the model when using the Python bindings / cactus serve (v2.1.0, macOS ARM64). Severity scales with multi-byte character density: English is unaffected, German nearly so, French partially broken, Japanese/Korean unusable.

Root cause

python/cactus/bindings/cactus.py::_to_json marshals messages/tools with json.dumps(obj). Python's default ensure_ascii=True rewrites every non-ASCII character as \uXXXX, and the engine's messages parser does not decode \u escapes: cactus-engine/src/engine.h::extract_json_string (line ~305) handles only \n \t \r \" \\ and falls through to value += json[pos] for anything else, so \u660e becomes the literal text u660e — the model receives u660eu65e5u306e… for 明日の. Each escaped character costs ~3 tokens and is semantic noise.

Evidence (model-independent)

Same messages+tools through the model's HF tokenizer vs usage.prompt_tokens from cactus serve (prebuilt-graph bundle of LiquidAI/LFM2.5-230M):

A Python replica of bpe.cpp (its byte map + the bundle's vocab/merges + whole-segment BPE) matches HF exactly for en/fr, so the tokenizer algorithm itself is fine — the corruption happens upstream of encode().

With ensure_ascii=False the excess is a constant +42 in all four languages (en/fr/ja/ko: +42/+42/+44/+42), and on an 808-row multilingual tool-calling eval, ja/ko false-action rates drop from 88–100% back to the bf16 reference (0–16%).

10-second repro

POST any Japanese sentence to /v1/chat/completions and compare usage.prompt_tokens with len(AutoTokenizer.from_pretrained(<same model>)(text).input_ids).

Fix

One line in _to_json: json.dumps(obj, ensure_ascii=False).encode() — PR: #804 (https://github.com/cactus-compute/cactus/pull/804). Alternative/complementary: have the C++ messages JSON reader decode \uXXXX — utils.h already contains a correct decoder used elsewhere.

Second, minor bug found on the way (C++, separate) — now tracked in #806, fixed by #805

bpe.cpp::init_byte_mappings maps bytes 161..255 to identity code points; GPT-2 byte-level BPE excludes 173 (U+00AD), which must map to U+0143 like the other non-printables. UTF-8 sequences containing byte 0xAD tokenize differently from the reference (a Japanese test sentence: 17 tokens with 1 unknown vs 15 in HF). Concrete symptom: 歯 is e6 ad af; cactus run on the stock lfm2.5-230m-cq4 bundle echoes it as �丈 (U+FFFD + wrong kanji) — this mapping bug on the decode side, independent of the escaping bug above. Fix: exclude 173 from the identity byte list AND from the identity branch of the assignment loop (else if (byte >= 161 && byte <= 255 && byte != 173)) — the branch is chosen by byte value, so moving the byte between lists alone is a no-op — and append 173 to remaining_bytes after 160 (→ U+0143). Verified: cactus_tokenize("歯") → [33687] (HF reference) after the change; before: [672, 0, 617]. PR: https://github.com/cactus-compute/cactus/pull/805

Related (decode side, Gemma): #684.

Environment: cactus 2.1.0 (brew + pip cactus-compute[convert]), macOS ARM64 (Darwin 25.4.0).