From c1b455097b79173a431d5903725571af6381f1f2 Mon Sep 17 00:00:00 2001
From: Rasaboun <40967731+Rasaboun@users.noreply.github.com>
Date: Wed, 2 Sep 2026 10:26:27 +0200
Subject: [PATCH] bindings/python: keep non-ASCII text as UTF-8 when
 marshalling JSON to the engine

json.dumps defaults to ensure_ascii=True, which rewrites every non-ASCII
character as \uXXXX. The engine's messages/chat-template path does not
decode these escapes, so the model receives literal escape text: ~3
tokens per character and semantic noise. Accented Latin degrades,
Japanese/Korean become unusable.

With ensure_ascii=False, prompt_tokens exceed the HF tokenizer by a
constant template offset in all tested languages (en/fr/ja/ko), and an
808-row multilingual tool-calling eval returns to the bf16 reference.

Fixes #803

Signed-off-by: Rasaboun <40967731+Rasaboun@users.noreply.github.com>
---
 python/cactus/bindings/cactus.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/python/cactus/bindings/cactus.py b/python/cactus/bindings/cactus.py
index fc9115713..7128d737f 100644
--- a/python/cactus/bindings/cactus.py
+++ b/python/cactus/bindings/cactus.py
@@ -992,7 +992,7 @@ def _to_json(obj):
     if obj is None:
         return None
     if isinstance(obj, (dict, list)):
-        return json.dumps(obj).encode()
+        return json.dumps(obj, ensure_ascii=False).encode()
     if isinstance(obj, str):
         return obj.encode()
     return obj