Skip to content

fix(engine): decode every JSON escape in message content - #820

Open
breken-ai wants to merge 1 commit into
cactus-compute:mainfrom
breken-ai:fix/message-content-escapes
Open

breken-ai wants to merge 1 commit into
cactus-compute:mainfrom
breken-ai:fix/message-content-escapes

Conversation

@breken-ai

Copy link
Copy Markdown

What

parse_messages_json decoded only \n and \" in a message's content. It did so by running string replaces over the raw JSON text. This PR moves the full decoder that json_string_field already had into read_json_string and uses it for content. json_string_field behaves the same as before.

Why

Every other escape reached the chat template verbatim, and the replace order also corrupted escaped backslashes. Parsed on main (fa094ce8):

input content:  "func main() {\n\tprint(\"C:\\temp\\new\")\n} caf\u00e9"
parsed content: func main() {
                \tprint("C:\\temp\
                ew")
                } caf\u00e9
  • Tabs arrive as a backslash followed by t.
  • Escaped backslashes stay doubled.
  • The \n inside \\new becomes a real newline.
  • \u00e9 stays 6 ASCII bytes.

Code and Windows paths are hit most, since every JSON encoder escapes tabs and backslashes.

A content string that ends in an escaped backslash was also read past its closing quote. "the path is C:\\" parsed as the path is C:\"},{.

This is the C++ side that #804 / #803 mention as follow-up hardening. That PR changes only the Python binding. This one fixes the reader for every binding (Swift, Kotlin, Flutter, and C callers too).

Tests

I added cactus-engine/tests/test_messages_json_escapes.cpp. It needs no model.

  • On main, both tests fail: content_decodes_json_escapes and content_ending_in_backslash.
  • With this PR, both pass.
  • All engine test targets build on macOS arm64. test_bucket (10/10) and test_kv_compress (29/29) still pass. The model-backed suites were not run: they need a bundle.

AI disclosure

I found and fixed this with help from an AI coding assistant. I checked the code path by hand and reproduced the failure on main before writing the fix. The commit is signed off for the DCO.

🤖 Generated with Claude Code

parse_messages_json only turned \n and \" back into characters, so any
other escape reached the chat template verbatim: a tab stayed a
backslash and t, "C:\\temp" kept both backslashes, and \u00e9 stayed six
ASCII bytes. Replacing \n after the fact also rewrote the \n inside an
escaped backslash ("C:\\new" became "C:\" plus a newline plus "ew"), and
a content string ending in an escaped backslash was read past its closing
quote into the next key.

Move the decoder json_string_field already had into read_json_string and
use it for message content.

Signed-off-by: breken <hello@breken.ai>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant