Repository navigation
Conversation
parse_messages_json only turned \n and \" back into characters, so any
other escape reached the chat template verbatim: a tab stayed a
backslash and t, "C:\\temp" kept both backslashes, and \u00e9 stayed six
ASCII bytes. Replacing \n after the fact also rewrote the \n inside an
escaped backslash ("C:\\new" became "C:\" plus a newline plus "ew"), and
a content string ending in an escaped backslash was read past its closing
quote into the next key.
Move the decoder json_string_field already had into read_json_string and
use it for message content.
Signed-off-by: breken <hello@breken.ai>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
parse_messages_jsondecoded only\nand\"in a message'scontent. It did so by running string replaces over the raw JSON text. This PR moves the full decoder thatjson_string_fieldalready had intoread_json_stringand uses it forcontent.json_string_fieldbehaves the same as before.Why
Every other escape reached the chat template verbatim, and the replace order also corrupted escaped backslashes. Parsed on
main(fa094ce8):t.\ninside\\newbecomes a real newline.\u00e9stays 6 ASCII bytes.Code and Windows paths are hit most, since every JSON encoder escapes tabs and backslashes.
A content string that ends in an escaped backslash was also read past its closing quote.
"the path is C:\\"parsed asthe path is C:\"},{.This is the C++ side that #804 / #803 mention as follow-up hardening. That PR changes only the Python binding. This one fixes the reader for every binding (Swift, Kotlin, Flutter, and C callers too).
Tests
I added
cactus-engine/tests/test_messages_json_escapes.cpp. It needs no model.main, both tests fail:content_decodes_json_escapesandcontent_ending_in_backslash.test_bucket(10/10) andtest_kv_compress(29/29) still pass. The model-backed suites were not run: they need a bundle.AI disclosure
I found and fixed this with help from an AI coding assistant. I checked the code path by hand and reproduced the failure on
mainbefore writing the fix. The commit is signed off for the DCO.🤖 Generated with Claude Code