fix: correct Jet Unicode compression decoding for CJK text fields - #38
Open
ilolita945 wants to merge 1 commit into
Open
fix: correct Jet Unicode compression decoding for CJK text fields#38ilolita945 wants to merge 1 commit into
ilolita945 wants to merge 1 commit into
Conversation
When BOM (\xff\xfe) is present in TEXT fields, the data uses Jet Unicode compression: 0x00-separated runs alternating between UTF-16LE and compressed ASCII (single bytes expanded to 0x00XX in UTF-16LE). Previously get_decoded_text() was called for BOM-marked fields, which tried UTF-8 then Latin-1 decoding. This produced garbled CJK text. Added decompress_jet_text() to properly handle the Jet compression format, and modified parse_type() to use it when leading 0x00 is present. Fixes claroty#21, claroty#26
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fix Jet Unicode compression decoding for CJK text fields.
Problem
When BOM (
\\xff\\xfe) is present in TEXT fields, the database uses Jet Unicode compression. The format stores text as 0x00-separated runs alternating between UTF-16LE pairs and compressed ASCII (single bytes expanded to 0x00XX in UTF-16LE).The original code called
get_decoded_text()for BOM-marked fields, which tried UTF-8 then Latin-1 decoding. This produced garbled CJK text (e.g. Chinese characters displayed as garbled ASCII).Solution
decompress_jet_text()function that correctly handles the Jet compression formatparse_type()BOM branch to use it when leading 0x00 is presentTesting
Tested with real-world Chinese archive management MDB files containing mixed CJK and ASCII text fields.
Related
Closes #21, Closes #26