Skip to content

fix: Hindi and Bengali paragraphs misclassified as Title - #4695

Open
shlokbam wants to merge 4 commits into
Unstructured-IO:mainfrom
shlokbam:shlokbam/indic-narrative-text
Open

shlokbam wants to merge 4 commits into
Unstructured-IO:mainfrom
shlokbam:shlokbam/indic-narrative-text

Conversation

@shlokbam

@shlokbam shlokbam commented Nov 11, 2026 •

Copy link
Copy Markdown

Problem

Text in Hindi, Bengali and other Indic scripts is misclassified in both directions:

  1. Paragraphs become Title. partition_text on six two-sentence Hindi paragraphs returns 3 Titles, and on three Bengali paragraphs returns 2 Titles. chunk_by_title then starts a new chunk at each false title:

    Document Before After
    Hindi (6 paragraphs) 3/6 Title, 4 chunks 0/6 Title, 1 chunk
    Bengali (3 paragraphs) 2/3 Title, 3 chunks 0/3 Title, 1 chunk
    English control (3 paragraphs) 0/3 Title, 1 chunk 0/3 Title, 1 chunk (unchanged)

    (chunk_by_title(elements, max_characters=2000, combine_text_under_n_chars=0))

  2. Real headings can't be detected as Title. is_possible_title("हिंदी") ("Hindi") returns False, because under_non_alpha_ratio("हिंदी") says the word is mostly non-letters.

Root causes

  1. Vowel signs aren't letters to str.isalpha(). In Devanagari and Bengali, vowel signs (matras), viramas and anusvaras are Unicode combining marks (category M*), so "हिंदी".isalpha() is False: only 2 of its 5 code points are letters.
    • under_non_alpha_ratio counts the marks as non-letters, so short words like "हिंदी" fall under the 0.5 threshold.
    • exceeds_cap_ratio keeps only tokens where tk.isalpha() is true. In a Hindi sentence where every word carries a vowel sign, nothing is left, and the "no tokens" branch returns True ("too many capitals"). The paragraph then fails the narrative check.
  2. The danda (।, ॥) isn't a reliable sentence end. The English spaCy model sometimes splits at । and sometimes doesn't. When it doesn't, a two-sentence paragraph counts as one sentence. That sends it into the single-sentence branch of exceeds_cap_ratio above, and into the English verb check.

Fix

  • text_type.py: a character counts as a letter if it is alphabetic or a combining mark.
    • under_non_alpha_ratio: counts marks as letters, written inline for speed (char.isalpha() or unicodedata.category(char)[0] == "M").
    • exceeds_cap_ratio: a token counts as a word if it has at least one letter and every character is a letter or a mark (_is_word), instead of tk.isalpha().
  • nlp/tokenize.py: after spaCy's sentence segmentation, each sentence is split again on (?<=[।॥])\s+. This is inside the existing lru_cached _tokenize_for_cache, so caching is unchanged and sent_tokenize/sentence_count pick it up.

Unchanged: thresholds, the English verb check, is_possible_narrative_text/is_possible_title logic, and behavior for text without combining marks or dandas. All existing tests in the touched files pass.

Tests

New tests in test_text_type.py, nlp/test_tokenize.py and partition/test_text.py:

  • under_non_alpha_ratio("हिंदी") is False; is_possible_title("हिंदी") is True
  • exceeds_cap_ratio is False for "हिंदी सबसे अधिक बोली जाने वाली भाषा है।" (every word has a vowel sign)
  • sentence_count of "हिंदी सबसे अधिक बोली जाने वाली भाषा है। गंगा पवित्र नदी है।" is 2 (was 1); sent_tokenize splits on । and ॥
  • is_possible_narrative_text is True for each Hindi/Bengali paragraph; partition_text returns only NarrativeText for both documents
  • Still works: under_non_alpha_ratio on full paragraphs, is_possible_title("परिचय"), and the existing English title/narrative cases

12 of the new tests fail on main and pass with this change; the rest guard behavior that already worked. test_text_type.py, nlp/test_tokenize.py and partition/test_text.py: 246 passed. ruff check and ruff format --check pass on the changed files.

To reproduce:

from unstructured.partition.text import partition_text
paras = ["दिल्ली में सरकार की बैठकें होती हैं। संसद भी यहीं है।",
         "কলকাতা পশ্চিমবঙ্গের রাজধানী। এখানে অনেক মানুষ থাকে।"]
print([type(e).__name__ for e in partition_text(text="\n\n".join(paras))])
# main: ['Title', 'Title']   this PR: ['NarrativeText', 'NarrativeText']

Performance

10,000 calls each, best of 5, Apple M2, Python 3.13:

Text Function main This PR
English paragraph under_non_alpha_ratio 8.20 µs 8.26 µs
English paragraph exceeds_cap_ratio 6.47 µs 6.00 µs
Hindi paragraph under_non_alpha_ratio 3.15 µs 4.16 µs
Hindi paragraph exceeds_cap_ratio 5.98 µs 5.90 µs

English is within noise. Hindi under_non_alpha_ratio does about 1 µs more work because it now looks up the category of each mark. The exceeds_cap_ratio numbers are the cached path; classifying a new element is dominated by the spaCy run (milliseconds), which this PR doesn't change.

Known limitation (possible follow-up)

A single-sentence Hindi/Bengali paragraph can still become a Title, e.g. "दिल्ली में सरकार की बैठकें होती हैं।", "भारत एक विशाल देश है।" or "রবীন্দ্রনাথ ঠাকুর নোবেল পুরস্কার পেয়েছিলেন।". Both character checks now pass, but with fewer than 2 sentences is_possible_narrative_text runs the English-only verb check (contains_verb), whose result for Hindi depends on spaCy happening to tag some word as an English verb.

partition_text detects hin (it's in element.metadata.languages) but doesn't pass languages to is_possible_narrative_text; is_possible_narrative_text(t, languages=["hin"]) returns True. I'm happy to send that as a separate PR.

Overlap with #4679

#4679 reorders the under_non_alpha_ratio and exceeds_cap_ratio calls inside is_possible_narrative_text. This PR only changes the bodies of those two functions (plus new helpers above is_possible_narrative_text), so the two shouldn't conflict, and the reorder doesn't change any result here since both checks can only make the function return False early.

View guided diff

… signs as letters, split sentences on danda)

@chrikrah chrikrah left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shlokbam blocking: unicodedata.category(char)[0] == "M" also counts U+FE0F (variation selector, Mn) and U+20E3 (keycap, Me), so an emoji-only line now passes under_non_alpha_ratio. is_possible_title never reaches _is_word, so these lines come out as Title and open a new chunk in chunk_by_title.

# trees from `git archive`: head 6544f77 and base 6544f77~1, CPython 3.12.14, clone .venv
# probe.py prints unstructured.__file__, then for each line: is_possible_title(s), element types of partition_text(text=s)
$ PYTHONPATH=/tmp/p34e/base .venv/bin/python probe.py
/tmp/p34e/base/unstructured/__init__.py
'⭐️⭐️⭐️⭐️⭐️' False ['Text']
'1️⃣ 2️⃣ 3️⃣' False ['Text']
'✅️ ✅️ ✅️' False ['Text']
'⚠️' False ['Text']
$ PYTHONPATH=/tmp/p34e/head .venv/bin/python probe.py
/tmp/p34e/head/unstructured/__init__.py
'⭐️⭐️⭐️⭐️⭐️' True ['Title']
'1️⃣ 2️⃣ 3️⃣' True ['Title']
'✅️ ✅️ ✅️' True ['Title']
'⚠️' True ['Title']

# head 6544f77, your tests pass and fail when the source hunks are reverted
$ .venv/bin/python -m pytest test_unstructured/partition/test_text_type.py test_unstructured/nlp/test_tokenize.py -q -p no:cacheprovider
151 passed in 25.04s
# same tree, unstructured/ reverted with `git apply -R` of the PR's source hunks
$ .venv/bin/python -m pytest test_unstructured/partition/test_text_type.py test_unstructured/nlp/test_tokenize.py -q -p no:cacheprovider
10 failed, 141 passed in 21.70s
# not run: ruff, test_unstructured/ outside these files

A fix that keeps the Indic cases: count a mark only when the previous non-space character is a letter or an attached mark. Matras, viramas and decomposed accents still count. A test that partition_text(text="⭐️⭐️⭐️⭐️⭐️") stays Text would pin it. Can you add that guard in this PR?

… letter

Counting every Unicode mark (M*) as a letter also counted the variation
selector U+FE0F and the keycap mark U+20E3, so emoji-only lines like
"⭐️⭐️⭐️⭐️⭐️" or "1️⃣ 2️⃣ 3️⃣" passed under_non_alpha_ratio and were
classified as Title.

Add one helper, _letter_flags, that counts a mark only when the previous
non-space character is a letter or a counted mark, and use it in both
_is_word and under_non_alpha_ratio. Indic vowel signs and viramas still
count; emoji marks do not.
@shlokbam

Copy link
Copy Markdown
Author

@chrikrah thanks for catching this, and for the probe. Added the guard in ffae7b8.

  • A combining mark now counts as a letter only when the previous non-space character is a letter or a counted mark. Matras, viramas and decomposed accents still count; U+FE0F and U+20E3 on an emoji don't.
  • _is_word and under_non_alpha_ratio now share one helper (_letter_flags), so they can't drift apart.
  • New tests: partition_text on ⭐️⭐️⭐️⭐️⭐️ and 1️⃣ 2️⃣ 3️⃣ gives [Text] (same as the base commit), plus under_non_alpha_ratio / is_possible_title on those and on ✅️ ✅️ ✅️ and ⚠️.
$ pytest test_unstructured/partition/test_text_type.py test_unstructured/nlp/test_tokenize.py test_unstructured/partition/test_text.py -q
256 passed
# same files, unstructured/partition/text_type.py from 6544f77
10 failed, 246 passed   (the 10 new emoji cases)
$ ruff check . && ruff format --check .
All checks passed! / 547 files already formatted

The existing Hindi and Bengali cases still pass.

@cragwolfe cragwolfe left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Medium — Keep the non-alpha check's memory use bounded

_letter_flags() builds a list entry for every non-space character, and under_non_alpha_ratio() builds the whole list before counting it. The previous implementation used two counters. Large paragraphs or extracted text blocks can reach this check without an element-length cap, so this adds a potentially large allocation and can exhaust a partition worker. The spaCy input limit does not bound the original string passed to the ratio check. Could the ratio use streaming counts and the word check use a small amount of state while preserving the new combining-mark rules? A focused long-input regression would help protect this behavior.

Compatibility decision: This also changes the public sentence tokenizer for danda-separated text, including its use by translation, and applies the combining-mark classification rule across Unicode scripts. The intended classification changes can alter element types and by-title chunk boundaries. Please confirm that this broader behavior is acceptable as part of the fix.

The earlier emoji regression is addressed by the new mark-handling rule and checked-in cases. This is a static review; exact-head CI requires approval and is not a successful test result.

(authored by codex)

_letter_flags built a list with an entry for every non-space character,
and under_non_alpha_ratio built the whole list before counting it. Large
extracted text blocks can reach this check, so the allocation grew with
the input.

Replace it with _iter_letter_flags, a generator that keeps only whether
the previous non-space character counted. under_non_alpha_ratio uses two
running counters again and _is_word stops at the first character that
does not count. Results are unchanged.
@shlokbam

Copy link
Copy Markdown
Author

@cragwolfe thanks for the review.

Memory: fixed in 72cbaf3. _letter_flags is now _iter_letter_flags, a generator that keeps only whether the previous non-space character counted. under_non_alpha_ratio uses two running counters again, and _is_word stops at the first character that doesn't count. On an 8 MB mixed input (3.7M characters), the tracemalloc peak of under_non_alpha_ratio went from 24.4 MB to under 0.01 MB; time is 0.34 s (0.20 s on main, 0.29 s with the list). Results are identical to the list version on 200k random strings mixing letters, combining marks, emoji selectors, digits and whitespace. New test test_letter_checks_use_bounded_memory_on_long_input runs both checks on ~5 MB and asserts a peak under 100 KB (the list version peaks at 15.2 MB on it).

Version / changelog: CONTRIBUTING says ordinary PRs don't edit the frozen CHANGELOG.md or bump __version__.py, so I didn't. The branch was based on the 0.27.37 release commit; I merged main (no rebase), so it now carries 0.27.38-dev, and scripts/check-new-release-version.sh passes locally.

Scope:

  • The danda split only changes text that contains । or ॥; other text tokenizes as before.
  • The mark rule applies to any script with combining marks. With partition_text I found no element-type change outside Hindi/Bengali in my samples (Tamil, Thai, Arabic with harakat, Hebrew with niqqud, Latin and Vietnamese in NFD), with detected or explicit languages. What does change is is_possible_narrative_text(text, languages=[lang]), from False to True, for a Tamil heading, Thai, Arabic with harakat, Hebrew with niqqud and NFD Latin: on main, exceeds_cap_ratio dropped every word containing a mark and then reported the caps threshold as exceeded. Plain Arabic, a Tamil paragraph and NFD Vietnamese are unchanged.

I think that change is a fix too, but if you'd rather keep this PR narrow, I can limit the mark rule to the Indic blocks.

$ pytest test_unstructured/partition/test_text_type.py test_unstructured/nlp/test_tokenize.py test_unstructured/partition/test_text.py -q
257 passed
$ ruff check . && ruff format --check .
All checks passed! / 550 files already formatted

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants