Skip to content

fix(html): leave ruby text out of the paragraph text - #4837

Open
gagan-baghel wants to merge 1 commit into
Unstructured-IO:mainfrom
gagan-baghel:fix-html-ruby-text
Open

gagan-baghel wants to merge 1 commit into
Unstructured-IO:mainfrom
gagan-baghel:fix-html-ruby-text

Conversation

@gagan-baghel

@gagan-baghel gagan-baghel commented Nov 21, 2026 •

Copy link
Copy Markdown

Summary

partition_html() glues ruby text to the base text it annotates. Japanese pages use ruby for furigana, and Chinese pages use it for pinyin:

from unstructured.partition.html import partition_html

partition_html(text="<p>首都は<ruby>東京<rt>とうきょう</rt></ruby>です</p>")[0].text
# before: '首都は東京とうきょうです'
# after:  '首都は東京です'

<rt> and <rp> had no element class, so DefaultElement treated them as transparent. Their text was then merged into the sentence, which corrupts search, embeddings and chunk text for 東京. This change maps both to RemovedPhrasing, like other content that isn't rendered as body text. The tail after each tag is kept. <rp> holds the fallback parentheses that browsers hide when they support ruby.

Validation

  • Added and_it_is_used_for_ruby_text_so_it_is_not_glued_to_the_base_text to DescribeRemovedPhrasing. It covers <rp> and two ruby pairs in one element. It fails on main and passes with this change.
  • pytest test_unstructured/partition/html test_unstructured/documents: 667 passed.
  • ruff check and ruff format --check pass on the changed files.

Checklist

  • The PR title is a user-facing release-note entry with a conventional commit prefix.
  • Apply a category label (bug): I can't add labels, so please apply it.
  • Not a breaking change.

View guided diff

<rt> and <rp> had no element class, so they were transparent and their
text was glued to the base text: "東京<rt>とうきょう</rt>" became
"東京とうきょう". Skip them like other content that is not shown as
body text.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant