Skip to content

Nutrient DWS graph body: keep bold across line wraps, follow the product's line-end hyphen decision - #3

Merged
etienne-alby merged 1 commit into
mainfrom
etienne/nutrient-dws-bold-hyphen
Oct 26, 2026
Merged

etienne-alby merged 1 commit into
mainfrom
etienne/nutrient-dws-bold-hyphen

Conversation

@etienne-alby

@etienne-alby etienne-alby commented Oct 25, 2026 •

Copy link
Copy Markdown

The Nutrient DWS adapter rebuilds each page's Markdown from the parse output's elements and per-word style flags (the "graph" body that CI grades). This fixes two bugs in that rendering. A bold sentence that wraps onto a second line now comes out as one bold span instead of two. And the adapter can now follow the product's own decision about a hyphen at the end of a line, once the API sends it.

Bug 1: style runs were cut at every visual line

The adapter grouped words into visual lines and styled each line on its own, so every line got its own **...** pair. A bold sentence over two lines became two bold spans. The is_bold rule looks for the whole sentence inside one span, so it failed even though every word was bold.

Before:

**There was no further clarification in the survey to specify if the visits to the doctor**
**were in person or online.**

After:

**There was no further clarification in the survey to specify if the visits to the doctor
were in person or online.**

A run now continues into the next line when the line break is a soft wrap and the next line starts in the same style. Visual lines stay as they are. Only the span markers move.

This is a rendering fix. The product's Markdown exporter builds style runs over the whole paragraph, so a wrapped bold sentence is already one span there. The grader is untouched.

How a soft wrap is told apart from a hard break

Some elements hold several short units in one style, for example a name over a line of credentials. Joining those would turn two title lines into one span and cost title rules. So a break counts as a soft wrap when the line above doesn't end with . ! ? : ;, a closing quote or a closing parenthesis, and one of these holds:

  • the next line starts in lower case, or
  • the line above is a full line: at least 35 characters, and the next line's first word wouldn't have fit at its end.

Inside a heading every break is a wrap, because the heading is rendered as one line anyway. ## **The Living** **Word** becomes ## **The Living Word**.

On the dev split this rule beat the simpler choices. Joining at every break gained less and dropped 7 pages. The width test alone dropped 5 pages, mostly two-line blocks whose first line is the widest.

Bug 2: compound hyphens were dropped at a line end

PR #1 rejoins a word split across lines by a hyphen. It drops every such hyphen when the next line starts in lower case. So performance- + based became performancebased, and net-zero became netzero. The hosted product Markdown keeps both. The SDK decides each case with a per-language lexicon (HyphenatedLineJoiner). On the dev text pages, it joins 305 of these line-end hyphens and keeps 42.

The JSON output doesn't carry that decision yet. An SDK change will add an optional word field, hyphenAtLineEnd, on the last word of a line that ends in a hyphen. The adapter now follows it:

hyphenAtLineEnd Rendering
wrap fuse without the hyphen: concentrations
compound fuse and keep it: performance-based
suspended keep both words apart: pre- + and post-tax
absent or unknown the PR #1 rule, unchanged

Until the SDK ships the field, this part changes no output and no score.

Measured effect

I re-rendered the stored API payloads of CI build #430 (understand mode) with this branch and regraded them with this repo's grader. Without the change, the re-render matches CI's Markdown on every page. I chose the rule on dev pages and scored held-out pages once, at the end.

Dimension Dev Held-out
Semantic formatting 72.48 to 74.24 (+1.76, 22 pages up, 1 down) 68.91 to 69.85 (+0.94)
Content faithfulness 84.98 to 84.96 (-0.02) 86.50 to 86.49 (-0.01)
Visual grounding (layout) +0.00 +0.00
Tables +0.00 +0.00
Charts 61.31 to 60.96 (-0.35, 1 page) 47.20 to 47.12 (-0.07, 1 page)

That is about +0.22 overall points. The small losses are side effects of how the grader reads Markdown:

  • Content. The sentence check splits Markdown at line breaks before it strips emphasis. A span over two lines leaves a lone * in each part, and those parts count as unexpected sentences.
  • Charts. The dev chart page passed because a chart label fuzzy-matched POLARISATION – SELECTED COUNTRIES, a bold fragment of a split heading. With the heading in one span, that match goes away.

Testing

  • New provider tests cover bold and italic spans that cross a wrap, spans that start and end mid-line, a line ending a sentence, short title-case lines, a line with room left, and a heading.
  • All three hyphenAtLineEnd values, a missing field and an unknown value are tested on plain and styled text.
  • uv run pytest -q passes (5068 tests). ruff check and ruff format --check are clean.

Upstream

We'll offer this to run-llama/ParseBench after it has run in our CI for a while.

Part of XT-600.

…ow the product's line-end hyphen decision

A bold (or italic, underlined, ...) run that wraps onto the next visual line of an
element now stays one span when the break is a soft wrap. The break is soft when the
line above doesn't end a sentence or label and either the next line continues in lower
case, or the line above is a full line (35+ characters, no room left for the next
word). Inside a heading every break is a wrap, since the heading is one line.

A line-end hyphen now follows the word's hyphenAtLineEnd field when the payload has
it: "wrap" fuses without the hyphen, "compound" fuses keeping it, "suspended" keeps
the words apart. Without the field the rendering is unchanged.
@etienne-alby
etienne-alby deleted the etienne/nutrient-dws-bold-hyphen branch October 26, 2026 06:27
@etienne-alby
etienne-alby restored the etienne/nutrient-dws-bold-hyphen branch October 26, 2026 06:28
@etienne-alby etienne-alby reopened this Oct 26, 2026
@etienne-alby
etienne-alby merged commit 2dcf53c into main Oct 26, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant