Repository navigation
Nutrient DWS graph body: keep bold across line wraps, follow the product's line-end hyphen decision - #3
Merged
Conversation
…ow the product's line-end hyphen decision A bold (or italic, underlined, ...) run that wraps onto the next visual line of an element now stays one span when the break is a soft wrap. The break is soft when the line above doesn't end a sentence or label and either the next line continues in lower case, or the line above is a full line (35+ characters, no room left for the next word). Inside a heading every break is a wrap, since the heading is one line. A line-end hyphen now follows the word's hyphenAtLineEnd field when the payload has it: "wrap" fuses without the hyphen, "compound" fuses keeping it, "suspended" keeps the words apart. Without the field the rendering is unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Nutrient DWS adapter rebuilds each page's Markdown from the parse output's elements and per-word style flags (the "graph" body that CI grades). This fixes two bugs in that rendering. A bold sentence that wraps onto a second line now comes out as one bold span instead of two. And the adapter can now follow the product's own decision about a hyphen at the end of a line, once the API sends it.
Bug 1: style runs were cut at every visual line
The adapter grouped words into visual lines and styled each line on its own, so every line got its own
**...**pair. A bold sentence over two lines became two bold spans. Theis_boldrule looks for the whole sentence inside one span, so it failed even though every word was bold.Before:
After:
A run now continues into the next line when the line break is a soft wrap and the next line starts in the same style. Visual lines stay as they are. Only the span markers move.
This is a rendering fix. The product's Markdown exporter builds style runs over the whole paragraph, so a wrapped bold sentence is already one span there. The grader is untouched.
How a soft wrap is told apart from a hard break
Some elements hold several short units in one style, for example a name over a line of credentials. Joining those would turn two title lines into one span and cost title rules. So a break counts as a soft wrap when the line above doesn't end with
. ! ? : ;, a closing quote or a closing parenthesis, and one of these holds:Inside a heading every break is a wrap, because the heading is rendered as one line anyway.
## **The Living** **Word**becomes## **The Living Word**.On the dev split this rule beat the simpler choices. Joining at every break gained less and dropped 7 pages. The width test alone dropped 5 pages, mostly two-line blocks whose first line is the widest.
Bug 2: compound hyphens were dropped at a line end
PR #1 rejoins a word split across lines by a hyphen. It drops every such hyphen when the next line starts in lower case. So
performance-+basedbecameperformancebased, andnet-zerobecamenetzero. The hosted product Markdown keeps both. The SDK decides each case with a per-language lexicon (HyphenatedLineJoiner). On the dev text pages, it joins 305 of these line-end hyphens and keeps 42.The JSON output doesn't carry that decision yet. An SDK change will add an optional word field,
hyphenAtLineEnd, on the last word of a line that ends in a hyphen. The adapter now follows it:hyphenAtLineEndwrapconcentrationscompoundperformance-basedsuspendedpre-+and post-taxUntil the SDK ships the field, this part changes no output and no score.
Measured effect
I re-rendered the stored API payloads of CI build #430 (understand mode) with this branch and regraded them with this repo's grader. Without the change, the re-render matches CI's Markdown on every page. I chose the rule on dev pages and scored held-out pages once, at the end.
That is about +0.22 overall points. The small losses are side effects of how the grader reads Markdown:
*in each part, and those parts count as unexpected sentences.POLARISATION – SELECTED COUNTRIES, a bold fragment of a split heading. With the heading in one span, that match goes away.Testing
hyphenAtLineEndvalues, a missing field and an unknown value are tested on plain and styled text.uv run pytest -qpasses (5068 tests).ruff checkandruff format --checkare clean.Upstream
We'll offer this to run-llama/ParseBench after it has run in our CI for a while.
Part of XT-600.