fix(diff): keep paragraph-spanning fields balanced and entry tabs in place (#1156)

[JSv4]

Closes #1156

Second of a three-PR stack for the comparison engine: #1169 (#1157), then this one, then the #1149 PR. It is based on #1169's branch and will be retargeted to main when #1169 merges.

Why

Word writes a table of contents as one complex field spread over several paragraphs: the first entry paragraph holds the field's begin, its TOC instruction and its separator, and the last entry paragraph holds the field's end. Each entry is a hyperlink with a nested PAGEREF field after a tab. At body level, comparing two such documents broke the field in three ways:

• Deleting the last entry: reject kept 5 field ends for 4 begins.
• Appending an entry: reject kept 2 ends for 3 begins.
• Editing an entry's text: accept moved the entry's tab inside the PAGEREF field.

How it works

There were two separate defects.

(a) A field end that moved between paragraphs carried no revision mark.

• The IR reader ignored a field end that had no begin in its own paragraph. A paragraph that gained or lost the table of contents' closing end therefore compared as unchanged, and the renderer copied the revised side's paragraph bare.
• IrReader.UnmatchedFieldCharacters now records each paragraph's unmatched field characters, in document order: depth-0 ends and separators, plus one begin per field left open. That list is folded into the paragraph's field-envelope digest.
• An existing rule then applies: a paragraph that is otherwise unchanged but whose structure changed is replaced whole, so accept and reject each keep their own field characters.
• Only the characters' order is recorded, not where they sit in the text. So a text edit in a paragraph that keeps its spanning field is still diffed word by word.
• The cost is that a paragraph which gains or loses the closing end is shown deleted and reinserted, even when its text didn't change.
• Such a paragraph is also no longer considered for split/merge pairing or cross-paragraph streaming. The digest's version tag moved from v2 to v3; nothing persists it.

(b) The tab moved inside the page-number field.

• When the renderer slices a run of source text for one edit operation, it always keeps zero-width markers (field characters, bookmark and comment endpoints). Before this change it took them anywhere in the span, both ends inclusive.
• After "Term" → "Term and renewal", the insertion ended at the character where the tab and then the PAGEREF begin/instruction/separator all sit. The tab belongs to the next, unchanged span, but the insertion grabbed the field characters behind it.
• SourceRunModel.Slice now leaves markers at a span's end to the next operation, but only when that operation:
  • starts there,
  • is in the same rendering call,
  • keeps the token there unchanged,
  • and, on the original's side, either reads that side itself or re-emits a matching marker.
• An unchanged span that emits the revised side's copy of a marker claims the original's copy, so a later deletion cannot emit it a second time.

Validation

• New BodyFieldSpanDiffTests (44 cases). For every case, every paragraph's sequence of field characters, instructions, tabs and text must match in both accept and reject, and the redline runs through the OOXML validator. The cases are:
  • the issue's twelve body-level cases, plus the reverse edit and a replaced entry, each in both shapes (end in the last entry, or in its own trailing paragraph);
  • the real Word-authored table of contents HC022-Table-Of-Contents.docx with its content control stripped;
  • a table-cell variant and a Consolidate case;
  • a bookmark or comment marker behind a tab after an edit;
  • a marker only the original carries;
  • a bookmark moved behind inserted text;
  • field-result changes, and page-number changes with and without a text edit.
• Mutation checks. Each half fails its own tests when reverted: 7 and 8 failures on the first commit, 10 on the final renderer. The claim, the original-side narrowing and the unchanged-owner condition are each guarded too.
• Full suite on the stacked branch: 6,925 passed, 0 failed, 2 skipped. The corpus baseline doesn't move, but it pins revision text, so it is not evidence for either half. The Release build has no warnings.

Also fixed in passing (#1165)

The same boundary rule fixes part of #1165. A field whose cached result changes directly after unchanged text no longer gains a second begin, instruction and separator on reject. That covers the common case of a table-of-contents page-number update. Two shapes of #1165 remain open, and that issue stays open for them:

• the result change merged with deleting the text before the field;
• the entry that opens the table of contents.

Adversarial review

One round of hostile review ran on the first commit. It confirmed that part (a) doesn't over-trigger: self-contained fields, editing the first or last entry, and split/merge all stay word-level. It found one real regression in part (b): markers left for the next operation could be lost or misplaced on the original's side.

• A bookmark or comment that only the original carried, after a tab, vanished on reject.
• A comment end could be re-emitted after inserted text, so the comment's range grew.

The second commit fixes that with the original-side narrowing and the claim rule, and adds tests for both shapes. A probe set of 54 body and 6 Consolidate cases is now no worse than main in any case and fixes 49 of main's failing checks. The CHANGELOG was corrected to claim only what holds.

Stacking on #1157 then exposed one interaction: an edited inline w:customXml made a shared comment reference a revised copy. The third commit fixes it by deferring only to an operation that keeps the token unchanged; #1157's test now passes unchanged.

Pre-existing gaps the review found are filed as #1165 and #1166.