test(qa): hold a DOCX fidelity line where it stands across, as well as down

.github/workflows/ci.yml

@@ -538,7 +538,7 @@ jobs:
     # Every template preset exported to DOCX, set by LibreOffice, and held line by
     # line to qa/src/test/resources/docx-fidelity/libreoffice-linux*.tsv
     # (DocxFidelityCorpusTest): a change that sets any line of any preset further
-    # from the engine's PDF, at other words, or on more pages fails here. Gated on
+    # from the engine's PDF, down or across, at other words, or on more pages fails here. Gated on
     # `docx` — what an export of the corpus is made from.
     #
     # The image is pinned, not ubuntu-latest: the baseline is LibreOffice's setting

CHANGELOG.md

@@ -1693,6 +1693,21 @@ follow semantic versioning; release dates are ISO 8601.
 
 ### Tests
 
+- **The DOCX fidelity corpus holds a line where it stands across, as well as down.** It measured
+  how far from the page's an editor set each line up or down, and nothing across: a table's
+  left margin the export dropped (fixed in #831) stood `PaymentsInvoice`'s bank details 36.7pt
+  and `VioletGrid`'s education lines 51pt left of the page's in both editors, no line further
+  down than its baseline held, and the gate passed both. The line file now records each line's
+  first letter's drift right of the page's (`FidelityMeasurement.Found.across`), and a line set
+  more than half a point further from the page across fails as one set further up or down does.
+  With #831 reverted, the LibreOffice gate now fails those two documents and `PlatformInvoice`'s
+  footer. A line file written before keeps holding lines down alone; one measured across
+  refuses a row without the column. "More than half a point" is counted in the hundredths
+  drifts are written in, down and across: in doubles, 22 of the 501 steps of exactly half a
+  point between 0 and 5pt counted as further. Across the corpus the median line stands 0.07pt off across in Word and 0.10pt in
+  LibreOffice on Windows; 175 and 130 lines stand more than 2pt off, and 427 in LibreOffice on
+  Linux, held where they are.
+
 - **The DOCX fidelity corpus measures a line from its first letter.** A space opened a line
   (`PdfLines`), and Word writes one for an empty paragraph: drawn just before a line of text
   beside it, a few points off its baseline, it gave that line its own. The lines measured past

CONTRIBUTING.md

@@ -348,9 +348,9 @@ Choose the smallest tests that match the change:
 - For DOCX export (`render-docx`) changes:
   [DocxFidelityCorpusTest.java](qa/src/test/java/com/demcha/compose/document/templates/fidelity/DocxFidelityCorpusTest.java).
   It exports every template preset to DOCX, has LibreOffice set each one, and holds every
-  line of every document to the drift its baseline records: a line set further from the page,
-  a line set at other words, or a page more fails it. It needs LibreOffice and runs only when
-  asked for — install first, as for any standalone `qa` run:
+  line of every document to the drift its baseline records, down and across: a line set further
+  from the page either way, a line set at other words, or a page more fails it. It needs
+  LibreOffice and runs only when asked for — install first, as for any standalone `qa` run:
   `./mvnw -B -ntp test -f qa/pom.xml -Dtest=DocxFidelityCorpusTest -Dgraphcompose.docxFidelity=libreoffice`.
   A change that moves documents nearer the page rewrites the baseline with
   `-Dgraphcompose.docxFidelity.update=true`, and commits it with the change: its diff shows

qa/src/test/java/com/demcha/compose/document/templates/fidelity/DocxFidelityGateTest.java

@@ -93,6 +93,86 @@ void theBaselineFailsALineSetFurtherFromThePage() {
                 .as("a line already 2.8pt off, set 0.6pt further").singleElement().asString().contains("further");
     }
 
+    @Test
+    void linesSetFurtherRightDriftAcrossByAsMuch() throws Exception {
+        // PaymentsInvoice's bank details stood 36.7pt left of the page's in Word, no line lower.
+        Path page = pdf("page", flow -> LINES.forEach(flow::addParagraph));
+        Path right = pdf("right", flow -> LINES.forEach(text -> flow.addParagraph(
+                p -> p.text(text).margin(new DocumentInsets(0, 0, 0, 8)))));
+
+        FidelityMeasurement measured = FidelityMeasurement.of("probe", page, right);
+
+        assertThat(measured.matched()).isEqualTo(3);
+        assertThat(measured.median()).as("no line lower").isCloseTo(0, within(0.05));
+        assertThat(measured.found().values()).allSatisfy(line ->
+                assertThat(line.across()).as("each line 8pt right").isCloseTo(8, within(0.05)));
+    }
+
+    @Test
+    void theBaselineFailsALineFurtherAcross() {
+        FidelityBaseline baseline = FidelityBaseline.of(List.of(doc("cv-probe", 1, 1,
+                line(1, "aa", 0.1, 0.2), line(1, "bb", 0.1, -3.0))));
+
+        assertThat(baseline.regressions(List.of(doc("cv-probe", 1, 1, line(1, "aa", 0.1, -0.71),
+                line(1, "bb", 0.1, -3.0))))).as("0.51pt further, to the other side")
+                .singleElement().asString().contains("1 lines further").contains("across");
+        assertThat(baseline.regressions(List.of(doc("cv-probe", 1, 1, line(1, "aa", 0.1, 0.69),
+                line(1, "bb", 0.1, -0.5))))).as("within half a point, and nearer").isEmpty();
+    }
+
+    @Test
+    void aLineFurtherBothWaysIsNamedOnceWithBoth() {
+        FidelityBaseline baseline = FidelityBaseline.of(List.of(doc("cv-probe", 1, 1, line(1, "aa", 0.1, 0.1))));
+
+        assertThat(baseline.regressions(List.of(doc("cv-probe", 1, 1, line(1, "aa", 2.0, -3.0)))))
+                .singleElement().asString().contains("1 lines further")
+                .contains("+0.10 -> +2.00pt, +0.10 -> -3.00pt across");
+    }
+
+    @Test
+    void aBaselineWrittenBeforeLinesWereMeasuredAcrossHoldsThemDownAlone() {
+        Map<String, Map<String, FidelityMeasurement.Found>> read = new LinkedHashMap<>();
+        FidelityMeasurement.parseLine("cv-a\t1:aa\t0.25\taa", read);
+        FidelityMeasurement.Found found = read.get("cv-a").get("1:aa");
+
+        assertThat(found.preview()).isEqualTo("aa");
+        assertThat(found.drift()).isEqualTo(0.25);
+        assertThat(found.across()).isNaN();
+        FidelityBaseline baseline = FidelityBaseline.of(List.of(doc("cv-a", 1, 1, Map.entry("1:aa", found))));
+        assertThat(baseline.regressions(List.of(doc("cv-a", 1, 1, line(1, "aa", 0.25, 40)))))
+                .as("no place across to hold it to").isEmpty();
+        assertThat(baseline.regressions(List.of(doc("cv-a", 1, 1, line(1, "aa", 1.0, 0)))))
+                .as("and still held down").singleElement().asString().contains("further").doesNotContain("across");
+    }
+
+    @Test
+    void aLineFileMeasuredAcrossRefusesARowWithoutItsDriftAcross() throws Exception {
+        Path file = dir.resolve("baseline.tsv");
+        FidelityBaseline.write(file, "probe", List.of(doc("cv-a", 1, 1, line(1, "aa", 0.25, 1.5))));
+        Path lines = dir.resolve("baseline-lines.tsv");
+        List<String> rows = Files.readAllLines(lines);
+
+        Files.write(lines, List.of(rows.get(0), "cv-a\t1:aa\t0.25\taa"));
+        assertThatThrownBy(() -> FidelityBaseline.read(file)).as("a row cut short, not one written before")
+                .isInstanceOf(IllegalArgumentException.class).hasMessageContaining("5 cells");
+        Files.write(lines, List.of("cv-a\t1:aa\t0.25\t1.50\taa", "cv-a\t1:bb\t0.50\tbb"));
+        assertThatThrownBy(() -> FidelityBaseline.read(file)).as("no header, and a row measured across beside one cut short")
+                .isInstanceOf(IllegalArgumentException.class).hasMessageContaining("5 cells");
+    }
+
+    @Test
+    void halfAPointFurtherIsRoundingForEveryDrift() {
+        // In doubles, 0.18 + 0.5 is less than 0.68: half a point further failed 22 of these drifts.
+        for (int hundredths = 0; hundredths <= 500; hundredths++) {
+            double was = hundredths / 100.0;
+            double now = (hundredths + 50) / 100.0;
+            FidelityBaseline baseline = FidelityBaseline.of(List.of(doc("cv-probe", 1, 1, line(1, "aa", was, was))));
+
+            assertThat(baseline.regressions(List.of(doc("cv-probe", 1, 1, line(1, "aa", now, now)))))
+                    .as("%.2f -> %.2f", was, now).isEmpty();
+        }
+    }
+
     @Test
     void theBaselineFailsALineNoLongerFound() {
         FidelityBaseline baseline = FidelityBaseline.of(List.of(doc("cv-probe", 1, 1, line(1, "aa", 0.4), line(1, "bb", 2.8))));
@@ -124,12 +204,16 @@ void theBaselineLetsThroughALineWithinRoundingOrNearerThePage() {
     @Test
     void aBaselineIsReadAsItWasWritten() throws Exception {
         Path file = dir.resolve("baseline.tsv");
-        List<FidelityMeasurement> rows = List.of(doc("cv-a", 1, 1, line(1, "aa", 0.25)),
+        List<FidelityMeasurement> rows = List.of(doc("cv-a", 1, 1, line(1, "aa", 0.25, -1.25)),
                 doc("cv-b", 2, 3, line(1, "aa", -1.5), line(2, "cc", 3)));
 
         FidelityBaseline.write(file, "probe", rows);
 
         assertThat(FidelityBaseline.read(file).regressions(rows)).isEmpty();
+        assertThat(FidelityBaseline.read(file).regressions(List.of(doc("cv-a", 1, 1, line(1, "aa", 0.25, -1.86)),
+                doc("cv-b", 2, 3, line(1, "aa", -1.5), line(2, "cc", 3)))))
+                .as("a drift across read back as written: -1.25 -> -1.86 is further")
+                .singleElement().asString().contains("across");
         assertThat(FidelityBaseline.read(file).regressions(List.of(doc("cv-a", 1, 1, line(1, "aa", 0.86)),
                 doc("cv-b", 2, 3, line(1, "aa", -1.5), line(2, "cc", 3)))))
                 .as("a drift read back as written: 0.25 -> 0.86 is further").singleElement().asString().contains("further");
@@ -310,7 +394,12 @@ private static FidelityMeasurement doc(String stem, int enginePages, int editorP
     }
 
     private static Map.Entry<String, FidelityMeasurement.Found> line(int page, String letters, double drift) {
-        return Map.entry(page + ":" + letters, new FidelityMeasurement.Found(page, letters, drift));
+        return line(page, letters, drift, 0);
+    }
+
+    private static Map.Entry<String, FidelityMeasurement.Found> line(int page, String letters, double drift,
+                                                                     double across) {
+        return Map.entry(page + ":" + letters, new FidelityMeasurement.Found(page, letters, drift, across));
     }
 
     private Path pdf(String name, Consumer<PageFlowBuilder> content) throws Exception {

qa/src/test/java/com/demcha/compose/document/templates/fidelity/FidelityBaseline.java

@@ -20,21 +20,27 @@
  * document fails when the editor sets it on a page count further from the engine's, when a
  * line the baseline found is no longer found — the editor breaks it at another word, or puts it
  * on another page — or when a found line drifts more than {@value #LINE_SLACK}pt further from
- * the page than it did. Line by line, because a document already set a few points off
+ * the page than it did, down or across. Across as well as down: a table's left margin the
+ * export dropped stood {@code PaymentsInvoice}'s bank details 36.7pt left of the page's and
+ * moved no line up or down. Line by line, because a document already set a few points off
  * throughout hides a new defect in its counts. A document the baseline does not hold fails
  * until the baseline is written with it, and so does one the baseline holds and the corpus no
  * longer has. A document whose lines the editor mostly sets at other words is held to the few
  * it finds: the gate keeps what an export has, and adds nothing for what it lacks.</p>
  *
  * <p>The baseline is two files: {@code <name>.tsv}, a row per document, and
  * {@code <name>-lines.tsv}, a row per found line, named by its page and a digest of its
- * letters, with a few of them to read it by.</p>
+ * letters, with its drift down and across and a few of its letters to read it by. A line file
+ * written before lines were measured across holds them to their drift down alone.</p>
  */
 final class FidelityBaseline {
 
     /** How much further a line may drift, in points: rounding, not room for a change to spend. */
     static final double LINE_SLACK = 0.5;
 
+    /** The line file's header names this column when its lines were measured across. */
+    private static final String ACROSS_COLUMN = "drift right of the page (pt)";
+
     /** How many of a document's failing lines a failure names. */
     private static final int NAMED = 4;
 
@@ -60,9 +66,15 @@ static FidelityBaseline read(Path documents) throws IOException {
             throw new IllegalStateException("the baseline " + documents + " has no line file " + lineFile);
         }
         Map<String, Map<String, FidelityMeasurement.Found>> lines = new LinkedHashMap<>();
-        for (String row : Files.readAllLines(lineFile, StandardCharsets.UTF_8)) {
+        List<String> lineRows = Files.readAllLines(lineFile, StandardCharsets.UTF_8);
+        // Its lines were measured across, for the whole file, when its header says so or any
+        // row holds a drift across: a row without one there is refused, not read as old.
+        boolean across = !lineRows.isEmpty() && lineRows.get(0).startsWith("#")
+                         && lineRows.get(0).contains(ACROSS_COLUMN)
+                         || lineRows.stream().anyMatch(row -> !row.startsWith("#") && row.split("\t", 5).length == 5);
+        for (String row : lineRows) {
             if (!row.isBlank() && !row.startsWith("#")) {
-                FidelityMeasurement.parseLine(row, lines);
+                FidelityMeasurement.parseLine(row, lines, across);
             }
         }
         for (String row : Files.readAllLines(documents, StandardCharsets.UTF_8)) {
@@ -91,7 +103,8 @@ static void write(Path documents, String note, Collection<FidelityMeasurement> m
         List<FidelityMeasurement> sorted = measurements.stream()
                 .sorted(Comparator.comparing(FidelityMeasurement::stem)).toList();
         List<String> rows = new ArrayList<>(List.of("# " + note, FidelityMeasurement.HEADER));
-        List<String> lines = new ArrayList<>(List.of("# " + note + " — document, line, drift below the page (pt), letters"));
+        List<String> lines = new ArrayList<>(List.of("# " + note
+                + " — document, line, drift below the page (pt), " + ACROSS_COLUMN + ", letters"));
         for (FidelityMeasurement measured : sorted) {
             rows.add(measured.row());
             lines.addAll(measured.lineRows());
@@ -126,9 +139,18 @@ List<String> regressions(Collection<FidelityMeasurement> measurements) {
                 FidelityMeasurement.Found there = now.found().get(id);
                 if (there == null) {
                     lost.add("\"" + line.preview() + "\"");
-                } else if (Math.abs(there.drift()) > Math.abs(line.drift()) + LINE_SLACK) {
-                    further.add(String.format(Locale.ROOT, "\"%s\" %+.2f -> %+.2fpt",
-                            line.preview(), line.drift(), there.drift()));
+                    return;
+                }
+                // Named once, with every way it moved further.
+                List<String> ways = new ArrayList<>(2);
+                if (further(line.drift(), there.drift())) {
+                    ways.add(String.format(Locale.ROOT, "%+.2f -> %+.2fpt", line.drift(), there.drift()));
+                }
+                if (!Double.isNaN(line.across()) && further(line.across(), there.across())) {
+                    ways.add(String.format(Locale.ROOT, "%+.2f -> %+.2fpt across", line.across(), there.across()));
+                }
+                if (!ways.isEmpty()) {
+                    further.add("\"" + line.preview() + "\" " + String.join(", ", ways));
                 }
             });
             if (!lost.isEmpty()) {
@@ -142,6 +164,15 @@ List<String> regressions(Collection<FidelityMeasurement> measurements) {
         return found;
     }
 
+    /**
+     * Whether a drift is more than {@link #LINE_SLACK}pt further from the page than it was,
+     * counted in the hundredths both are written in: in doubles, half a point further was
+     * further for some drifts and not for others.
+     */
+    private static boolean further(double was, double now) {
+        return Math.round(Math.abs(now) * 100) > Math.round(Math.abs(was) * 100) + Math.round(LINE_SLACK * 100);
+    }
+
     private static String named(List<String> lines) {
         return String.join(", ", lines.subList(0, Math.min(NAMED, lines.size())))
                + (lines.size() > NAMED ? ", and " + (lines.size() - NAMED) + " more" : "");