@@ -0,0 +1,35 @@
+# Findings — 2026-09-09 run
+
+Corpus: 107 ground-truth renders (14 docs × docx/odt/pptx/html/md/pdf, 3 two-column PDFs, 4 workbooks × xlsx/xls/xlsb/ods/csv), 30 real-world files, 14 robustness cases. Raw metrics in `REPORT.md`. Reproduce with the scripts in this directory (see `PLAN.md`).
+
+Content recall is 0.99–1.00 in every prose format and PDF text matches PyMuPDF at NED 0.996–1.000 on six real documents. The problems are structure, boilerplate, and typed cells.
+
+| # | Finding | Where | Evidence |
+|---|---|---|---|
+| 1 | PDF text flattened to one line before chunking (both modes) | `pdf-parser.ts` `.replace(/\s+/g, ' ')` | paragraph retention 0.06, heading retention 0.00; IRS p17 → 246 chunks, none at a paragraph |
+| 2 | Spreadsheet dates/percent/currency indexed raw; Google Sheets sync inherits it | `xlsx-parser.ts` (no `raw:false`/`cellDates`) | `2026-03-04` → `46085`, `20%` → `0.2`; ODS dates → JS local-time string |
+| 3 | Running headers/footers/page numbers leak into every PDF | `pdf-parser.ts` | absence 0.00 on 17/17 renders |
+| 4 | Tables exploded one cell per line in docx/pptx/odt/odp | mammoth `extractRawText`, officeparser | table adjacency 0.00; mammoth HTML computed but unused |
+| 5 | Words glued at line/column/cell boundaries in PDFs | items joined without separator when `hasEOL` false | irs-p17 39 glued tokens, omnidocbench 19, 2 of 3 two-column renders |
+| 6 | Legacy .doc/.ppt fallback emits ZIP names, XML, master-slide placeholders | `doc-parser.ts`, `pptx-parser.ts` fallback | 7/7 real files degraded; KB and workspace-files search honour the flag |
+| 7 | Slide numbers, footer placeholders, review comments indexed as body | officeparser | `poi-notes.pptx` bare `1..11` + `testdoc`; odt comment spliced mid-sentence |
+| 8 | Non-UTF-8 text silently stripped | `txt-parser.ts`/`md-parser.ts` + `sanitizeTextForUTF8` | Latin-1 "Café résumé naïve £" → "Caf rsum nave" |
+| 9 | Mislabelled inputs accepted; corrupt PDF error untyped (transient → OCR) | `index.ts` extension routing; pdf.js `Invalid PDF structure.` | CSV-as-xlsx mojibake + serials; HTML-as-txt raw markup |
+
+Fix order: PDF line structure + spacing → PDF furniture suppression → SheetJS formatted text → DOCX via mammoth HTML → officeparser post-processing → transcode fallback for text → type the PDF structure error → magic-byte sniffing.
+
+## Validation pass (9 parallel investigators, 2026-09-09)
+
+All nine findings confirmed. Corrections to the original framing:
+
+| # | Correction | Precedent |
+|---|---|---|
+| 1 | The whitespace collapse was copied from unpdf 1.4.0 for byte-identical output in #6425; unpdf fixed it in 1.7.0 (PR #58) before #6425 landed. No consumer or test depends on single-line text. pdf.js `hasEOL` arrives on an empty item whose y is the NEXT line. | unpdf #58, pdf.js text_layer, pdfplumber y_tolerance, pdfminer line_margin |
+| 2 | `raw:false` alone is not enough: Excel's General format truncates 16-digit numbers to `4.11111E+15` and dates render locale-shaped; a pre-pass rewriting `w` for `t:'d'` and General cells fixes both. The Google Sheets and Microsoft Excel connectors ALREADY request formatted text, so Drive-synced Sheets disagree with Sheets-connector Sheets today. `xlsx-preview-data.ts` (file viewer) has the same defect. | SheetJS `raw`/`cellDates` docs, MarkItDown #53 |
+| 3 | 62% of IRS p17 chunks carry the running footer. A frequency rule alone misses footers whose chapter title changes; Marker's consecutive-streak rule (>=3 pages) recovers it. Requires #1 first (needs reconstructed lines + y). | Marker IgnoreTextProcessor, OmniDocBench 'abandon', pymupdf4llm margins |
+| 4 | DOCX via mammoth HTML -> existing HtmlParser walker prototyped: 8/8 adjacency, footnotes recovered, zero new deps. mammoth `convertToMarkdown` drops tables (do not use). officeparser 7.8 fixes tables but pulls tesseract.js + pdfjs-dist@6 (126 MB) and drops ODT header rows. | mammoth README, MarkItDown, unstructured, Docling |
+| 5 | Fusions are NOT missing-hasEOL at line ends; they are (a) Form XObject boundaries (pdf.js resets prevTransform) and (b) backwards x-move on the same baseline (pdf.js flushes without EOL). Geometry join rule prototyped: catalog 15->0 fusions, IRS 56->25. Dehyphenation must check doc-local compounds or it breaks `open-source`. | pdf.js evaluator constants, MuPDF stext-device, pdfplumber |
+| 6 | Worse than reported: two of four real .doc files return 3% and 17% of the body (UCS-2 text invisible to the ASCII regex). KB and workspace search honour `degraded`; Copilot file-reader, chat upload reader, File block (`internal/file/parser.ts`) and `get content` do NOT and hand the scrape to the model. `.xls` is fine (SheetJS BIFF). `word-extractor` (pure JS, frozen 2021) gets 89-100% on the POI .doc files; no viable pure-JS .ppt extractor exists. | word-extractor, Tika, Docling/unstructured shell out to soffice |
+| 7 | The leaked `1..11` + `testdoc` come from NOTES PAGES (`ppt/notesSlides`), which officeparser dumps because we pass no options; not slide-level footers. officeparser 7.8 does not fix it. ODT splice includes `text:sender-initials`; tracked-change deletions also leak. Same JSZip walker as #4 fixes both. | python-pptx placeholder types, MarkItDown, POI SlideShowExtractor, pandoc ODT reader |
+| 8 | Also: UTF-8 BOM leaks into content; UTF-16 'pass' was an ASCII accident; connectors keep U+FFFD as mojibake. Bun 1.3.14 TextDecoder supports `fatal` + `windows-1252` natively. Truncated-UTF-8 downloads need a tail retry before falling back. | Tika EncodingDetector chain, unstructured encoding.py, LangChain autodetect_encoding |
+| 9 | `Invalid PDF structure.` is a named `InvalidPDFException`; the classifier just never checks it. docx-as-pdf never reaches OCR (`assertOcrSourceSupported` sniffs `%PDF-`). Truncated PDFs with a header DO reach OCR, deliberately, pinned by `pdf-ocr-triage.test.ts:439,580`. `ArchiveIntegrityError` is already classified permanent. `file-type@16.5.4` (CJS) is already in the tree via officeparser. | Tika detection precedence, unstructured detect_filetype, pdf.js exception names |
@@ -0,0 +1,52 @@
+# Knowledge-base parser quality evaluation
+
+## Why
+
+Every file a connector (Drive, OneDrive, SharePoint, Box, Dropbox, S3, SFTP, Bitbucket, Gmail/Outlook attachments) or an upload delivers as bytes goes through `apps/sim/lib/file-parsers` before chunking and embedding. If a parser emits noise (XML internals, placeholder sentences, boilerplate), drops content, destroys paragraph structure, or scrambles reading order, every downstream search result inherits it silently: the document row still reads "success".
+
+## What the state of the art measures
+
+| Benchmark | What it scores | How |
+|---|---|---|
+| OmniDocBench (CVPR 2025) | text, tables, formulas, reading order across 10 doc types | Normalized Edit Distance on text and reading order, TEDS on tables; headers/footers/page numbers are an "abandon" class excluded from scoring |
+| olmOCR-bench (Ai2) | 1,403 PDFs, 7,010 binary unit tests | text presence, text absence (headers/footers/page numbers must NOT appear), natural reading order pairs, table cell adjacency, math |
+| READoc / opendataloader-bench | PDF to structured markdown | heading detection, reading order, table structure, F1 on blocks |
+
+Two design ideas transfer directly: (1) score with **binary unit tests per document** (presence, absence, order, adjacency), because fuzzy whole-document similarity hides localized failures; (2) treat **boilerplate leakage as a first-class failure**, not a rounding error.
+
+## Framework
+
+### Corpus (two tiers)
+
+**Tier A — ground truth by construction.** Source documents are authored as Markdown with a machine-readable spec (paragraphs, headings, list items, table cells, sentinel sentences, ordering pairs). Each source is rendered by pandoc to DOCX, ODT, PPTX, HTML and (via typst) PDF, so the same known content arrives in every container our parsers handle. PDFs are additionally rendered with running headers, footers and page numbers so absence tests are meaningful. Tabular specs are written with SheetJS to XLSX, XLS, XLSB, ODS and CSV.
+
+**Tier B — real-world documents with no gold text.** Public PDFs (two-column papers, forms, reports), DOCX/PPTX/XLSX/DOC/PPT/XLS/ODT files from open-source test corpora, and HTML pages. Scored by agreement against independent reference extractors (PyMuPDF, pdfplumber, python-docx, python-pptx, openpyxl) plus reference-free noise heuristics.
+
+### Metrics per (document, format)
+
+| Metric | Definition | Catches |
+|---|---|---|
+| `ned` | 1 − Levenshtein(norm(out), norm(gt)) / max(len) | gross content loss or gain |
+| `presence` | share of sentinel sentences found (partial ratio ≥ 90) | dropped paragraphs, cells, slide bodies |
+| `absence` | share of boilerplate strings (running header/footer/page numbers/"Sheet:" wrappers) NOT found | leakage into the index |
+| `order` | share of (a before b) pairs preserved | column/slide/cell reordering |
+| `table_adjacency` | share of (left cell, right cell) pairs appearing on one output line | tables exploded one cell per line |
+| `noise_ratio` | share of output word tokens absent from gt vocabulary | XML names, placeholders, scraped bytes |
+| `paragraph_retention` | output paragraph breaks / gt paragraphs | whitespace collapse that starves the chunker |
+| `heading_retention` | share of headings appearing on their own line | headings glued into paragraphs |
+| `junk_chars` | control/replacement/private-use chars per 1k chars | encoding damage |
+| `chunk_sentence_boundary` | share of TextChunker chunks ending at sentence punctuation | how the parse degrades chunking |
+| `metadata` | degraded/truncated/pageCount agree with reality | wrong flags either poison the index or skip good files |
+| `latency_ms` | wall time | regressions |
+
+Robustness cases (empty, truncated, mislabeled extension, encrypted, non-UTF8) are scored pass/fail on whether a typed `FileParserError` is raised rather than garbage returned.
+
+### Execution
+
+1. `generate-corpus.py` builds Tier A sources, specs and renders (pandoc + typst + SheetJS).
+2. `fetch-real-world.sh` downloads Tier B.
+3. `run-parsers.ts` (bun, inside apps/sim so `@/` resolves) runs `parseBuffer` for every file exactly as the ingestion path does (`pdfTextMode: 'complete'` for PDFs) and writes JSON outputs.
+4. `reference-extract.py` runs the reference extractors on the same files.
+5. `score.py` computes the metric table, aggregated by format and by parser, and lists the worst documents.
+
+Everything reproducible from `apps/sim/scripts/parser-eval/`.
@@ -0,0 +1,102 @@
+# Parser quality report
+
+Tier A: 107 files, Tier B: 30 files, robustness: 14 cases
+
+## Tier A — ground truth by construction (mean per format)
+
+| format | n | ned | presence | absence | order | table_adjacency | noise_ratio | glued_words | paragraph_retention | heading_retention | chunk_sentence_boundary | junk_per_1k | ms |
+|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
+| csv | 4 | 0.70 | 0.98 | — | — | 0.95 | 0.00 | 0.00 | — | — | — | 0.00 | 1.65 |
+| docx | 14 | 0.96 | 1.00 | — | 1.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.91 | 1.00 | 0.00 | 11.35 |
+| html | 14 | 0.91 | 1.00 | — | 1.00 | 1.00 | 0.01 | 0.00 | 1.00 | 0.85 | 1.00 | 0.00 | 1.69 |
+| md | 14 | 0.94 | 1.00 | — | 1.00 | 1.00 | 0.00 | 0.00 | 0.99 | 1.00 | 1.00 | 0.00 | 0.13 |
+| ods | 4 | 0.78 | 0.87 | — | — | 0.81 | 0.14 | 0.00 | — | — | — | 0.00 | 2.17 |
+| odt | 14 | 0.96 | 1.00 | — | 1.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.91 | 1.00 | 0.00 | 2.81 |
+| pdf | 14 | 0.89 | 0.99 | 0.00 | 1.00 | 0.87 | 0.07 | 0.14 | 0.06 | 0.00 | 0.00 | 0.00 | 16.11 |
+| pdf-2col | 3 | 0.93 | 1.00 | 0.00 | 1.00 | 0.72 | 0.05 | 0.67 | 0.03 | 0.00 | 0.00 | 0.00 | 8.43 |
+| pptx | 14 | 0.96 | 1.00 | — | 1.00 | 0.00 | 0.00 | 0.00 | 1.00 | 0.91 | 1.00 | 0.00 | 2.26 |
+| xls | 4 | 0.81 | 0.87 | — | — | 0.81 | 0.12 | 0.00 | — | — | — | 0.00 | 2.25 |
+| xlsb | 4 | 0.81 | 0.87 | — | — | 0.81 | 0.12 | 0.00 | — | — | — | 0.00 | 1.60 |
+| xlsx | 4 | 0.81 | 0.87 | — | — | 0.81 | 0.12 | 0.00 | — | — | — | 0.00 | 3.27 |
+
+### Worst Tier A files by sentinel presence / adjacency / noise
+
+| file | presence | absence | order | adjacency | noise | para | heading | noise sample |
+|---|---|---|---|---|---|---|---|---|
+| sheet-typed.ods | 0.464 | None | None | 0.25 | 0.515 | None | None | 0700 0800 085 1063 12000 1250 |
+| sheet-typed.xls | 0.464 | None | None | 0.25 | 0.455 | None | None | 085 1063 12000 1250 1500 46085 |
+| sheet-typed.xlsb | 0.464 | None | None | 0.25 | 0.455 | None | None | 085 1063 12000 1250 1500 46085 |
+| sheet-typed.xlsx | 0.464 | None | None | 0.25 | 0.455 | None | None | 085 1063 12000 1250 1500 46085 |
+| unicode-multilingual.pdf | 0.833 | 0.0 | 1.0 | 1.0 | 0.185 | 0.083 | None | acme confidential corp distribute do draft |
+| sheet-typed.csv | 0.929 | None | None | 0.792 | 0.0 | None | None |  |
+| changelog.pdf | None | 0.0 | None | None | 0.121 | 0.062 | 0.0 | acme confidential corp distribute do draft |
+| product-catalog.2col.pdf | 1.0 | 0.0 | 1.0 | 0.686 | 0.094 | 0.021 | None | 2027hw acme confidential corp discounthw distribute |
+| product-catalog.pdf | 1.0 | 0.0 | 1.0 | 0.686 | 0.094 | 0.021 | None | 2027hw acme confidential corp discounthw distribute |
+| memo.pdf | 1.0 | 0.0 | 1.0 | None | 0.091 | 0.143 | None | acme confidential corp distribute do draft |
+| onboarding-guide.pdf | 1.0 | 0.0 | 1.0 | None | 0.081 | 0.062 | 0.0 | acme confidential corp distribute do draft |
+| sop-access-review.pdf | 1.0 | 0.0 | 1.0 | 0.75 | 0.077 | 0.05 | 0.0 | acme confidential corp distribute do draft |
+| meeting-notes.pdf | 1.0 | 0.0 | 1.0 | 0.875 | 0.068 | 0.062 | 0.0 | acme confidential corp distribute do internal |
+| tech-spec.pdf | 1.0 | 0.0 | 1.0 | 1.0 | 0.062 | 0.038 | 0.0 | acme confidential corp distribute do draft |
+| faq-benefits.pdf | 1.0 | 0.0 | 1.0 | None | 0.061 | 0.091 | 0.0 | acme confidential corp distribute draft infra |
+
+## Tier B — real-world files vs reference extractors
+
+| file | fmt | len | degraded | pages (ours/ref) | reference | ned | ref line recall | out line precision | noise | noise sample |
+|---|---|---|---|---|---|---|---|---|---|---|
+| attention.pdf | pdf | 39642 | False | 15/15 | pymupdf (39495) | 0.996 | 1.0 | 1.0 | 0.001 | df epos visualizationsinput |
+| attention.pdf | pdf | 39642 | False | 15/15 | pdfplumber (35525) | 0.849 | 0.185 | 1.0 | 0.001 | df epos visualizationsinput |
+| bitcoin.pdf | pdf | 21227 | False | 9/9 | pymupdf (21220) | 0.999 | 1.0 | 1.0 | 0.0 | blockblock |
+| bitcoin.pdf | pdf | 21227 | False | 9/9 | pdfplumber (21216) | 0.906 | 0.915 | 1.0 | 0.0 | blockblock |
+| irs-f1040.pdf | pdf | 10151 | False | 2/2 | pymupdf (10156) | 1.0 | 1.0 | 1.0 | 0.001 | 2025u |
+| irs-f1040.pdf | pdf | 10151 | False | 2/2 | pdfplumber (10152) | 0.8 | 0.739 | 0.0 | 0.001 | 2025u |
+| irs-p17.pdf | pdf | 959987 | False | 142/142 | pymupdf (960116) | 1.0 | 1.0 | 1.0 | 0.0 | 000caution 16caution 2025get 4vtip 8815records andcaution |
+| irs-p17.pdf | pdf | 959987 | False | 142/142 | pdfplumber (431054) | 0.295 | 0.215 | 0.0 | 0.0 | 000caution 16caution 2025get 4vtip 8815records andcaution |
+| lo-fdo38244.odt | odt | 32 | False |  | pandoc (16) | 0.5 | None | None | 0.2 | mfirst |
+| lo-lists.odt | odt | ERROR | | | | | | | | No text could be extracted from this OpenDocument file |
+| lo-simple.odp | odp | 28 | False |  | (none) | | | |  | |
+| lo-simple.ods | ods | 12322 | False |  | (none) | | | |  | |
+| lo-tables.odt | odt | 17 | False |  | (none) | | | |  | |
+| mdn-fetch.html | html | 6810 | False |  | (none) | | | |  | |
+| omnidocbench.pdf | pdf | 103150 | False | 32/32 | pymupdf (102111) | 0.999 | 1.0 | 1.0 | 0.005 | 10190 2011年1月1日 7000 aaaaa ajhb annotationsfigure |
+| omnidocbench.pdf | pdf | 103150 | False | 32/32 | pdfplumber (100495) | 0.354 | 0.365 | 1.0 | 0.005 | 10190 2011年1月1日 7000 aaaaa ajhb annotationsfigure |
+| pdf-reference-excerpt.pdf | pdf | 14 | False | 1/1 | pymupdf (14) | 1.0 | None | None | 0.0 |  |
+| pdf-reference-excerpt.pdf | pdf | 14 | False | 1/1 | pdfplumber (14) | 1.0 | None | None | 0.0 |  |
+| poi-basic.ppt | ppt | 907 | True |  | (none) | | | |  | |
+| poi-bug-tables.doc | doc | 416 | True |  | (none) | | | |  | |
+| poi-bullets.ppt | ppt | 767 | True |  | (none) | | | |  | |
+| poi-footnotes.docx | docx | 35 | False |  | python-docx (33) | 1.0 | 1.0 | 1.0 | 0.0 |  |
+| poi-footnotes.docx | docx | 35 | False |  | pandoc (47) | 0.702 | 1.0 | 1.0 | 0.0 |  |
+| poi-header-footer.doc | doc | 660 | True |  | (none) | | | |  | |
+| poi-headerfooter.docx | docx | ERROR | | | | | | | | No text could be extracted from this DOCX file |
+| poi-layouts.pptx | pptx | 645 | False |  | python-pptx (650) | 0.983 | 1.0 | 1.0 | 0.0 |  |
+| poi-lists.doc | doc | 1198 | True |  | (none) | | | |  | |
+| poi-multisheet.xls | xls | 133 | False |  | (none) | | | |  | |
+| poi-notes.pptx | pptx | 2393 | False |  | python-pptx (2357) | 0.968 | 1.0 | 1.0 | 0.037 | 10 testdoc |
+| poi-sample.docx | docx | 1548 | False |  | python-docx (1542) | 1.0 | 1.0 | 1.0 | 0.0 |  |
+| poi-sample.docx | docx | 1548 | False |  | pandoc (1542) | 1.0 | 1.0 | 1.0 | 0.0 |  |
+| poi-sample.pptx | pptx | 140 | False |  | python-pptx (152) | 0.908 | 1.0 | 1.0 | 0.0 |  |
+| poi-sample.xlsx | xlsx | 391 | False |  | openpyxl (292) | 0.753 | 1.0 | 0.5 | 0.019 | empty |
+| poi-sampledoc.doc | doc | 1463 | True |  | (none) | | | |  | |
+| poi-simple.xls | xls | 144 | False |  | (none) | | | |  | |
+| poi-tables.ppt | ppt | 6359 | True |  | (none) | | | |  | |
+| w3c-html-spec-intro.html | html | 54667 | False |  | (none) | | | |  | |
+| wiki-rag.html | html | 28081 | False |  | (none) | | | |  | |
+
+## Robustness
+
+| case | expected | passed | outcome |
+|---|---|---|---|
+| csv-labelled-xlsx | typed error OR correct UTF-8 text | ❌ | ok 3122 chars |
+| docx-bytes-labelled-pdf | typed error | ❌ | UNTYPED: Invalid PDF structure. |
+| docx-labelled-doc | correct text | ✅ | ok 843 chars |
+| docx-labelled-xlsx | typed error | ✅ | typed invalid_format: Failed to parse XLSX buffer: Could not find workbook |
+| empty.docx | typed error | ✅ | typed empty_input: Empty buffer provided |
+| html-labelled-txt | markup stripped or typed error | ❌ | ok 4904 chars |
+| latin1-txt | text decodes to "Café résumé naïve £" | ❌ | ok 17 chars |
+| pdf-bytes-labelled-docx | typed error OR correct text | ✅ | ok 933 chars |
+| png-labelled-doc | typed error, never placeholder prose | ❌ | ok 87 chars DEGRADED |
+| pptx-labelled-ppt | correct text | ✅ | ok 843 chars |
+| random-bytes-labelled-ppt | typed error, never placeholder prose | ❌ | ok 99 chars DEGRADED |
+| truncated-docx | typed error (invalid_format) | ❌ | UNTYPED: Unable to inspect ZIP central directory; refusing to parse an unverifiable ZIP-shaped arch |
+| truncated-pdf | typed error (invalid_format) | ❌ | UNTYPED: Invalid PDF structure. |
+| utf16-txt | text decodes to "Hello UTF-16 world" | ✅ | ok 18 chars |