Import PT9 interlinearizations end-to-end (parser → InterlinearProject)

[imnasnainaec]

Umbrella for the goal raised in #94: fully support importing PT9 interlinearizations into an InterlinearProject. The XML parser exists but is wired into nothing — this issue is the end-to-end pipeline from PT9 data to a working, editable project.

Companions: #148 (model-text suggestion source, incl. the major-language model PT9 glosses against) and #149 (output project). This issue consumes both — it discovers a PT9 project's model/output relationships and attaches them to the imported project.

Design principles this follows

• Simplify project creation — import produces an ordinary project, not a locked "PT9 mode." The model/output relationships it discovers are attached to the result (per Wire a model project into the suggestion engine (surface-form tier) #148/Wire the output project (back-translation / adaptation) into export, not creation #149), never baked into a rigid creation mode.
• Generalize the interlinearizer — an imported PT9 project behaves like any other: its source, model, and output can be changed or dropped afterward.
• First-contact discoverable — import is offered where a user would look for it (e.g. from the project picker / "open" flow), not as a required up-front decision.

Current state

• src/parsers/pt9/interlinearXmlParser.ts — InterlinearXmlParser.parse(xml) → InterlinearData (ScrTextName, GlossLanguage, BookId, Verses: verse key → VerseData of Clusters + Punctuations). Schema in pt9-xml.md; one sample in test-data/Interlinear_en_MAT.xml.
• Not wired into anything — no import command, no cluster→analysis mapping, no gloss-text resolution, no UI. grep confirms the parser and its types are referenced only by their own tests.

Scope — the pipeline

1. Import command + UI. An interlinearizer.importPt9 command and an entry point (file/folder picker for Interlinear_<lang>_<book>.xml, or selection of a PT9 project). Creates an InterlinearProject via projectStorage / interlinearizer.createProject.
2. Cluster → Token mapping (the crux). PT9 Cluster.Range is an Index/Length span in the verse text; our tokens carry charStart/charEnd within Segment.baselineText from bookTokenizer. Reconcile the two so each cluster resolves to the token(s) it covers. A single-lexeme cluster over one token → a TokenAnalysis; a cluster spanning multiple tokens → a PhraseAnalysis; a multi-lexeme cluster → a morpheme breakdown (TokenAnalysis.morphemes).
3. Gloss-text resolution (the big unknown). The XML stores only Lexeme.Id + GlossId/SenseId — not gloss text. Resolving to a displayable gloss needs a lexicon. Options: resolve to TokenAnalysis.glossSenseRef (a SenseRef into the Lexicon extension, resolved lazily — the model already supports this) and/or resolve to literal gloss text at import time from PT9's lexicon files. Decide the strategy (see Open questions).
4. Punctuation / Excluded / Hash. Punctuation tokens come from the tokenizer already; PT9 Punctuation change records are likely informational. Excluded clusters map to a suppressed/location-specific state. Verse Hash present ⇒ approved: map to status: 'approved' (+ confidence), absent ⇒ suggested.
5. Keys & offsets. Verse key ("MAT 1:1") → Segment.id; cluster ranges → token offsets (step 2). Non-matching ranges (baseline drift) → stale handling.
6. Multi-file merge. One file per book per gloss language (Interlinear_<lang>_<book>.xml). Merge across books and languages into one project whose analysisLanguages union the GlossLanguages.
7. Link model & output. Attach the discovered major-language model project (Wire a model project into the suggestion engine (surface-form tier) #148) and any output project (Wire the output project (back-translation / adaptation) into export, not creation #149) to the imported project.

Dependencies

• Wire a model project into the suggestion engine (surface-form tier) #148 — major-language model linkage & the alignment path PT9 relies on.
• Wire the output project (back-translation / adaptation) into export, not creation #149 — output-project linkage for imported BT/adaptation setups.
• Lexicon extension — gloss-text / sense resolution (see the EntryRef/SenseRef gaps documented in interlinearizer.d.ts).
• bookTokenizer — the token grid clusters must map onto.

Out of scope

• The model-text suggestion engine itself (Wire a model project into the suggestion engine (surface-form tier) #148) and output/export pipeline (Wire the output project (back-translation / adaptation) into export, not creation #149) — this issue links them, doesn't build them.
• Exporting back to PT9 XML (separate, if ever needed).

Open questions

• Gloss resolution: import to glossSenseRef (lazy Lexicon resolution) vs. literal gloss text (from PT9 lexicon files) vs. both? What is the source of gloss text when the Lexicon extension lacks the entry?
• Where PT9 data comes from: a mounted PT9 project directory, individual XML files, or a PAPI-exposed PT9 project? This decides the import UI and how ScrTextName maps to a Platform.Bible sourceProjectId.
• Discovering model/output projects: PT9 stores the model text and BT/daughter project in project settings, not the interlinear XML. How do we read that config to populate Wire a model project into the suggestion engine (surface-form tier) #148/Wire the output project (back-translation / adaptation) into export, not creation #149 linkage on import?
• Should this go to user-questions.md for review outside the dev team? (per AGENTS.md UX-decisions guidance)

Size: L (multi-stage pipeline; the cluster↔token mapping and gloss resolution are each substantial).
Priority: P1 — this is the concrete goal ("fully support importing PT9 interlinearizations"); #148/#149 are its enabling pieces.

[imnasnainaec]

Related: #144

[alex-rawlings-yyc]

My thought on the open questions:

• Both; import ref to lexicon extension and literal gloss text to our model
• Seems outside of our jurisdiction on first read
• Not quite sure about this one
• Probably not a user-facing issue

[alex-rawlings-yyc]

Absorbing #144 ("Import PT9 project"), closed as a duplicate of this issue. Its only content beyond the title was a scheduling note:

Further support of RTL in core may be necessary to begin on this

Recorded here so it is not lost. Note that the note refers to RTL support in paranext-core, not to our own #97 — a blocked by #97 edge has been added as the closest trackable stand-in, but if the real constraint is upstream, that edge should be replaced with a link to the relevant paranext-core issue.

Assignee carried over from #144.