Investigate whether PT9 import can produce morphemes with blank forms

[alex-rawlings-yyc]

What prompted this

Reviewing the suggestion breakdown work (#213), a Devin finding pointed at breakdownOf in src/utils/suggestion-engine.ts. Verifying it turned up a reachable path where a TokenAnalysis can carry morphemes whose form is the empty string. The breakdown render is only where it happens to be visible; the record itself is the problem, and it originates in the PT9 importer.

What is established

parseLexemeKeyId accepts an id with an empty form. LEXEME_KEY_ID_RE in src/parsers/pt9/lexemeKey.ts uses a lazy, zero-width-capable form group, so parseLexemeKeyId('Stem:') returns { Type: 'Stem', Form: '' } rather than undefined (verified by a throwaway probe against HEAD).

Nothing on the bare-word path rejects that. src/converters/pt9/bareWordAnalyses.ts:122 copies key.Form onto the morpheme verbatim; its only rejection is a keys.length === 0 check, which counts keys rather than inspecting their content. So a word parse with ids ["Stem:", "Suffix:"] yields a TokenAnalysis with two blank-form morphemes.

The cluster path in src/converters/pt9/analysisMerger.ts is incidentally protected: clusterAnchoring.ts matches the concatenation of a parse's forms against the token's surface text, and an all-blank parse concatenates to '', which matches nothing. That protection is a side effect of anchoring, not a guard — a mixed parse like ["", "ba"] still anchors to token "ba" and persists the blank morpheme alongside the real one.

The in-app write paths cannot produce this. MorphemeEditor.tsx normalizes and blocks empty commits, and TokenChip.tsx independently re-filters with .filter(Boolean), so import is the only source.

What is not established

Whether PT9 ever actually emits an id of this shape. The reachability above is a code-path argument, not an observed input — no real project data was examined. That is the question this issue exists to answer.

Investigation

• Determine whether PT9 emits bare Type: ids in practice, and if so what they mean. src/parsers/pt9/pt9-xml.md documents the schema; real interlinear data from a PT9 project is the better evidence.
• Decide whether the parse boundary should reject an empty form regardless of the answer. The machinery already exists but is never reached: returning undefined routes the cluster path to the existing unparseableLexemeId drop counter, and makes bareWordAnalyses' keys.length !== ids.length check catch it too.
• If blank forms turn out to be meaningful rather than malformed, the fix belongs downstream instead, and the consumers below need to handle them deliberately.

Downstream consumers if such a record exists

• breakdownOf (src/utils/suggestion-engine.ts) joins forms on a space, so all-blank morphemes yield " " and a mixed parse yields " ba". Both are !== undefined, so the suggestion row renders a blank annotation and speaks an incomplete aria-label.
• MorphemeBox renders blank forms as empty grid cells that are still clickable edit triggers.
• morphemeIdentity (src/utils/analysis-identity.ts) folds morphemes into the payload identity, so all-blank breakdowns for different words collapse into one dedup bucket.

Patching any one of these in isolation would hide the symptom at one surface while leaving the malformed record in the store, which is why this is filed as an import-level investigation rather than a render fix.

[imnasnainaec]

WIP (to be deleted once finished and applied in the right place)
Destined for: comment on this issue (#285)

---

Investigated: blank forms are real PT9 data, and the reachable defect is next door

Paratext 9.5.110.9 with a live project (SIT, glossing, Interlinear_es), plus reflection probes of ParatextData 9.5.0.24 — the version paranext-core pins. Note SIT is linked to a FieldWorks project, and PT9's Interlinearizer opts into that link (InterlinearForm.LoadInterlinearizer calls IntegratedLexicalProvider.GetLexicon(project, allowFlexData: true)), so everything below is the FLEx-integrated path. The merge is additive — LexiconWithFlexAdditions holds both lexicons and user-created lexemes still land in Lexicon.xml — so the observations are not a non-FLEx fallback.

1. PT9's parse editor cannot produce a blank form

It splits on whitespace, not hyphens, trims the ends, collapses interior runs, and rejects an empty submit. Typing thanksgiving, thanks-giving, thanks-, -giving, -, thanks--giving, thanks - giving, thanks giving for "Thanksgiving" produced Form="" zero times — hyphen-bearing forms instead:

<Lexeme Type="Stem" Form="-giving" Homograph="1" />
<Lexeme Type="Stem" Form="thanks-" Homograph="1" />
<Lexeme Type="Stem" Form="thanks--giving" Homograph="1" />
<Lexeme Type="Stem" Form="-" Homograph="1" />

Same rule as ours (MorphemeEditor.tsx:112 collapses and trims; TokenChip.tsx:225 splits on /\s+/ and filters). A hyphen is an ordinary character in a form on both sides. This also settles how PT9's parse dropdown reads: slashes wrap each morpheme, so /-giving/ is one morpheme named -giving, never ["", "giving"].

2. But PT9 treats a blank form as first-class data

probe	result
new LexemeKey(scrText, "Stem:")	Form="", Homograph=1; .Id round-trips "Stem:"
new LexemeKey(scrText, "Stem::2")	Form="", Homograph=2
"Stem" / ":x" / ""	ArgumentException: Match on id failed
"Bogus:"	ArgumentException — unknown type names rejected
CreateLexeme(Stem, ""), FindOrCreateLexeme(Stem, null)	succeed, .Id = "Stem:" — no form validation in ParatextData

Hand-wrote a blank-form Stem with one sense, a mixed parse ["Stem:", "Stem:giving"], an all-blank parse ["Stem:", "Stem:"], a Stem::2, and a cluster ["Stem:", "Stem:giving"] into SIT. PT9 then:

• rendered the blank morpheme as an empty // slot and auto-selected its single sense as the gloss — the defaultSingle outcome pt9GlossSource already models;
• let that gloss be edited like any other, persisting the new sense onto the blank-form lexeme;
• wrote a sense selection for it into the cluster (<Lexeme Id="Stem:" GlossId="bb8QBehB" />);
• preserved every blank form across its own save, rewriting all three files with them intact.

A zero morpheme is ordinary PT9 data: meaningful, glossable, durable. LEXEME_KEY_ID_RE is faithful to PT9's grammar rather than lax, so rejecting Stem: at the parse boundary would be wrong twice over — it diverges from the grammar the doc comment claims to implement, and it discards data PT9 shows and lets users curate. Any rejection belongs at the conversion boundaries.

paranext-core delivers it on every inbound path (real reader, DummyScrText harness, Form="" seeded):

entry           id="Word:" type="Word" form="" hom=1
legacyAnalysis  word="runs"   analyses=["Stem:","Suffix:s"]
wordAnalysis    word="walked" analyses=["Stem:","Suffix:ed"]
cluster         verse=JAS 1:2 lexemes=["Stem:"]

A missing Form attribute differs: PT9 parses it to LexicalForm = null and the read fails with InvalidDataException: Could not read PT9 interlinear file 'Lexicon.xml'. One such lexeme costs the project every gloss — filed separately against paranext-core.

3. Withdrawing one claim from the issue body

The morphemeIdentity bullet does not hold. It has a single call site, inside analysesAreIdentical, which also compares folded surfaceText (analysis-identity.ts:100-105), so all-blank breakdowns of different words never share a dedup bucket.

4. What is actually reachable

PT9 anchors a cluster by <Range Index Length>; we re-anchor by matching concatenated forms to the token surface, because PT9's offsets index marker-bearing USFM. So PT9 accepts any parse the user enters, segmenting or not, while we require the concatenation to equal the surface. Three casualties in this one small project, all dropped as formMismatch:

PT9 cluster	concatenation	token	source
["Stem:thanks", "Stem:-", "Stem:giving"] @ 78-12	thanks-giving	thanksgiving	PT9 editor
["Stem:fr", "Stem:-om"] @ 28-4	fr-om	from	PT9 editor
["Stem:", "Stem:giving"] @ 78-12	giving	thanksgiving	hand-injected

PT9 displays all three. Tracking that as its own issue, since it outranks this one.

Disposition

1. The anchoring gap is the priority and is filed separately.
2. Blank forms: preserve, don't drop. No new drop counters — they are zero morphemes PT9 glosses, links senses to, and keeps.
3. Render them deliberately downstream. breakdownOf joins on a space, so a zero morpheme yields " giving" and a blank annotation with an incomplete aria-label; MorphemeBox gives an empty clickable cell with no affordance. Adopt a visible placeholder (PT9 uses an empty // slot) and a spoken label.
4. Editor round-trip: MorphemeEditor collapses and filters, so editing an imported analysis likely destroys its zero morpheme. Read from code, not yet observed in the app.

Next steps

• Confirm (4) in the running app before acting on it.
• Run the importer end-to-end over the real SIT project and publish the report — turns "we'd drop these" into counts.
• Decide preserve-vs-strict for blank forms. Note the asymmetry: the anchoring fix is required by the two editor-produced cases regardless, and once forms need not concatenate to the surface, blank forms ride along free — strictness costs extra code.

SIT still carries the injected test data plus PT9's own rewrites, so it is a usable fixture for the above.

[imnasnainaec]

WIP (to be deleted once finished and applied in the right place)
Destined for: comment on #130

---

Real PT9 data and PT9's own code answer both halves of this issue's implicit model, and one row of the table is wrong (from #285; Paratext 9.5.110.9, ParatextData 9.5.0.24).

PT9 does derive indicators from type

Lexeme.DisplayString decorates the stored form:

type	form om	form -om	form ""
Prefix	om-	-om-	""
Suffix	-om	--om	""
Infix	<om>	<-om>	""
Phrase / Word / Lemma / Stem	om	-om	""

Three consequences:

• Infix is <om>, not -om-. The table's Infix row disagrees with PT9.
• No doubling guard. Suffix + form -om gives --om in PT9 itself.
• A blank form is never decorated — which is why an imported zero morpheme renders as a bare empty slot.

The type inventory is 7, not 20

PT9's LexemeType is Phrase, Word, Lemma, Stem, Prefix, Suffix, Infix. The other 13 rows here (BoundRoot, Circumfix, Simulfix, Suprafix, the interfixes, clitics, Particle, DiscontiguousPhrase) have no PT9 equivalent, and cannot arrive from a PT9 file: even a FieldWorks-backed lexeme reaches PT9 through WrappedFlexLexeme.Type, which returns that same 7-value enum, so FLEx's richer inventory is lossy at the boundary. Their indicator scheme is ours to define, not to import — worth stating explicitly here, because it changes what this issue is for.

Forms carry indicators too

PT9's parse editor splits a typed breakdown on whitespace only and assigns Stem to every morpheme, so users encode affixhood by typing the character into the form. All written by PT9 itself:

<Lexeme Type="Stem" Form="-giving" Homograph="1" />
<Lexeme Type="Stem" Form="thanks-" Homograph="1" />
<Lexeme Type="Stem" Form="-" Homograph="1" />
<Lexeme Type="Stem" Form="-om" Homograph="1" />

Project-wide PT9 wrote only Word (19), Stem (10), Phrase (1) — no affix type at all. So an imported morpheme's type does not predict whether its form already carries an indicator, and rendering one over -om gives --om. Whatever normalization adds or strips an indicator should be the same one import anchoring uses to compare a parse's concatenated forms against a token surface, or the two will disagree about what a form is — today fr + -om fails to anchor to "from" and is dropped (see #285 and the anchoring issue).

Next steps

The project I tested is FLEx-linked, and the Interlinearizer does consult that link (InterlinearForm.LoadInterlinearizer passes allowFlexData: true), but no FLEx-backed lexeme was ever selected in it — so the affix-typed path is still untested. Two tests will close it:

• Add a suffix entry in the linked FieldWorks project, select it for a word in the Interlinearizer, and read the id PT9 writes into Interlinear_<lang>_<BOOK>.xml. That shows whether affix types reach the files at all, whether the form carries the hyphen, and whether a FLEx-backed id is even Type:Form-shaped.
• Run PT9's morphology guesser and inspect the types it writes — the manual editor only ever wrote Stem.

Suggested body wording

PT9 supports only 7 of these types (Phrase, Word, Lemma, Stem, Prefix, Suffix, Infix) and decorates three of them itself via Lexeme.DisplayString: Prefix om-, Suffix -om, Infix <om>. The remaining rows cannot come from a PT9 file, so their indicators are ours to define.

Imported forms may already carry an indicator character: PT9 assigns type Stem while the user types the hyphen into the form — observed Stem:-om, Stem:-giving, Stem:thanks-, Stem:-. So an indicator cannot be derived from type alone for imported data, and rendering one must not double it (PT9 itself does not guard against --om).

[imnasnainaec]

WIP (to be deleted once finished and applied in the right place)
Destined for: new issue in this repo - PT9 cluster anchoring

---

Title

PT9 import drops any parse whose morpheme forms don't concatenate to the token surface

Body

PT9 anchors a cluster to a word with <Range Index="78" Length="12" />. We ignore those offsets — correctly, since they index marker-bearing USFM rather than our baseline text — and re-anchor by matching the concatenation of a parse's lexeme forms against the token's folded surface (clusterAnchoring.ts:181; normalizeSurfaceForm NFC-folds and lowercases only, analysis-identity.ts:35). The range survives as ordering and a proportional prior.

The asymmetry is the bug: PT9 never needs the forms to know which word a parse belongs to, so it accepts any parse the user enters, segmenting or not. We require the concatenation to equal the surface, so a legal PT9 analysis anchors to nothing and drops as formMismatch.

Observed casualties

One small test project, Paratext 9.5.110.9:

PT9 cluster	concatenation	token	how it got there
["Stem:thanks", "Stem:-", "Stem:giving"] @ 78-12	thanks-giving	thanksgiving	typed thanks - giving in PT9's parse editor
["Stem:fr", "Stem:-om"] @ 28-4	fr-om	from	typed fr -om in PT9's parse editor
["Stem:", "Stem:giving"] @ 78-12	giving	thanksgiving	hand-written (zero morpheme, see #285)

PT9 displays all three correctly. The first two came from ordinary use of PT9's editor, which splits a typed breakdown on whitespace only — so a hyphen is an ordinary character inside a morpheme form, and users encode affixhood that way (#130).

Why it matters

A deliberate user analysis disappears from the import. The only trace is a formMismatch count in Pt9ImportModal, which shows the top two reasons and never says which verses or words were lost (#140).

Candidate fixes

• Separator folding. Fold indicator characters (-, =, ~) out of both sides before comparing. Recovers rows 1 and 2 exactly. Must share its normalization with Handle morph-type indicators #130's indicator rendering, or the two disagree about what a form is.
• Range-derived fallback. When no concatenation matches, anchor by the range's ordering and proportional prior among unclaimed tokens instead of dropping. Recovers non-segmenting parses such as row 3. Riskier — it can place a parse on the wrong word.
• Surface the drop. Whatever the matcher does, make an unanchored cluster reportable per verse rather than a bare count (Report analysis-invariant violations instead of silently repairing them #140).

Relations

• Investigate whether PT9 import can produce morphemes with blank forms #285 — origin; also establishes that blank forms are legal, glossable PT9 data, so row 3 is not corruption.
• Handle morph-type indicators #130 — shares the normalization question; its table is also wrong about Infix and about PT9's type inventory.
• Report analysis-invariant violations instead of silently repairing them #140 — the reporting half.
• Not Decide the token/segment identity re-anchoring strategy #136, which is about re-keying our own token refs across text edits. This is about matching PT9's clusters onto tokens at import time.

Next steps before choosing a fix

1. Run the importer end-to-end over the real test project and publish per-reason counts, so the fix is chosen against numbers.
2. Test whether PT9's own Range survives a text edit: insert a word before an analyzed one, reopen the Interlinearizer, and see whether PT9 keeps the analysis on the right word. If PT9 re-anchors reliably, the range fallback is defensible; if PT9 mis-anchors too, restrict the fix to separator folding.

Size: M — matcher change plus fixtures.
Priority: P1 — silently loses user data from every import that meets it.

[imnasnainaec]

WIP (to be deleted once finished and applied in the right place)
Destined for: new issue in this repo - FLEx-linked projects

---

Title

A FLEx-linked PT9 project imports without its lexicon, and hasAssociatedLexicalProject is unused

Body

When a PT9 project is associated with a FieldWorks project, its lexicon does not live only in Lexicon.xml, and PT9's Interlinearizer reads the merged view. We import the files alone and never check the flag that says so.

Established (Paratext 9.5.110.9, ParatextData 9.5.0.24)

• The Interlinearizer opts in. Paratext.Interlinear.InterlinearForm.LoadInterlinearizer calls IntegratedLexicalProvider.GetLexicon(project, allowFlexData: true) and ParatextLexicalAnalyser.Get(project, true) — the model text gets false — plus an explicit RequiredFieldWorksProjectNotAvailable guard. UseFileImplementation returns "files only" when allowFlexData is false, so this is a real opt-in.
• The merge is additive. LexiconWithFlexAdditions holds both paratextLexicon and flexLexicon; EncodesKeyInId delegates to the Paratext half. User-created lexemes still land in Lexicon.xml, so a FLEx-linked project has both halves.
• FLEx-backed lexemes are FieldWorks' own. WrappedFlexLexeme delegates Id, Type, LexicalForm, and DisplayString to the FieldWorks Lexeme.
• We read neither half of FLEx. paranext-core reads only Lexicon.xml, WordAnalyses.xml, and the interlinear book files, and its own docs say so: "hasAssociatedLexicalProject is true when the project's lexicon lives in an associated external lexical project (e.g. FieldWorks) rather than in Lexicon.xml, so an absent lexicon does not mean the project has no gloss data." FLEx word analyses (WordAnalysesWithFlexAdditions) are equally invisible.
• We ignore the flag. hasAssociatedLexicalProject appears in this repo only as a test-fixture default (convertPt9Project.test.ts:19).

Consequences

• Morphemes whose lexeme lives in FLEx import unglossed, and entryRefsUnresolved / senseRefsUnresolved climb with no explanation available to the user.
• Parses that exist only in FLEx never appear at all.
• If a FieldWorks lexeme's id is not Type:Form, every cluster selecting one drops as unparseableLexemeId; if the id happens to contain a colon, it parses into a wrong Type/Form. Untested — see next steps.

Next steps

1. Determine the id shape. In a FLEx-linked project, select a FieldWorks-backed entry for a word in PT9's Interlinearizer and read the id written into Interlinear_<lang>_<BOOK>.xml. This decides whether FLEx-linked projects are partially or almost entirely unimportable, and it gates everything else here.
2. Depending on (1): reject with an explicit reason, or resolve FLEx refs through the platform's Lexicon extension as paranext-core's docs suggest.
3. Regardless of (1): read hasAssociatedLexicalProject and tell the user their lexicon is external, rather than reporting unresolved refs as if the data were missing.

Note for whoever picks this up: the project used in #285 is FLEx-linked, so those findings are already on the merged path — but no FLEx-backed lexeme was ever selected in it, which is why (1) is still open.

[imnasnainaec]

WIP (to be deleted once finished and applied in the right place)
Destined for: new issue in paranext-core - absent Form attribute

---

Title

A Lexeme element with no Form attribute fails the whole PT9 lexicon read

Body

One malformed lexeme costs a project every gloss it has.

Behavior

Lexicon.xml containing <Lexeme Type="Stem" Homograph="1" /> — no Form attribute — fails the read:

InvalidDataException: Could not read PT9 interlinear file 'Lexicon.xml'

Pt9InterlinearReader then returns no lexicon at all, so every gloss in the project is unavailable to consumers, not just the one entry.

Why it happens

PT9 itself is tolerant here: LexemeKey parses the missing attribute to LexicalForm = null and its Id getter still composes "Stem:". CleanPt9LexiconData passes that null into scrText.Normalize(...), and the resulting NullReferenceException is contained by ReadTyped — deliberate, per its own doc ("PT9's dictionary reader admits nil entries that the projection code cannot carry, and such a file is corrupt") — but the blast radius is the file, and that file is the project's whole lexicon.

An empty Form="" behaves differently and correctly: it survives, arriving as form: "" with id "Stem:", matching what PT9 does with it.

Reproduction

Seed a project through the DummyScrText harness with the lexeme above and call GetPt9InterlinearData. Verified against ParatextData 9.5.0.24.

Why it's worth fixing

• PT9 opens the same file without complaint, so a project usable in Paratext 9 is unusable through this interface.
• Pt9LexiconEntry.Form is declared non-nullable in both the C# record and the TypeScript types, and that contract currently holds only because this read fails. Nothing names that dependency, so a future tolerance fix would silently start emitting form: null to consumers.

Suggested fix

Normalize an absent Form to "" at the parse boundary, matching PT9's own treatment of an empty form, so the non-nullable type holds by construction rather than by failure. Failing just that entry with a count is the alternative; failing the whole file is the outcome to avoid.

Found while investigating blank morpheme forms for sillsdev/interlinearizer-extension#285.

[imnasnainaec]

WIP (to be deleted once finished and applied in the right place)
Destined for: a one-line doc fix in src/parsers/pt9/pt9-xml.md

---

pt9-xml.md:164 says duplicate lexicon entry keys cause a parse error. They don't: PT9's own deserializer silently collapses them, keeping the last occurrence (probed at ParatextData 9.5.0.24 - two identical blank-form keys reduced 18 entries to 17 with no error). paranext-core's reader docs agree: "a duplicate verse reference, lexicon key, or wordform keeps the last occurrence."

--- a/src/parsers/pt9/pt9-xml.md
+++ b/src/parsers/pt9/pt9-xml.md
@@ -164 +164 @@
-    - **`Lexeme`** (required): The entry's key as an attribute triple. A missing key element causes a parse error; duplicate keys (treating an absent `Homograph` as homograph 1) cause a parse error.
+    - **`Lexeme`** (required): The entry's key as an attribute triple. A missing key element causes a parse error; duplicate keys (treating an absent `Homograph` as homograph 1) silently keep the last occurrence.

The missing-key-element half of that sentence is correct and stays. Not verified through the full reader, only the deserializer, so confirm before relying on it for anything but the doc text.