Skip to content

Warn when persisted segment boundaries are lost on load (reversification / source edits) #156

Description

@alex-rawlings-yyc

Problem

Segment boundaries are persisted as a SegmentationDelta anchored to token refs of the form "<verse SID>:<charStart>" (e.g. "PSA 3:2:14"). On load, resegmentBookeffectiveStarts/normalize (src/utils/segmentation.ts) silently drop any anchor whose ref isn't found in the freshly-tokenized book, reverting that region to the default one-segment-per-verse segmentation.

This is the intended, non-destructive degradation — but it's silent. When the source text is reversified (classically the Psalms verse-0↔verse-1 superscription shift, which renumbers SIDs) or edited upstream (shifting char offsets), a user's hand-placed boundaries vanish with no signal. They lose work and may not notice.

Decisions made by Alex (up for debate)

Detect-and-warn — not remap. Reframed as a general "lost boundaries" safety net (reversification is just one trigger; ordinary edits are the same event to an anchor):

  • Keep the existing silent drop (non-destructive). Add a warning.
  • Detection signal: any non-empty SegmentationDelta that loses anchors on load. Do not try to isolate reversification from ordinary edits — this deliberately sidesteps the host's versificationStr, which is documented as unreliable (InterlinearNavContext.tsx:51). A project with default/empty segmentation never warns.
  • Do NOT prune the dead anchors from storage. Keeping them preserves the model's revival property: if the source reverts and the token ref reappears, the boundary comes back on its own.
  • Explicitly out of scope: remapping boundaries across versifications, and any versification-mapping API. That's a much larger effort with poor cost/benefit for a rare event; not this issue.

Implementation notes (from a prior investigation)

  • Set-difference site: the book memo in src/components/InterlinearizerLoader.tsx (~line 272) is where the freshly-tokenized verseBook and the delta (draft?.segmentation) are both in scope. Detection must run in a useEffect, not the pure book memo (notifications are a side effect).
  • Add a new pure helper (e.g. lostAnchors(verseBook, delta): string[]) that intersects [...removedVerseStarts, ...addedStarts] against the book's token refs and returns the missing ones. Do not make effectiveStarts/normalize report drops — keep segmentation.ts pure. (Note: bookLookups in that file already computes an all token-ref set you can mirror.)
  • Delivery: papi.notifications.send({ message, severity: 'warning' }) (PlatformNotification; Severity = 'info' | 'warning' | 'error'). No frontend code sends notifications yet, but the mock already exposes papi.notifications.send.
  • Copy: generic — "source text changed," not "versification changed" (we can't reliably attribute the cause). Optionally include a count. Add a key to contributions/localizedStrings.json, resolve via useLocalizedStrings, interpolate the count with .replace('{count}', () => …) (pattern in MorphemeBox.tsx / TokenChip.tsx).
  • Dedup: the extension re-tokenizes on many events (duplicate GetText re-fetches), so a naive per-load warning would spam. useInterlinearizerBookData returns a reference-stable verseBook across duplicate USJ payloads (it dedupes by JSON), making the verseBook reference a natural dedup anchor (track the last-warned one in a useRef).

⚠️ Open decision — resolve before implementing

Dedup semantics were not settled:

  • (A) Per-tokenization: warn again on each genuinely new re-tokenization that still has lost anchors; duplicate re-fetches suppressed via the reference-stable verseBook. (Better matches "warn until something meaningful changes"; recommended in the prior investigation.)
  • (B) Strict once-per-session: a projectId+book Set that never re-warns for that book again this session — even across a reversify → revert → reversify cycle.

Pick one before building.

Acceptance / tests

  1. A lossy non-empty delta → warning sent (positive).
  2. A delta whose anchors all still resolve → no warning (negative).
  3. Default/empty segmentation → no warning (negative).
  4. Repeat render with the same verseBook → no repeat warning (dedup).
  5. Dead anchors are not pruned — the persisted/draft segmentation is untouched by the warning path.

100% coverage enforced; npm run lint-fix at the end.
`

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions