PR2b: read-side flip + ensureLoaded + UI config + flag ON (#2337)

[paynejd]

Summary

PR2b of the unified candidate/concept data model refactor for OpenConceptLab/ocl_issues#2337. Flips reads from the legacy allCandidates shape to the new rowMatchState + conceptCache model via structured tuples, lands $resolveReference-backed $lookup as ensureLoaded, surfaces canonical-URL / namespace / bridge_repos[] in the project config UI, debounces the rerank trigger, and turns UNIFIED_MODEL_ENABLED on as the last step.

Builds on #16 (PR1) and #17 (PR2a). Spec: plans/unified-mapper-model.md.

---

⚠️ DO NOT MERGE TO main UNTIL STAGING VERIFICATION COMPLETES

This PR turns UNIFIED_MODEL_ENABLED to true. Bridge, scispacy, and AI Assistant code paths only attach at production build time via PRIVATE_PACKAGES_GIT and could not be exercised in local OSS dev (BridgeMatchStub.canBridge() === false). They must be verified on the staging deployment before this PR is merged to main.

Required staging checks (details in the test plan below):

1. Bridge cascade fan-out end-to-end (CIEL → LOINC)
2. Scispacy $resolveReference flow
3. AI Assistant recommendation with v2 payload

If any of these fail on staging, hold the merge and triage before promotion to prod.

Suggested workflow: merge this branch to a staging branch (or however the staging build is triggered), exercise the unchecked items below on the staging URL, and only then promote to main.

---

What's in this PR

Stage A — write-side foundations

• Fix latent PR1 keying bug in mergeIntoRowMatchState: def.url→def.key, cr.concept_url→cr.concept_key. Was silent because the flag was OFF.
• Extract normalizeLegacyAllCandidates into normalizers.js and call it from fetchAndSetProject so reloaded v1 projects backfill rowMatchState under the flag. Necessary precondition for the flag flip; PR3's schema-v2 load subsumes this.

Stage B — ensureLoaded over $resolveReference

• ensureLoaded(conceptKeys): batched POST /$resolveReference/?namespace=... followed by per-resolved concept fetch, with in-flight Promise dedup keyed by concept_key. Replaces lookupCandidates / lookupCode. Direct ocl_url fetch when available, else canonical resolution path.

Stage C — UI config

• MultiAlgoSelector: new required canonical_url field for custom algos with URL validation; bridge canonical_url surface; lookup_required checkbox removed (no longer a property of the algorithm).
• ConfigurationForm: target_repo canonical URL displayed with "Auto-derived" badge when canonical_url_source === 'derived'; bridge_repos[] summary with same badge convention; Resolution Namespace under an Advanced disclosure (defaulted to project owner).
• namespace state plumbed through MapProject; persisted on save/load.

Stage D — read flip to structured tuples

• New viewBuilders.js (pure JS, node:test loadable): buildAlgorithmRowViews, buildQualityRowViews, candidateToRowView, sortRowViews, conceptForMapping, getScoreDetails, resolveAICandidateID.
• Candidates.jsx: getCandidates rewritten to iterate RowState.candidates (algo view) / RowState.concept_rows (quality view); passes {candidate, conceptDefinition, conceptRow, bridgeConceptDefinition?, bridgeChildren?, contributingCandidates?} tuples downward.
• Concept.jsx: receives tuples directly; ad-hoc mapping.cascade_target_* extraction removed; bridge children nested under their parent bridge in algorithm view, surface as their target concept in quality view.
• Score.jsx: reads conceptRow.rerank_score + candidate.score; thin wrapper over the pure getScoreDetails adds the bucketColor UI affordance.
• setAutoMatched and setStateViews rewritten to read from rowMatchState + conceptCache via a pickTopRowView helper that filters by target_repo.canonical_url so bridge intermediaries are excluded from auto-mapping.
• Bulk match path (processBatch.then) now normalizes and merges into rowMatchState before setStateViews consumes it.

Stage E — rerank debounce + in-flight check

• scheduleRerank(rowIndex): debounce timer per row + in-flight Set + rerun-needed flag for the case where new ConceptRows arrive while an earlier rerank is still flying. Triggered from mergeIntoRowMatchState whenever any ConceptRow has rerank_score === undefined. Replaces the "wait for every algo" implicit batching.
• Rerank writes rerank_score directly onto ConceptRow; legacy allCandidates write preserved for save format compatibility (until PR3).
• Normalizer Stage E lift: newConceptRow now extracts result.search_meta.search_normalized_score onto ConceptRow.rerank_score so the single-algo reranker:true $match path doesn't need a separate $rerank round-trip.

Stage F — AI Assistant response resolution

• resolveAICandidateID: prefer primary_candidate.concept_key resolved via conceptCache[concept_key].reference.code; falls back to PR2a's canonical_reference.code shim, then legacy concept_id / id. Used in both Candidates.jsx's UI highlight match and MapProject's CSV export.

Stage G — flag flip

• UNIFIED_MODEL_ENABLED = true at MapProject.jsx.

Test plan

✅ Local verification (done)

• npm test — 79/79 (was 27, +52 new tests)
• npm run eslint — clean
• NODE_ENV=production npm run build — green
• docker compose build — green (production target)
• Save/load round-trip end-to-end against prod API from local dev:
  • 4 matched rows preserved across save → hard refresh → reload
  • rerank_score values byte-identical (e.g. 81.87800645828247 → 81.87800645828247)
  • allCandidates envelopes restored with full results arrays
  • rowMatchState rehydrated with same candidate/concept_row counts
  • No $match re-fire on row open for previously-matched rows
• Single-algo ocl-search: candidates render, scores show, mapping persists
• Configuration form: canonical URL badge, namespace under Advanced, bridge canonical surfaces
• Bridge fan-out at the view layer unit-tested (data shape exercised via fixtures; live algo waits for staging)

🟡 Staging verification — REQUIRED BEFORE MERGE TO main

These code paths only attach at production build time. Each must pass on staging before this PR is promoted to prod.

• Bridge (CIEL → LOINC) — per PR2a's verification notes:
  • rowMatchState populates with bridge intermediary + cascade targets at correct canonicals (CIEL: https://CIELterminology.org, LOINC: http://loinc.org)
  • AI payload bridge_context populates with target_concept_keys pointing at the LOINC entries in recommendable_concepts
  • Quality view shows the LOINC target with both contributing candidates while the CIEL bridge appears as its own bucketed concept
  • Selecting a bridge_child candidate maps to the LOINC target (not the CIEL intermediary)
  • Auto-Match never selects a bridge intermediary (the pickTopRowView filter by target_repo.canonical_url excludes them)
• Scispacy — $resolveReference/?namespace=... fires when scispacy returns LOINC codes; per-concept GET requests follow the resolved repo URL; concept cards fill in with names/descriptions (not just bare IDs)
• AI Assistant — request body shows v2 fields (recommendable_concepts + bridge_context + target_repo) alongside legacy candidates; payload_version: 'v2' present; resolveAICandidateID highlights the recommended candidate in the list with the sparkle icon

Regression check (also on staging)

• Auto-match a project across all rows (multi-algo + bridge) → verify Mapped/Ready/Unmapped counts match expectations
• Open AI Assistant recommendation on a multi-algo + bridge row → verify primary_candidate is from the target repo (not a bridge intermediary)
• Save → reload an existing legacy v1 project → verify candidates render identically (the normalize-on-load path)

New tests (52)

• __tests__/views.test.js (21) — algorithm grouping, quality grouping with bridge fan-out, multi-algo convergence, multi-bridge namespaces, sort orderings, missing-cache-entry handling.
• __tests__/normalizeLegacyAllCandidates.test.js (11) — load-path backfill including bridge cascade fan-out, multi-algo dedup, identity injection fallback, dedup across rows.
• __tests__/viewHelpers.test.js (20) — getScoreDetails bucketing for recommended/available/low_ranked, conceptForMapping legacy projection incl. bridge_child, resolveAICandidateID resolution chain.

Not in this PR (PR3 scope)

• Drop legacy allCandidates state
• Schema-v2 save format + normalizeLegacy.js for v1 backward-compatible load
• Drop legacy candidates field + payload_version + concept_id/id shims from AI Assistant payload
• Pagination append path under the unified model

Coordination

• The ocl-ai-assistant prompt template revision (separate session) reads the v2 payload fields and structurally excludes bridges from the recommendation pool. PR2b can ship without it — legacy candidates is still in the payload (PR2a Option A).
• The _to_essential allow-list addition in ocl-ai-assistant for recommendable_concepts + bridge_context ships alongside the prompt template revision.

🤖 Generated with Claude Code

[paynejd]

Review-feedback follow-up commit (d7180d7)

Addresses correctness gaps + UX polish found while testing the read-flip locally against a pre-PR2b legacy project. No new tests required — covered by the existing 79/79 suite. Bridge / scispacy / AI Assistant staging verification still gates merge per the original PR description.

Correctness

• Language zh added #1 — pickTopRowView canonical filter: now reads target_repo.canonical_url from buildProjectContext(), the same source the normalizer uses to stamp ConceptDefinition.reference.url. Eliminates the derived-vs-explicit mismatch that could silently empty auto-match.
• [Snyk] Security upgrade nginx from 1.19-alpine to 1.29.4-alpine #2 — Custom-algo canonical_url: was decorative — user could enter a URL but nothing wired it into concept_identity. Centralized injection in new ensureConceptIdentity() helper (algorithms.jsx), used by getAlgoDef, bulk processBatch.then, lookupCandidates, and the legacy load path. Also adds Save-time validation: Save button disabled and an Alert lists offending algos when canonical_url is missing/malformed.
• OpenConceptLab/ocl_issues#2315 | Updated Search Highlights dialog #7 — Rerank fail-loud: matchRerankResultToKey throws on canonical-identity miss instead of silently falling back to fuzzy ocl_url / id+source matches. The rerank loop collects errors, logs a single summary entry, and surfaces an alert to the user.
• OpenConceptLab/ocl_issues#2388 | custom encoder model for reranking #8 — State-driven ensureLoaded: mergeIntoRowMatchState now auto-triggers ensureLoaded for newly-arrived ConceptDefinitions with lookup_status !== 'full' (bridge cascade targets, sparse algorithm results). Uses a forward-ref pattern mirroring scheduleRerankRef.
• OpenConceptLab/ocl_issues#2485 | Copy project #13 — Missing target canonical on load: when a saved project has match data but no resolvable target canonical, surface an error alert and force-open the configuration drawer instead of silently rendering an empty candidate list.

Click-path wiring (rowView ↔ legacy concept shape at the seam)

• Candidate row click → details modal: Candidates.jsx decorates rowViews with top-level id/url/version_url so SearchResults.handleRowClick can resolve clicks back to the rowView and fire onShowItemSelect.
• Score chip click → Matched Attributes: Concept.jsx passes the conceptForMapping projection (not the raw tuple) to setShowHighlights so SearchHighlightsDialog finds search_meta.search_highlight.
• Score chip unified score regression: viewBuilders.getScoreDetails now accepts either the {candidate, conceptRow} tuple or the legacy {search_meta} shape so the dialog works regardless of which form the caller sends.

CSV export

• __map_repo_url__ and __map_repo_id__ gated on concept (the mapSelected entry) so they're blank on unmapped rows — matches the rest of the __map_* namespace. Dropped the asymmetric project-repo fallback that predated PR2b.

UI polish + i18n

• Compact canonical-URL caption under target repo: info icon + monospace URL with ellipsis truncation + tooltip + muted · derived suffix. Bridge repos use the same line per bridge. Replaces the orange warning-colored chip.
• All PR2b-introduced t() calls switched from t(key) || 'default' (which never falls through — i18next returns the truthy key string when missing) to t(key, 'default') / t(key, {defaultValue, ...interp}).
• Added the PR2b string set to en/translations.json. resolution_namespace_description now spells out the default behavior: "When blank, defaults to {{owner}}" — answers a reviewer question about whether namespace defaults to global or to the project owner (it's the project owner). Other locales inherit via fallbackLng: 'en'.

Verified

• 79/79 unit tests pass, eslint clean, NODE_ENV=production npm run build green.
• Live in-browser: legacy project loads, candidates render, row clicks open details, score chips open the highlights dialog with unified score + matched attributes, Auto-Match maps a row whose __map_algorithm__ correctly serializes to ocl-semantic via the unified-model projection.

🤖 Generated with Claude Code

[snyaggarwal]

Tried against 2 rows with CIEL bridge algo for ICD-11-WHO

1. Score is not coming on top

2. mapping to bridge target (ICD) is not getting logged 3. Candidates table view not complete 4. Target code on left is always empty 5. Fetch More in Candidates is not doing anything (no calls in network, infinite loading candidates...)

6. Auto Match not firing any calls with just bridge algo. With two algos (bridge + ES Token Search) it did.

7. Map a CIEL candidate from Unified candidates list and then go to algorithm view of candidates, no indicator in candidates that which CIEL concept was mapped

[snyaggarwal]

Another thing:
This PR may send the rows to rerank API with no name (None) because bridge targets lookup may fail.
Reranker will assign -100000 score to those and they look weird in candidates list

[snyaggarwal]

Trying 8 rows with only Semantic Algo, target CIEL -- Automatch worked here.

1. every candidate is duplicated:

2. Fetch More button fires the call but doesnt update the candidates list

[paynejd]

Review-feedback follow-up commit (2d96c09) — @snyaggarwal

Thanks for the detailed report. Addressed 6 of 10 in this commit; 2 are expected to resolve transitively; 1 needs your diagnostics.

#	Issue	Status
9	Every candidate duplicated	Fixed — mergeIntoRowMatchState now drops same-algo candidates on each invocation (mirrors the legacy reject(...) on allCandidates). Concept_rows pruned when no surviving candidate references them.
5/10	Fetch More re-fires + doesn't update	Fixed — pagination append branch feeds appended results into rowMatchState via a new {append: true} option so earlier pages stay put.
4	Target code on left always empty	Fixed — Concept.jsx now wraps legacy concept-shape input through a legacyToRowView() synthesizer at the top. One change covers Target Code column, decision tables, Search results, anywhere Concept is invoked with a legacy projection.
7	Mapped CIEL concept indicator missing in algorithm view	Fixed — bridge intermediary's algoScoreFirst row was hard-coded isSelectedForMap={false}. Now passes the real function so the Mapped indicator renders when the user mapped the intermediary from Unified view.
8	Rerank sent rows with no name → -100000	Fixed — buildRerankRowsForRow filters out ConceptRows whose ConceptDefinition has no usable display_name (typically pending bridge cascade targets). scheduleRerank stays re-eligible so the row reranks once ensureLoaded completes.
2	Bridge target mapping not logged	Fixed — _onMap no longer gates the log entry on concept?.url. Bridge cascade targets can lack ocl_url until $resolveReference resolves them; log now fires on url || id and surfaces object_id as fallback.
1	Score not on top (bridge case)	Expected to resolve transitively via #8: once bridge targets get rerank_scores from the (now non-poisoned) rerank pass, they sort to the correct position. Worth re-checking after this commit.
3	Candidates table view not complete	Expected to resolve transitively via #4: same root cause (Concept bailing on legacy shape). Worth re-checking.
6	Auto Match doesn't fire calls with bridge-only algo	Pending — need diagnostics from you. Code review doesn't show a smoking gun; fetchBulkBridgeCandidates looks correct and canBridge should be true in your env. When you re-test, could you share: (a) the browser console + network tab during the bridge-only Auto Match, (b) whether the row was Unmapped when you clicked it (the bulk path filters rowStatuses.unmapped when autoMatchUnmappedOnly is true — a stale readyForReview entry would skip the row silently), and (c) whether bridge-only with autoMatchUnmappedOnly=false works?

Notes on the duplicate fix (#9)

The merge now does replace-by-algorithm, not stack. If you had any reliance on the old behavior (e.g. multiple algorithms returning the same concept building up cumulative candidates), that still works — candidates from different algorithms are unaffected; only same-(rowIndex, algorithm) re-runs replace. Multi-algo convergence in Quality view continues to dedup by concept_key.

Notes on the pagination fix (#10)

mergeIntoRowMatchState now takes an options arg: mergeIntoRowMatchState(rowIndex, normalized, {append: true}). The append path skips the same-algo drop and the orphan concept_row pruning. All existing callers default to append=false; only the offset > 0 branch in fetchAllCandidatesForRow.onResponse uses append=true.

🤖 Generated with Claude Code

[snyaggarwal]

11. (with bridge) on candidates visit, it makes rerank call every time.

• rerank -> save -> reload -> row click -> rerank again
• rerank -> close -> open -> rerank

12. No candidates rendered even when api returned candidates (this is the row I never ran earlier to get candidates)

13. AutoMatch ran and got the candidates but dint selected map when all criterias for top candidates matched

14. Score is still NAN

[paynejd]

Round-2 review fixes (104a7c9) — @snyaggarwal

Addresses the 4 issues you reported on the previous commit plus 4 from local testing. Net result: Quality view shows only target-repo concepts (bridge intermediaries appear only in Algorithm view, per spec), no more NaN% chips, no rerank re-fire loop, table view renders.

#	Issue	Status
11	Rerank fires on every row visit	Fixed — scheduleRerank now gates on whether the ConceptDefinition has a usable display_name. Rows that the previous-commit's buildRerankRowsForRow filter skips no longer stay eligible forever. writeConceptCachePatch chains into scheduleRerank when a lookup transitions to having a display_name, so once ensureLoaded completes for a pending bridge cascade target, the row gets a score.
14	Score shows NaN%	Fixed — getScoreDetails returns rerankScore: '' / algoScore: '' when no numeric value is present. The Target Code panel's chip will now render empty (or you can choose to hide it) rather than NaN%.
13	AutoMatch ran but didn't auto-select	Expected to resolve transitively via #11 — cascade targets now receive rerank scores via the chained scheduleRerank, so pickTopRowView's sort no longer parks them at -1. Worth re-testing.
12	First-time row click w/ bridge: API returned data, UI shows none	Fixed — buildProjectContext now falls back to '/orgs/CIEL/sources/CIEL/' when bridgeAlgo.target_repo_url is missing (matches the MultiAlgoSelector placeholder). Without this, the normalizer's resolveReference returned null for every bridge primary and rowState ended up empty. Auto-Match happened to work because of state-ordering quirks; manual click surfaced the underlying bug.
	Quality view should only show target-repo concepts	Fixed — buildQualityRowViews output is now filtered by conceptDefinition.reference.url === target_canonical (sourced from buildProjectContext()). Bridge intermediaries are gone from this view; they only appear in Algorithm view as parent groupings for their cascade children.
	Algorithm view shows Map button on every candidate	Fixed — Map action gated on the same target_canonical everywhere (action column + Concept.jsx Item). Bridge intermediary's algoScoreFirst row reverts to the placeholder spacer it had before; cascade children retain their Map action since they're in target_repo.
	Table view broken for all candidates	Fixed — rowsForTable now spreads the conceptForMapping legacy projection AND the rowView fields. Table cells read legacy fields (names, descriptions, source, etc.); card view renderer reads the tuple. Click-lookup still works (id/url/version_url set explicitly at the end).

Things still pending diagnostics

• S6 (Auto Match no API calls with bridge-only algo) — your console/network log would help. If OpenConceptLab/ocl_issues#2483 | Throttling Error UI #12's target_repo_url-fallback fix happens to also resolve S6 (same bridgeAlgo state inspection), please confirm.

Re-test request

When you have a minute, would you re-run your previous flows and confirm:

1. Bridge cascade in Algorithm view — bridge intermediary visible, no Map button on it, cascade children below with their Map buttons.
2. Quality view — only target-repo concepts (no CIEL intermediaries).
3. Open a row that hasn't been matched yet → candidates render (previously empty).
4. Open a previously-matched row → no extra rerank API call fires (previously fired every time).
5. Map a row, then re-open the Mapping Decisions panel — score chip shows a number (previously NaN%).
6. Switch to table view — full columns populated (previously blank).

🤖 Generated with Claude Code