Repository navigation
ndi_common re-sync from NDI-matlab: probe-geometry library, missing document/schema pairs, and the probe_geometry matrix invariant #96
Description
Part of #90 (PR #61 catch-up). Items M5, S5, and the read-side half of S6.
1. Probe-geometry library data is absent
src/ndi/ndi_common/probe/ contains exactly one file, probetype2object.json. NDI-matlab carries a geometry library that Python has none of:
probe/geometry/README.md
probe/geometry/generic/linear16_25um.json
probe/geometry/generic/tetrode.json
probe/geometry/neuronexus/A1x32-Poly2-5mm-50s-177.json
probe/geometry/neuropixels/NP2_1shank.json
probe/geometry/ucla/UCLAd64.json
probe/geometry/ucla/UCLAe64.json
probe/geometry/ucla/UCLAf64.json
Vendoring the data is mechanical (byte-for-byte from NDI-matlab, plus consistency tests over the layouts — PR #61 carried 29). The geometry package that consumes it (S5) is a separate, larger job and is not in scope here.
2. Missing document / schema pairs
Present in NDI-matlab, absent here — each is a database_documents/*.json plus its schema_documents/*_schema.json:
data/filter(+filter.md)data/pyraview(+pyraview.md)ingestion/daqreader_image_epochdata_ingestedtreatment/treatment_transferapps/kilosort/kilosort_clusters
kilosort_clusters is a dependency of the Kilosort import pipeline — see the separate issue for that; the others stand alone.
3. probe_geometry matrix invariant in ndi.validate (S6, read side)
MATLAB bf85e24b7 establishes that a matrix-valued schema field must be [] rather than null, and must be rectangular. src/ndi/validate.py enforces neither.
Porting the read-side check is small. Two deliberate exclusions to preserve:
- The emit-side half is blocked on the absent geometry package (S5).
- Declared-dimension enforcement is deliberately excluded, pending the
t0_t1orientation decision — see the divergences issue.element_epoch/epochclocktimesschemas declare[2, NaN]while Python emits N×2, and flat JSON cannot distinguish the orientation. Enforcing declared dimensions before that is settled would break valid documents.
4. Re-sync the 7 known-divergent schema files
PR #61 commit 549c90f synced 20 files against NDI-matlab 7509dc96c, including the 7 pre-existing divergent schema files — among them the EMPTY 7-digit ids that Python's own ontology resolver requires. That commit's manifest is the reference for what was synced byte-for-byte and what was deliberately left alone.
Explicitly out of scope: V_eta-coupled files. Those stay untouched and recorded, as PR #61 had them.
What it would take
Mostly vendoring plus a consistency-test sweep; the only real code change is item 3. The sync should be re-run against NDI-matlab's current head rather than replaying 7509dc96c, and a tests/test_ndi_common_matlab_sync.py equivalent should pin the result so the next drift is visible.
Activity
Sync EMPTY-id schema divergences from NDI-matlab
Sync calc v2 refactor and probetype2object.json from NDI-matlab
Merge pull request #177 from Waltham-Data-Science/claude/issue-96-ndi…
stevevanhooser commentedon Sep 3, 2026
Closing.
Fully addressed by PR #177. Every item the issue asked for is done, and the pinning test (
tests/test_ndi_common_matlab_sync.py) now has an emptyEXPECTED_DIVERGENCES— Python and NDI-matlab agree byte-for-byte acrossndi_common/(aside from short, declared exclusion lists for MATLAB-only and Python-only trees).What landed in the PR:
daqreader_image_epochdata_ingested,treatment_transfer); the other 3 were already present._check_matrix_shapeinvalidate.pyenforces[]-not-nulland rectangularity, with 10 tests. The two exclusions the issue itself called out (emit-side, declared-dimension enforcement pendingt0_t1) stay excluded.ontologyTableRow.jsondocument synced byte-for-byte after verifying againstndi-ontology-pythonthat itsEMPTYProviderpads to 7 digits (Python's 8-digit IDs likeEMPTY:00000090don't resolve at all).EXPECTED_DIVERGENCES.Also cleared during the same PR (drift discovered during the audit, not in the issue's original list, but easy to trace and fix):
apps/calculations/simple_calc.jsonwhen the file lives atapps/calculators/simple_calc.json); the v2 form fixes that.validate.pyusesPath(definition).stemfor schema lookup and never reads the validation URL, so v1 documents keep validating unchanged.probetype2object.jsonrenames — turned out to be a bug fix: the stale entries mapped tondi_probe.timage/ndi.probe.timage, neither of which exists in Python, while the code that actually uses these mappings (test_vhprairieview,epochprobemap_daqsystem_vhlab) already used MATLAB's new names.rayo_stim.jsonregex — NDI-matlab commita05a0a8documents this as a bug fix:#was a file-navigator wildcard thatregexpitreated as a literal. Python already had the more robust fix (RayoLabStims.readmetadatareturns the constant[{"stimid": 1}]and ignoresepochfiles); no Python code readsMetadataReaderFileParametersat all. Synced defensively.Not covered here (the issue itself called these out of scope):
validate.py(blocked on thet0_t1orientation decision).Generated by Claude Code