Tracking: land the remaining parts of PR #61 (2026-08 catch-up) onto current main #90

Why this exists

PR #61 (catchup/ndi-python-2026-08, 113 commits off ed78f8b) never merged, and main has since moved 42 commits in an independent direction (DID file services, syncgraph rebuild, download correctness, ndi.fun.ensemble, parseText, VHSB, navigator panes). git cherry reports 0 of 113 commits patch-equivalent, so nothing can be assessed by commit identity — every item needed a content check against today's main.

This issue records the result of that check and links the issues that carry the remaining work.

Status snapshot 2026-09-04

• PR NDI-python 2026-08 catch-up — hardening bundle + MATLAB parity + CI safety #61 is closed (2026-09-01, mergeable_state: dirty, never merged).
• Fifteen of the eighteen recovered issues have landed: CI wiring: cloud tests missing from the nightly, PR CI limited to main, and a test file that never runs #91, Security hardening backlog: committed 7 MB tarball, XXE-prone XML parsing, no zip-bomb guard, unsanitised stored file names #92, Four correctness bugs still live on main: query De Morgan, ngrid column-major order, fail-open validation, unparseable schemas #93, setup.lab() drops every metadata reader's file parameter and installs no syncgraph; port ndi.setup.sync and vendor sync_rules/ #94, getCloudDatasetIdForLocalDataset is dead for every real dataset; port dataset.isInCloud() #95, ndi_common re-sync from NDI-matlab: probe-geometry library, missing document/schema pairs, and the probe_geometry matrix invariant #96, Ontology: add Uberon, NCIT, EDAM, IAO, STATO and SchemaOrg providers #98, Opening a dataset mutates it: the shared dabrowska test corpus lost a session document, and nothing prevents a recurrence #99, Cloud sync index: write truncates before locking, read takes no lock, and module-level fcntl breaks the import on Windows #101, ndi.neuron and ndi.element.timeseries report the wrong ndi_element_class and are unregistered: neurons are silently lost in both directions #133, ndi.app.oridirtuning is a framework: all five compute methods raise NotImplementedError #134, dataset.isIngested() and convertLinkedSessionToIngested() are missing: uploadDataset cannot make the check MATLAB refuses to skip #136, Cloud client retries nothing: a 502/503/504 on an idempotent request fails the whole operation #137, ndi.database.fun.readtablechar / writetablechar are missing, and doc_table.treatment drops the three mixture columns MATLAB always produces #138, Decision needed: epochgraph() and matchedepochtable() share MATLAB's names but not its signatures or semantics #139.
• Three remain open: Kilosort import pipeline, interactive spike-sorter GUI, and the .vsw / KlustaKwik file formats #97 (Kilosort/GUI/.vsw/KlustaKwik), Three recorded MATLAB/Python divergences that need a decision, not a port: epochdir ANY-vs-ALL, non-functional cedspike2, t0_t1 wire orientation #100 (three MATLAB/Python divergence decisions), ndi.app.spikesorter and ndi.app.spikeextractor are frameworks, and are not tracked by #97 or anywhere else #135 (ndi.app.spikesorter / ndi.app.spikeextractor headless apps — still needs branch reading before catchup/ndi-python-2026-08 goes).
• Of the two "not filed separately" residuals, the completeness half of the porting-log gap has been closed (see below); a smaller hash-freshness half remains. Dep pins are deferred while cross-repo development is still rapid.

At this point the tracker mostly indexes work that has closed on its own tickets. See the recommendation at the bottom.

Already on main — do not re-port

These landed independently and are equivalent to or better than the PR #61 version:

Already present at the fork point — CORRECTED 2026-09-02

Also note: a large block of PR #61's older commits (markgarbage identifyvalidintervals, oridirtuning double-gaussian, tuning_response compute, isIngested, convertLinkedSessionToIngested, epochgraph, matchedepochtable, readtablechar, constructable ndi.neuron, cloud 5xx retry) were already present at the fork point ed78f8b — they are replays of work main already carries, not new content. This is the main reason the PR looks larger than the gap actually is.

This was wrong, and it was the load-bearing claim for closing #61. Nine of those ten items are still missing from main today, and the tenth is half-done. The error was reading "the file and the method name exist at ed78f8b" as "the work is done at ed78f8b" — but at the fork point those methods were raise NotImplementedError stubs, and five of the files have not been touched since:

$ for f in app/oridirtuning.py app/markgarbage.py epoch/epochset.py cloud/client.py neuron.py; do
    a=$(git show ed78f8b:src/ndi/$f | git hash-object --stdin)
    b=$(git show origin/main:src/ndi/$f | git hash-object --stdin)
    [ "$a" = "$b" ] && echo "IDENTICAL ed78f8b==main : $f"
  done
IDENTICAL ed78f8b==main : app/oridirtuning.py
IDENTICAL ed78f8b==main : app/markgarbage.py
IDENTICAL ed78f8b==main : epoch/epochset.py
IDENTICAL ed78f8b==main : cloud/client.py
IDENTICAL ed78f8b==main : neuron.py

Item by item, re-checked against main at 429c062:

So the tally is half an item on main, two divergences needing a decision, and eight genuinely missing.

(That table is a snapshot taken against 429c062 and is left as written; the live status is the one below.)

The ndi.neuron one is a live data bug, not just a gap

ndi_neuron does not override ndi_element_class(), so it inherits "ndi.element", and "ndi.neuron" is not in the class registry. ndi_element_timeseries has the identical defect:

$ python -c "
import ndi
from ndi.class_registry import get_class
print(ndi.ndi_neuron().ndi_element_class())
print(ndi.ndi_element_timeseries().ndi_element_class())
for n in ('ndi.neuron', 'ndi.element.timeseries'):
    print(n, '->', get_class(n))
"
ndi.element
ndi.element
ndi.neuron -> None
ndi.element.timeseries -> None

Both directions lose data, neither loudly:

• Python-written neurons are stored labelled ndi.element and read back as plain ndi_element — no error, wrong class.
• MATLAB-written neurons are labelled ndi.neuron, so _document_to_object raises Unknown element class — and getelements wraps that in except Exception: pass (session_base.py:901), so the neurons are silently dropped from the result.

main already knows about this and works around it in one place rather than fixing it — fun/ensemble_db.py:70 falls back to ndi_element_timeseries explicitly because "ndi_element_timeseries does not override it, so the registry alone would strand them without readtimeseries". That fallback rescues ensemble loading; getelements has no such fallback, so neurons still vanish there.

The fix on the branch is a 14-line ndi_element_class() override on ndi_neuron plus registry entries for both classes. Carried by #133. Landed 2026-09-02 in #143.

Why this matters for #61

The disposition below recommends closing #61. That recommendation rested on this paragraph. With the paragraph corrected, the branch is the only place these implementations exist:

The last two are worth calling out separately: #97 covers the interactive sorter GUI (spikesorter_gui.py, spikesorter_clustermodel.py) and the .vsw/KlustaKwik formats, but the headless ndi.app.spikesorter and ndi.app.spikeextractor implementations are not named in #97 or anywhere else in this triage. They are unaccounted for.

Extract or re-file before closing #61. Done as of 2026-09-02: #134 (oridirtuning) and #135 (spikesorter/spikeextractor) now carry these; read them against the branch before it goes. #134 has since landed, so oridirtuning no longer depends on the branch; #135 still does.

Recovered work — one issue per cluster (final tally 2026-09-04)

Fifteen of eighteen have landed. Three of the fifteen came with cross-language work worth recording: #133 landed with VH-Lab/NDI-matlab#913 (the elementClass symmetry battery), the #136 port uncovered a MATLAB-side bug of its own — uploadDataset's "not fully ingested" refusal returned before cloudDatasetId was ever assigned, so the message it wrote was unreachable through every real caller (VH-Lab/NDI-matlab#914, fixed in VH-Lab/NDI-matlab#915).

Ordered as originally filed:

#97 overlaps with #78 (the navigator/GUI direction main is already taking) — those two should not proceed independently. (#78 has since closed; the overlap consideration for #97 stands against whatever navigator work follows it.)

Not filed separately — status 2026-09-04

Two residual items originally noted as small enough to fold into whatever work touched them:

• Bridge-contract / porting-log hygiene. Substantially addressed. The completeness half — every MATLAB function has a bridge entry, or an exclusion with a reason — is now enforced in CI by tests/test_matlab_bridge_completeness.py, built in Port ndi.cloud.helloMatlab, record the five unbridged +cloud functions, and add a test that the bridge files stay complete #131 and extended package-by-package through Port helloMatlab and add bridge completeness guard #142 / Guard six more packages with the bridge completeness check #150 / Register the remaining ten packages with the bridge completeness guard #154 / Guard every remaining package, and close the gaps it found #155, closing Register the remaining ten packages with the bridge completeness guard #154 on 2026-09-03. All 17 packages are registered; unrecorded MATLAB functions now fail the CI check rather than sitting silently stale. What remains open is narrower: the guard does not check that matlab_last_sync_hash values reflect a recent examination, so an entry can carry a hash from months ago and no test complains. PR NDI-python 2026-08 catch-up — hardening bundle + MATLAB parity + CI safety #61's convention — "on a partial sync bump the hash and record examined_at: ⟨hash⟩; deferred: ⟨items⟩" — is still the right one, and adopting it means either (a) sweeping the pre-guard entries once so every hash is a recent examination, and/or (b) adding a lint that flags entries whose recorded hash is older than the current MATLAB tip beyond some threshold. Small, mechanical, not urgent — a good ticket to file if it starts biting during the next MATLAB sync, otherwise leave it as this note.
• Dependency pins. Deferred. pyproject.toml floats did, ndr, vhlab-toolbox-python, vhlab-library-python and ndi-compress at @main. The failure mode is real (PR NDI-python 2026-08 catch-up — hardening bundle + MATLAB parity + CI safety #61's June "green" numbers were untrustworthy for this reason; a container with a stale vhlab-library-python on 2026-09-03 failed ten test_app_oridirtuning tests that CI passed on the same SHA), but cross-repo development is still moving too fast for SHA-pinning to be worth the bump ceremony. Revisit once co-development across did / ndr / vhlab-* / ndi-compress stabilises; until then the workaround is a fresh pip install --force-reinstall before trusting a local run, and treating local-vs-CI disagreements as "did a dep move?" rather than a code issue.

Disposition of PR #61 itself

The branch is 41k insertions / 15k deletions away from today's main and mergeable_state is dirty. Rebasing it is not realistic — several of its files no longer exist on main (e.g. database.get_binary_path, the whole pre-DID file-storage layer), so some of its patches have no target. Recommend closing #61 and #60 in favour of the issues above, each re-derived against current main.

Amended (2026-09-02): closing #61 is still the right call. The correction above briefly blocked it; that block is now lifted — every recovered item is carried by an issue (#133 – #139), and markgarbage.identifyvalidintervals and the tuning_response compute methods are implemented in #124. The branch is still the only place working implementations of oridirtuning, spikesorter and spikeextractor exist, so #134 and #135 should be read against it before it goes.

Amended (2026-09-03): #134 has landed, so only #135 (spikesorter / spikeextractor) still needs reading against the branch before it is closed.

Amended (2026-09-04): PR #61 was closed 2026-09-01. The branch itself (catchup/ndi-python-2026-08) still exists and remains the only place working spikesorter / spikeextractor implementations exist — do not delete it until #135 has read from it.

Recommendation on this tracker

Fifteen of the eighteen recovered issues have closed; the remaining three — #97, #100, #135 — are independent, well-scoped, and stand on their own without this index. The completeness half of the porting-log residual has been closed on its own tickets; the hash-freshness half and the dep-pin residual are both now documented above with an explicit posture (file if it bites; revisit when co-development stabilises). At that point this issue has nothing left to index that isn't already tracked or explicitly parked.

Suggested closure gate: nothing further is required — the three remaining recovered issues track themselves and the two residuals are parked with clear conditions for revisit. Reasonable to close as completed now, or hold open as a lightweight reference until #135 lands (so the branch-preservation note has a natural home).