Always bind an epoch's binary, empty or not

[stevevanhooser]

main went red at 12:02:34Z, when VH-Lab/DID-python#86 closed the fail-open in check_files' step 3 — four minutes after main's last green run (11:45:34Z → 11:58:54Z). The failures are all one shape:

did.validate.ValidationError: Required file "epoch_binary_data.vhsb" is declared in
the file_list of element_epoch doc <id> but no file is bound to that name
(files.file_info has no entry for it)

What DID started rejecting is real, and it is ours.

The divergence

ndi_element_timeseries.addepoch skipped writing and attaching the binary when timepoints or datapoints was None, and fell through to the base class — which added an element_epoch document with nothing bound to that name:

has_data = timepoints is not None and datapoints is not None
if not has_data or self._session is None:
    return super().addepoch(epoch_id, epoch_clock, t0_t1, add_to_database=add_to_database)

MATLAB has no such branch. In ndi.element.timeseries/addepoch the write and the attach sit after the argument handling rather than inside it, and timepoints/datapoints are required positional arguments, so the class cannot produce a fileless epoch document at all:

fname = [ndi.common.PathConstants.TempFolder filesep epochdoc.id() '.vhsb'];
vlt.file.custom_file_formats.vhsb_write(fname,timepoints,datapoints,'use_filelock',0);
epochdoc = epochdoc.add_file('epoch_binary_data.vhsb',fname);

An epoch may have no data; it may not have no file

An epoch with no samples is a supported case, not an error — ndi/+mock/+fun/subject_stimulator_neuron.m:76 does exactly that for a stimulator element that marks a time range and carries no samples:

nde_stimulator.addepoch('mockepoch',ndi.time.clocktype('UTC'),[0 100], [], []);

That still writes a vhsb and still attaches it. The file is there and empty. vhsb_write handles empty input fine — 1,836 bytes of header that reads back as zero samples.

So the distinction element_epoch's schema draws with mustbenotempty on epoch_binary_data.vhsb is exactly right, and it stays. My first reading of this was that the schema was over-claiming and should be relaxed to mustbenotempty: 0; that was wrong, and checking NDI-matlab is what showed it. Relaxing the schema would have papered over a Python bug and, worse, desynchronised ndi_common from NDI-matlab to do it.

The tests asserted the divergence

One of them was named test_addepoch_without_samples_stores_no_binary — it asserted a document MATLAB cannot produce. TestEpochWithoutData now holds the contract instead:

• the binary is bound when an epoch has no samples;
• it is declared on the document — bound, not merely written, since what DID rejects is a name in file_list with no matching file_info entry;
• reading an epoch with no samples gives nothing rather than raising, which is what the old second test was really about.

Measured

Against main in the same working tree, with did and vhlab-toolbox-python both installed from main as CI installs them (my container's copies of both were stale, which initially hid this behind unrelated vhsb_write errors on short arrays):

	before	after
tests/test_fun_ensemble_db.py	26 errors	0
tests/test_gui_app_ensemble_maker.py	3 errors	0
tests/test_element_timeseries_vhsb.py	5 failed	0

Nothing newly broken — the before/after failure lists differ only by removals.

What this does NOT fix

A second family survives, and it is a different kind of problem: test fixtures that construct documents declaring a required file and never bind one. tests/test_session_database_rm.py:39 adds a data/generic_file with no generic_file.ext; several matlab_tests cases add a demoNDI with no filename1.ext (the conftest.py helper attaches one correctly — the individual tests do not). test_did_integration's allowlist also records error codes that have changed.

That is roughly twenty sites across a dozen files, and none of it is a product bug: those documents were only ever built by tests. It belongs in its own change rather than bundled with a fix to shipping code, and I did not want to hold this one behind it. Happy to follow up with it.

Related

• VH-Lab/NDI-matlab#977 dealt with the MATLAB-side counterpart of the same DID hardening one step higher, where makeSourceFile moved off generic_file because a described-only file cannot supply the generic_file.ext that class requires.
• Waltham-Data-Science/NDI-python#291 is blocked by this breakage and is unrelated to it.

🤖 Generated with Claude Code

https://claude.ai/code/session_01YHxsZ56GZsZ3hJ98TJ81t9

---

Generated by Claude Code

[stevevanhooser]

CI result on 4296ddc: 21 failed, 5196 passed, against 26 failed + 29 errors on main. The 34 this fixes are gone and nothing new appeared.

Every one of the 21 survivors is the fixture family named under What this does NOT fix — documents that declare a required file and bind nothing:

test_session_database_rm.py (5)	generic_file.ext
matlab_tests/*, test_extract_doc_files_series.py	filename1.ext on demoNDI
test_did_integration.py (1)	the allowlist records ValidationFieldInteger/ValidationFieldDouble where DID now reports ValidationFiles

So this PR cannot reach green by itself, and that is by design rather than an oversight — the remaining work is test-side and touches ~20 sites across a dozen files, which does not belong bundled with a fix to shipping code. Tracked separately; this PR is complete as scoped.

---

Generated by Claude Code