Move file storage and retrieval onto DID's file services #65

Why

NDI-python rolled its own binary file storage before DID-python had any, and still uses it. DID-python now has the services (open_doc with custom_file_handler, a files table, did.file.FileCache), and NDI-matlab already routes through them. Python should too.

MATLAB is the source of truth: imitate it. Python has no use history and no customers yet, so the design is free to change on both the NDI-python and DID-python sides. Do not preserve Python's current scheme for its own sake, and do not carry a compatibility shim for it.

Where things stand

NDI-python's own store — a flat namespace beside the database:

# src/ndi/database.py
self._binary_dir = self.session_path / db_name / "files"

def get_binary_path(self, document, file_name):
    return self._binary_dir / f"{document.id}_{file_name}"

Ingest is a shutil.copy2 into that directory (session_base._ingest_binary_files); database_openbinarydoc opens the path directly. Cloud retrieval is separate again — ndi.cloud.filehandler / session._try_cloud_fetch — written before DID had a seam to hang it on.

NDI-python never calls DID's open_doc, never reads the files table, and does not use did.file.FileCache.

What changed underneath it. Since DID-python 9997412, add_docs writes files rows automatically. NDI acquired rows it never asked for and does not read. (That is also what produced the foreign-key bug on removal, fixed in VH-Lab/DID-python#39.)

So there are two records of where each file is, and they disagree. DID's files table records what _file_entries reads out of file_info[].locations — the original source path. NDI's copy sits under its own binary directory, named DOCID_FILENAME. Nothing writes NDI's ingested location back into the table.

This matters for interop specifically: MATLAB's do_open_doc resolves a file by querying docs, files and never reads locations out of the document JSON. A session written by NDI-python today has files rows MATLAB will follow — straight to a source path that may not exist on that machine.

Target shape

+ndi/+database/+implementations/+database/didsqlite.m is the reference to copy:

do_add                -> db.add_docs(doc, 'a', 'customFileHandler', @download_file_from_cloud)
do_openbinarydoc      -> db.open_doc(id, filename, 'customFileHandler', @download_file_from_cloud)
                         followed by fopen()
check_exist_binarydoc -> db.exist_doc(id, filename)

download_file_from_cloud is NDI's, satisfying DID's customFileHandler contract — DID downloads nothing itself. Python's equivalent already exists in ndi.cloud.filehandler; it just is not wired to DID.

Prerequisites in DID-python

Two pieces MATLAB has and Python does not. Both are needed before the NDI side can be finished, and both are ports rather than designs — DID-matlab already defines the behaviour:

open_doc(doc_id, filename, custom_file_handler=…) is already there.

Decisions to make

No migration path is required. Sessions written by the current Python scheme can be abandoned; a straight cutover is acceptable. (Recorded because the first draft of this issue treated existing on-disk sessions as a constraint — they are not.)

What still needs deciding, and should follow MATLAB unless there is a reason not to:

1. What the files row points at after ingest. It must be the ingested copy, not the original source, or MATLAB keeps following a path that only exists on the machine that did the ingest. Check what MATLAB's do_add records and match it.
2. Whether NDI keeps a binary directory at all, or hands the whole question to did.file.FileCache as MATLAB does.

Scope

36 call sites across 7 files: src/ndi/database.py, src/ndi/session/session_base.py, src/ndi/database_fun.py, src/ndi/element_timeseries.py, src/ndi/dataset/_dataset.py, src/ndi/daq/mfdaq.py, src/ndi/fun/data.py.

Related divergences, cheap to fold in

• OnMissing on remove. did.database.remove_docs defaults to OnMissing='error'; NDI-matlab's do_remove overrides it to 'ignore', on the reasoning that "a document someone else already deleted still satisfies the request that it be gone". NDI-python's SQLiteDriver.delete_by_id instead does its own existence check and returns False. Same intent, different mechanism.
• Hard delete vs unlink. DID-python's _do_remove_doc deletes docs/doc_data/files once the last branch reference goes; DID-matlab only unlinks branch_docs and leaves the rest behind a % TODO. Recorded in DID-python's bridge as of Add dataset symmetry tests for artifact generation and verification #39. If MATLAB is the source of truth, either MATLAB implements its TODO or Python stops hard-deleting — worth settling rather than leaving as a silent asymmetry.

Background

Analysis that surfaced this: docs/developer_notes/DEPENDENCY_DRIFT_2026-08.md (§2), added in #64.