Enumerate a document's binaries the way NDI-matlab does

[stevevanhooser]

Closes #306. Replaces the closed #307, which patched around the symptom at the wrong layer and never covered series MEMBERS.

The bug, in two parts

1. Manifest. did.document.Document.add_file_series writes the series manifest to a tempfile.NamedTemporaryFile(prefix="did_manifest_"), then calls add_file(name, manifest_path). add_file's default is delete_original=True, so DID copies the manifest into <FileDir>/<manifest_uid> and removes the source. The doc's authoring record (files.file_info[].locations[].location) still names the tempfile, which is gone. file_uploads_for_document read only that record, so nothing was uploaded for the manifest -- the failure mode run 227 on TEST_USER_1 hit ("Manifest for file series (uid ...) is not a file of this dataset").

2. Members. A series' MEMBERS have no file_info entry at all. They're named NAME_i for i in 1..series_info[k].count and their bytes live at <FileDir>/<member_uid>. A caller that only walks file_info never sees them, so an upload sent zero member files regardless of what happened to the manifest. This half was undetected until this reshape.

The fix

NDI-matlab's +ndi/+database/+internal/list_binary_files.m has never had this trouble because it goes through the storage-side source of truth:

stored_names = database_documents{i}.current_file_list();
for f = 1:numel(stored_names)
    [file_exists, full_file_path] = ndi_dataset.database_existbinarydoc(ndi_document_id, this_filename);
    ...
end
seriesInfo = database_documents{i}.document_properties.files.series_info;
for s = 1:numel(seriesInfo)
    for m = 1:seriesInfo(s).count
        member_name = sprintf('%s_%d', series_name, m);
        [member_exists, member_path] = ndi_dataset.database_existbinarydoc(ndi_document_id, member_name);
        ...
    end
end

database_existbinarydoc is DID's exist_doc, which returns the storage-side path (<FileDir>/<uid>) whether the caller ingested seconds ago or the doc was loaded from disk. It also resolves series members through the manifest without needing them in file_info.

Ported the same shape:

• file_uploads_for_document(doc, *, dataset=None) takes an optional dataset. When given, it walks file_info[i].name (which yields the manifest name for a series document) and iterates series_info[k].count for members, asking dataset.database_existbinarydoc(doc_id, name) for each. The uid is the basename of the returned path -- DID keys files by uid, so fileparts(full_file_path) is enough (same as MATLAB).
• Threaded the dataset through uploadFilesForDatasetDocuments, through sync.operations._upload_binaries (all three callers -- uploadNew, mirrorToRemote, twoWaySync), and through orchestration.uploadDataset.
• A caller that hands only a raw dict without a dataset gets the pre-uploadDataset silently skips a series manifest's bytes (uploaded doc references a nonexistent tempfile after DID ingests) #306 shape: read files.file_info[].locations[].location, skip anything that isn't on disk. Kept for isolated tests and legacy callers.

Tests

tests/test_cloud_upload_enumerates_via_dataset.py covers:

• The manifest name is looked up via database_existbinarydoc (not by reading the doc's stale location).
• Series members are enumerated from series_info[k].count, one-based, in slot order.
• A sparse series (holes between present slots) yields only the present slots -- absent is normal.
• uid = basename(path), matching MATLAB.
• MATLAB-shaped single-element dicts for both file_info and series_info still work.
• A doc with no id yields nothing (no probe on the empty string).
• Raw-dict fallback: live location returned, gone location skipped, top-level file_uid/file_path shape still returned, and an object that isn't a real dataset (no database_existbinarydoc) falls back rather than raising.
• End-to-end with a real dataset: build a series doc, database_add it, prove the recorded manifest location is gone (that IS the shape the test targets), and check that file_uploads_for_document(doc, dataset=local) returns manifest + all 4 members with uid == basename(path) and each path pointing at a real file under DID's FileDir.

Live-cloud round-trip coverage (tests/test_cloud_file_series_round_trip.py) is untouched -- it's what will pass on TEST_USER_1 with this fix applied.

Why not fix DID?

Considered. The recorded location in file_info[].locations[] is the doc's authoring record, and DID keeps its own storage-side record separately (file_locations table's cached_location column) precisely so both remain readable. Rewriting the authoring record on ingest would break other consumers that read the doc as originally-authored. add_file_series writing to a persistent path instead of a tempfile is possible, but it just moves the same "recorded path can go stale" problem elsewhere -- any user calling add_file(delete_original=True) on their own file hits it too. The right answer is the one NDI-matlab already has: on the upload side, ask the storage-side API where the bytes actually live.

Files

• src/ndi/cloud/upload.py -- reshape.
• src/ndi/cloud/sync/operations.py -- thread dataset through _upload_binaries + three callers.
• src/ndi/cloud/orchestration.py -- pass dataset through uploadDataset.
• src/ndi/cloud/ndi_matlab_python_bridge.yaml -- decision log for the new kwarg.
• tests/test_cloud_upload_enumerates_via_dataset.py -- new offline test.

Bridge check passes (128 tests). Sync/upload/cloud tests pass (570 non-live). The 7 pre-existing failures in test_cloud_profile_secret_field_names.py reproduce on main too -- unrelated sandbox keyring issue.

🤖 Generated with Claude Code

https://claude.ai/code/session_01AdGZ8fRPeY9JSpUHgXm9ao

---

Generated by Claude Code