NDI Cloud file CRC-32: lookupFiles, checksummed uploads (ndi-cloud-node#162)

[stevevanhooser]

This is the Python half of VH-Lab/NDI-matlab#1025. It is the client for Waltham-Data-Science/ndi-cloud-node#162, implemented in ndi-cloud-node#163. That server change is deployed to dev and prod and tested with the MATLAB client.

NDI Cloud now records a CRC-32 for every uploaded file. An importer can therefore check which cloud copies match its local files and upload only the rest.

Changes

• ndi.fun.file.crc32(path) and crc32ToBase64(value) compute a file's CRC-32 in the cloud's encoding: base64 of the 4-byte big-endian value, the same format S3 uses for x-amz-checksum-crc32. The file is read in chunks with zlib.crc32.
• Checksummed uploads:
  • getFileUploadURL(..., checksum_crc32=crc) sends checksumCRC32, and the returned URL signs the x-amz-checksum-crc32 header.
  • putFiles / putFileBytes(..., checksum_crc32=crc) send that header, so S3 rejects content that does not match. A malformed value raises ValueError before any request is made.
  • ndi.cloud.upload.uploadSingleFile now does both by default; pass checksum=False to turn it off.
• New ndi.cloud.api.files.lookupFiles(dataset_id, uids) calls POST /datasets/{id}/files/lookup and returns {"files": [{uid, uploaded, size, crc32}], "missing": [...]}. Duplicate uids are collapsed, and lists longer than the server's 5000-uid limit are split into several calls and merged.
• getSignedURLSetAll merges each page's fileInfo into one list. getSignedURLSet, listFiles and getFileDetails already return the server's JSON unchanged, so crc32 and fileInfo come through without other changes.
• Bridge files:
  • lookupFiles, crc32 and crc32ToBase64 are recorded with matlab_branch: claude/file-crc32-lookup. That marker must be removed once NDI-matlab#1025 merges, and the check will say so.
  • getFileUploadURL, putFiles, uploadSingleFile, listFiles(All), getSignedURLSet(All) and url are synced to NDI-matlab 8fa6ea5e, with a note in each decision log.
• CHANGELOG.md has an entry under Unreleased/Added.

Tests

• New tests/test_cloud_api_files_crc32.py (25 tests, no network). It covers:

  • the encoding, checked against zlib reference values, including chunk boundaries and the empty file;
  • the checksumCRC32 query parameter and the x-amz-checksum-crc32 header;
  • rejection of malformed values;
  • lookupFiles: deduplication, splitting at 5000 uids, and found/missing results;
  • merging of fileInfo;
  • uploadSingleFile sending the checksum by default and not sending it with checksum=False.

• New TestFileChecksums in tests/test_cloud_live.py, mirroring the MATLAB FilesTest:

  • an upload with a checksum, with crc32 read back from lookupFiles, getFileDetails and listFiles;
  • an upload with a wrong checksum, which S3 rejects;
  • uploadSingleFile recording the checksum;
  • the server returning 400 for more than 5000 uids or for a malformed body;
  • a lookup split across two calls.

  These tests fail (they do not skip) on a server without /files/lookup, so a missing or broken route is reported rather than hidden.

• Two existing tests in tests/test_phase2_gaps.py call uploadSingleFile on a file that doesn't exist, with the upload API mocked. They now pass checksum=False, because the default now reads the file.

Local results

• black --check, ruff check and pytest tests/ all pass. The live tests need credentials and were not run locally.
• I ran the bridge checks with NDI_MATLAB_PATH pointing at a shallow NDI-matlab clone on claude/file-crc32-lookup. None of the entries this PR changes fail. The checks that do fail here fail the same way on unmodified main in this environment: the clone lacks old commits and the lightsheet branch. CI checks out the full NDI-matlab branch of the same name, so it isn't affected by this.

Merge order

1. Deploy ndi-cloud-node#163 to prod. Done.
2. Merge NDI-matlab#1025.
3. Merge this PR, removing the three matlab_branch markers if #1025 merged first.

🤖 Generated with Claude Code

https://claude.ai/code/session_01SwsqFA9Zs1HGm6YM4g2XRV

---

Generated by Claude Code

[stevevanhooser]

MATLAB/Python bridge completeness fails here, but on entries this PR doesn't touch.

Everything this PR changes passes, including drift, the hash-justification check and the branch-pending checks against claude/file-crc32-lookup. The single failure is test_no_entry_has_drifted, and it names three lightsheet entries:

Entry	Recorded	Moved by (on NDI-matlab claude/lightsheet-zarr-ndi-viewer-djp5vk)
fun/doc_lightsheet: makePyramid	5c9e8a83	04ce18f9 (2026-10-12 13:28): "fromSPIM -- pyramid straight from SmartSPIM All_Channels" (+243/−36)
fun/doc_lightsheet: view	a7ba07a	2f9b0eaf (2026-10-12 12:30): "launch the napari viewer with debugging on and its log kept" (+35/−12)
gui/app/lightsheetZarr: LightsheetZarrManager	60a79cf0	2f9b0eaf (+8/−16)

Those commits went onto the lightsheet MATLAB branch after main last passed this check, at 00:47 UTC today. So main will fail the same way on its next run. These are real behaviour changes, a new fromSPIM path and a change to how the viewer is launched, not just bookkeeping. They need porting, or a decision that they don't need porting, in the lightsheet work (e.g. #320), not in this checksum PR. No open PR carries that port yet.

The fix: once the lightsheet changes are ported or reviewed, bump those three matlab_last_sync_hash values to 04ce18f9 / 2f9b0eaf, adding a decision_log note for any that need no Python change. I'll re-run the job once in case the branch state changes.

---

Generated by Claude Code