Repository navigation
Add ndr.format.blosc encoder/decoder over a self-installing venv - #139
Merged
Merged
Conversation
Introduces a Blosc container codec for NDR-matlab that works without a
system zstd binary and without touching MATLAB's `pyenv`. Encode/decode
delegate to numcodecs.Blosc, invoked as a subprocess in a private venv
that NDR sets up on first use.
New packages:
+ndr/+format/+blosc/
encode.m raw bytes / typed array -> Blosc v1 container
decode.m Blosc v1 container -> raw bytes
isBlosc.m informed header sniff (version + cbytes)
header.m parse 16-byte header without decoding
+ndr/+util/+blosc/
setup.m creates <NDR>/private_env/blosc, pip installs numcodecs
(accepts 'python', 'wheelDir', 'force')
pythonExe.m cached path to the venv's python
toolScript.m path to tools/blosc_tool.py
runTool.m stdin/stdout tempfile subprocess wrapper
version.m numcodecs/libblosc versions inside the venv
venvDir.m canonical venv directory
tools/blosc_tool.py
Tiny stdin/stdout worker with `encode`, `decode`, and `version`
subcommands. Runs under the private venv only.
Why the sonpipe pattern rather than MATLAB's `py.` bridge: MATLAB's
pyenv is process-global. Setting it steals the customer's Python
configuration and prompts a restart. This scaffold runs Python as an
external process against a directory NDR owns, so nothing in the
customer's environment is disturbed. First-time bootstrap needs a
`python3` (3.9+) on PATH and PyPI access; after that neither is
required. An offline path exists via `wheelDir`.
README gains a self-installing-codec section; .gitignore adds
`private_env/` so the venv never lands in git. TestBloscRoundtrip
covers uint16/single round-trips, explicit typesize, shuffle-off,
header read-back, and a length-mismatch error path; it skips cleanly
when the venv cannot be provisioned.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| txt = strtrim(fread(fid, inf, '*char')'); | ||
| fclose(fid); | ||
| catch | ||
| try, fclose(fid); catch, end %#ok<CTCH> |
Contributor
runTool already prepends the subcommand (`encode`) between the script path and the args; encode.m was also putting `encode` at the start of args, so argparse saw `blosc_tool encode encode --cname ...` and rejected the second `encode` as an unrecognized positional. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #139 +/- ##
==========================================
- Coverage 53.32% 53.26% -0.06%
==========================================
Files 142 160 +18
Lines 8623 9190 +567
==========================================
+ Hits 4598 4895 +297
- Misses 4025 4295 +270 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Modern MATLAB Code Analyzer recognises the onCleanup idiom (an assignment kept only to hold the cleanup object alive) and does not warn. Preemptively matches the same cleanup landing on the NDI-matlab side. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w
Blosc's Python/C API caps clevel at 9 regardless of the inner codec. numcodecs.Blosc raises `clevel parameter must be between 0 and 9!` past that, so 22 in the docstring / no upper bound on the validator was inviting a runtime failure. Updated the arguments block with mustBeLessThanOrEqual(9), and rewrote the docstring to say 1..9 for every codec and include the observation from a lightsheet-like benchmark that 5 is the sweet spot: 9 buys ~4% smaller files at ~3x encode time. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w
The hand-rolled decompressBlosc.m only knew Zstd -- anything else inside the Blosc container hit "UnsupportedCodec". Zarr v2's default compressor is Blosc/LZ4 (dask, xarray, ome-zarr, spim2zarr all take that default), so reading a typical OME-Zarr from another tool failed on the first chunk. Replace the codec-per-block loop with a one-line delegation to ndr.format.blosc.decode, which routes through numcodecs.Blosc.decode in the private venv we already set up. numcodecs reads the codec identity out of the container header directly, so we get zstd, lz4, lz4hc, blosclz and snappy for free -- plus byte-shuffle, bit-shuffle and the memcpy fast-path -- without reimplementing them one by one. compressorName stays in the signature for backward compatibility; callers can still pass it, we just do not need it. The header truncation check is retained as a fast rejection for obviously bad inputs before we hand them to numcodecs. decompressZstd.m stays on disk unused: harmless, and a follow-up can delete it once the follow-up churn settles. That is a separate PR. Motivating case: a SmartSPIM stitched volume written by ferret-lightsheet/pyanalysis/spim2zarr uses zarr's default compressor and would have failed the NDI ingest path (fromOMEZarr -> makePyramid -> readArray -> readChunk) mid-stream on this file. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w
…awns Every Blosc encode/decode was spawning a fresh Python process. On macOS with numcodecs' import chain that costs ~200-500 ms per spawn. A single OME-Zarr readArray of a 2 GB region touches hundreds of source chunks -- minutes of pure startup overhead per read, one worker doing them all serially in the async prefetch path. Two mechanisms fix this, and both new APIs use them transparently: 1. Batch API: ndr.format.blosc.encodeMany / decodeMany take a cell array of items and process the whole batch in ONE round-trip. blosc_tool.py grows encode_batch / decode_batch subcommands whose stdin/stdout carry a length-prefixed batched wire format (packBatch / unpackBatch mirror it on the MATLAB side). One subprocess spawn for N items. 2. Persistent server: blosc_tool.py's new `server` verb accepts ENCB/DECB/QUIT requests over stdin and writes responses to stdout. ndr.util.blosc.serverConnect lazily spawns one per MATLAB process (persistent scope, so each parpool worker gets its own on first use) and reuses it for the rest of that worker's lifetime. readArray on a 4-hour ingest that touches 100 k chunks spawns ~19 Pythons total (one client + 18 workers), not 100 k. encodeMany / decodeMany try the server first; failure (Java bridge unavailable, server refused to start, protocol error) falls through to a one-shot tempfile subprocess -- still batched. The singular encode / decode route through the batch API as 1-item batches so they benefit from the persistent server automatically. readArray in +ndr/+format/+omezarr is restructured into three passes: (1) plan every source chunk's file path and destination geometry, (2) fread all present chunks into a cell array of raw containers, (3) one decodeMany, then reshape+permute+place. Same observed behaviour on missing chunks (still fill_value). Tests cover: encodeMany + decodeMany round-trip parity against singular encode/decode, empty batch is a noop, packBatch / unpackBatch bit-exact roundtrip (the wire format the client and server both rely on). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w
The persistent server was deadlocking on the first request: Python's
child stderr had no drain, so any import-time warning (numcodecs has
been noisy about deprecations across recent releases) filled the OS
pipe buffer, Python blocked on the next stderr write, and MATLAB
waited forever on stdout.
Two fixes together:
1. ProcessBuilder.Redirect.INHERIT sends the child's stderr to the
parent's (MATLAB's) stderr. Nothing fills the buffer; imports
that print warnings just print them.
2. NDR_BLOSC_NO_SERVER env var forces the fallback path so a user
who still hits a hang in the wild can `setenv('NDR_BLOSC_NO_SERVER',
'1')` and get a working (if slower) ingest immediately, without
waiting for a code push.
encodeMany / decodeMany already treat a serverConnect error as a
signal to use runTool, so the env-var path is a one-line escape
hatch.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w
The Python subprocess round-trip costs 100+ ms per invocation and was
the reason encodeMany/decodeMany existed at all. blosc-matlab
(Waltham-Data-Science/blosc-matlab) ships a small MEX that does the
same encode/decode as one direct C call in ~microseconds, so the
whole "make Python fast" apparatus can go away:
- New ndr.util.blosc.ensureMex downloads the pinned blosc-matlab
prebuilt for this platform on first use, extracts under prefdir(),
and adds it to the path. Idempotent, memoized per-process, and
safe to call from a parfor worker (each worker fetches the
tarball once). Failure to download or install returns
available=false with a diagnostic reason instead of erroring, so
the caller falls through to the Python path cleanly.
- ndr.format.blosc.encodeMany and decodeMany try the MEX first;
on unavailability they use runTool (the one-shot Python
subprocess) exactly as before. Both are SERIAL over the batch:
the MEX is fast enough on a single MATLAB thread for typical
workloads, and callers with an I/O-bound batch (the lightsheet
ingest) wrap parfor around encodeMany at whatever granularity
suits them. No parpool is required.
- encode.m and decode.m route through the batch APIs as before;
only their docstrings change to describe the new fast path.
- The persistent Python server (serverConnect, serverCall) is
deleted. Its raison d'etre was hiding Python startup latency
across many calls; with MEX at ~us per call the latency it hid
no longer exists. packBatch / unpackBatch stay -- runTool's
encode_batch / decode_batch subcommands still speak that wire
format.
Version-pinned at v0.1.0 of blosc-matlab. Bumping
BLOSC_MATLAB_VERSION in ensureMex forces every worker to re-fetch
on next call. If a customer's platform has no prebuilt (Linux arm64,
macOS Intel), the MEX path is unavailable and the Python fallback
runs; that keeps the previous behaviour for those users while the
common cases (macOS Apple Silicon, Linux x64, Windows x64) skip
Python entirely.
A follow-up we may want in blosc-matlab v0.2.0: an optional
`nthreads` name-value on blosc.encode/decode so a caller with a
large in-memory batch and no parpool can turn on blosc-internal
threading for that specific call. Left off by default because
mixing it with parfor over-subscribes cores.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w
| tmpDir = fullfile(tempdir, ['ndr-blosc-fetch-' ... | ||
| char(matlab.lang.internal.uuid())]); | ||
| mkdir(tmpDir); | ||
| cleaner = onCleanup(@() safeRmdir(tmpDir)); %#ok<NASGU> |
The old shortcut said "if blosc.encode resolves, MEX must be there". Not necessarily -- a user with a blosc-matlab clone on their path but no MEX installed would satisfy that check, and the first encode call would then error inside +blosc/private/. Look for the mex file explicitly under the same package folder as blosc.encode. If it's missing, fall through to the download path just as if +blosc weren't on the path at all. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w
blosc-matlab v0.2.0 ships encodeChunk / decodeChunk, which do pad + axis-transpose + typecast + (de)encode in one C call with zero intermediate MATLAB allocations. NDI's padEncodeAndWrite uses this fast path when it detects v0.2.0 is available; on a v0.1.0 install it falls back to the old pad-permute-typecast path in MATLAB (so old NDR customers keep working while the new install propagates through prefdir cache). Two changes here: 1. Bump BLOSC_MATLAB_VERSION to v0.2.0. Because the cache dir includes the version (prefdir/blosc-matlab/v0.2.0/), users who already have v0.1.0 cached at prefdir/blosc-matlab/v0.1.0/ will fetch v0.2.0 fresh into the new subdir on their next ingest without disturbing the v0.1.0 tree. 2. discoverExistingMex now requires the v0.2.0 marker `blosc.encodeChunk` to resolve before claiming an on-path install is usable. Prevents a user who has v0.1.0 on their path (from a clone with a manual install, or a previous cache) from satisfying the check and then hitting an "Unknown verb 'encode_chunk'" error inside the MEX. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w
stevevanhooser
pushed a commit
to VH-Lab/NDR-python
that referenced
this pull request
Sep 16, 2026
Follow-up to VH-Lab/NDR-matlab#139 (merged there): the probe/reduce helpers and the omezarr/smartspim reader classes were already ported; this commit adds the pytest coverage that was missing on the Python side and bumps the four bridge entries that had drifted. Tests mirror the MATLAB unit tests function-for-function: - tests/test_omezarr_probe.py mirrors TestOMEZarrProbe.m: ok / n_pyramids / pyramid_names / axes_order / level0_* on the shared programmatic fixture, plus the not-ok paths (missing dir, empty dir). - tests/test_omezarr_reduce.py mirrors TestOMEZarrReduce.m: mean/max, dtype (max preserves, mean promotes to float64), per-axis factor vector, trailing partial block kept, arity + reduction validation, factor-1 pass-through. - tests/test_omezarr_reader.py mirrors TestOMEZarrReader.m: pinned / unpinned / unknown / sidecar resolveepoch, all metadata methods, single-frame / multi-frame / Level>1 pixel reads against ground truth, and mean-vs-max routing proof at level 2. - tests/test_smartspim_reader.py mirrors TestSmartSPIMReader.m: pinned / unpinned-channel / unpinned-tile / sidecar resolveepoch, all metadata methods, single-frame / multi-frame / cross-channel reads against the synthetic fixture, out-of-range error. Programmatic fixtures mirror ndr.test.format.{omezarr,smartspim} .makeExampleFixture.m: dual-pyramid Zarr v2 store with a shared level-0 array (uncompressed C-order chunks are enough since the read path is zarr.open); a synthetic SmartSPIM tree with real TIFF slices whose pixels encode (channelIndex, z) so cross-channel routing has a byte-for-byte proof. Bridge yaml updates (all drift picked up by test_matlab_bridge_completeness.py): - +ndr/+format/+blosc/encode.m 9a87c0a -> aa89463 - +ndr/+format/+blosc/decode.m 9a87c0a -> aa89463 - +ndr/+format/+omezarr/readArray.m a847431 -> 8bea2d2 - +ndr/+format/+omezarr/private/ decompressBlosc.m a847431 -> 5a44303 Every one is MATLAB-side blosc plumbing (batch API, persistent server, MEX preference, hand-rolled codec dispatch replaced with one-line delegation). Python has always used numcodecs.Blosc through zarr directly, so all four are hash-bump-only per the decision_log notes. +ndr/+util/+blosc/* MEX bootstrapping stays deliberately unported (numcodecs.Blosc on the Python side; see the matlab_only entries in src/ndr/format/blosc/ndr_matlab_python_bridge.yaml). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UdLHfApBhntCz64BatnwY5
3 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a Blosc v1 container codec to NDR-matlab that works without a system
zstdbinary and without touching MATLAB'spyenv. The MATLAB surface is thin; the actual encode/decode is delegated tonumcodecs.Bloscinvoked as a subprocess in a private venv NDR sets up on first use. Same "sonpipe pattern" NDR-matlab already uses for CED sonpy.Motivation:
VH-Lab/NDI-matlab#979(the lightsheet OME-Zarr document builder) needs a portable Blosc+Zstd chunk encoder for Windows/macOS/Linux. The previous approach shelled out to a systemzstdbinary; this replaces that with a codec every platform can install identically. NDI-matlab'smakePyramid.mon the same branch now callsndr.format.blosc.encodedirectly.What's in the package
Public API —
+ndr/+format/+blosc/:encode(bytes, ...)— raw bytes or typed array → Blosc v1 containerdecode(container)— container → raw bytesisBlosc(bytes)— informed header sniffheader(container)— parse the 16-byte header without decodingToolchain —
+ndr/+util/+blosc/:setup()— create<NDR>/private_env/blosc, pip install numcodecs (acceptspython,wheelDir,force)pythonExe()— cached path to the venv's pythonrunTool()— stdin/stdout tempfile subprocess wrapperversion()— numcodecs and libblosc versions inside the venvPlus
tools/blosc_tool.py(the stdin/stdout worker) andTestBloscRoundtrip.m(8 tests, cleanly skip when the venv can't be provisioned).Try it
Requirements the first time: a
python33.9+ on PATH and network access to PyPI. After that, no network and no hostpython3are needed. Offline install:ndr.util.blosc.setup('wheelDir', '/path/to/wheels').private_env/is in.gitignoreso the venv never lands in git.Test plan
ndr.util.blosc.setup()on a fresh machine creates the venv and pip-installs numcodecsndr.util.blosc.version()returns non-emptynumcodecsandbloscfieldsTestBloscRoundtrip— uint16 and single arrays round-trip byte-identicalTestBloscRoundtrip—header(container)reportstypesize=2,nbytes=1024,hasByteShuffle=trueafter encoding 512 uint16 valuesTestBloscRoundtrip— length-mismatch (odd bytes withtypesize=2) errors rather than truncating silentlyndi.test.lightsheet.ingestBlobFixture('codec','blosc-zstd')produces a viewable pyramid viandi.fun.doc.lightsheet.view🤖 Generated with Claude Code
https://claude.ai/code/session_01N67xH9BejMzZp4GNAq8m7w
Generated by Claude Code