Skip to content

Add Dockerized basic Holoscan ingest pipeline#1

Open
dleshchev wants to merge 1 commit into
electronmicroscopy:mainfrom
dleshchev:feat/holoscan-pipeline-basic
Open

Add Dockerized basic Holoscan ingest pipeline#1
dleshchev wants to merge 1 commit into
electronmicroscopy:mainfrom
dleshchev:feat/holoscan-pipeline-basic

Conversation

@dleshchev

Copy link
Copy Markdown
Collaborator

Summary

  • Add a Docker-built Holoscan v4.4.0 CUDA 13 pipeline image.
  • Add native C++/CUDA Holoscan operators:
    • ZmqRxOp: ZMQ receive plus fast CBOR bslz4 extraction
    • DecompressGpuOp: GPU bslz4 path using nvCOMP LZ4 plus CUDA bit-unshuffle
  • Add the Python-composed pipeline:
    ZmqRxOp -> DecompressGpuOp -> StatsSinkOp
  • Emit decompressed frames as CUDA tensors and consume them from Python with PyTorch DLPack.
  • Keep the existing Python pipeline in place while the Holoscan path is validated.

Validation

  • python3 -m py_compile holoscan_pipeline/python/dectris_holoscan/*.py
  • docker compose -f docker/compose.holoscan.yaml config
  • git diff --check
  • docker build -f docker/Dockerfile.holoscan -t dectris-holoscan-basic .
  • GPU container import smoke test for dectris_holoscan_ops
  • Full end-to-end 1,000,000 image run:
    • simulator source rate: 139,136.5 frames/s
    • pipeline processed 1,000,000 frames in 1000 batches
    • final sink rate: 139,608.3 fps
    • output tensor: shape=(1000, 96, 96), dtype=torch.uint16, device=cuda:0
    • pipeline errors: 0

Current Limitations

  • Fixed initial detector assumptions: 96x96 uint16 bslz4 frames.
  • nvCOMP is inherited from the Holoscan base image (libnvcomp5-cuda-13 at 5.0.0.6-1 in the validated image).
  • CompressedBatch payloads point into a three-slot pinned host pool owned by ZmqRxOp; the basic pipeline relies on downstream consumption before slot reuse.

Follow-ups

  • Add focused correctness tests against known frames.
  • Generalize detector shape, dtype, and compression settings beyond the initial binned ARINA path.
  • Replace the implicit pinned-pool producer/consumer contract with explicit slot ownership or scheduler backpressure.
  • Consider preallocating nvCOMP per-batch workspace once the target batch/shape set is finalized.

@dleshchev dleshchev marked this pull request as ready for review July 11, 2026 03:34
@cophus cophus requested review from bobleesj and kucukogluberk July 13, 2026 15:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant