A modern toolkit for handling larger-than-memory omics datasets using hydra-zen and dask on a Slurm cluster with Docker/Apptainer containerization.
- 🧪 Concatenation of multiple sub-datasets/cohorts into one larger dataset
- 🔀 Assigning train/val dataset splits and shuffling
- 💾 Sharding of the folds across multiple smaller-than-memory files
- 🚀 Dask + Slurm integration for efficient parallelization
- 🧩 Reproducibility and modular type-checked configs via hydra-zen
- 🔑 Container Registry Authentication
- 🐳 Container Setup
- 🧬 Running Preprocessing
- 📦 Package Management
- 📚 Documentation
- 👥 Contributions
(container-registry-authentication)=
- Create a new GitHub token at Settings → Developer settings → Personal access tokens with:
read:packagespermissionwrite:packagespermission
With Apptainer:
apptainer remote login --username <your GitHub username> docker://ghcr.ioWith Docker:
docker login ghcr.io -u <your GitHub username>When prompted, enter your token as the password.
(container-setup)=
Choose one of the following methods to set up your environment:
(option-1-apptainer-slurm-cluster)=
-
Install VSCode Remote Tunnels Extension
First, install the Remote Tunnels extension in VSCode.
-
Connect to compute resources
For CPU resources:
srun --partition=cpu-2h --pty bash
For GPU resources:
srun --partition=gpu-2h --gpus-per-task=1 --pty bash
-
Launch container
To open a tunnel to connect your local VSCode to the container on the cluster:
apptainer run --nv --writable-tmpfs oras://ghcr.io/marvinsxtr/omics-utils:latest-sif code tunnel
💡 You can specify a version tag (e.g.,
v0.0.1) instead oflatest. Available versions are listed at GitHub Container Registry.In VSCode press
Shift+Alt+P(Windows/Linux) orShift+Cmd+P(Mac), type "connect to tunnel", select GitHub and select your named node on the cluster. Your IDE is now connected to the cluster.To open a shell in the container on the cluster:
apptainer run --nv --writable-tmpfs oras://ghcr.io/marvinsxtr/omics-utils:latest-sif /bin/bash
💡 This may take a few minutes on the first run as the container image is downloaded.
(option-2-docker-local-machine)=
-
Install VSCode Dev Containers Extension
First, install the Dev Containers extension in VSCode.
-
Open the Repository in the Dev Container
Click the
Reopen in Containerbutton in the pop-up that appears once you open the repository in VSCode.Alternatively, open the command palette in VSCode by pressing
Shift+Alt+P(Windows/Linux) orShift+Cmd+P(Mac), and typeDev Containers: Reopen in Container.
In order to access Slurm with submitit from within the container, you first need to set up passwordless SSH to the login node.
On the cluster, create a new SSH key pair in case you don't have one yet:
ssh-keygen -t ed25519 -C "your_email@example.com"and add your public key to the authorized_keys:
cat ~/.ssh/id_ed25519.pub >> ~/.ssh/authorized_keysYou can verify that this works by running
ssh $USER@$HOST exitwhich should return without any prompt.
(running-preprocessing)=
Make sure you followed the previous sections beforehand and run these commands in the container.
The preprocessing pipeline supports multiple omics datasets:
Before running the CellXGene preprocessing, download the h5ad files from AWS into a folder as described in the documentation and set DATA_DIR to point to this folder.
The current version was tested on the
2024-07-01release of CellXGene.
python omics_utils/preprocess.py pipeline=cellxgenepython omics_utils/preprocess.py pipeline=hlca(package-management)=
-
Update dependencies
This project uses uv for Python dependency management.
Inside the container (!):
# Add a specific package uv add <package-name> # Update all dependencies from pyproject.toml uv sync
-
Commit changes to the repository:
Use tags for versioning:
git add pyproject.toml uv.lock git commit -m "Updated dependencies" git tag v0.0.1 git push && git push --tags
-
Use the updated image:
The GitHub Actions workflow automatically builds a new image when changes are pushed.
With Apptainer:
apptainer run --nv --writable-tmpfs oras://ghcr.io/marvinsxtr/omics-utils:v0.0.1-sif /bin/bash
With Docker:
docker run -it --rm --platform=linux/amd64 ghcr.io/marvinsxtr/omics-utils:v0.0.1 /bin/bash
(documentation)=
To build the documentation locally, run the following command in your container:
uv run sphinx-build docs/source docs/build(contributions)=
Contributions to this project are very welcome! Feel free to open a PR or reach out with suggestions.