Skip to content

About

Utilities for handling large-scale omics data

Topics

Resources

Stars

0 stars

Watchers

1 watching

Forks

Repository files navigation

🧬 omics-utils

A modern toolkit for handling larger-than-memory omics datasets using hydra-zen and dask on a Slurm cluster with Docker/Apptainer containerization.

Python 3.12 Docker Hydra Zen Dask

🌟 Key Features

  • 🧪 Concatenation of multiple sub-datasets/cohorts into one larger dataset
  • 🔀 Assigning train/val dataset splits and shuffling
  • 💾 Sharding of the folds across multiple smaller-than-memory files
  • 🚀 Dask + Slurm integration for efficient parallelization
  • 🧩 Reproducibility and modular type-checked configs via hydra-zen

📋 Table of Contents

(container-registry-authentication)=

🔑 Container Registry Authentication

Generate Token

  1. Create a new GitHub token at Settings → Developer settings → Personal access tokens with:
    • read:packages permission
    • write:packages permission

Log In

With Apptainer:

apptainer remote login --username <your GitHub username> docker://ghcr.io

With Docker:

docker login ghcr.io -u <your GitHub username>

When prompted, enter your token as the password.

(container-setup)=

🐳 Container Setup

Choose one of the following methods to set up your environment:

(option-1-apptainer-slurm-cluster)=

Option 1: Apptainer (Slurm Cluster)

  1. Install VSCode Remote Tunnels Extension

    First, install the Remote Tunnels extension in VSCode.

  2. Connect to compute resources

    For CPU resources:

    srun --partition=cpu-2h --pty bash

    For GPU resources:

    srun --partition=gpu-2h --gpus-per-task=1 --pty bash
  3. Launch container

    To open a tunnel to connect your local VSCode to the container on the cluster:

    apptainer run --nv --writable-tmpfs oras://ghcr.io/marvinsxtr/omics-utils:latest-sif code tunnel

    💡 You can specify a version tag (e.g., v0.0.1) instead of latest. Available versions are listed at GitHub Container Registry.

    In VSCode press Shift+Alt+P (Windows/Linux) or Shift+Cmd+P (Mac), type "connect to tunnel", select GitHub and select your named node on the cluster. Your IDE is now connected to the cluster.

    To open a shell in the container on the cluster:

    apptainer run --nv --writable-tmpfs oras://ghcr.io/marvinsxtr/omics-utils:latest-sif /bin/bash

    💡 This may take a few minutes on the first run as the container image is downloaded.

(option-2-docker-local-machine)=

Option 2: Docker (Local Machine)

  1. Install VSCode Dev Containers Extension

    First, install the Dev Containers extension in VSCode.

  2. Open the Repository in the Dev Container

    Click the Reopen in Container button in the pop-up that appears once you open the repository in VSCode.

    Alternatively, open the command palette in VSCode by pressing Shift+Alt+P (Windows/Linux) or Shift+Cmd+P (Mac), and type Dev Containers: Reopen in Container.

Using Slurm within Apptainer

In order to access Slurm with submitit from within the container, you first need to set up passwordless SSH to the login node.

On the cluster, create a new SSH key pair in case you don't have one yet:

ssh-keygen -t ed25519 -C "your_email@example.com"

and add your public key to the authorized_keys:

cat ~/.ssh/id_ed25519.pub >> ~/.ssh/authorized_keys

You can verify that this works by running

ssh $USER@$HOST exit

which should return without any prompt.

(running-preprocessing)=

🧬 Running Preprocessing

Make sure you followed the previous sections beforehand and run these commands in the container.

The preprocessing pipeline supports multiple omics datasets:

CellXGene Preprocessing

Before running the CellXGene preprocessing, download the h5ad files from AWS into a folder as described in the documentation and set DATA_DIR to point to this folder.

The current version was tested on the 2024-07-01 release of CellXGene.

python omics_utils/preprocess.py pipeline=cellxgene

HLCA Preprocessing

python omics_utils/preprocess.py pipeline=hlca

(package-management)=

📦 Package Management

  1. Update dependencies

    This project uses uv for Python dependency management.

    Inside the container (!):

    # Add a specific package
    uv add <package-name>
    
    # Update all dependencies from pyproject.toml
    uv sync
  2. Commit changes to the repository:

    Use tags for versioning:

    git add pyproject.toml uv.lock 
    git commit -m "Updated dependencies"
    git tag v0.0.1
    git push && git push --tags
  3. Use the updated image:

    The GitHub Actions workflow automatically builds a new image when changes are pushed.

    With Apptainer:

    apptainer run --nv --writable-tmpfs oras://ghcr.io/marvinsxtr/omics-utils:v0.0.1-sif /bin/bash

    With Docker:

    docker run -it --rm --platform=linux/amd64 ghcr.io/marvinsxtr/omics-utils:v0.0.1 /bin/bash

(documentation)=

📚 Documentation

To build the documentation locally, run the following command in your container:

uv run sphinx-build docs/source docs/build

(contributions)=

👥 Contributions

Contributions to this project are very welcome! Feel free to open a PR or reach out with suggestions.

About

Utilities for handling large-scale omics data

Topics

Resources

Stars

0 stars

Watchers

1 watching

Forks

Releases

Packages

Contributors

Languages